Detect Indexers in Maintenance Mode

This query will show any clustered indexers that are currently in maintenance mode. For it to work as an alert you will need to schedule it. It will not work if you run it in real time.   sourcetype=splunkd reason=”‘Maintenance mode*” | dedup host | eval maintenance_mode_enabled=if(reason=”‘Maintenance mode started'”, “true”, “false”) | where maintenance_mode_enabled=”true” | […]

Continue Reading →

Timestamp vs Indextime of Events (Diagnostic Query)

This query has in the past help me track down issues between forwarders and indexers, and even on occasion finding some time sync issues. Feel free to tweak, modify, and improve upon this query as I’m not 100% certain the math will work in your favor outside of highlighting (positive or negative) time differences! index=* […]

Continue Reading →

Splunk Server Restart Duration

As titled, the following Splunk search query will show the restart duration (using the transaction command) of the Splunk service itself.   index=_audit (action=”splunkShuttingDown” OR action=”splunkStarting”) | eval Date=strftime(_time, “%Y/%m/%d”) | transaction splunk_server startswith=action=”splunkShuttingDown” endswith=action=”splunkStarting” | eval duration=round(duration/60, 2) |table Date splunk_server duration| rename duration as “Splunk Restart Duration” splunk_server as “Splunk Server”

Continue Reading →

Total Number of Hosts reporting in.

This query will list the total number of hosts reporting to the indexer for any specified time range. This only works for universal forwarders. If you have hosts reporting in over syslog (typically port 514) they will not be listed. index=_internal sourcetype=splunkd | stats dc(hostname) as “Number of Hosts”

Continue Reading →

Detailed list of Universal Forwarders Reporting to Indexer

The following query will list in detail information on the universal forwarders checking into the indexer. I’ve renamed some of the fields to be more user-friendly. index=_internal sourcetype=splunkd destPort!=”-“| stats sparkline count by hostname, sourceHost, host, destPort, version | rename destPort as “Destination Port” | rename host as “Indexer” | rename sourceHost as “Universal Forwarder […]

Continue Reading →