Forwarder Diagnostics – Last time Data Was Received by Index and Sourcetype

The following Splunk query was modified from the Splunk Health Overview app on Splunkbase. This particular Splunk search returns a list of hosts with their indexes and sourcetypes and determines when each last sent data. This query can help diagnose which Splunk Universal Forwarders may be having issues sending data, or simply stopped sending a […]

Continue Reading →

Detailed list of Errors Per Host

The following Splunk search will return a detailed list (by message) of errors associated with hosts running a universal forwarder: index=_internal sourcetype=”splunkd” log_level=”ERROR” | stats sparkline count dc(host) as uniqhosts last(message) as message last(_time) as last first(_time) as first by punct  | convert ctime(last) ctime(first) | table message count uniqhosts sparkline first last | sort […]

Continue Reading →

Splunk Server Restart Duration

As titled, the following Splunk search query will show the restart duration (using the transaction command) of the Splunk service itself.   index=_audit (action=”splunkShuttingDown” OR action=”splunkStarting”) | eval Date=strftime(_time, “%Y/%m/%d”) | transaction splunk_server startswith=action=”splunkShuttingDown” endswith=action=”splunkStarting” | eval duration=round(duration/60, 2) |table Date splunk_server duration| rename duration as “Splunk Restart Duration” splunk_server as “Splunk Server”

Continue Reading →

Total Number of Hosts reporting in.

This query will list the total number of hosts reporting to the indexer for any specified time range. This only works for universal forwarders. If you have hosts reporting in over syslog (typically port 514) they will not be listed. index=_internal sourcetype=splunkd | stats dc(hostname) as “Number of Hosts”

Continue Reading →