The following Splunk query was modified from the Splunk Health Overview app on Splunkbase. This particular Splunk search returns a list of hosts with their indexes and sourcetypes and determines when each last sent data. This query can help diagnose which Splunk Universal Forwarders may be having issues sending data, or simply stopped sending a […]
List of Sourcetypes Sent by Forwarder
The following Splunk search will show a list of sourcetypes (or perhaps sources I need to confirm) being sent by each Forwarder: index=”_internal” |where host!=splunk_server | stats values(series) as Sourcetypes by host
Detailed list of Errors Per Host
The following Splunk search will return a detailed list (by message) of errors associated with hosts running a universal forwarder: index=_internal sourcetype=”splunkd” log_level=”ERROR” | stats sparkline count dc(host) as uniqhosts last(message) as message last(_time) as last first(_time) as first by punct | convert ctime(last) ctime(first) | table message count uniqhosts sparkline first last | sort […]
Count of Splunk Errors Per Host
The following Splunk query will list the number of errors associated with each host over a given time range: index=_internal sourcetype=”splunkd” log_level=”ERROR” host!=splunk_server | stats count by host | sort – count
Splunk Server Restart Duration
As titled, the following Splunk search query will show the restart duration (using the transaction command) of the Splunk service itself. index=_audit (action=”splunkShuttingDown” OR action=”splunkStarting”) | eval Date=strftime(_time, “%Y/%m/%d”) | transaction splunk_server startswith=action=”splunkShuttingDown” endswith=action=”splunkStarting” | eval duration=round(duration/60, 2) |table Date splunk_server duration| rename duration as “Splunk Restart Duration” splunk_server as “Splunk Server”
Failed Attempts to Logon to Splunk Web
The following Splunk Search Query will return all users who have failed to logon to the Splunk Web console. This query will also include an average (from eventstats). index=_audit action=”login attempt” info=failed | timechart count(user) as Failed_Attempts| eventstats avg(Failed_Attempts) as Average
Total Number of Hosts reporting in.
This query will list the total number of hosts reporting to the indexer for any specified time range. This only works for universal forwarders. If you have hosts reporting in over syslog (typically port 514) they will not be listed. index=_internal sourcetype=splunkd | stats dc(hostname) as “Number of Hosts”
