As titled, the following Splunk search query will show the restart duration (using the transaction command) of the Splunk service itself. index=_audit (action=”splunkShuttingDown” OR action=”splunkStarting”) | eval Date=strftime(_time, “%Y/%m/%d”) | transaction splunk_server startswith=action=”splunkShuttingDown” endswith=action=”splunkStarting” | eval duration=round(duration/60, 2) |table Date splunk_server duration| rename duration as “Splunk Restart Duration” splunk_server as “Splunk Server”
List Ports Forwarders are Using
Use the following Splunk Search Query to list what ports your Universal Forwarders are using to communicate to the Indexer: index=”_internal” source=”*metrics.lo*” group=tcpin_connections NOT eventType=* | dedup sourceHost |stats count by destPort
Introspection – Memory used by SID (Search ID)
Chart memory used by each SID (search ID) using the Introspection logs index=”_introspection” “data.process”=splunkd | timechart max(data.mem_used) by data.search_props.sid usenull=f useother=f
Splunk Query Count by users
index=_audit search=* NOT (search_id=’scheduler* OR search_id=’Summary*) user=admin | timechart span=1d count by user usenull=f
Failed Attempts to Logon to Splunk Web
The following Splunk Search Query will return all users who have failed to logon to the Splunk Web console. This query will also include an average (from eventstats). index=_audit action=”login attempt” info=failed | timechart count(user) as Failed_Attempts| eventstats avg(Failed_Attempts) as Average
Splunk Query to report on users logging on to the Splunk Web Console
index=_audit tag=authentication info=succeeded |dedup user | table user timestamp
License Usage by Index per Day
The following Splunk search query will output license usage for each index for each day for the week to date. It will also output an average for each index over the course of the given time period. index=_internal source=*license_usage.log type=”Usage” splunk_server=* earliest=-1w@d | eval Date=strftime(_time, “%Y/%m/%d”) | eventstats sum(b) as volume by idx, Date […]
Percentage of Daily License Usage
This Splunk search query will indicate the percentage of license used for the current day. This is already shown in the licensing tab under settings, however this query is extracted if you would want to use it within a dashboard or any other reason. NOTE – splunk_server= should be set to your license master. […]
Top 5 License Consuming Hosts
The following Splunk search query will return the top five licensing consuming hosts: index=_internal source=*license_usage.log type=”Usage” | stats sum(b) AS volume by h | eval GB=round(volume/1024/1024/1024,5) | table h GB | sort 5 – GB
License Usage by Sourcetypes
The following Splunk query will return results for license usage by sourcetype: index=_internal source=”*license_usage.lo*” type=Usage | stats sum(b) as bytes by st | eval Megabytes=bytes/1048576 |eval Megabytes=round(Megabytes,2) | fieldformat Megabytes=tostring(Megabytes,”commas”)| rename st as sourcetype | fields – bytes | sort – Megabytes
Last Time a Forwarder Checked In
The following Splunk Search Query will return results based on the last time a forwarder (universal forwarder, heavy forwarder, or otherwise) checked in. The query is a modified version of a query that was packaged with the Deployment Monitor app. index=”_internal” source=”*metrics.lo*” group=tcpin_connections NOT eventType=* | eval sourceHost=if(isnull(hostname), sourceHost,hostname) | eval connectionType=case(fwdType==”uf”,”universal forwarder”, fwdType==”lwf”, “lightweight […]
List of Universal Forwarders with Version
The following Splunk query will return results of any host using a universal forwarder to transmit data back to a Splunk indexer. The query will return hostname, version, as well as architecture (64-bit vs 32-bit). index=”_internal” sourcetype=splunkd group=tcpin_connections NOT eventType=* | eval Hostname=if(isnull(hostname), sourceHost,hostname) | eval version=if(isnull(version),”pre 4.2″,version) | eval architecture=if(isnull(arch),”n/a”,arch) | stats count […]
List of hosts and sourcetypes not sending data in last 24 Hours
|noop |append [ |metadata type=hosts | table *] | append [|metadata type=sourcetypes | table *] | eval t = now() – lastTime | where t > 86400 | eval name = coalesce(host,sourcetype)| table name t lastTime totalCount type |rename t as “Seconds since Event” | convert ctime(lastTime) timeformat=”%m/%d/%Y %H:%M:%S %z”
Comparing Stats Time Over Time
index=_internal earliest=-48h latest=-24h | bin _time span=10m | stats count by _time | eval window=”yesterday” | append [ search index=_internal earliest=-24h | bin _time span=10m | stats count by _time| eval window=”today” | eval _time=(_time-(60*60*24))] | timechart span=10m sum(count) by window This search will lay a count of something (in this case, just a count) […]
Detailed list of Universal Forwarders Reporting to Indexer
The following query will list in detail information on the universal forwarders checking into the indexer. I’ve renamed some of the fields to be more user-friendly. index=_internal sourcetype=splunkd destPort!=”-“| stats sparkline count by hostname, sourceHost, host, destPort, version | rename destPort as “Destination Port” | rename host as “Indexer” | rename sourceHost as “Universal Forwarder […]
