Ever wonder how your search performance is across search heads? Try this query. Depending on your environment you’ll want to specify the host=* section to better represent your environment. Say if you have a naming convention that includes “shc” and a number representing searchheads in a cluster (distributed environment) you can use (host=shc1.fq.dn OR host=shc2.fq.dn […]
Search for all errors in splunkd
index=_internal sourcetype=”splunkd” log_level=”ERROR” | stats sparkline count dc(host) as hosts last(_raw) as last_raw_msg values(sourcetype) as sourcetype last(_time) as last_msg_time first(_time) as first_msg_time values(index) as index by punct | eval delta=round((first_msg_time-last_msg_time),2) | eval msg_per_sec=round((count/delta),2) | convert ctime(last_msg_time) ctime(first_msg_time) | table last_raw_msg count hosts sparkline msg_per_sec sourcetype index first_msg_time last_msg_time delta | sort -count
List skipped searches by name, reason
index=_internal sourcetype=scheduler savedsearch_name=* status=skipped | stats count by savedsearch_name reason Look at the reason to know how to TB.
Detect Scheduler Running Twice a Search
There is a bug that make a search being executed 2 times or more. index=_internal sourcetype=scheduler scheduled_time=* savedsearch_name=* |stats count by scheduled_time, savedsearch_name | where count > 1
Find queues that are nearly full
Easily find Splunk Instances (both forwarders and full installs) that have queues that are over 80% full! index=_internal group=queue | eval percfull=((current_size_kb/max_size_kb)*100) | search percfull>80 | dedup host, name | table _time host name current_size_kb max_size_kb
License Usage by Pool per hour for last 24 hours
Have a misbehaving License Pool that the admin swears he fixed this morning? index=_internal source=*license_usage.log* type=Usage earliest=-24h| bucket _time span=1h | stats sum(b) as bytes by _time, pool | eval GB = round(bytes/1024/1024/1024,5) | fields _time pool GB
List All Hosts Associated with All Indexes
Using the Splunk Tstats command you can quickly list all hosts associated with all indexes: |tstats values(host) where index=* by index
Events Sent to Null Que – Internal Logs
This will show events that have been sent to the null que within the splunk internal logs. index=_internal component=metrics processor=nullqueue group=pipeline sourcetype=splunkd | table _time log_level name processor cpu_seconds executes, cumulative_hits
Average Splunk Web requests by hour
This query is pretty awesome! It helped enlighten us to exactly when our splunk infrastructure is being hit with users index=_internal sourcetype=splunk_web_access [ rest / splunk_server=local | fields splunk_server | rename splunk_server as host ] | bin _time span=1d | stats count by date_hour _time | appendpipe [ fields _time | dedup _time | eval […]
Forwarder Diagnostics – Last time Data Was Received by Index and Sourcetype
The following Splunk query was modified from the Splunk Health Overview app on Splunkbase. This particular Splunk search returns a list of hosts with their indexes and sourcetypes and determines when each last sent data. This query can help diagnose which Splunk Universal Forwarders may be having issues sending data, or simply stopped sending a […]
Count of Host added to Splunk by Month
Can we get a Splunk Query that list of hosts added to Splunk in a month. Like Month 1 : 200 devices added Month 2: 400 devices added You would do this: host=* | stats dc(host) as host by date_month (Edits Made and query provided by the GoSplunk Ninja)
List of Sourcetypes Sent by Forwarder
The following Splunk search will show a list of sourcetypes (or perhaps sources I need to confirm) being sent by each Forwarder: index=”_internal” |where host!=splunk_server | stats values(series) as Sourcetypes by host
Detailed list of Errors Per Host
The following Splunk search will return a detailed list (by message) of errors associated with hosts running a universal forwarder: index=_internal sourcetype=”splunkd” log_level=”ERROR” | stats sparkline count dc(host) as uniqhosts last(message) as message last(_time) as last first(_time) as first by punct | convert ctime(last) ctime(first) | table message count uniqhosts sparkline first last | sort […]
Count of Splunk Errors Per Host
The following Splunk query will list the number of errors associated with each host over a given time range: index=_internal sourcetype=”splunkd” log_level=”ERROR” host!=splunk_server | stats count by host | sort – count
Traffic Volume by Forwarder
This Splunk search query will show you the top 10 “chattiest” forwarders on your network. I’ve used this query to determine why some forwarders were sending more data than others. The results are displayed in kilobits, you could use an eval to change it to the appropriate size for your network. index=”_internal” source=”*metrics.lo*” group=tcpin_connections NOT […]
User Activity in DBConnect
The following Splunk query is for the DBConnect app. This will return all user activity using this particular app. I’ve provided the regex in the search. index=_audit sourcetype=audittrail action=”db_connect*” |eval Date=strftime(_time, “%Y/%d/%m”) |rex “user=(?<user>\S+),” | stats count by Date, user, info, action
rangemap command with single value string
index=_internal sourcetype=splunkd OR sourcetype=splunkd_access | stats latest(sourcetype) as sourcetype | eval sourcetypeidx=case(sourcetype=”splunkd”,2,sourcetype=”splunkd_access”,1) | rangemap field=sourcetypeidx severe=0-1 low=2-4 default=low
How to Check When Splunk is finished Indexing a log file
How can I tell when Splunk is finished indexing a log file? (Credit for this one goes to learnsplunk.com author who originally posted it on his website) By watching data from splunk’s metrics log in real-time. ************************************************************ index=”_internal” source=”*metrics.log” group=”per_sourcetype_thruput” series=”your_sourcetype_here” | eval MB=kb/1024 | chart sum(MB) ************************************************************ or to watch everything happening split by […]
Search to show what apps are ready to be updated
| rest splunk_server=local /services/apps/local | search update.version=* | table title version update.version If that Splunk has internet access, it’ll have the update.* fields filled with the latest version if there is an update available for any app installed on that system. The splunk_server filter should be usable for querying search peers as well. Using that […]
Permissions for splunk users
Another view for which splunk user can do what in your splunk environment | rest /services/authentication/users | mvexpand roles | table realname, title, roles, email | join roles [ rest /services/authorization/roles | rename title as roles | search srchIndexesAllowed=* | table roles srchIndexesAllowed]
