Ok that one is a big one so be prepared ;) The following will (on a SH / SH Cluster): list all users and their roles list inherited roles list all indexes allowed by the shown roles list all indexes allowed for inherited roles (one level!) inherited allowed indexes will show the originator (which inherited […]
List forwarders generating socket errors due to unkown SSL protocol
If you are using SSL on port 9997 or 9998 (or other port) to send logs from your forwarders to your indexers, you know that a misconfigured forwarder or one with incorrect certs can generate these errors. This search gives you a succinct summary of those hosts. Use Today in the timepicker. Note: Add an […]
count all events for 1 or multiple index(es)
Total count of all events for 1 or more index(es) Approach 1 (fastest) | eventcount index=foo or | eventcount index=foo index=bar does *not* support time ranges in the time picker tested on: splunk v6.6 Approach 2 (fast – especially when tsidx are *not* reduced) | tstats count where index=foo OR index=bar by span=1d _time index […]
Windows security daily domain activities
sourcetype=WinEventLog:Security src_nt_domain!=”NT AUTHORITY” EventCode=4720 OR EventCode=4726 OR EventCode=4738 OR EventCode=4767 OR EventCode=4781 OR EventCode=4727 OR EventCode=4730 OR EventCode=4731 OR EventCode=4734 OR EventCode=4735 OR EventCode=4737 OR EventCode=4744 OR EventCode=4745 OR EventCode=4748 OR EventCode=4749 OR EventCode=4750 OR EventCode=4753 OR EventCode=4754 OR EventCode=4755 OR EventCode=4758 OR EventCode=4759 OR EventCode=4760 OR EventCode=4763 OR EventCode=4764 OR EventCode=4728 OR EventCode=4729 OR […]
License Usage Prediction
There is an older Splunk query here that had previously predicted license usage. I’m not sure why (perhaps the predict command has changed since the original post in 2015?), but the query is no longer working. I’ve updated the query to predict Splunk license usage using the Splunk predict command as shown below: index=_internal source=”*license_usage.lo*” […]
Indexes in Splunk
For those who have more than a few indexes (we’ve got 27 non-administrative indexes) I wrote this search so people could figure-out what we have and what it is used for. The search requires that there be a file called indexdescriptions.csv located in $SPLUNK_HOME/etc/apps/search/lookups (or “Program Files”\splunk\etc\apps\search\lookups\indexdescriptions.csv ). That file should have “index,description” on the […]
Saved Search Scheduler Activity
I use this query a lot to tune and adjust scheduling, find out what searches need attention: index=_internal sourcetype=scheduler result_count | extract pairdelim=”,”, kvdelim=”=”, auto=f | stats avg(result_count) min(result_count) max(result_count), sparkline avg(run_time) min(run_time) max(run_time) sum(run_time) values(host) AS hosts count AS execution_count by savedsearch_name, app | join savedsearch_name type=outer [| rest /servicesNS/-/-/saved/searches | fields title eai:acl.owner […]
Show indexing queue sizes
Use a linechart with this search to show you the indexing queue sizes: index=_internal source=*metrics.log group=queue (name=parsingqueue OR name=indexqueue OR name=typingqueue OR name=aggqueue) | timechart avg(current_size) by name
Percentage of skipped searches
This query will give you a table with a percentage of skipped searches and an evaluation with 3 ranges index=_internal sourcetype=scheduler | stats count as total, count(eval(status=”skipped”)) as skipped | eval pct=round(skipped/total * 100, 0) | rangemap field=pct low=0-10, elevated=10-20 severe=20-100 | eval pct = pct . “%” | fields pct, range
Retention Period in days per index
This query will give you a table of all indexes and their respective retention period in days: | rest splunk_server=* /services/data/indexes | join type=outer title [ | rest splunk_server=* /services/data/indexes-extended ] | eval retentionInDays=frozenTimePeriodInSecs/86400 | table title retentionInDays
Show Splunk User to Role mapping
The following Splunk REST query shows all roles, number of capabilities, and landing app for each user. | rest /services/authentication/users | eval name=coalesce(realname, title) | stats values(roles) as Role first(defaultApp) as “Landing App” count(capabilities) as “Number of Capabilities” by name
Bucket Count by indexer/index
This search displays the amount of buckets per indexer/index To learn more about the | dbinspect command go to: http://docs.splunk.com/Documentation/Splunk/7.1.2/SearchReference/Dbinspect |dbinspect index=* | search index!=_* | chart dc(bucketId) over splunk_server by index
Bucket Count by State over Index
This search counts the amount of buckets per state for each index. To learn more about | dbinspect go to: http://docs.splunk.com/Documentation/Splunk/7.1.2/SearchReference/Dbinspect |dbinspect index=* | eval state=case(state==”warm” OR state==”hot”,”hot/warm”,1=1, state) | chart dc(bucketId) over index by state
Memory Usage (MB) per Splunk Process Class
Use the following search with a column chart visualisation. It will give you a good overview of what Splunk processes use the most memory: index=_introspection sourcetype=splunk_resource_usage component=PerProcess host=* | eval process = ‘data.process’ | eval args = ‘data.args’ | eval sid = ‘data.search_props.sid’ | eval elapsed = ‘data.elapsed’ | eval mem_used = ‘data.mem_used’ | eval mem […]
Show all Indexes and Sourcetypes via REST
The following Splunk query uses REST to display non internal indexes associated with sourcetypes. It is my understanding that this is all time (such is the way of REST searches) | rest /services/data/inputs/all | search index!=_* | stats values(sourcetype) by index
Universal Forwarder Throughput Limit Hit Count
This search counts the amount of times the UF’s throughput limit is hit. I also threw in a sparkline: index=_internal sourcetype=splunkd “current data throughput” | rex “Current data throughput \((?<kb>\S+)” | eval rate=case(kb < 500, “256”, kb > 499 AND kb < 520, “512”, kb > 520 AND kb < 770 ,”768″, kb>771 AND kb<1210, […]
Top 10 Accessed Dashboards
Where “host=”your_sh_host”” you could specify a host, or put a wildcard * in place. index=”_internal” source=*access.log user!=”-” */app/* (host=”your_sh_host”) | rex field=referer “/en-US/app/(?<app>[^/]+)/(?<dashboard>[^?/\s]+)” | top 10 dashboard
Universal Forwarder Throughput Statistics
This search creates a table to list all Universal Forwarders. There is also an eval in there that classifies hosts based on their average Kbps. You can modify this as needed. index=_internal source=*metrics.log group=tcpin_connections splunk_server=* | eval host=if(isnull(hostname), sourceHost,hostname) | search (host=*) AND (host!=”(ALL)”) | eval connectionType=case(fwdType==”uf”,”univ fwder”, fwdType==”lwf”, “lightwt fwder”,fwdType==”full”, “heavy fwder”, connectionType==”cooked” or […]
Timechart of the status of an Locked Out Account
This query will show a timechart of the status of an Locked Out Account sourcetype=”WinEventLog:Security” EventCode=4625 AND Status=0xC0000234 | timechart count by user | sort -count
Active Directory Password change attempts
Use the following search to create a stacked barchart of AD Password change attempts: source=”WinEventLog:Security” “EventCode=4723″ src_user!=”*$” src_user!=”_svc_*” | eval daynumber=strftime(_time,”%Y-%m-%d”) | chart count by daynumber, status | eval daynumber = mvindex(split(daynumber,”-“),2)
