Just sharing a query I found useful lately when licensing dashboards are being silly. Mileage may vary. index=”_introspection” component=”licensing.stack”| bucket _time span=1d | stats latest(“data.consumption”) as dataConsumption latest(“data.pools{}.quota”) as poolQuota by _time| eval pctUsed=(dataConsumption/poolQuota * 100)| timechart span=1d max(pctUsed)
Original posted in 2015 Testing an issue with brackets
The following search was copied from 2015: index=_internal earliest=-48h latest=-24h | bin _time span=10m | stats count by _time | eval window=”yesterday” | append [ | timechart span=10m sum(count) by window]
Indexes size and EPS
Description: SPL request to display by index : Index name Index size Events sum, min, avg, max, perc95 Events sum, min, avg, max, perc95 to work hours (8am-6pm) Required: Splunk license Query: index=_internal source=*license_usage.log idx=z* | fields b idx _time| eval GB=b/1024/1024/1024, index=idx | stats sum(GB) as “Volume GB” by index | append extendtimerange=t […]
Identifying Hosts not sending data for more than 6 hours
| tstats latest(_time) as latest where index!=”*_” earliest=-9h by host index sourcetype | eval recent = if(latest > relative_time(now(),”-360m”),”1″,”0″), LastReceiptTime = strftime(latest,”%c”) | where recent=0 | sort LastReceiptTime | eval age=now()-latest | eval age=round((age/60/60),1) | eval age=age.”hour” | fields – recent latest
Datamodel Search Performance
See how well your DM searches are running. Run this search using the Line Chart visualization: index=_internal sourcetype=scheduler component=SavedSplunker ACCELERATE NOT skipped run_time=* | rex field=savedsearch_id “ACCELERATE_(?:[A-F0-9\-]{36}_)?(?<acceleration>.*?)_ACCELERATE” | timechart span=5m max(run_time) AS run_time by acceleration
Data Usage for Indexer and Forwarders
In my previous role I created this dashboard to identify how much data a Splunk forwarder had sent to my indexers. This was a daily check that either myself of someone on my team would review. This check helped us identify a misconfiguration across all of my production Windows servers. I was able to drilldown […]
Searches to check search concurrency for historical or real time
The following Splunk search will output historical or real time concurrency in a timechart by host. *NOTE* Change the text <search_head> to your search heads name, alternatively use a *. index=_internal host= source=*metrics.log group=search_concurrency “system total” NOT user=* | timechart max(active_hist_searches) by host index=_internal host= source=*metrics.log group=search_concurrency “system total” NOT user=* | timechart max(active_realtime_searches) by […]
Apps Deployed from Deployment Server
Want to show what apps have been deployed to forwarders from a deployment server (DS)? Try this Splunk Search: index=_internal sourcetype=splunkd component=DeployedApplication installing | stats count latest(_time) AS latest_time by host app | convert ctime(latest_time)
List of Forwarders that are Deployment Clients
Need a list of Forwarders that are talking to a Deployment Server? Try this: index=_internal sourcetype=splunkd component=DC* Handshake | stats count by host Additional REST query (performed on the DS) will return desired results (Thanks to Lyx for pointing this out!): | rest /services/deployment/server/clients splunk_server=local | table hostname applications.*.serverclasses | untable hostname, applications, serverclass | […]
See who is using Splunk by user, app and view
########## Admin Notes This query is a modified version of one submitted by tokenwander here: https://gosplunk.com/whos-using-splunk/ ########## index=_internal sourcetype=”splunk_web_access” method=”GET” status=”200″ user!=- | stats count latest(_time) as ViewTime by user app view | sort -count | eventstats sum(count) as countByApp list(view) as view list(count) as count list(ViewTime) as ViewTime by user app | convert timeformat=”%a […]
Host not sending logs for x days
This Splunk Query will show hosts that stopped sending logs for at least 48 hours. You’ll want to change the time range to be relevant to your environment, and you may need to tweak the 48 hour range to something that is more appropriate for your environment. | tstats count as countAtToday latest(_time) as lastTime […]
Build License usage by Group
This was cobbled together from multiple searches I found. This search feeds the license and storage dashboard posted here: It relies on the Chargeback app for the customers.csv form. index=_internal source=*license_usage.log type=”Usage” earliest=-30d@d latest=@d | eval indexname = if(len(idx)=0 OR isnull(idx),”(UNKNOWN)”,idx) | eval sourcetypename = st | bin _time span=1d | stats values(poolsz) as poolsz […]
Detailed User Activity
index=_* search=* user=* user!=- user!=splunk-system-user | rex field=search max_match=0 “index\s*=[\s\”]*(?<idx1>.*?)[\|\s\”\)]” | rex field=search max_match=0 “[\+\(|\+]index\%3D(?<idx2>.*?)[\+|\)\+]” | eval idx=if(isnull(idx1), idx2, idx1) | eval frequency=if(source=”/opt/splunk/var/log/splunk/splunkd_access.log”, “scheduled”, “ad-hoc”) | eval type=if(match(search, “summary*”), “summary”, type1) | eval idx=if(isnull(idx), “NONE”, idx) | eval end_type=if(frequency=”ad-hoc”, “ad-hoc”, type) | rename end_type as type | table _time frequency type source user idx search […]
Who’s Using Splunk?
I often get asked how much a certain dashboard gets looked at, or how many times a user looks at a specific app. I wrote this quick query to answer that question. index=_internal sourcetype=”splunk_web_access” method=”GET” status=”200″ | stats count as count by user, view | appendpipe [stats sum(count) as count by user | eval view […]
Find unused dashboards
Use this search to find unused dashboards: | rest /servicesNS/-/-/data/ui/views splunk_server=* | search isDashboard=1 | rename eai:acl.app as app | fields title app | join type=left title [| search index=_internal sourcetype=splunk_web_access host=* user=* | rex field=uri_path “.*/(?<title>[^/]*)$” | stats latest(_time) as Time latest(user) as user by title ] | where isnotnull(Time) | eval Now=now() | […]
skipped searches and why
Quickly identify high amounts of skipped searches in your cluster or standalone SH(s): index = _internal skipped sourcetype=scheduler status=skipped host=[your splunk SH(s)] | stats count by app search_type reason savedsearch_name | sort -count Adjust “[your splunk SH(s)]” to the SH(s) you want to check obviously ;)
Internal Splunk User Stats
This simple Splunk query will show us unique Splunk user logged into Splunk per day, as well as total count of log-ons. index=_audit info=succeeded | timechart span=1d dc(user) as unique_users count(user) as logons_all_users
Use TSTATS to find hosts no longer sending data
This is a simple tstats query shows all hosts and sourcetypes that have reported data, and shows the time in seconds since anything was sent. Be sure to run the query over a lengthy period of time in order to include machines that haven’t sent data for sometime. Don’t worry about the search-time so much, […]
Show Searches with Details (Who | When | What)
The following Splunk search will show a list of searches ran on a splunk server with the following details: Who ran the search What sourcetype was used What index was used What the search string was When the search was last ran index=_audit action=search sourcetype=audittrail search_id=* NOT (user=splunk-system-user) search!=”‘typeahead*” | rex “search\=\'(search|\s+)\s(?P<search>[\n\S\s]+?(?=\’))” | rex field=search […]
Forwarder TCP Connections info
This search should help identify which forwarders are connected and give you more information on the forwarders. index=”_internal” sourcetype=”splunkd” source=”*metrics.lo*” group=tcpin_connections component=Metrics | eval sourceHost=if(isnull(hostname), sourceHost,hostname) | eval connectionType=case(fwdType==”uf”,”universal forwarder”, fwdType==”lwf”, “lightweight forwarder”,fwdType==”full”, “heavy forwarder”, connectionType==”cooked” or connectionType==”cookedSSL”,”Splunk forwarder”, connectionType==”raw” or connectionType==”rawSSL”,”legacy forwarder”) | eval version=if(isnull(version),”pre 4.2″,version) | eval guid=if(isnull(guid),sourceHost,guid) | eval os=if(isnull(os),”n/a”,os)| eval arch=if(isnull(arch),”n/a”,arch) […]
