Splunk License Consumption via _introspection

Just sharing a query I found useful lately when licensing dashboards are being silly. Mileage may vary. index=”_introspection” component=”licensing.stack”| bucket _time span=1d | stats latest(“data.consumption”) as dataConsumption latest(“data.pools{}.quota”) as poolQuota by _time| eval pctUsed=(dataConsumption/poolQuota * 100)| timechart span=1d max(pctUsed)

Continue Reading →

Indexes size and EPS

Description: SPL request to display by index : Index name Index size Events sum, min, avg, max, perc95 Events sum, min, avg, max, perc95 to work hours (8am-6pm) Required: Splunk license Query: index=_internal source=*license_usage.log idx=z* |  fields b idx _time| eval GB=b/1024/1024/1024, index=idx | stats sum(GB) as “Volume GB” by index |  append extendtimerange=t     […]

Continue Reading →

Identifying Hosts not sending data for more than 6 hours

| tstats latest(_time) as latest where index!=”*_” earliest=-9h by host index sourcetype | eval recent = if(latest > relative_time(now(),”-360m”),”1″,”0″), LastReceiptTime = strftime(latest,”%c”) | where recent=0 | sort LastReceiptTime | eval age=now()-latest | eval age=round((age/60/60),1) | eval age=age.”hour” | fields – recent latest

Continue Reading →

Datamodel Search Performance

See how well your DM searches are running. Run this search using the Line Chart visualization: index=_internal sourcetype=scheduler component=SavedSplunker ACCELERATE NOT skipped run_time=* | rex field=savedsearch_id “ACCELERATE_(?:[A-F0-9\-]{36}_)?(?<acceleration>.*?)_ACCELERATE” | timechart span=5m max(run_time) AS run_time by acceleration

Continue Reading →

Data Usage for Indexer and Forwarders

In my previous role I created this dashboard to identify how much data a Splunk forwarder had sent to my indexers.  This was a daily check that either myself of someone on my team would review.  This check helped us identify a misconfiguration across all of my production Windows servers.  I was able to drilldown […]

Continue Reading →

Searches to check search concurrency for historical or real time

The following Splunk search will output historical or real time concurrency in a timechart by host. *NOTE* Change the text <search_head> to your search heads name, alternatively use a *. index=_internal host= source=*metrics.log group=search_concurrency “system total” NOT user=* | timechart max(active_hist_searches) by host index=_internal host= source=*metrics.log group=search_concurrency “system total” NOT user=* | timechart max(active_realtime_searches) by […]

Continue Reading →

List of Forwarders that are Deployment Clients

Need a list of Forwarders that are talking to a Deployment Server? Try this: index=_internal sourcetype=splunkd component=DC* Handshake | stats count by host Additional REST query (performed on the DS) will return desired results (Thanks to Lyx for pointing this out!): | rest /services/deployment/server/clients splunk_server=local | table hostname applications.*.serverclasses | untable hostname, applications, serverclass | […]

Continue Reading →

See who is using Splunk by user, app and view

########## Admin Notes This query is a modified version of one submitted by tokenwander here: https://gosplunk.com/whos-using-splunk/ ########## index=_internal sourcetype=”splunk_web_access” method=”GET” status=”200″ user!=- | stats count latest(_time) as ViewTime by user app view | sort -count | eventstats sum(count) as countByApp list(view) as view list(count) as count list(ViewTime) as ViewTime by user app | convert timeformat=”%a […]

Continue Reading →

Host not sending logs for x days

This Splunk Query will show hosts that stopped sending logs for at least 48 hours. You’ll want to change the time range to be relevant to your environment, and you may need to tweak the 48 hour range to something that is more appropriate for your environment. | tstats count as countAtToday latest(_time) as lastTime […]

Continue Reading →

Build License usage by Group

This was cobbled together from multiple searches I found. This search feeds the license and storage dashboard posted here: It relies on the Chargeback app for the customers.csv form. index=_internal source=*license_usage.log type=”Usage” earliest=-30d@d latest=@d | eval indexname = if(len(idx)=0 OR isnull(idx),”(UNKNOWN)”,idx) | eval sourcetypename = st | bin _time span=1d | stats values(poolsz) as poolsz […]

Continue Reading →

Detailed User Activity

index=_* search=* user=* user!=- user!=splunk-system-user | rex field=search max_match=0 “index\s*=[\s\”]*(?<idx1>.*?)[\|\s\”\)]” | rex field=search max_match=0 “[\+\(|\+]index\%3D(?<idx2>.*?)[\+|\)\+]” | eval idx=if(isnull(idx1), idx2, idx1) | eval frequency=if(source=”/opt/splunk/var/log/splunk/splunkd_access.log”, “scheduled”, “ad-hoc”) | eval type=if(match(search, “summary*”), “summary”, type1) | eval idx=if(isnull(idx), “NONE”, idx) | eval end_type=if(frequency=”ad-hoc”, “ad-hoc”, type) | rename end_type as type | table _time frequency type source user idx search […]

Continue Reading →

Who’s Using Splunk?

I often get asked how much a certain dashboard gets looked at, or how many times a user looks at a specific app. I wrote this quick query to answer that question. index=_internal sourcetype=”splunk_web_access” method=”GET” status=”200″ | stats count as count by user, view | appendpipe [stats sum(count) as count by user | eval view […]

Continue Reading →

Find unused dashboards

Use this search to find unused dashboards: | rest /servicesNS/-/-/data/ui/views splunk_server=* | search isDashboard=1 | rename eai:acl.app as app | fields title app | join type=left title [| search index=_internal sourcetype=splunk_web_access host=* user=* | rex field=uri_path “.*/(?<title>[^/]*)$” | stats latest(_time) as Time latest(user) as user by title ] | where isnotnull(Time) | eval Now=now() | […]

Continue Reading →

skipped searches and why

Quickly identify high amounts of skipped searches in your cluster or standalone SH(s): index = _internal skipped sourcetype=scheduler status=skipped host=[your splunk SH(s)] | stats count by app search_type reason savedsearch_name | sort -count Adjust “[your splunk SH(s)]” to the SH(s) you want to check obviously ;)

Continue Reading →

Use TSTATS to find hosts no longer sending data

This is a simple tstats query shows all hosts and sourcetypes that have reported data, and shows the time in seconds since anything was sent. Be sure to run the query over a lengthy period of time in order to include machines that haven’t sent data for sometime. Don’t worry about the search-time so much, […]

Continue Reading →

Show Searches with Details (Who | When | What)

The following Splunk search will show a list of searches ran on a splunk server with the following details: Who ran the search What sourcetype was used What index was used What the search string was When the search was last ran index=_audit action=search sourcetype=audittrail search_id=* NOT (user=splunk-system-user) search!=”‘typeahead*” | rex “search\=\'(search|\s+)\s(?P<search>[\n\S\s]+?(?=\’))” | rex field=search […]

Continue Reading →

Forwarder TCP Connections info

This search should help identify which forwarders are connected and give you more information on the forwarders. index=”_internal” sourcetype=”splunkd” source=”*metrics.lo*” group=tcpin_connections component=Metrics | eval sourceHost=if(isnull(hostname), sourceHost,hostname) | eval connectionType=case(fwdType==”uf”,”universal forwarder”, fwdType==”lwf”, “lightweight forwarder”,fwdType==”full”, “heavy forwarder”, connectionType==”cooked” or connectionType==”cookedSSL”,”Splunk forwarder”, connectionType==”raw” or connectionType==”rawSSL”,”legacy forwarder”) | eval version=if(isnull(version),”pre 4.2″,version) | eval guid=if(isnull(guid),sourceHost,guid) | eval os=if(isnull(os),”n/a”,os)| eval arch=if(isnull(arch),”n/a”,arch) […]

Continue Reading →