See how well your DM searches are running. Run this search using the Line Chart visualization:
|
1 2 3 |
index=_internal sourcetype=scheduler component=SavedSplunker ACCELERATE NOT skipped run_time=* | rex field=savedsearch_id "ACCELERATE_(?:[A-F0-9\-]{36}_)?(?<acceleration>.*?)_ACCELERATE" | timechart span=5m max(run_time) AS run_time by acceleration |
|
1 |
index=_internal sourcetype=dbx_server Starting dbx_task_server |
Will return events that display a little dragon ascii art:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
|\___/| (,\ /,)\ / / \ (@_^_@)/ \ W//W_/ \ (//) | \ (/ /) _|_ / ) \ (// /) '/,_ _ _/ (~^-. (( // )) ,-{ _ `. (( /// )) '/\ / | (( ///)) `. { } ((/ )) .----~-.\ \-' ///.----..> \ ///-._ _ _ _} |
This search shows all the alerts that where triggered in your splunk environment:
|
1 |
index=_audit action=alert_fired ss_app=* | eval ttl=expiration-now() | search ttl>0 | convert ctime(trigger_time) | table trigger_time ss_name severity | rename trigger_time as "Alert Time" ss_name as "Alert Name" severity as "Severity" |
Quick snippet to evaluate temperature Fahrentheit to Celsius:
|
1 |
| eval Temperature_Fahrenheit=Temperature_Celsius*1.8+32 |
Use this search to find unused dashboards:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
| rest /servicesNS/-/-/data/ui/views splunk_server=* | search isDashboard=1 | rename eai:acl.app as app | fields title app | join type=left title [| search index=_internal sourcetype=splunk_web_access host=* user=* | rex field=uri_path ".*/(?<title>[^/]*)$" | stats latest(_time) as Time latest(user) as user by title ] | where isnotnull(Time) | eval Now=now() | eval "Days since last accessed"=round((Now-Time)/86400,2) | sort - "Days since last accessed" | convert ctime(Time) | fields - Now |
Admin Notes – Fantastic query! I modified the SPL slightly as I had an issue when I copied it to my two test environments.
A simple method on checking if your strftime (TIME_FORMAT=) in the props.conf matches your log file timestamp format. strftime(X,Y) This function takes a UNIX time value, X, as the first argument and renders the time as a string using the format specified by Y. The UNIX time must be in seconds. Use the first 10 […]
I use this query a lot to tune and adjust scheduling, find out what searches need attention:
|
1 |
index=_internal sourcetype=scheduler result_count | extract pairdelim=",", kvdelim="=", auto=f | stats avg(result_count) min(result_count) max(result_count), sparkline avg(run_time) min(run_time) max(run_time) sum(run_time) values(host) AS hosts count AS execution_count by savedsearch_name, app | join savedsearch_name type=outer [| rest /servicesNS/-/-/saved/searches | fields title eai:acl.owner cron_schedule dispatch.earliest_time dispatch.latest_time search | rename title AS savedsearch_name eai:acl.app AS App eai:acl.owner AS Owner cron_schedule AS "Cron Schedule" dispatch.earliest_time AS "Dispatch Earliest Time" dispatch.latest_time AS "Dispatch Latest Time"]| rename savedsearch_name AS "Saved Search Name" search AS "SPL Query" app AS App | makemv delim="," values(host) | sort - avg(run_time) | table "Saved Search Name", App, Owner, "SPL Query", "Dispatch Earliest Time" "Dispatch Latest Time" "Cron Schedule" hosts, execution_count, sparkline, *(result_count), sum(run_time) *(run_time) |
Use a linechart with this search to show you the indexing queue sizes:
|
1 |
index=_internal source=*metrics.log group=queue (name=parsingqueue OR name=indexqueue OR name=typingqueue OR name=aggqueue) | timechart avg(current_size) by name |
This query will give you a table with a percentage of skipped searches and an evaluation with 3 ranges
|
1 |
index=_internal sourcetype=scheduler | stats count as total, count(eval(status="skipped")) as skipped | eval pct=round(skipped/total * 100, 0) | rangemap field=pct low=0-10, elevated=10-20 severe=20-100 | eval pct = pct . "%" | fields pct, range |
This query will give you a table of all indexes and their respective retention period in days:
|
1 2 3 4 |
| rest splunk_server=* /services/data/indexes | join type=outer title [ | rest splunk_server=* /services/data/indexes-extended ] | eval retentionInDays=frozenTimePeriodInSecs/86400 | table title retentionInDays |
This search displays the amount of buckets per indexer/index To learn more about the | dbinspect command go to: http://docs.splunk.com/Documentation/Splunk/7.1.2/SearchReference/Dbinspect
|
1 |
|dbinspect index=* | search index!=_* | chart dc(bucketId) over splunk_server by index |
This search counts the amount of buckets per state for each index. To learn more about | dbinspect go to: http://docs.splunk.com/Documentation/Splunk/7.1.2/SearchReference/Dbinspect
|
1 |
|dbinspect index=* | eval state=case(state=="warm" OR state=="hot","hot/warm",1=1, state) | chart dc(bucketId) over index by state |
Use the following search with a column chart visualisation. It will give you a good overview of what Splunk processes use the most memory:
|
1 |
index=_introspection sourcetype=splunk_resource_usage component=PerProcess host=* | eval process = 'data.process' | eval args = 'data.args' | eval sid = 'data.search_props.sid' | eval elapsed = 'data.elapsed' | eval mem_used = 'data.mem_used' | eval mem = 'data.mem' | eval pct_memory = 'data.pct_memory' | eval app = 'data.search_props.app' | eval type = 'data.search_props.type' | eval mode = 'data.search_props.mode' | eval user = 'data.search_props.user' | eval role = 'data.search_props.role' | eval process_class = case( process=="splunk-optimize","index service", process=="sh" OR process=="ksh" OR process=="bash" OR like(process,"python%") OR process=="powershell","scripted input", process=="mongod", "KVStore") | eval process_class = case( process=="splunkd" AND (like(args,"-p %start%") OR like(args,"service")),"splunkd server", process=="splunkd" AND isnotnull(sid),"search", process=="splunkd" AND (like(args,"fsck%") OR like(args,"recover-metadata%") OR like(args,"cluster_thing")),"index service", process=="splunkd" AND args=="instrument-resource-usage", "scripted input", (like(process,"python%") AND like(args,"%/appserver/mrsparkle/root.py%")) OR like(process,"splunkweb"),"Splunk Web", isnotnull(process_class), process_class) | eval process_class = if(isnull(process_class),"other",process_class) | stats latest(data.mem_used) AS resource_usage_dedup latest(process_class) AS process_class by data.pid, _time | stats sum(resource_usage_dedup) AS resource_usage by _time, process_class | timechart minspan=10s median(resource_usage) AS "Resource Usage" by process_class |
|
1 |
This search counts the amount of times the UF’s throughput limit is hit. I also threw in a sparkline:
|
1 |
index=_internal sourcetype=splunkd "current data throughput" | rex "Current data throughput \((?<kb>\S+)" | eval rate=case(kb < 500, "256", kb > 499 AND kb < 520, "512", kb > 520 AND kb < 770 ,"768", kb>771 AND kb<1210, "1024", 1=1, ">1024") | stats count as Count sparkline as Trend by host, rate | where Count > 4 | rename host as "Host" rate as "Throughput rate(kb)" count as "Hit Count"| sort -"Throughput rate(kb)",-Count |
Where “host=”your_sh_host”” you could specify a host, or put a wildcard * in place.
|
1 2 3 |
index="_internal" source=*access.log user!="-" */app/* (host="your_sh_host") | rex field=referer "/en-US/app/(?<app>[^/]+)/(?<dashboard>[^?/\s]+)" | top 10 dashboard |
This search creates a table to list all Universal Forwarders. There is also an eval in there that classifies hosts based on their average Kbps. You can modify this as needed.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
index=_internal source=*metrics.log group=tcpin_connections splunk_server=* | eval host=if(isnull(hostname), sourceHost,hostname) | search (host=*) AND (host!="(ALL)") | eval connectionType=case(fwdType=="uf","univ fwder", fwdType=="lwf", "lightwt fwder",fwdType=="full", "heavy fwder", connectionType=="cooked" or connectionType=="cookedSSL","Splunk fwder", connectionType=="raw" or connectionType=="rawSSL","legacy fwder") | eval Ver=if(isnull(version),"pre 4.2",version) | fields connectionType sourceIp host kb tcp_eps tcp_KBps splunk_server Ver | stats min(_time) as first_time, max(_time) as last_time, max(tcp_KBps) as max_tcp_KBps, avg(tcp_KBps) as avg_tcp_KBps, avg(tcp_eps) as avg_tcp_eps, sum(kb) as sum_kb by connectionType sourceIp host Ver | dedup host | eval sum_mb = round(sum_kb/1024,0) | fields - sum_kb | eval avg_tcp_KBps = round(avg_tcp_KBps,0) | eval max_tcp_KBps = round(max_tcp_KBps,0) | eval class=case( avg_tcp_KBps>0 AND avg_tcp_KBps<=128, "Standard Hosts", avg_tcp_KBps>128 AND avg_tcp_KBps<=256, "Better Hosts", avg_tcp_KBps>256, "Special Hosts") | convert ctime(*time) | rename first_time as "First seen", last_time as "Last seen", avg_tcp_KBps as "AVG KB/s", avg_tcp_eps as "AVG Events/s per 30 seconds", sum_mb as "Total MB", max_tcp_KBps as "Peak Kbps" connectionType AS "Forwarder Type" sourceIp as "Source IP" host AS "Host" Ver As "Splunk Version" | sort - "Peak Kbps" |
This query will show a timechart of the status of an Locked Out Account
|
1 |
sourcetype="WinEventLog:Security" EventCode=4625 AND Status=0xC0000234 | timechart count by user | sort -count |
Use the following search to create a stacked barchart of AD Password change attempts:
|
1 |
source="WinEventLog:Security" "EventCode=4723" src_user!="*$" src_user!="_svc_*" | eval daynumber=strftime(_time,"%Y-%m-%d") | chart count by daynumber, status | eval daynumber = mvindex(split(daynumber,"-"),2) |
The streamstats count command creates a field called eventCount that displays the amount of events from the fieldname you specify:
|
1 |
| streamstats count as eventCount by fieldname |
A few different queries / methods to list all fields for indexes.
|
1 |
index=yourindex| fieldsummary | table field |
or
|
1 |
index=yourindex | stats values(*) AS * | transpose | table column | rename column AS Fieldnames |
or
|
1 |
index=yourindex | stats dc() as * | transpose |
or ;-)
|
1 |
index=yourindex | table * |
