Use the following search with a column chart visualisation. It will give you a good overview of what Splunk processes use the most memory: index=_introspection sourcetype=splunk_resource_usage component=PerProcess host=* | eval process = ‘data.process’ | eval args = ‘data.args’ | eval sid = ‘data.search_props.sid’ | eval elapsed = ‘data.elapsed’ | eval mem_used = ‘data.mem_used’ | eval mem […]
Universal Forwarder Throughput Limit Hit Count
This search counts the amount of times the UF’s throughput limit is hit. I also threw in a sparkline: index=_internal sourcetype=splunkd “current data throughput” | rex “Current data throughput \((?<kb>\S+)” | eval rate=case(kb < 500, “256”, kb > 499 AND kb < 520, “512”, kb > 520 AND kb < 770 ,”768″, kb>771 AND kb<1210, […]
Top 10 Accessed Dashboards
Where “host=”your_sh_host”” you could specify a host, or put a wildcard * in place. index=”_internal” source=*access.log user!=”-” */app/* (host=”your_sh_host”) | rex field=referer “/en-US/app/(?<app>[^/]+)/(?<dashboard>[^?/\s]+)” | top 10 dashboard
Universal Forwarder Throughput Statistics
This search creates a table to list all Universal Forwarders. There is also an eval in there that classifies hosts based on their average Kbps. You can modify this as needed. index=_internal source=*metrics.log group=tcpin_connections splunk_server=* | eval host=if(isnull(hostname), sourceHost,hostname) | search (host=*) AND (host!=”(ALL)”) | eval connectionType=case(fwdType==”uf”,”univ fwder”, fwdType==”lwf”, “lightwt fwder”,fwdType==”full”, “heavy fwder”, connectionType==”cooked” or […]
Timechart of the status of an Locked Out Account
This query will show a timechart of the status of an Locked Out Account sourcetype=”WinEventLog:Security” EventCode=4625 AND Status=0xC0000234 | timechart count by user | sort -count
Active Directory Password change attempts
Use the following search to create a stacked barchart of AD Password change attempts: source=”WinEventLog:Security” “EventCode=4723″ src_user!=”*$” src_user!=”_svc_*” | eval daynumber=strftime(_time,”%Y-%m-%d”) | chart count by daynumber, status | eval daynumber = mvindex(split(daynumber,”-“),2)
Add a count of events by fieldname
The streamstats count command creates a field called eventCount that displays the amount of events from the fieldname you specify: | streamstats count as eventCount by fieldname
List all fields for an index
A few different queries / methods to list all fields for indexes. index=yourindex| fieldsummary | table field or index=yourindex | stats values(*) AS * | transpose | table column | rename column AS Fieldnames or index=yourindex | stats dc() as * | transpose or ;-) index=yourindex | table *
Search for duplicate events in Splunk
index=<indexname> | stats count values(host) values(source) values(sourcetype) values(index) by _raw | WHERE count>1
Search for all errors in splunkd
index=_internal sourcetype=”splunkd” log_level=”ERROR” | stats sparkline count dc(host) as hosts last(_raw) as last_raw_msg values(sourcetype) as sourcetype last(_time) as last_msg_time first(_time) as first_msg_time values(index) as index by punct | eval delta=round((first_msg_time-last_msg_time),2) | eval msg_per_sec=round((count/delta),2) | convert ctime(last_msg_time) ctime(first_msg_time) | table last_raw_msg count hosts sparkline msg_per_sec sourcetype index first_msg_time last_msg_time delta | sort -count
Easter egg that created sample data
| windbag This command creates a set of sample data of 100 events
