Disk Usage per Index by Indexer

Summary: Instead of grabbing data from all time, using the dbinspect command will allow administrators to quickly determine how big an index is.  There are additional fields in the dbinspect, so explore that to gain other data pivots.   |dbinspect index=_internal | stats sum(sizeOnDiskMB) by splunk_server

Continue Reading →

Apdex Score

Apdex Score Apdex is a measure of response time based against a set threshold. It measures the ratio of satisfactory response times to unsatisfactory response times. The response time is measured from an asset request to completed delivery back to the requestor. It determines user satisfaction, and is based on request type & response time. All […]

Continue Reading →

Data Usage for Indexer and Forwarders

In my previous role I created this dashboard to identify how much data a Splunk forwarder had sent to my indexers.  This was a daily check that either myself of someone on my team would review.  This check helped us identify a misconfiguration across all of my production Windows servers.  I was able to drilldown […]

Continue Reading →

Windows Sysmon Process Dashboard

(updated on 8/26/2020) Working with a customer I started this dashboard to give a high level overview of Windows Sysmon data.  I have been evolving the dashboard in my home environment and will take any feedback to improve the effectiveness of this dashboard. First is getting sysmon data into your splunk environment.  My home computers […]

Continue Reading →

Get KV Store Metrics

This Splunk REST query will return KV Store Metrics: | rest /services/server/introspection/kvstore/collectionstats | mvexpand data | spath input=data | rex field=ns “(?<App>.*)\.(?<Collection>.*)” | eval dbsize=round(size/1024/1024, 2) | eval indexsize=round(totalIndexSize/1024/1024, 2) | stats first(count) AS “Number of Objects” first(nindexes) AS Accelerations first(indexsize) AS “Acceleration Size (MB)” first(dbsize) AS “Collection Size (MB)” by App, Collection

Continue Reading →

Searches to check search concurrency for historical or real time

The following Splunk search will output historical or real time concurrency in a timechart by host. *NOTE* Change the text <search_head> to your search heads name, alternatively use a *. index=_internal host= source=*metrics.log group=search_concurrency “system total” NOT user=* | timechart max(active_hist_searches) by host index=_internal host= source=*metrics.log group=search_concurrency “system total” NOT user=* | timechart max(active_realtime_searches) by […]

Continue Reading →

LDAP Search Dashboard

Description: This dashboard is designed to simplify Splunk’s LDAPSEARCH command. LDAP must be configured in your Splunk instance for this to work.   <form> <label>LDAP objectClass/CN/OU Search</label> <description>LDAPSEARCH Dashboard.</description> <fieldset submitButton=”true” autoRun=”false”> <input type=”radio” token=”objectClass_field”> <label>objectClass</label> <default>*</default> <choice value=”*”>Any objectClass</choice> <choice value=”user”>Users</choice> <choice value=”computer”>Computers</choice> </input> <input type=”text” token=”cn_field”> <label>CN</label> <default>*</default> </input> <input type=”text” token=”ou_field”> <label>OU</label> […]

Continue Reading →

Show your triggered alerts

This search shows all the alerts that where triggered in your splunk environment: index=_audit action=alert_fired ss_app=* | eval ttl=expiration-now() | search ttl>0 | convert ctime(trigger_time) | table trigger_time ss_name severity | rename trigger_time as “Alert Time” ss_name as “Alert Name” severity as “Severity”

Continue Reading →

List of Forwarders that are Deployment Clients

Need a list of Forwarders that are talking to a Deployment Server? Try this: index=_internal sourcetype=splunkd component=DC* Handshake | stats count by host Additional REST query (performed on the DS) will return desired results (Thanks to Lyx for pointing this out!): | rest /services/deployment/server/clients splunk_server=local | table hostname applications.*.serverclasses | untable hostname, applications, serverclass | […]

Continue Reading →

Successful Logons to WordPress Admin Area

Ever want more detailed information on authentications to your WordPress Admin Area? This Splunk Query will show detailed information on successful authentications to the wp-admin section of your site: sourcetype=”access_combined” uri=”/wp-admin/admin-ajax.php?_fs_blog_admin=*” | iplocation clientip | stats sparkline latest(_time) as Latest_Date count(status) as count values(status) by uri, Country, Region, City, clientip | convert ctime(Latest_Date) | sort […]

Continue Reading →

See who is using Splunk by user, app and view

########## Admin Notes This query is a modified version of one submitted by tokenwander here: https://gosplunk.com/whos-using-splunk/ ########## index=_internal sourcetype=”splunk_web_access” method=”GET” status=”200″ user!=- | stats count latest(_time) as ViewTime by user app view | sort -count | eventstats sum(count) as countByApp list(view) as view list(count) as count list(ViewTime) as ViewTime by user app | convert timeformat=”%a […]

Continue Reading →

Host not sending logs for x days

This Splunk Query will show hosts that stopped sending logs for at least 48 hours. You’ll want to change the time range to be relevant to your environment, and you may need to tweak the 48 hour range to something that is more appropriate for your environment. | tstats count as countAtToday latest(_time) as lastTime […]

Continue Reading →

License and Storage Usage Dashboard

This relies on the search posted earlier: This will display storage and license usage broken down by groups, predefined in the chargeback app customers.csv <form> <label>License and Storage Usage</label> <fieldset submitButton=”false”> <input type=”dropdown” token=”grouppicker”> <label>Group</label> <choice value=”Group1″>Group1</choice> <choice value=”Group2″>Group2</choice> <choice value=”Group3″>Group3</choice> <choice value=”Group4″>Group4</choice> <choice value=”Group5″>Group5</choice> <choice value=”Group6″>Group6</choice> <choice value=”*”>All Groups</choice> <default>*</default> </input> </fieldset> <row> <panel> […]

Continue Reading →

Build License usage by Group

This was cobbled together from multiple searches I found. This search feeds the license and storage dashboard posted here: It relies on the Chargeback app for the customers.csv form. index=_internal source=*license_usage.log type=”Usage” earliest=-30d@d latest=@d | eval indexname = if(len(idx)=0 OR isnull(idx),”(UNKNOWN)”,idx) | eval sourcetypename = st | bin _time span=1d | stats values(poolsz) as poolsz […]

Continue Reading →

Detailed User Activity

index=_* search=* user=* user!=- user!=splunk-system-user | rex field=search max_match=0 “index\s*=[\s\”]*(?<idx1>.*?)[\|\s\”\)]” | rex field=search max_match=0 “[\+\(|\+]index\%3D(?<idx2>.*?)[\+|\)\+]” | eval idx=if(isnull(idx1), idx2, idx1) | eval frequency=if(source=”/opt/splunk/var/log/splunk/splunkd_access.log”, “scheduled”, “ad-hoc”) | eval type=if(match(search, “summary*”), “summary”, type1) | eval idx=if(isnull(idx), “NONE”, idx) | eval end_type=if(frequency=”ad-hoc”, “ad-hoc”, type) | rename end_type as type | table _time frequency type source user idx search […]

Continue Reading →