In my previous role I created this dashboard to identify how much data a Splunk forwarder had sent to my indexers. This was a daily check that either myself of someone on my team would review. This check helped us identify a misconfiguration across all of my production Windows servers. I was able to drilldown […]
Windows Sysmon Process Dashboard
Working with a customer I started this dashboard to give a high level overview of Windows Sysmon data. I have been evolving the dashboard in my home environment and will take any feedback to improve the effectiveness of this dashboard. First is getting sysmon data into your splunk environment. My home computers are running Windows […]
Get KV Store Metrics
This Splunk REST query will return KV Store Metrics:
|
1 2 3 4 5 6 7 |
| rest /services/server/introspection/kvstore/collectionstats | mvexpand data | spath input=data | rex field=ns "(?<App>.*)\.(?<Collection>.*)" | eval dbsize=round(size/1024/1024, 2) | eval indexsize=round(totalIndexSize/1024/1024, 2) | stats first(count) AS "Number of Objects" first(nindexes) AS Accelerations first(indexsize) AS "Acceleration Size (MB)" first(dbsize) AS "Collection Size (MB)" by App, Collection |
List of installed non-core applications
This Splunk REST query will return all non-core applications:
|
1 |
| rest /services/apps/local | search disabled=0 core=0 | table label title version |
Searches to check search concurrency for historical or real time
The following Splunk search will output historical or real time concurrency in a timechart by host. *NOTE* Change the text <search_head> to your search heads name, alternatively use a *.
|
1 |
<span class="il">index</span>=<span class="il">_internal</span> host=<search_head> source=*metrics.log group=search_concurrency "system total" NOT user=* | timechart max(active_hist_searches) by host |
|
1 |
<span class="il">index</span>=<span class="il">_internal</span> host=<search_head> source=*metrics.log group=search_concurrency "system total" NOT user=* | timechart max(active_realtime_searches) by host |
LDAP Search Dashboard
Description: This dashboard is designed to simplify Splunk’s LDAPSEARCH command. LDAP must be configured in your Splunk instance for this to work.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
<form> <label>LDAP objectClass/CN/OU Search</label> <description>LDAPSEARCH Dashboard.</description> <fieldset submitButton="true" autoRun="false"> <input type="radio" token="objectClass_field"> <label>objectClass</label> <default>*</default> <choice value="*">Any objectClass</choice> <choice value="user">Users</choice> <choice value="computer">Computers</choice> </input> <input type="text" token="cn_field"> <label>CN</label> <default>*</default> </input> <input type="text" token="ou_field"> <label>OU</label> <default>*</default> </input> </fieldset> <row> <panel> <table> <search> <query>| ldapsearch search="(&(cn=$cn_field$)(objectClass=$objectClass_field$))" | rex field=distinguishedName max_match=50 "(?<ou_extraction>OU=([^,]+))" | search ou_extraction="OU=$ou_field$" | rename ou_extraction AS "ou" | table cn, dNSHostName, operatingSystem, ou, objectClass, isCriticalSystemObject, lastLogon, pwdLastSet, description</query> <earliest>-15m</earliest> <latest>now</latest> </search> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> </row> </form> |
Show your triggered alerts
This search shows all the alerts that where triggered in your splunk environment:
|
1 |
index=_audit action=alert_fired ss_app=* | eval ttl=expiration-now() | search ttl>0 | convert ctime(trigger_time) | table trigger_time ss_name severity | rename trigger_time as "Alert Time" ss_name as "Alert Name" severity as "Severity" |
Evaluate Fahrenheit to Celsius
Quick snippet to evaluate temperature Fahrentheit to Celsius:
|
1 |
| eval Temperature_Fahrenheit=Temperature_Celsius*1.8+32 |
Apps Deployed from Deployment Server
Want to show what apps have been deployed to forwarders from a deployment server (DS)? Try this Splunk Search:
|
1 |
index=_internal sourcetype=splunkd component=DeployedApplication installing <br />| stats count latest(_time) AS latest_time by host app <br />| convert ctime(latest_time) |
List of Forwarders that are Deployment Clients
Need a list of Forwarders that are talking to a Deployment Server? Try this:
|
1 |
index=_internal sourcetype=splunkd component=DC* Handshake | stats count by host |
Successful Logons to WordPress Admin Area
Ever want more detailed information on authentications to your WordPress Admin Area? This Splunk Query will show detailed information on successful authentications to the wp-admin section of your site:
|
1 2 3 4 5 |
sourcetype="access_combined" uri="/wp-admin/admin-ajax.php?_fs_blog_admin=*" | iplocation clientip | stats sparkline latest(_time) as Latest_Date count(status) as count values(status) by uri, Country, Region, City, clientip | convert ctime(Latest_Date) | sort - count |
Screenshot: Notes: Please comment if this is successful or unsuccessful for you, I have limited access to WordPress data. That said this worked for me.
See who is using Splunk by user, app and view
########## Admin Notes This query is a modified version of one submitted by tokenwander here: https://gosplunk.com/whos-using-splunk/ ##########
|
1 2 3 4 5 6 7 8 |
index=_internal sourcetype="splunk_web_access" method="GET" status="200" user!=- | stats count latest(_time) as ViewTime by user app view | sort -count | eventstats sum(count) as countByApp list(view) as view list(count) as count list(ViewTime) as ViewTime by user app | convert timeformat="%a %m/%d/%Y %I:%M:%S %p" ctime(ViewTime) | dedup app | appendpipe [stats sum(count) as count by user | eval view = "Total Views"] | sort + user -countByApp |
Host not sending logs for x days
This Splunk Query will show hosts that stopped sending logs for at least 48 hours. You’ll want to change the time range to be relevant to your environment, and you may need to tweak the 48 hour range to something that is more appropriate for your environment.
|
1 2 3 4 5 6 7 8 |
| tstats count as countAtToday latest(_time) as lastTime where index!="*_" by host sourcetype index | eval age=now()-lastTime | sort age d | fieldformat lastTime=strftime(lastTime,"%Y/%m/%d %H:%M:%S") | eval age=round((age/60/60),1) | search age>=48 | eval age=age."hour" | dedup host |
Blocked Firewall Scanning Activity with indicator if Source has been allowed.
This search is still a work in progress, but thought I would go ahead and post it. Currently use OPNsense firewall in my house. The purpose of the search is to identify blocked scanning activity on my firewall that does a 2nd search via a join to add if any src_ip that had been blocked […]
Windows Logon Dashboard
Windows dashboard to help identify users that have either failed or successfully logged in. At the top you have a box I called “Filter” that allows you to insert search parameters in the base search (ex: user=thall). Each panel has it’s own TimeRangePicker and a field selection box which allows you to decide what fields […]
License and Storage Usage Dashboard
This relies on the search posted earlier: This will display storage and license usage broken down by groups, predefined in the chargeback app customers.csv
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 |
<form> <label>License and Storage Usage</label> <fieldset submitButton="false"> <input type="dropdown" token="grouppicker"> <label>Group</label> <choice value="Group1">Group1</choice> <choice value="Group2">Group2</choice> <choice value="Group3">Group3</choice> <choice value="Group4">Group4</choice> <choice value="Group5">Group5</choice> <choice value="Group6">Group6</choice> <choice value="*">All Groups</choice> <default>*</default> </input> </fieldset> <row> <panel> <title>All Storage Usage</title> <single> <search> <query>| inputlookup size_license_group_usage.csv | search group=$grouppicker$ | stats sum(currentDBSizeMB) AS SplunkIndexSizeMB | eval SplunkIndexSize=case( SplunkIndexSizeMB>=(1024*1024),round(SplunkIndexSizeMB/(1024*1024),2)."TB", SplunkIndexSizeMB>=(1024),round(SplunkIndexSizeMB/(1024),2)."GB", 1=1,SplunkIndexSizeMB."B") | fields SplunkIndexSize</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="drilldown">none</option> <option name="rangeColors">["0x53a051","0x0877a6","0xf8be34","0xf1813f","0xdc4e41"]</option> <option name="refresh.display">progressbar</option> </single> </panel> <panel> <title>Yesterday's License Usage</title> <single> <search> <query>| inputlookup size_license_group_usage.csv | search group=$grouppicker$ | rename sum(Yesterday) AS ydaydata | stats sum(ydaydata) as ydaydata | eval volume_converted=case( ydaydata>=(1024),round(ydaydata/(1024),2)."TB", 1=1,round(ydaydata,2)."GB") | rename volume_converted AS "Yesterday's License Usage" | fields "Yesterday's License Usage"</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="drilldown">none</option> <option name="rangeColors">["0x53a051","0x0877a6","0xf8be34","0xf1813f","0xdc4e41"]</option> <option name="refresh.display">progressbar</option> </single> </panel> </row> <row> <panel> <title>License and Storage for Selected Group by Index</title> <table> <search> <query>| inputlookup size_license_group_usage.csv | search group=$grouppicker$ | rename sum(Average) AS "Average GB License Daily" | rename sum(Yesterday) AS "Yesterdays GB License Usage" | eval "Index Storage Usage MB" = tostring(currentDBSizeMB, "commas") | fields group indexname "Average GB License Daily" "Yesterdays GB License Usage" "Index Storage Usage MB"</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="count">100</option> <option name="dataOverlayMode">none</option> <option name="drilldown">none</option> <option name="percentagesRow">false</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> <format type="number" field="Average GB License Daily"></format> <format type="number" field="Yesterdays GB License Usage"></format> <format type="color" field="group"> <colorPalette type="sharedList"></colorPalette> <scale type="sharedCategory"></scale> </format> </table> </panel> </row> </form> |
Build License usage by Group
This was cobbled together from multiple searches I found. This search feeds the license and storage dashboard posted here: It relies on the Chargeback app for the customers.csv form.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
index=_internal source=*license_usage.log type="Usage" earliest=-30d@d latest=@d | eval indexname = if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx) | eval sourcetypename = st | bin _time span=1d | stats values(poolsz) as poolsz sum(b) as b by _time, pool, indexname, sourcetypename | eval GB=(b/1024/1024/1024) | eval pool=(poolsz/1024/1024/1024) | fields _time, indexname, sourcetypename, GB, pool | eval day=strftime(_time,"%a") | eval yesterday=tostring(strftime(relative_time(now(),"-d@d"),"%Y-%m-%d")) | eval time=tostring(strftime(_time,"%Y-%m-%d")) | eval isyesterday=if(yesterday==time,"true","is_average") | eval Yesterday=if(isyesterday="true",GB,0) | stats values(pool) as pool sum(Yesterday) as Yesterday avg(GB) as Average by indexname, sourcetypename | stats values(pool) as pool list(sourcetypename) sum(Yesterday) sum(Average) by indexname | join indexname [| inputlookup customers.csv | rename idx AS indexname | fields group indexname ] | join indexname [| rest /services/data/indexes search="totalEventCount!=0" | stats sum(currentDBSizeMB) AS currentDBSizeMB by title | rename title AS indexname | fields indexname currentDBSizeMB] | fields group indexname sum(Yesterday) sum(Average) currentDBSizeMB | sort - sum(Yesterday) | outputlookup size_license_group_usage.csv |
Detailed User Activity
|
1 2 3 4 5 6 7 8 9 10 11 12 |
index=_* search=* user=* user!=- user!=splunk-system-user | rex field=search max_match=0 "index\s*=[\s\"]*(?<idx1>.*?)[\|\s\"\)]" | rex field=search max_match=0 "[\+\(|\+]index\%3D(?<idx2>.*?)[\+|\)\+]" | eval idx=if(isnull(idx1), idx2, idx1) | eval frequency=if(source="/opt/splunk/var/log/splunk/splunkd_access.log", "scheduled", "ad-hoc") | eval type=if(match(search, "summary*"), "summary", type1) | eval idx=if(isnull(idx), "NONE", idx) | eval end_type=if(frequency="ad-hoc", "ad-hoc", type) | rename end_type as type | table _time frequency type source user idx search | bin _time span=1h | stats count as count by _time idx user frequency type search |
Exclude single event type from logs
Do this on HF transforms.conf:
|
1 2 3 4 5 6 |
[discard_gotoips] REGEX = <<<use regex,URL>>> DEST_KEY = queue FORMAT = nullQueue |
props.conf:
|
1 2 3 4 5 |
[default] TRANSFORMS-null = discard_gotoips File location: /etc/system/local |
Who’s Using Splunk?
I often get asked how much a certain dashboard gets looked at, or how many times a user looks at a specific app. I wrote this quick query to answer that question.
|
1 2 3 4 |
index=_internal sourcetype="splunk_web_access" method="GET" status="200" | stats count as count by user, view | appendpipe [stats sum(count) as count by user | eval view = "Total Views"] | sort + user |
