This Splunk REST query will return KV Store Metrics:
|
1 2 3 4 5 6 7 |
| rest /services/server/introspection/kvstore/collectionstats | mvexpand data | spath input=data | rex field=ns "(?<App>.*)\.(?<Collection>.*)" | eval dbsize=round(size/1024/1024, 2) | eval indexsize=round(totalIndexSize/1024/1024, 2) | stats first(count) AS "Number of Objects" first(nindexes) AS Accelerations first(indexsize) AS "Acceleration Size (MB)" first(dbsize) AS "Collection Size (MB)" by App, Collection |
This Splunk REST query will return all non-core applications:
|
1 |
| rest /services/apps/local | search disabled=0 core=0 | table label title version |
The following Splunk search will output historical or real time concurrency in a timechart by host. *NOTE* Change the text <search_head> to your search heads name, alternatively use a *.
|
1 |
<span class="il">index</span>=<span class="il">_internal</span> host=<search_head> source=*metrics.log group=search_concurrency "system total" NOT user=* | timechart max(active_hist_searches) by host |
|
1 |
<span class="il">index</span>=<span class="il">_internal</span> host=<search_head> source=*metrics.log group=search_concurrency "system total" NOT user=* | timechart max(active_realtime_searches) by host |
Description: This dashboard is designed to simplify Splunk’s LDAPSEARCH command. LDAP must be configured in your Splunk instance for this to work.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
<form> <label>LDAP objectClass/CN/OU Search</label> <description>LDAPSEARCH Dashboard.</description> <fieldset submitButton="true" autoRun="false"> <input type="radio" token="objectClass_field"> <label>objectClass</label> <default>*</default> <choice value="*">Any objectClass</choice> <choice value="user">Users</choice> <choice value="computer">Computers</choice> </input> <input type="text" token="cn_field"> <label>CN</label> <default>*</default> </input> <input type="text" token="ou_field"> <label>OU</label> <default>*</default> </input> </fieldset> <row> <panel> <table> <search> <query>| ldapsearch search="(&(cn=$cn_field$)(objectClass=$objectClass_field$))" | rex field=distinguishedName max_match=50 "(?<ou_extraction>OU=([^,]+))" | search ou_extraction="OU=$ou_field$" | rename ou_extraction AS "ou" | table cn, dNSHostName, operatingSystem, ou, objectClass, isCriticalSystemObject, lastLogon, pwdLastSet, description</query> <earliest>-15m</earliest> <latest>now</latest> </search> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> </row> </form> |
This search shows all the alerts that where triggered in your splunk environment:
|
1 |
index=_audit action=alert_fired ss_app=* | eval ttl=expiration-now() | search ttl>0 | convert ctime(trigger_time) | table trigger_time ss_name severity | rename trigger_time as "Alert Time" ss_name as "Alert Name" severity as "Severity" |
Quick snippet to evaluate temperature Fahrentheit to Celsius:
|
1 |
| eval Temperature_Fahrenheit=Temperature_Celsius*1.8+32 |
Want to show what apps have been deployed to forwarders from a deployment server (DS)? Try this Splunk Search:
|
1 |
index=_internal sourcetype=splunkd component=DeployedApplication installing <br />| stats count latest(_time) AS latest_time by host app <br />| convert ctime(latest_time) |
Need a list of Forwarders that are talking to a Deployment Server? Try this:
|
1 |
index=_internal sourcetype=splunkd component=DC* Handshake | stats count by host |
Ever want more detailed information on authentications to your WordPress Admin Area? This Splunk Query will show detailed information on successful authentications to the wp-admin section of your site:
|
1 2 3 4 5 |
sourcetype="access_combined" uri="/wp-admin/admin-ajax.php?_fs_blog_admin=*" | iplocation clientip | stats sparkline latest(_time) as Latest_Date count(status) as count values(status) by uri, Country, Region, City, clientip | convert ctime(Latest_Date) | sort - count |
Screenshot: Notes: Please comment if this is successful or unsuccessful for you, I have limited access to WordPress data. That said this worked for me.
########## Admin Notes This query is a modified version of one submitted by tokenwander here: https://gosplunk.com/whos-using-splunk/ ##########
|
1 2 3 4 5 6 7 8 |
index=_internal sourcetype="splunk_web_access" method="GET" status="200" user!=- | stats count latest(_time) as ViewTime by user app view | sort -count | eventstats sum(count) as countByApp list(view) as view list(count) as count list(ViewTime) as ViewTime by user app | convert timeformat="%a %m/%d/%Y %I:%M:%S %p" ctime(ViewTime) | dedup app | appendpipe [stats sum(count) as count by user | eval view = "Total Views"] | sort + user -countByApp |
This Splunk Query will show hosts that stopped sending logs for at least 48 hours. You’ll want to change the time range to be relevant to your environment, and you may need to tweak the 48 hour range to something that is more appropriate for your environment.
|
1 2 3 4 5 6 7 8 |
| tstats count as countAtToday latest(_time) as lastTime where index!="*_" by host sourcetype index | eval age=now()-lastTime | sort age d | fieldformat lastTime=strftime(lastTime,"%Y/%m/%d %H:%M:%S") | eval age=round((age/60/60),1) | search age>=48 | eval age=age."hour" | dedup host |
This search is still a work in progress, but thought I would go ahead and post it. Currently use OPNsense firewall in my house. The purpose of the search is to identify blocked scanning activity on my firewall that does a 2nd search via a join to add if any src_ip that had been blocked […]
Windows dashboard to help identify users that have either failed or successfully logged in. At the top you have a box I called “Filter” that allows you to insert search parameters in the base search (ex: user=thall). Each panel has it’s own TimeRangePicker and a field selection box which allows you to decide what fields […]
This relies on the search posted earlier: This will display storage and license usage broken down by groups, predefined in the chargeback app customers.csv
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 |
<form> <label>License and Storage Usage</label> <fieldset submitButton="false"> <input type="dropdown" token="grouppicker"> <label>Group</label> <choice value="Group1">Group1</choice> <choice value="Group2">Group2</choice> <choice value="Group3">Group3</choice> <choice value="Group4">Group4</choice> <choice value="Group5">Group5</choice> <choice value="Group6">Group6</choice> <choice value="*">All Groups</choice> <default>*</default> </input> </fieldset> <row> <panel> <title>All Storage Usage</title> <single> <search> <query>| inputlookup size_license_group_usage.csv | search group=$grouppicker$ | stats sum(currentDBSizeMB) AS SplunkIndexSizeMB | eval SplunkIndexSize=case( SplunkIndexSizeMB>=(1024*1024),round(SplunkIndexSizeMB/(1024*1024),2)."TB", SplunkIndexSizeMB>=(1024),round(SplunkIndexSizeMB/(1024),2)."GB", 1=1,SplunkIndexSizeMB."B") | fields SplunkIndexSize</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="drilldown">none</option> <option name="rangeColors">["0x53a051","0x0877a6","0xf8be34","0xf1813f","0xdc4e41"]</option> <option name="refresh.display">progressbar</option> </single> </panel> <panel> <title>Yesterday's License Usage</title> <single> <search> <query>| inputlookup size_license_group_usage.csv | search group=$grouppicker$ | rename sum(Yesterday) AS ydaydata | stats sum(ydaydata) as ydaydata | eval volume_converted=case( ydaydata>=(1024),round(ydaydata/(1024),2)."TB", 1=1,round(ydaydata,2)."GB") | rename volume_converted AS "Yesterday's License Usage" | fields "Yesterday's License Usage"</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="drilldown">none</option> <option name="rangeColors">["0x53a051","0x0877a6","0xf8be34","0xf1813f","0xdc4e41"]</option> <option name="refresh.display">progressbar</option> </single> </panel> </row> <row> <panel> <title>License and Storage for Selected Group by Index</title> <table> <search> <query>| inputlookup size_license_group_usage.csv | search group=$grouppicker$ | rename sum(Average) AS "Average GB License Daily" | rename sum(Yesterday) AS "Yesterdays GB License Usage" | eval "Index Storage Usage MB" = tostring(currentDBSizeMB, "commas") | fields group indexname "Average GB License Daily" "Yesterdays GB License Usage" "Index Storage Usage MB"</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="count">100</option> <option name="dataOverlayMode">none</option> <option name="drilldown">none</option> <option name="percentagesRow">false</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> <format type="number" field="Average GB License Daily"></format> <format type="number" field="Yesterdays GB License Usage"></format> <format type="color" field="group"> <colorPalette type="sharedList"></colorPalette> <scale type="sharedCategory"></scale> </format> </table> </panel> </row> </form> |
This was cobbled together from multiple searches I found. This search feeds the license and storage dashboard posted here: It relies on the Chargeback app for the customers.csv form.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
index=_internal source=*license_usage.log type="Usage" earliest=-30d@d latest=@d | eval indexname = if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx) | eval sourcetypename = st | bin _time span=1d | stats values(poolsz) as poolsz sum(b) as b by _time, pool, indexname, sourcetypename | eval GB=(b/1024/1024/1024) | eval pool=(poolsz/1024/1024/1024) | fields _time, indexname, sourcetypename, GB, pool | eval day=strftime(_time,"%a") | eval yesterday=tostring(strftime(relative_time(now(),"-d@d"),"%Y-%m-%d")) | eval time=tostring(strftime(_time,"%Y-%m-%d")) | eval isyesterday=if(yesterday==time,"true","is_average") | eval Yesterday=if(isyesterday="true",GB,0) | stats values(pool) as pool sum(Yesterday) as Yesterday avg(GB) as Average by indexname, sourcetypename | stats values(pool) as pool list(sourcetypename) sum(Yesterday) sum(Average) by indexname | join indexname [| inputlookup customers.csv | rename idx AS indexname | fields group indexname ] | join indexname [| rest /services/data/indexes search="totalEventCount!=0" | stats sum(currentDBSizeMB) AS currentDBSizeMB by title | rename title AS indexname | fields indexname currentDBSizeMB] | fields group indexname sum(Yesterday) sum(Average) currentDBSizeMB | sort - sum(Yesterday) | outputlookup size_license_group_usage.csv |
|
1 2 3 4 5 6 7 8 9 10 11 12 |
index=_* search=* user=* user!=- user!=splunk-system-user | rex field=search max_match=0 "index\s*=[\s\"]*(?<idx1>.*?)[\|\s\"\)]" | rex field=search max_match=0 "[\+\(|\+]index\%3D(?<idx2>.*?)[\+|\)\+]" | eval idx=if(isnull(idx1), idx2, idx1) | eval frequency=if(source="/opt/splunk/var/log/splunk/splunkd_access.log", "scheduled", "ad-hoc") | eval type=if(match(search, "summary*"), "summary", type1) | eval idx=if(isnull(idx), "NONE", idx) | eval end_type=if(frequency="ad-hoc", "ad-hoc", type) | rename end_type as type | table _time frequency type source user idx search | bin _time span=1h | stats count as count by _time idx user frequency type search |
Do this on HF transforms.conf:
|
1 2 3 4 5 6 |
[discard_gotoips] REGEX = <<<use regex,URL>>> DEST_KEY = queue FORMAT = nullQueue |
props.conf:
|
1 2 3 4 5 |
[default] TRANSFORMS-null = discard_gotoips File location: /etc/system/local |
I often get asked how much a certain dashboard gets looked at, or how many times a user looks at a specific app. I wrote this quick query to answer that question.
|
1 2 3 4 |
index=_internal sourcetype="splunk_web_access" method="GET" status="200" | stats count as count by user, view | appendpipe [stats sum(count) as count by user | eval view = "Total Views"] | sort + user |
Use this search to find unused dashboards:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
| rest /servicesNS/-/-/data/ui/views splunk_server=* | search isDashboard=1 | rename eai:acl.app as app | fields title app | join type=left title [| search index=_internal sourcetype=splunk_web_access host=* user=* | rex field=uri_path ".*/(?<title>[^/]*)$" | stats latest(_time) as Time latest(user) as user by title ] | where isnotnull(Time) | eval Now=now() | eval "Days since last accessed"=round((Now-Time)/86400,2) | sort - "Days since last accessed" | convert ctime(Time) | fields - Now |
Admin Notes – Fantastic query! I modified the SPL slightly as I had an issue when I copied it to my two test environments.
Quickly identify high amounts of skipped searches in your cluster or standalone SH(s):
|
1 2 3 |
index = _internal skipped sourcetype=scheduler status=skipped host=[your splunk SH(s)] | stats count by app search_type reason savedsearch_name | sort -count |
Adjust “[your splunk SH(s)]” to the SH(s) you want to check obviously ;)
