The following Splunk query will show a table of all users and their roles:
|
1 |
| rest /services/authentication/users | stats values(roles) as Roles by user |
Dashboard and App views by user
This Splunk query / search shows historical access to dashboards and apps on a local splunk server.
|
1 2 3 4 5 6 7 8 9 |
index=_internal sourcetype=splunk_web_access host=* user=* | rex field=uri_path ".*/(?<title>[^/]*)$" | join title [| rest /servicesNS/-/-/data/ui/views splunk_server=* | search isDashboard=1 isVisible=1 | rename eai:acl.app as app | fields title app ] | rename title as dashboard | stats count by _time user dashboard app host |
Show how much disk space is used by _internal
The following Splunk query will return disk space used by the _internal index.
|
1 |
index=_internal source=*license_usage.log type=Usage | eval gb=b/1024/1024/1024 | timechart span=1d sum(gb) as GB by host useother=false | untable _time host gb | top limit=1 host | join time [search index=_internal source=*license_usage.log type=Usage | eval gb=b/1024/1024/1024 | timechart span=1d sum(gb) as GB by sourcetype useother=false |untable _time sourcteype sum(GB)] |
Find queues that are nearly full
Easily find Splunk Instances (both forwarders and full installs) that have queues that are over 80% full!
|
1 |
index=_internal group=queue | eval percfull=((current_size_kb/max_size_kb)*100) | search percfull>80 | dedup host, name | table _time host name current_size_kb max_size_kb |
Events Sent to Null Que – Internal Logs
This will show events that have been sent to the null que within the splunk internal logs.
|
1 |
index=_internal component=metrics processor=nullqueue group=pipeline sourcetype=splunkd | table _time log_level name processor cpu_seconds executes, cumulative_hits |
REST Call for Memory & CPU usage on Splunk Servers
This Splunk search will show you use and available CPU and Memory statistics. Depending on your environment you may see multiple Splunk servers:
|
1 |
| rest /services/server/status/resource-usage/hostwide | eval cpu_count = if(isnull(cpu_count), "N/A", cpu_count) | eval cpu_usage = cpu_system_pct + cpu_user_pct | eval mem_used_pct = round(mem_used / mem * 100 , 2) | eval mem_used = round(mem_used, 0) | eval mem = round(mem, 0) |eval mem=tostring(mem, "commas") | eval mem_used=tostring(mem_used, "commas")| fields splunk_server, cpu_count, cpu_usage, mem, mem_used, mem_used_pct | sort - cpu_usage, -mem_used | rename splunk_server AS Instance, cpu_count AS "CPU Cores", cpu_usage AS "CPU Usage (%)", mem AS "Physical Memory Capacity (MB)", mem_used AS "Physical Memory Usage (MB)", mem_used_pct AS "Physical Memory Usage (%)" |
Slightly modified from: http://www.brainfold.net/2016/03/frequently-used-rest-api-calls-in-splunk.html
REST Call for a list of Lookup Files
Use this splunk search to get a list of all lookup files:
|
1 |
| rest /services/data/transforms/lookups | table eai:acl.app eai:appName filename title fields_list updated id |
REST Call for Splunk Server Role Status
This REST Splunk search returns the status of roles on each Splunk server in your environment.
|
1 |
| rest /services/server/introspection | table title splunk_server status updated |
Splunk Objects With Permissions Granted to Non-existent Roles
Useful search to show a bit of detail on roles and user permissions.
|
1 2 3 4 5 6 7 8 9 10 11 |
| rest /servicesNS/-/-/admin/directory count=0 splunk_server=local | fields eai:acl.app, eai:acl.owner, eai:acl.perms.*, eai:acl.sharing, eai:location, title | eval perms=mvappend('eai:acl.perms.read','eai:acl.perms.write') | fields - eai:acl.perms.* | mvexpand perms | where perms!="*" AND NOT [ | rest /servicesNS/-/-/authorization/roles count=0 splunk_server=local | fields title | rename title as perms ] |
I found this at: https://gist.github.com/acharlieh/3254a7ab13297c760376 Credit goes to acharlieh!
Every index explicitly granted to a role
Self explanatory, maps roles to indexes. Useful if you have a lot of indexes!
|
1 2 3 4 5 |
| rest /servicesNS/-/-/authorization/roles count=0 splunk_server=local | fields title,srchIndexesAllowed | rename srchIndexesAllowed as index title as role | mvexpand index | where NOT match(index,".*\*.*") |
I found this at: https://gist.github.com/acharlieh/3254a7ab13297c760376 Credit goes to acharlieh!
Average Splunk Web requests by hour
This query is pretty awesome! It helped enlighten us to exactly when our splunk infrastructure is being hit with users
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
index=_internal sourcetype=splunk_web_access [ rest / splunk_server=local | fields splunk_server | rename splunk_server as host ] | bin _time span=1d | stats count by date_hour _time | appendpipe [ fields _time | dedup _time | eval date_hour=mvrange(0,24,1) | eval count=0 | mvexpand date_hour ] | stats sum(count) as count by date_hour _time | stats avg(count) as avg by date_hour | sort date_hour |
I found this at: https://gist.github.com/acharlieh/3254a7ab13297c760376 Credit goes to acharlieh!
All indexes not explicitly granted to a role
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
| rest /servicesNS/-/-/data/indexes count=0 | stats max(isInternal) as internal, max(disabled) as disabled, max(isReadOnly) as readonly by title | fillnull | where internal=0 AND disabled=0 AND readonly=0 | fields title | rename title as index | join index type=left [ rest /servicesNS/-/-/authorization/roles count=0 splunk_server=local | fields title,srchIndexesAllowed | rename srchIndexesAllowed as index title as role | mvexpand index | where NOT match(index,".*\*.*") ] | search NOT role=* | fields index |
I found this at: https://gist.github.com/acharlieh/3254a7ab13297c760376 Credit goes to acharlieh!
List of Sourcetypes Sent by Forwarder
The following Splunk search will show a list of sourcetypes (or perhaps sources I need to confirm) being sent by each Forwarder:
|
1 |
index="_internal" |where host!=splunk_server | stats values(series) as Sourcetypes by host |
Detailed list of Errors Per Host
The following Splunk search will return a detailed list (by message) of errors associated with hosts running a universal forwarder:
|
1 |
index=_internal sourcetype="splunkd" log_level="ERROR" | stats sparkline count dc(host) as uniqhosts last(message) as message last(_time) as last first(_time) as first by punct | convert ctime(last) ctime(first) | table message count uniqhosts sparkline first last | sort -count | rename message as "Error Output" count as Count uniqhosts as "Number of Hosts" first as "First Occurance" last as "Most Recent Occurance" |
Failed Attempts to Logon to Splunk Web
The following Splunk Search Query will return all users who have failed to logon to the Splunk Web console. This query will also include an average (from eventstats).
|
1 |
index=_audit action="login attempt" info=failed | timechart count(user) as Failed_Attempts| eventstats avg(Failed_Attempts) as Average |
Detailed list of Universal Forwarders Reporting to Indexer
The following query will list in detail information on the universal forwarders checking into the indexer. I’ve renamed some of the fields to be more user-friendly.
|
1 |
index=_internal sourcetype=splunkd destPort!="-"| stats sparkline count by hostname, sourceHost, host, destPort, version | rename destPort as "Destination Port" | rename host as "Indexer" | rename sourceHost as "Universal Forwarder IP" | rename version as "Splunk Forwarder Version" | rename hostname as "Universal Forwarder Host Name" | rename sparkline as "Traffic Frequency" | sort - count |
