Show how much disk space is used by _internal

The following Splunk query will return disk space used by the _internal index. index=_internal source=*license_usage.log type=Usage | eval gb=b/1024/1024/1024 | timechart span=1d sum(gb) as GB by host useother=false | untable _time host gb | top limit=1 host | join time [ search index=_internal source=*license_usage.log type=Usage | eval gb=b/1024/1024/1024 | timechart span=1d sum(gb) as GB by […]

Continue Reading →

License Usage Prediction

There is an older Splunk query here that had previously predicted license usage. I’m not sure why (perhaps the predict command has changed since the original post in 2015?), but the query is no longer working. I’ve updated the query to predict Splunk license usage using the Splunk predict command as shown below: index=_internal source=”*license_usage.lo*” […]

Continue Reading →

Average Search Duration

Ever wonder how your search performance is across search heads? Try this query. Depending on your environment you’ll want to specify the host=* section to better represent your environment. Say if you have a naming convention that includes “shc” and a number representing searchheads in a cluster (distributed environment) you can use (host=shc1.fq.dn OR host=shc2.fq.dn […]

Continue Reading →

User Info Dashboard – Using REST

I found this very useful user statistics/information splunk dashboard on www.function1.com/2016/06/rest-easy-with-the-splunk-rest-api. They have additional Splunk REST queries and examples worth checking out! <dashboard>       <label>REST API: access control</label>       <row>         <panel>           <single>             <title>You are</title>             <searchString>| rest /services/authentication/current-context | where NOT username=”splunk-system-user” | fields username</searchString>             <earliestTime>0</earliestTime>             <latestTime/>             <option name=”drilldown”>none</option> […]

Continue Reading →

Use REST to gather Index Info

Here is some SPL to get useful information via REST on indexes within your Splunk environment: | REST /services/data/indexes | eval currentDBSizeMB=tostring(currentDBSizeMB, “commas”) | eval totalEventCount=tostring(totalEventCount, “commas”) | eval frozenTimePeriodInHours=(frozenTimePeriodInSecs/60/60) | table title splunk_server currentDBSizeMB frozenTimePeriodInHours maxTime minTime totalEventCount

Continue Reading →

List Inputs using REST

As the title says. Pretty nice Splunk Search if you’ve forgotten what inputs you have configured and need a central place to list them. | rest /services/data/inputs/all | convert ctime(starttime) AS “Start Time”  | convert ctime(endtime) AS “End Time” | table index interval source sourcetype title updated starttime endtime “Start Time” “End Time”

Continue Reading →

REST Call for Memory & CPU usage on Splunk Servers

This Splunk search will show you use and available CPU and Memory statistics. Depending on your environment you may see multiple Splunk servers: | rest /services/server/status/resource-usage/hostwide | eval cpu_count = if(isnull(cpu_count), “N/A”, cpu_count) | eval cpu_usage = cpu_system_pct + cpu_user_pct | eval mem_used_pct = round(mem_used / mem * 100 , 2) | eval mem_used = […]

Continue Reading →

Splunk Objects With Permissions Granted to Non-existent Roles

Useful search to show a bit of detail on roles and user permissions. | rest /servicesNS/-/-/admin/directory count=0 splunk_server=local | fields eai:acl.app, eai:acl.owner, eai:acl.perms.*, eai:acl.sharing, eai:location, title | eval perms=mvappend(‘eai:acl.perms.read’,’eai:acl.perms.write’) | fields – eai:acl.perms.* | mvexpand perms | where perms!=”*” AND NOT [ | rest /servicesNS/-/-/authorization/roles count=0 splunk_server=local | fields title | rename title as perms […]

Continue Reading →

Every index explicitly granted to a role

Self explanatory, maps roles to indexes. Useful if you have a lot of indexes! | rest /servicesNS/-/-/authorization/roles count=0 splunk_server=local | fields title,srchIndexesAllowed | rename srchIndexesAllowed as index title as role | mvexpand index | where NOT match(index,”.*\*.*”) I found this at: https://gist.github.com/acharlieh/3254a7ab13297c760376 Credit goes to acharlieh!

Continue Reading →

Average Splunk Web requests by hour

This query is pretty awesome! It helped enlighten us to exactly when our splunk infrastructure is being hit with users index=_internal sourcetype=splunk_web_access [ rest / splunk_server=local | fields splunk_server | rename splunk_server as host ] | bin _time span=1d | stats count by date_hour _time | appendpipe [ fields _time | dedup _time | eval […]

Continue Reading →

All indexes not explicitly granted to a role

| rest /servicesNS/-/-/data/indexes count=0 | stats max(isInternal) as internal, max(disabled) as disabled, max(isReadOnly) as readonly by title | fillnull | where internal=0 AND disabled=0 AND readonly=0 | fields title | rename title as index | join index type=left [ rest /servicesNS/-/-/authorization/roles count=0 splunk_server=local | fields title,srchIndexesAllowed | rename srchIndexesAllowed as index title as role | […]

Continue Reading →

Malware Detection

I’m reposting this query I stumbled upon in a blog here. The description states that it can be used to detect malware reporting out to the web. Check out the article it’s a decent read. search.goes.here | convert mktime(_time) as epoch | sort 0 uri_host,client_ip,epoch | delta epoch as epoch_delta | search epoch_delta>0 epoch_delta<30 | chart […]

Continue Reading →

Detailed list of Errors Per Host

The following Splunk search will return a detailed list (by message) of errors associated with hosts running a universal forwarder: index=_internal sourcetype=”splunkd” log_level=”ERROR” | stats sparkline count dc(host) as uniqhosts last(message) as message last(_time) as last first(_time) as first by punct  | convert ctime(last) ctime(first) | table message count uniqhosts sparkline first last | sort […]

Continue Reading →