List of Forwarders that are Deployment Clients

Need a list of Forwarders that are talking to a Deployment Server? Try this: index=_internal sourcetype=splunkd component=DC* Handshake | stats count by host Additional REST query (performed on the DS) will return desired results (Thanks to Lyx for pointing this out!): | rest /services/deployment/server/clients splunk_server=local | table hostname applications.*.serverclasses | untable hostname, applications, serverclass | […]

Continue Reading →

Host not sending logs for x days

This Splunk Query will show hosts that stopped sending logs for at least 48 hours. You’ll want to change the time range to be relevant to your environment, and you may need to tweak the 48 hour range to something that is more appropriate for your environment. | tstats count as countAtToday latest(_time) as lastTime […]

Continue Reading →

Use TSTATS to find hosts no longer sending data

This is a simple tstats query shows all hosts and sourcetypes that have reported data, and shows the time in seconds since anything was sent. Be sure to run the query over a lengthy period of time in order to include machines that haven’t sent data for sometime. Don’t worry about the search-time so much, […]

Continue Reading →

Universal Forwarder Throughput Limit Hit Count

This search counts the amount of times the UF’s throughput limit is hit. I also threw in a sparkline: index=_internal sourcetype=splunkd “current data throughput” | rex “Current data throughput \((?<kb>\S+)” | eval rate=case(kb < 500, “256”, kb > 499 AND kb < 520, “512”, kb > 520 AND kb < 770 ,”768″, kb>771 AND kb<1210, […]

Continue Reading →

Hosts Taking a Long Time to Scan – Qualys

The following Splunk query will show the hosts taking an abnormally lengthy time to scan (helps find that needle in a haystack) within the Qualys Sourcetype: sourcetype=”qualys:hostDetection” eventtype=qualys_host_summary_event SCAN_DURATION> 1800 | sort -SCAN_DURATION | table IP, DNS, OS, SCAN_DURATION I take no credit for this. These queries were discovered on Tarun Kumar’s blog.

Continue Reading →

Splunk Objects With Permissions Granted to Non-existent Roles

Useful search to show a bit of detail on roles and user permissions. | rest /servicesNS/-/-/admin/directory count=0 splunk_server=local | fields eai:acl.app, eai:acl.owner, eai:acl.perms.*, eai:acl.sharing, eai:location, title | eval perms=mvappend(‘eai:acl.perms.read’,’eai:acl.perms.write’) | fields – eai:acl.perms.* | mvexpand perms | where perms!=”*” AND NOT [ | rest /servicesNS/-/-/authorization/roles count=0 splunk_server=local | fields title | rename title as perms […]

Continue Reading →

Every index explicitly granted to a role

Self explanatory, maps roles to indexes. Useful if you have a lot of indexes! | rest /servicesNS/-/-/authorization/roles count=0 splunk_server=local | fields title,srchIndexesAllowed | rename srchIndexesAllowed as index title as role | mvexpand index | where NOT match(index,”.*\*.*”) I found this at: https://gist.github.com/acharlieh/3254a7ab13297c760376 Credit goes to acharlieh!

Continue Reading →

Average Splunk Web requests by hour

This query is pretty awesome! It helped enlighten us to exactly when our splunk infrastructure is being hit with users index=_internal sourcetype=splunk_web_access [ rest / splunk_server=local | fields splunk_server | rename splunk_server as host ] | bin _time span=1d | stats count by date_hour _time | appendpipe [ fields _time | dedup _time | eval […]

Continue Reading →

All indexes not explicitly granted to a role

| rest /servicesNS/-/-/data/indexes count=0 | stats max(isInternal) as internal, max(disabled) as disabled, max(isReadOnly) as readonly by title | fillnull | where internal=0 AND disabled=0 AND readonly=0 | fields title | rename title as index | join index type=left [ rest /servicesNS/-/-/authorization/roles count=0 splunk_server=local | fields title,srchIndexesAllowed | rename srchIndexesAllowed as index title as role | […]

Continue Reading →

Forwarder Diagnostics – Last time Data Was Received by Index and Sourcetype

The following Splunk query was modified from the Splunk Health Overview app on Splunkbase. This particular Splunk search returns a list of hosts with their indexes and sourcetypes and determines when each last sent data. This query can help diagnose which Splunk Universal Forwarders may be having issues sending data, or simply stopped sending a […]

Continue Reading →

Detailed list of Errors Per Host

The following Splunk search will return a detailed list (by message) of errors associated with hosts running a universal forwarder: index=_internal sourcetype=”splunkd” log_level=”ERROR” | stats sparkline count dc(host) as uniqhosts last(message) as message last(_time) as last first(_time) as first by punct  | convert ctime(last) ctime(first) | table message count uniqhosts sparkline first last | sort […]

Continue Reading →

Memory Usage and Information on Splunk Server

This Splunk Search Query will perform a rest call to indicate current memory consumption on the Splunk server(s) itself/themselves: *NOTE* The following has been modified from the “Distributed Management Console” to be more generic for a copy, paste, and search example. | rest splunk_server=* /services/server/status/resource-usage/hostwide  | stats first(normalized_load_avg_1min) as load_average first(cpu_system_pct) as system, first(cpu_user_pct) as […]

Continue Reading →

Timestamp vs Indextime of Events (Diagnostic Query)

This query has in the past help me track down issues between forwarders and indexers, and even on occasion finding some time sync issues. Feel free to tweak, modify, and improve upon this query as I’m not 100% certain the math will work in your favor outside of highlighting (positive or negative) time differences! index=* […]

Continue Reading →

Last Time a Forwarder Checked In

The following Splunk Search Query will return results based on the last time a forwarder (universal forwarder, heavy forwarder, or otherwise) checked in. The query is a modified version of a query that was packaged with the Deployment Monitor app. index=”_internal” source=”*metrics.lo*” group=tcpin_connections NOT eventType=* | eval sourceHost=if(isnull(hostname), sourceHost,hostname) | eval connectionType=case(fwdType==”uf”,”universal forwarder”, fwdType==”lwf”, “lightweight […]

Continue Reading →