Forwarder Diagnostics – Last time Data Was Received by Index and Sourcetype

The following Splunk query was modified from the Splunk Health Overview app on Splunkbase. This particular Splunk search returns a list of hosts with their indexes and sourcetypes and determines when each last sent data. This query can help diagnose which Splunk Universal Forwarders may be having issues sending data, or simply stopped sending a […]

Continue Reading →

Malware Detection

I’m reposting this query I stumbled upon in a blog here. The description states that it can be used to detect malware reporting out to the web. Check out the article it’s a decent read. search.goes.here | convert mktime(_time) as epoch | sort 0 uri_host,client_ip,epoch | delta epoch as epoch_delta | search epoch_delta>0 epoch_delta<30 | chart […]

Continue Reading →

Detailed list of Errors Per Host

The following Splunk search will return a detailed list (by message) of errors associated with hosts running a universal forwarder: index=_internal sourcetype=”splunkd” log_level=”ERROR” | stats sparkline count dc(host) as uniqhosts last(message) as message last(_time) as last first(_time) as first by punct  | convert ctime(last) ctime(first) | table message count uniqhosts sparkline first last | sort […]

Continue Reading →

Memory Usage and Information on Splunk Server

This Splunk Search Query will perform a rest call to indicate current memory consumption on the Splunk server(s) itself/themselves: *NOTE* The following has been modified from the “Distributed Management Console” to be more generic for a copy, paste, and search example. | rest splunk_server=* /services/server/status/resource-usage/hostwide  | stats first(normalized_load_avg_1min) as load_average first(cpu_system_pct) as system, first(cpu_user_pct) as […]

Continue Reading →

Hard Disk Usage and Information on Splunk Server

The following Splunk Query will utilize a “| REST” call to gather information related to disk usage on your Splunk server(s). The following has been modified from the “Distributed Management Console” to be more generic for a copy, paste, and search example.   | rest splunk_server=* /services/server/status/partitions-space | eval free = if(isnotnull(available), available, free) | […]

Continue Reading →

Timestamp vs Indextime of Events (Diagnostic Query)

This query has in the past help me track down issues between forwarders and indexers, and even on occasion finding some time sync issues. Feel free to tweak, modify, and improve upon this query as I’m not 100% certain the math will work in your favor outside of highlighting (positive or negative) time differences! index=* […]

Continue Reading →

Top Visited Pages in IIS Web Logs

There are a number of ways to track user behavior within web logs. One such method is to use the JSESSIONID which in this query is used. The variable you can/will change in this query is the reference to JSESSIONID as to better align with your web logs and web site(s) in general. This working […]

Continue Reading →

Traffic Volume by Forwarder

This Splunk search query will show you the top 10 “chattiest” forwarders on your network. I’ve used this query to determine why some forwarders were sending more data than others. The results are displayed in kilobits, you could use an eval to change it to the appropriate size for your network. index=”_internal” source=”*metrics.lo*” group=tcpin_connections NOT […]

Continue Reading →

Qualys Hosts not Scanned in 30 days+

The following Splunk Search (query) is for Qualys and will show hosts that have not been scanned in 30 days or more. This query assumes that your index is defined as qualys. index=qualys HOSTVULN earliest=-30d@d STATUS=”RE-OPENED” | dedup HOST_ID, QID sortby +_time | join HOST_ID [ search index=qualys HOSTSUMMARY OS=”Windows*” NOT “Windows Server*” | where […]

Continue Reading →