The following Splunk search will show a list of sourcetypes (or perhaps sources I need to confirm) being sent by each Forwarder: index=”_internal” |where host!=splunk_server | stats values(series) as Sourcetypes by host
Hard Disk Usage and Information on Splunk Server
The following Splunk Query will utilize a “| REST” call to gather information related to disk usage on your Splunk server(s). The following has been modified from the “Distributed Management Console” to be more generic for a copy, paste, and search example. | rest splunk_server=* /services/server/status/partitions-space | eval free = if(isnotnull(available), available, free) | […]
Timestamp vs Indextime of Events (Diagnostic Query)
This query has in the past help me track down issues between forwarders and indexers, and even on occasion finding some time sync issues. Feel free to tweak, modify, and improve upon this query as I’m not 100% certain the math will work in your favor outside of highlighting (positive or negative) time differences! index=* […]
Top Visited Pages in IIS Web Logs
There are a number of ways to track user behavior within web logs. One such method is to use the JSESSIONID which in this query is used. The variable you can/will change in this query is the reference to JSESSIONID as to better align with your web logs and web site(s) in general. This working […]
Convert non timestamp time to Epoch
Scenario: You have a non timestamp field that you need to convert to epoch time to perform statistics on within splunk. Here’s how you do it: your search goes here |eval Epoch_Time=strptime(Field_Date, “%Y-%m-%d %H:%M:%S”)
Failed Attempt to Login to a Disabled Account
This Splunk Search Query will indicate any user who attempted to login to a disabled account. (Tested only on Windows 7 / Server 2008 and newer Windows logs). source=”WinEventLog:security” EventCode=4625 (Sub_Status=”0xc0000072″ OR Sub_Status=”0xC0000072″) Security_ID!=”NULL SID” Account_Name!=”*$” | eval Date=strftime(_time, “%Y/%m/%d”)| rex “Which\sLogon\sFailed:\s+\S+\s\S+\s+\S+\s+Account\sName:\s+(?<facct>\S+)” | eval Date=strftime(_time, “%Y/%m/%d”) | stats count by Date, facct, host, Keywords | […]
Traffic Volume by Forwarder
This Splunk search query will show you the top 10 “chattiest” forwarders on your network. I’ve used this query to determine why some forwarders were sending more data than others. The results are displayed in kilobits, you could use an eval to change it to the appropriate size for your network. index=”_internal” source=”*metrics.lo*” group=tcpin_connections NOT […]
User Activity in DBConnect
The following Splunk query is for the DBConnect app. This will return all user activity using this particular app. I’ve provided the regex in the search. index=_audit sourcetype=audittrail action=”db_connect*” |eval Date=strftime(_time, “%Y/%d/%m”) |rex “user=(?<user>\S+),” | stats count by Date, user, info, action
CPU Thresholds, Warnings, and Risk Scoring
The following splunk query can be used to do as the title says. Risk scoring is done via color codes in the source information of the chart (see below for an example). The following query looks at the average CPU time for 30 minute chunks of time over a selected time range (ie: what was […]
Failed Attempts to Logon to Splunk Web
The following Splunk Search Query will return all users who have failed to logon to the Splunk Web console. This query will also include an average (from eventstats). index=_audit action=”login attempt” info=failed | timechart count(user) as Failed_Attempts| eventstats avg(Failed_Attempts) as Average
Network Traffic Sent in Megabytes over Time
The following splunk query will show a timechart of network traffic sent over a period of time for any host specified (make sure you edit the query to specify a host, this one defaults to all). The query also converts the default value of Bytes to Megabytes. sourcetype=”Perfmon:Network Interface” (host=”*”) counter=”Bytes Sent/sec” | eval MB=(Value/1024/1024) […]
Network Traffic Received in Megabytes over Time
The following splunk query will show a timechart of network traffic received over a period of time for any host specified (make sure you edit the query to specify a host, this one defaults to all). The query also converts the default value of Bytes to Megabytes. sourcetype=”Perfmon:Network Interface” (host=”*”) counter=”Bytes Received/sec”| eval MB=(Value/1024/1024)| […]
Free Disk Space for each Drive Letter
The following Splunk query will return results for all hosts reporting in Perfmon data on available disk space per assigned drive letter (NOTE you must make the change to include free diskspace per partition in your inputs.conf file) Query: sourcetype=”Perfmon:Free Disk Space” counter=”Free Megabytes” (instance!=”HarddiskVolume*”) (instance!=_Total) |eval FreeSpace=(Value/1024)| eval GB=tostring(FreeSpace,”commas”) | table host instance GB […]
License Usage by Index per Day
The following Splunk search query will output license usage for each index for each day for the week to date. It will also output an average for each index over the course of the given time period. index=_internal source=*license_usage.log type=”Usage” splunk_server=* earliest=-1w@d | eval Date=strftime(_time, “%Y/%m/%d”) | eventstats sum(b) as volume by idx, Date […]
Percentage of Daily License Usage
This Splunk search query will indicate the percentage of license used for the current day. This is already shown in the licensing tab under settings, however this query is extracted if you would want to use it within a dashboard or any other reason. NOTE – splunk_server= should be set to your license master. […]
Top 5 License Consuming Hosts
The following Splunk search query will return the top five licensing consuming hosts: index=_internal source=*license_usage.log type=”Usage” | stats sum(b) AS volume by h | eval GB=round(volume/1024/1024/1024,5) | table h GB | sort 5 – GB
License Usage by Sourcetypes
The following Splunk query will return results for license usage by sourcetype: index=_internal source=”*license_usage.lo*” type=Usage | stats sum(b) as bytes by st | eval Megabytes=bytes/1048576 |eval Megabytes=round(Megabytes,2) | fieldformat Megabytes=tostring(Megabytes,”commas”)| rename st as sourcetype | fields – bytes | sort – Megabytes
Last Time a Forwarder Checked In
The following Splunk Search Query will return results based on the last time a forwarder (universal forwarder, heavy forwarder, or otherwise) checked in. The query is a modified version of a query that was packaged with the Deployment Monitor app. index=”_internal” source=”*metrics.lo*” group=tcpin_connections NOT eventType=* | eval sourceHost=if(isnull(hostname), sourceHost,hostname) | eval connectionType=case(fwdType==”uf”,”universal forwarder”, fwdType==”lwf”, “lightweight […]
List of Universal Forwarders with Version
The following Splunk query will return results of any host using a universal forwarder to transmit data back to a Splunk indexer. The query will return hostname, version, as well as architecture (64-bit vs 32-bit). index=”_internal” sourcetype=splunkd group=tcpin_connections NOT eventType=* | eval Hostname=if(isnull(hostname), sourceHost,hostname) | eval version=if(isnull(version),”pre 4.2″,version) | eval architecture=if(isnull(arch),”n/a”,arch) | stats count […]
Failed Versus Successful Logon Attempts
This Splunk search query example will return results indicating failed vs successful login attempts in a Windows environment: source=”WinEventLog:security” (Logon_Type=2 OR Logon_Type=7 OR Logon_Type=10) (EventCode=528 OR EventCode=540 OR EventCode=4624 OR EventCode=4625 OR EventCode=529 OR EventCode=530 OR EventCode=531 OR EventCode=532 OR EventCode=533 OR EventCode=534 OR EventCode=535 OR EventCode=536 OR EventCode=537 OR EventCode=539) | eval status=case(EventCode=528, “Successful […]
