Windows dashboard to help identify users that have either failed or successfully logged in. At the top you have a box I called “Filter” that allows you to insert search parameters in the base search (ex: user=thall). Each panel has it’s own TimeRangePicker and a field selection box which allows you to decide what fields […]
License and Storage Usage Dashboard
This relies on the search posted earlier: This will display storage and license usage broken down by groups, predefined in the chargeback app customers.csv
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 |
<form> <label>License and Storage Usage</label> <fieldset submitButton="false"> <input type="dropdown" token="grouppicker"> <label>Group</label> <choice value="Group1">Group1</choice> <choice value="Group2">Group2</choice> <choice value="Group3">Group3</choice> <choice value="Group4">Group4</choice> <choice value="Group5">Group5</choice> <choice value="Group6">Group6</choice> <choice value="*">All Groups</choice> <default>*</default> </input> </fieldset> <row> <panel> <title>All Storage Usage</title> <single> <search> <query>| inputlookup size_license_group_usage.csv | search group=$grouppicker$ | stats sum(currentDBSizeMB) AS SplunkIndexSizeMB | eval SplunkIndexSize=case( SplunkIndexSizeMB>=(1024*1024),round(SplunkIndexSizeMB/(1024*1024),2)."TB", SplunkIndexSizeMB>=(1024),round(SplunkIndexSizeMB/(1024),2)."GB", 1=1,SplunkIndexSizeMB."B") | fields SplunkIndexSize</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="drilldown">none</option> <option name="rangeColors">["0x53a051","0x0877a6","0xf8be34","0xf1813f","0xdc4e41"]</option> <option name="refresh.display">progressbar</option> </single> </panel> <panel> <title>Yesterday's License Usage</title> <single> <search> <query>| inputlookup size_license_group_usage.csv | search group=$grouppicker$ | rename sum(Yesterday) AS ydaydata | stats sum(ydaydata) as ydaydata | eval volume_converted=case( ydaydata>=(1024),round(ydaydata/(1024),2)."TB", 1=1,round(ydaydata,2)."GB") | rename volume_converted AS "Yesterday's License Usage" | fields "Yesterday's License Usage"</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="drilldown">none</option> <option name="rangeColors">["0x53a051","0x0877a6","0xf8be34","0xf1813f","0xdc4e41"]</option> <option name="refresh.display">progressbar</option> </single> </panel> </row> <row> <panel> <title>License and Storage for Selected Group by Index</title> <table> <search> <query>| inputlookup size_license_group_usage.csv | search group=$grouppicker$ | rename sum(Average) AS "Average GB License Daily" | rename sum(Yesterday) AS "Yesterdays GB License Usage" | eval "Index Storage Usage MB" = tostring(currentDBSizeMB, "commas") | fields group indexname "Average GB License Daily" "Yesterdays GB License Usage" "Index Storage Usage MB"</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="count">100</option> <option name="dataOverlayMode">none</option> <option name="drilldown">none</option> <option name="percentagesRow">false</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> <format type="number" field="Average GB License Daily"></format> <format type="number" field="Yesterdays GB License Usage"></format> <format type="color" field="group"> <colorPalette type="sharedList"></colorPalette> <scale type="sharedCategory"></scale> </format> </table> </panel> </row> </form> |
Build License usage by Group
This was cobbled together from multiple searches I found. This search feeds the license and storage dashboard posted here: It relies on the Chargeback app for the customers.csv form.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
index=_internal source=*license_usage.log type="Usage" earliest=-30d@d latest=@d | eval indexname = if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx) | eval sourcetypename = st | bin _time span=1d | stats values(poolsz) as poolsz sum(b) as b by _time, pool, indexname, sourcetypename | eval GB=(b/1024/1024/1024) | eval pool=(poolsz/1024/1024/1024) | fields _time, indexname, sourcetypename, GB, pool | eval day=strftime(_time,"%a") | eval yesterday=tostring(strftime(relative_time(now(),"-d@d"),"%Y-%m-%d")) | eval time=tostring(strftime(_time,"%Y-%m-%d")) | eval isyesterday=if(yesterday==time,"true","is_average") | eval Yesterday=if(isyesterday="true",GB,0) | stats values(pool) as pool sum(Yesterday) as Yesterday avg(GB) as Average by indexname, sourcetypename | stats values(pool) as pool list(sourcetypename) sum(Yesterday) sum(Average) by indexname | join indexname [| inputlookup customers.csv | rename idx AS indexname | fields group indexname ] | join indexname [| rest /services/data/indexes search="totalEventCount!=0" | stats sum(currentDBSizeMB) AS currentDBSizeMB by title | rename title AS indexname | fields indexname currentDBSizeMB] | fields group indexname sum(Yesterday) sum(Average) currentDBSizeMB | sort - sum(Yesterday) | outputlookup size_license_group_usage.csv |
Detailed User Activity
|
1 2 3 4 5 6 7 8 9 10 11 12 |
index=_* search=* user=* user!=- user!=splunk-system-user | rex field=search max_match=0 "index\s*=[\s\"]*(?<idx1>.*?)[\|\s\"\)]" | rex field=search max_match=0 "[\+\(|\+]index\%3D(?<idx2>.*?)[\+|\)\+]" | eval idx=if(isnull(idx1), idx2, idx1) | eval frequency=if(source="/opt/splunk/var/log/splunk/splunkd_access.log", "scheduled", "ad-hoc") | eval type=if(match(search, "summary*"), "summary", type1) | eval idx=if(isnull(idx), "NONE", idx) | eval end_type=if(frequency="ad-hoc", "ad-hoc", type) | rename end_type as type | table _time frequency type source user idx search | bin _time span=1h | stats count as count by _time idx user frequency type search |
Exclude single event type from logs
Do this on HF transforms.conf:
|
1 2 3 4 5 6 |
[discard_gotoips] REGEX = <<<use regex,URL>>> DEST_KEY = queue FORMAT = nullQueue |
props.conf:
|
1 2 3 4 5 |
[default] TRANSFORMS-null = discard_gotoips File location: /etc/system/local |
Who’s Using Splunk?
I often get asked how much a certain dashboard gets looked at, or how many times a user looks at a specific app. I wrote this quick query to answer that question.
|
1 2 3 4 |
index=_internal sourcetype="splunk_web_access" method="GET" status="200" | stats count as count by user, view | appendpipe [stats sum(count) as count by user | eval view = "Total Views"] | sort + user |
Find unused dashboards
Use this search to find unused dashboards:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
| rest /servicesNS/-/-/data/ui/views splunk_server=* | search isDashboard=1 | rename eai:acl.app as app | fields title app | join type=left title [| search index=_internal sourcetype=splunk_web_access host=* user=* | rex field=uri_path ".*/(?<title>[^/]*)$" | stats latest(_time) as Time latest(user) as user by title ] | where isnotnull(Time) | eval Now=now() | eval "Days since last accessed"=round((Now-Time)/86400,2) | sort - "Days since last accessed" | convert ctime(Time) | fields - Now |
Admin Notes – Fantastic query! I modified the SPL slightly as I had an issue when I copied it to my two test environments.
skipped searches and why
Quickly identify high amounts of skipped searches in your cluster or standalone SH(s):
|
1 2 3 |
index = _internal skipped sourcetype=scheduler status=skipped host=[your splunk SH(s)] | stats count by app search_type reason savedsearch_name | sort -count |
Adjust “[your splunk SH(s)]” to the SH(s) you want to check obviously ;)
find blocking queues
Blocked queues are (obviously) bad for your environment so here a search to identify those:
|
1 2 3 4 5 6 |
index=_internal sourcetype=splunkd group=queue (name=parsingQueue OR name=indexqueue OR name=tcpin_queue OR name=aggqueue) | eval is_blocked=if(blocked=="true",1,0), host_queue=host." - ".name | stats sparkline sum(is_blocked) as blocked,count by host_queue | eval blocked_ratio=round(blocked/count*100,2) | sort 20 -blocked_ratio | eval requires_attention=case(blocked_ratio>50.0,"fix highly recommended!",blocked_ratio>40.0,"you better check..",blocked_ratio>20.0,"usually no need to worry but keep an eye on it",1=1,"not unusual") |
Example result:
Regex Extraction for WordPress Version from Apache Logs
The following Splunk search extracts the WordPress version from your Apache Web Logs. For fun I also did a time chart using 100% stacked bar chart to show by month each version of wordpress used. This was actually a pretty neat display of colors to show the upgrade path of WordPress over the years! I […]
Utilizing tstats for Page Views within Apache Web Logs
Here’s a Splunk query to show a timechart of page views from a website running on Apache. Due to the search utilizing tstats, the query will return results incredibly fast over a very LONG period of time if desired. Change the index to reflect yours, as well as the span to reflect a span you […]
Windows RDP sessions
Here is a dashboard I built to look at Windows Logon Type 2 & 10 (remote & remote interactive) that will help identify which users access which servers and how many times. Also when you click on a user it will run a 30 day search and a 24 hour search that produces a column […]
Successful File Access Attempts and Filename Accessed
Ever need to find when a user accessed a file within a Windows environment? The following Splunk query will show successful file accesses by each user for a given day. Depending on the size of your environment this can get out of hand quickly. You’ll want to tweak to best fit your environment. *Note* you […]
Internal Splunk User Stats
This simple Splunk query will show us unique Splunk user logged into Splunk per day, as well as total count of log-ons.
|
1 |
index=_audit info=succeeded | timechart span=1d dc(user) as unique_users count(user) as logons_all_users |
Apache Traffic Dashboard
Description: The following Dashboard is what I use to monitor traffic to GoSplunk. It uses the built in sourcetype of access_combined. No additional add-on’s or TA’s are required. I replaced my index with index=* so it’ll work out of the box. You’ll want to change this to your index for best practices. Also I have […]
Check your strftime is correct in the props.conf
A simple method on checking if your strftime (TIME_FORMAT=) in the props.conf matches your log file timestamp format. strftime(X,Y) This function takes a UNIX time value, X, as the first argument and renders the time as a string using the format specified by Y. The UNIX time must be in seconds. Use the first 10 […]
List all ES Correlation Searches
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 |
| rest splunk_server=local count=0 /services/saved/searches | where match('action.correlationsearch.enabled', "1|[Tt]|[Tt][Rr][Uu][Ee]") | rex field=action.customsearchbuilder.spec "datamodel\\\":\s+\\\"(?<Data_Model>\w+)" | rex field=action.customsearchbuilder.spec "object\\\":\s+\\\"(?<Dataset>\w+)" | rename action.correlationsearch.label as Search_Name title as Rule_Name eai:acl.app as Application_Context request.ui_dispatch_app as UI_Dispatch_Context description as Description Data_Model as Guided_Mode:Data_Model Dataset as Guided_Mode:Dataset action.customsearchbuilder.enabled as Guided_Mode action.customsearchbuilder.spec as Guided_Mode:Search_Logic search as Search dispatch.earliest_time as Earliest_Time dispatch.latest_time as Latest_Time cron_schedule as Cron_Schedule schedule_window as Schedule_Window schedule_priority as Schedule_Priority alert_type as Trigger_Conditions:Trigger_Alert_When alert_comparator as Trigger_Conditions:Alert_Comparator alert_threshold as Trigger_Conditions:Alert_Threshold alert.suppress.period as Throttling:Window_Duration alert.suppress.fields as Throttling:Fields_To_Group_By action.notable.param.rule_title as Notable:Title action.notable.param.rule_description as Notable:Description action.notable.param.security_domain as Notable:Security_Domain action.notable.param.severity as Notable:Severity action.notable.param.default_owner as Notable:Default_Owner action.notable.param.default_status as Notable:Default_Status action.notable.param.drilldown_name as Notable:Drill-down_Name action.notable.param.drilldown_search as Notable:Drill-down_Search action.notable.param.drilldown_earliest_offset as Notable:Drill-down_Earliest_Offset action.notable.param.drilldown_latest_offset as Notable:drill-down_Latest_Offset action.notable.param.next_steps as Notable:Next_Steps action.risk.param._risk_score as Risk_Analysis:Risk_Score action.risk.param._risk_object as Risk_Analysis:Risk_Object_Field action.risk.param._risk_object_type as Risk_Analysis:Risk_Object_Type | eval Guided_Mode:Enabled = if(Guided_Mode == 1, "Yes", "No") | eval Real-time_Scheduling_Enabled = if(realtime_schedule == 1, "Yes", "No") | table disabled Search_Name, Rule_Name, Application_Context, UI_Dispatch_Context, Description, Guided_Mode:Enabled, Guided_Mode:Data_Model, Guided_Mode:Dataset, Guided_Mode:Search_Logic, Search, Earliest_Time, Latest_Time, Cron_Schedule, Real-time_Scheduling_Enabled, Schedule_Window, Schedule_Priority, Trigger_Conditions:Trigger_Alert_When, Trigger_Conditions:Alert_Comparator, Trigger_Conditions:Alert_Threshold, Throttling:Window_Duration, Throttling:Fields_To_Group_By, Notable:Title, Notable:Description, Notable:Security_Domain, Notable:Severity, Notable:Default_Owner, Notable:Default_Status, Notable:Drill-down_Name, Notable:Drill-down_Search, Notable:Drill-down_Earliest_Offset, Notable:drill-down_Latest_Offset, Notable:Next_Steps, Risk_Analysis:Risk_Score, Risk_Analysis:Risk_Object_Field, Risk_Analysis:Risk_Object_Type |
Windows service activity & MSI installs
Here is a dashboard that built to look for Service activity and MSI installs in Windows. This dashboard utilizes Post Processing so there is only 2 searches that are launched when the dashboard is loaded to minimize impact on search queuing. Add-on’s: Splunk Add-on for Microsoft Windows – https://splunkbase.splunk.com/app/742/
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 |
<form> <label>Windows service activity & MSI installs</label> <search id="service_base1"> <query>index=wineventlog $filter1$ SourceName="*Service Control Manager" LogName=System | fillnull value=* Service_Name, Service_File_Name, Service_Type, Service_Start_Type, Service_Account| stats count by _time, host, Service_Name, Service_File_Name, Service_Type, Service_Start_Type, Service_Account, Message, EventCode</query> <earliest>$time1.earliest$</earliest> <latest>$time1.latest$</latest> <sampleRatio>1</sampleRatio> </search> <fieldset submitButton="true" autoRun="false"> <input type="time" token="time1"> <label></label> <default> <earliest>-24h@h</earliest> <latest>now</latest> </default> </input> <input type="text" token="filter1"> <label>Filter:</label> <default>*</default> </input> </fieldset> <row> <panel> <title>Service was installed</title> <table> <search base="service_base1"> <query>search EventCode=7045 OR EventCode=7035 Service_Name!=tenable_mw_scan | eval Message=split(Message,".") | eval Short_Message=mvindex(Message,0) | table _time, host, Service_Name, Service_File_Name, Service_Type, Service_Start_Type, Service_Account, Short_Message</query> </search> <option name="count">10</option> <option name="drilldown">none</option> </table> </panel> </row> <row> <panel> <title>Service entered Running/Stopped state</title> <table> <search base="service_base1"> <query>search EventCode=7036 | stats sum(count) as count by Message</query> </search> <option name="count">10</option> <option name="drilldown">cell</option> <drilldown> <set token="term1">$click.value$</set> </drilldown> </table> <table> <title>Looking at "$term1$"</title> <search> <query>index=wineventlog $term1$ SourceName="*Service Control Manager" EventCode=7036 | stats count by _time,host</query> <earliest>$time1.earliest$</earliest> <latest>$time1.latest$</latest> </search> <option name="count">10</option> <option name="drilldown">none</option> </table> </panel> <panel> <title>Service state changes</title> <table> <search base="service_base1"> <query>search EventCode=7040 Message!="The start type of the Windows Modules Installer service*" Message!="The start type of the Background Intelligent Transfer Service service*" | stats sum(count) as count by Message</query> </search> <option name="count">10</option> <option name="drilldown">cell</option> <drilldown> <set token="term2">$click.value$</set> </drilldown> </table> <table> <title>Looking at "$term2$"</title> <search> <query>index=wineventlog $term2$ SourceName="*Service Control Manager" LogName=System EventCode=7040 Message!="The start type of the Windows Modules Installer service*" Message!="The start type of the Background Intelligent Transfer Service service*" | stats count by _time,host</query> <earliest>$time1.earliest$</earliest> <latest>$time1.latest$</latest> </search> <option name="count">10</option> <option name="drilldown">none</option> </table> </panel> </row> <row> <panel> <title>Windows service activity</title> <table> <search base="service_base1"> <query>search EventCode!=7045 EventCode!=7036 EventCode!=7035 | stats sum(count) as count by Message</query> </search> <option name="count">10</option> <option name="drilldown">cell</option> <drilldown> <set token="term3">$click.value$</set> </drilldown> </table> <table> <title>Looking at "$term3$"</title> <search> <query>index=wineventlog $term3$ SourceName="*Service Control Manager" EventCode!=7045 EventCode!=7036 EventCode!=7035 | fillnull value=* user | stats count by _time,host,user</query> <earliest>$time1.earliest$</earliest> <latest>$time1.latest$</latest> </search> <option name="count">10</option> <option name="drilldown">none</option> </table> </panel> </row> <row> <panel> <title>MSI Installs</title> <table> <search> <query>index=wineventlog $filter1$ "SourceName=MsiInstaller" NOT EventCode="1015" | stats list(Message) by _time,host,User | sort -_time</query> <earliest>$time1.earliest$</earliest> <latest>$time1.latest$</latest> </search> <option name="count">10</option> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> </row> </form> |
Windows Account Management Dashboard
Here is a dashboard that I have built to look at Windows Account Management events. The dashboard utilizes a drill-down that will feed a multi-select which is using a dynamic search to give you fields that are available for the stats output in 2nd panel dependent on your selection. Add-on’s: Splunk Add-on for Microsoft Windows […]
Simple File Integrity Monitoring Management Dashboard
This is the code for my original reddit post at https://www.reddit.com/r/Splunk/comments/am3tgr/simple_file_integrity_monitoring/ This dashboard allows users to manage simple File Integrity Monitoring (FIM) within Splunk. Please note that this isn’t a full FIM suite as it only validates if a checksum has been changed on a file, but I have included a simple TA for Linux. However, if you […]
