In my previous role I created this dashboard to identify how much data a Splunk forwarder had sent to my indexers. This was a daily check that either myself of someone on my team would review. This check helped us identify a misconfiguration across all of my production Windows servers. I was able to drilldown into the Source/Sourcetypes of the affected Windows servers to understand where the increase of log data was origination from. For this event the logs were origination from WinEventLog:Security. With that information I was able to open a new search reviewing which EventCodes had the highest event count. This information quickly helped identify the issue and which teams to contact for resolution.
<dashboard theme="dark" version="1.0"> <label>Data Usage</label> <description>Dashboard to help see how much data is being used the past 7 days and to track how much each Forwarder is sending.</description> <row> <panel> <title>Data Indexed in GB for Last 7 days per Indexer.</title> <table> <search> <query>index=_internal source=*license_usage.log type="RolloverSummary" | eval _time=_time - 43200 | bin _time span=1d | eval GB=round(b/1024/1024/1024, 3) | stats sum(GB) by host _time | sort -_time</query> <earliest>-7d@h</earliest> <latest>now</latest> </search> <option name="count">10</option> <option name="dataOverlayMode">none</option> <option name="drilldown">cell</option> <option name="rowNumbers">true</option> <option name="wrap">true</option> </table> </panel> </row> <row> <panel> <title>Forwader License Usage information</title> <table> <title>Last 24 hours per host - click on host name to see Source and Sourcetype information.</title> <search> <query>index=_internal source=*license_usage.log | eval GB=b/1024/1024/1024 | stats sum(GB) by h | sort -sum(GB)</query> <earliest>-24h@h</earliest> <latest>now</latest> </search> <option name="dataOverlayMode">none</option> <option name="drilldown">cell</option> <option name="rowNumbers">false</option> <option name="wrap">true</option> <drilldown> <set token="h">$click.value$</set> </drilldown> </table> </panel> </row> <row> <panel> <title>$h$</title> <table> <title>Source/Sourcetype volume by GB in descending order. Clicking in a row will launch a new search to investigate further.</title> <search> <query>index=_internal source=*license_usage.log h="$h$" | eval GB=b/1024/1024/1024 | stats sum(GB) as volume_in_GB by idx s st | sort - volume_in_GB | rename idx AS index s AS source st AS sourcetype</query> <earliest>-24h@h</earliest> <latest>now</latest> </search> <option name="count">10</option> <option name="drilldown">cell</option> <option name="refresh.display">progressbar</option> <drilldown> <link target="_blank">search?q=index=$row.index$%20host=%22$h$%22%20source=%22$row.source$%22%20sourcetype=%22$row.sourcetype$%22&earliest=-24h@h&latest=now</link> </drilldown> </table> </panel> </row> </dashboard>