Dashboard
exploremydata - data explorer
Description
This dashboard provides and overview of the data that is available to query.
Click on the index below to review source types in that index, and then a sourcetype to review fields. Finally, you can click on a field to see sample values in that field.
Click "Show Filters" above to open a search window to search for specific index, sourcetype or field name.
When you run the build query, if a field/sourcetype/index is found, the entry will be updated with the date/time that is was found.
14 0
<form hideFilters="true" theme="dark">
<!--
Welcome!
Use this query for your saved search (uncomment the rolloff if you want to include that):
| tstats count WHERE index=* OR index=_* GROUPBY index, sourcetype
| rename index AS indexname, sourcetype AS sourcetypename
| map maxsearches=200 search="| search index=\"$indexname$\" sourcetype=\"$sourcetypename$\" | head 1000 | fieldsummary | eval index=\"$indexname$\", sourcetype=\"$sourcetypename$\" | fields index, sourcetype, field"
|dedup index sourcetype field
|eval last_found=(now())
|convert ctime(last_found)
|append [|inputlookup exploremydata.csv append=true]
|dedup index sourcetype field
| sort + last_found
```| eval rolloff=relative_time(now(),"-60d@d")
| eval remove_time=strptime(last_found, "%m/%d/%Y %H:%M:%S")
| where remove_time>rolloff```
| table index sourcetype field last_found
| outputlookup exploremydata.csv
Enjoy - James Callahan, www.professionalparanoid.com
-->
<label>exploremydata</label>
<description>version 1.1</description>
<init>
<unset token="index_token"></unset>
<unset token="sourcetype_token"></unset>
<unset token="field_token"></unset>
<unset token="search_for"></unset>
<unset token="last_run"></unset>
<unset token="nocsv">true</unset>
</init>
<search id="baserun">
<query>|inputlookup exploremydata.csv
|sort - last_found
|head 1
|eval last=strptime(last_found, "%m/%d/%Y %H:%M:%S")
|eval nower=(now())
|eval delta=(nower-last)
|eval rerunq=if(delta>604800,"Y","N")
<!-- (7 days = 604800) below for testing
|eval rerunq=if(delta>800,"Y","N")-->
|fillnull value="q" rerunq
|fields rerunq last
</query>
</search>
<fieldset submitButton="false">
<input type="time" token="lookback" searchWhenChanged="true">
<label>Timeframe for Field Review</label>
<default>
<earliest>-15d@d</earliest>
<latest>now</latest>
</default>
</input>
<input type="text" token="search_for">
<label>Search Index, Sourcetype, Field</label>
</input>
<input type="checkbox" token="trellis_stats" searchWhenChanged="true">
<label></label>
<choice value="true">Hide Stats</choice>
<change>
<condition label="Hide Stats">
<set token="trellis_stats">true</set>
<unset token="trellis_stats">false</unset>
</condition>
</change>
<default></default>
</input>
</fieldset>
<row depends="$nocsv$">
<panel>
<!-- NOTE if you make changes to this text, you'll have to update this text in a second location, below... for now -->
<html>
<h2>
<b>First Run of Dashboard</b>
</h2>
<p>This looks like the first time you've run this dashboard, or the csv file that makes this work isn't there anymore.</p>
<p>This dashboard is based off a csv file that is built via a query under the "Actions: Rerun Button" below. Pressing that should pop out in a new query window and provide an 'outputlookup' to create the exploremydata.csv file. There will be a warning about this query. If you're trepidatious about this, click 'investigate' on that warning and remove the outputlookup command - then run it and see what it does. </p>
<p>If this dashboard is used regularlly, consider creating a scheduled search from the 'Rerun Button' query to update the csv file on a recurring basis. </p>
</html>
</panel>
</row>
<row depends="$nevershow$">
<panel>
<table>
<search base="baserun">
<query>
|eval last=strftime(last, "%m/%d/%Y %H:%M")
|table last
</query>
<done>
<set token="last_run">$result.last$</set>
</done>
</search>
</table>
</panel>
</row>
<row>
<panel>
<html>
<details>
<summary>General Information and Actions</summary>
<details>
<summary>Overview</summary>
This dashboard provides and overview of the data that is available to query.<br/>
Click on the index below to review source types in that index, and then a sourcetype to review fields. Finally, you can click on a field to see sample values in that field.<br/>
Click "Show Filters" above to open a search window to search for specific index, sourcetype or field name.
<p>When you run the build query, if a field/sourcetype/index is found, the entry will be updated with the date/time that is was found. </p>
</details>
<details>
<summary>Actions</summary>
<table align="center" width="100%">
<td align="center" width="25%">
Lastest date/time in the lookup file:<br/>
<font color="cyan">
<b>$last_run$</b>
</font>
</td>
<td align="center" width="25%">
<a class="btn default edit-cancel" href="search?q=%7Cinputlookup%20exploremydata.csv" target="_seecsv">Review csv file</a>
</td>
<td align="center" width="25%">
<a class="btn default edit-cancel" href="./search?q=%7C%20tstats%20count%20WHERE%20index%3D*%20OR%20index%3D_*%20GROUPBY%20index%2C%20sourcetype%0A%7C%20rename%20index%20AS%20indexname%2C%20sourcetype%20AS%20sourcetypename%0A%20%7C%20map%20maxsearches%3D200%20search%3D%22%7C%20search%20index%3D%5C%22%24indexname%24%5C%22%20sourcetype%3D%5C%22%24sourcetypename%24%5C%22%20%7C%20head%201000%20%7C%20fieldsummary%20%7C%20eval%20index%3D%5C%22%24indexname%24%5C%22%2C%20sourcetype%3D%5C%22%24sourcetypename%24%5C%22%20%7C%20fields%20index%2C%20sourcetype%2C%20field%22%0A%7Cdedup%20index%20sourcetype%20field%0A%7Ceval%20last_found%3D(now())%0A%7Cconvert%20ctime(last_found)%0A%7Cappend%20%5B%7Cinputlookup%20exploremydata.csv%20append%3Dtrue%5D%0A%7Cdedup%20index%20sourcetype%20field%0A%0A%7C%20sort%20%2B%20last_found%0A%0A%7C%20eval%20rolloff%3Drelative_time(now()%2C%22-60d%40d%22)%0A%7C%20eval%20remove_time%3Dstrptime(last_found%2C%20%22%25m%2F%25d%2F%25Y%20%25H%3A%25M%3A%25S%22)%0A%7C%20where%20remove_time%3Erolloff%0A%7C%20table%20index%20sourcetype%20field%20last_found%0A%0A%7Coutputlookup%20exploremydata.csv" target="runit_">Rerun Build Query<br/>(or run for the first time)</a>
<br/>
</td>
<td align="center" width="25%">
<div stle="text-align: right;">
<a href="./exploremydata" class="btn btn-primary">Reload This Dashboard.
<i class="icon-rotate-counter"/>
</a>
</div>
</td>
</table>
</details>
<details>
<summary>First Run Info</summary>
<html>
<!-- NOTE if you make changes to this text, you'll have to update this text in a second location, below... for now -->
<h2>
<b>First Run of Dashboard</b>
</h2>
<p>This looks like the first time you've run this dashboard, or the csv file that makes this work isn't there anymore.</p>
<p>This dashboard is based off a csv file that is built via a query under the "Actions: Rerun Button" below. Pressing that should pop out in a new query window and provide an 'outputlookup' to create the exploremydata.csv file. There will be a warning about this query. If you're trepidatious about this, click 'investigate' on that warning and remove the outputlookup command - then run it and see what it does. </p>
<p>If this dashboard is used regularlly, consider creating a scheduled search from the 'Rerun Button' query to update the csv file on a recurring basis. </p>
</html>
</details>
</details>
</html>
</panel>
</row>
<row depends="$trellis_stats$">
<panel>
<html>
<!-- used to put the trellis in the center -->
<style>
#middle .facets-container{
display: flex !important;
justify-content: center !important;
}
</style>
</html>
<single id="middle">
<search>
<query>|inputlookup exploremydata.csv
|stats dc(index) as indexs dc(sourcetype) as sourcetypes dc(field) as fields count as combinations</query>
<earliest>-24h@h</earliest>
<latest>now</latest>
<sampleRatio>1</sampleRatio>
<progress>
<condition match="$result.indexs$=="0"">
<set token="nocsv">true</set>
</condition>
</progress>
</search>
<option name="colorBy">value</option>
<option name="colorMode">block</option>
<option name="drilldown">none</option>
<option name="height">94</option>
<option name="numberPrecision">0</option>
<option name="rangeColors">["0xdc4e41","0x2B65EC"]</option>
<option name="rangeValues">[0]</option>
<option name="showSparkline">1</option>
<option name="showTrendIndicator">1</option>
<option name="trellis.enabled">1</option>
<option name="trellis.scales.shared">1</option>
<option name="trellis.size">small</option>
<option name="trendColorInterpretation">standard</option>
<option name="trendDisplayMode">absolute</option>
<option name="unitPosition">after</option>
<option name="useColors">1</option>
<option name="useThousandSeparators">1</option>
</single>
<html>
<center>
<i>(for reference only - use panel below to explore the data available)</i> - last updated: $last_run$
</center>
</html>
</panel>
</row>
<row depends="$search_for$">
<panel>
<title>Search Results</title>
<table>
<search>
<query>
|inputlookup exploremydata.csv
|search field=$search_for$ OR sourcetype=$search_for$ OR index=$search_for$
|eval foundas=case(match(field,"$search_for$"),"field",sourcetype="$search_for$","sourcetype",index="$search_for$","index")
|stats dc(field) as fieldc, values(field) as fieldv, dc(index) as indexc, values(index) as indexv, dc(sourcetype) as sourcetypec values(sourcetype) as sourcetypev by foundas
|eval search_term=case(foundas="field",fieldv, foundas="sourcetype",sourcetypev, foundas="index",indexv)
|eval fields=if(fieldc>5,fieldc,fieldv)
|eval sourcetypes=if(sourcetypec>5,sourcetypec,sourcetypev)
|eval indexs=if(indexc>5,indexc,indexv)
|rename foundas AS found_in
|mvexpand sourcetypes
|mvexpand indexs
|table search_term found_in, indexs sourcetypes fields
</query>
<done>
<set token="foundin">$row.found_in$</set>
</done>
</search>
<option name="count">10</option>
<option name="dataOverlayMode">none</option>
<option name="drilldown">true</option>
<option name="percentagesRow">false</option>
<option name="rowNumbers">false</option>
<option name="totalsRow">false</option>
<option name="wrap">true</option>
<drilldown>
<condition match="$row.found_in$="index"">
<set token="index_token">$row.indexs$</set>
<unset token="sourcetype_token"></unset>
<unset token="field_token"></unset>
</condition>
<condition match="$row.found_in$="sourcetype"">
<set token="index_token">$row.indexs$</set>
<set token="sourcetype_token">$row.sourcetypes$</set>
<unset token="field_token"></unset>
</condition>
<condition match="$row.found_in$="field"">
<set token="index_token">$row.indexs$</set>
<set token="sourcetype_token">$row.sourcetypes$</set>
<set token="field_token">$row.fields$</set>
</condition>
</drilldown>
</table>
</panel>
</row>
<row>
<panel>
<title>Select Index to Explore</title>
<table>
<search>
<query>|inputlookup exploremydata.csv
|dedup index
|fields index
|eval sorting=case(match(index,"^_\S+"),1,
match(sourcetype,"stash"),3,
true(),5)
|sort - sorting
|table index
</query>
<earliest>$lookback.earliest$</earliest>
<latest>$lookback.latest$</latest>
<sampleRatio>1</sampleRatio>
</search>
<drilldown>
<set token="index_token">$click.value$</set>
</drilldown>
<option name="count">10</option>
<option name="dataOverlayMode">none</option>
<option name="percentagesRow">false</option>
<option name="rowNumbers">false</option>
<option name="totalsRow">false</option>
<option name="wrap">true</option>
</table>
</panel>
<panel depends="$index_token$">
<title>select sourcetype to review fields</title>
<table>
<search>
<query>|inputlookup exploremydata.csv
|search index=$index_token$
|dedup index sourcetype
|fields index sourcetype</query>
<earliest>$lookback.earliest$</earliest>
<latest>$lookback.latest$</latest>
<sampleRatio>1</sampleRatio>
<done>
<unset token="field_token"></unset>
<unset token="form.field_token"></unset>
<unset token="sourcetype_token"></unset>
<unset token="form.sourcetype_token"></unset>
<unset token="search_for"></unset>
<unset token="form.search_for"></unset>
</done>
</search>
<drilldown>
<set token="sourcetype_token">$row.sourcetype$</set>
<unset token="field_token"></unset>
</drilldown>
<option name="count">10</option>
<option name="dataOverlayMode">none</option>
<option name="percentagesRow">false</option>
<option name="rowNumbers">false</option>
<option name="totalsRow">false</option>
<option name="wrap">true</option>
</table>
</panel>
</row>
<row depends="$sourcetype_token$">
<panel>
<title>field values for index="$index_token$" sourcetype="$sourcetype_token$"</title>
<html>
Click on field name to see sample values or<br/>
<a href="search?q=search%20index%3D$index_token$%20sourcetype%3D$sourcetype_token$%20%7Chead%2020%20%7Cfields%20*&display.page.search.mode=verbose&dispatch.sample_ratio=1&workload_pool=&earliest=-24h%40h&latest=now" target="_sample">
<b>click here</b> for complete sample events for $index_token$ $sourcetype_token$ </a>
</html>
<table>
<search>
<query>|inputlookup exploremydata.csv
|search index="$index_token$"
sourcetype="$sourcetype_token$"
|streamstats count
|eval sets = count % 5
|eval fields_{sets} = field
|stats values(fields_*) as fields_*
|rename fields_0 as fields, fields_1 as "fields ", fields_2 as "fields ", fields_3 as "fields ", fields_4 as "fields "</query>
<earliest>$lookback.earliest$</earliest>
<latest>$lookback.latest$</latest>
<sampleRatio>1</sampleRatio>
</search>
<drilldown>
<set token="field_token">$click.value2$</set>
</drilldown>
<option name="count">10</option>
<option name="dataOverlayMode">none</option>
<option name="percentagesRow">false</option>
<option name="rowNumbers">false</option>
<option name="totalsRow">false</option>
<option name="wrap">true</option>
</table>
</panel>
<panel depends="$field_token$">
<title>Sample values for "$field_token$" in index="$index_token$" sourcetype="$sourcetype_token$"</title>
<table>
<search>
<query>index="$index_token$"
sourcetype="$sourcetype_token$"
|top "$field_token$"
|head 10
|table "$field_token$"</query>
<earliest>$lookback.earliest$</earliest>
<latest>$lookback.latest$</latest>
<sampleRatio>1</sampleRatio>
</search>
<option name="count">10</option>
<option name="dataOverlayMode">none</option>
<option name="drilldown">none</option>
<option name="percentagesRow">false</option>
<option name="rowNumbers">false</option>
<option name="totalsRow">false</option>
<option name="wrap">true</option>
</table>
<html>If there are no results, adjust the timeframe selector above. Not all fields are in all events.</html>
</panel>
</row>
</form>
Comments
3 total
Been using this for a while, great dashboard!
search used to generate csv. I think this would be helpful to add. I had to decode the url encoding. \r\n| tstats count WHERE index=* OR index=_* GROUPBY index, sourcetype\r\n| rename index AS indexname, sourcetype AS sourcetypename\r\n| map maxsearches=200 search=\"| search index=\\"$indexname$\\" sourcetype=\\"$sourcetypename$\\" | head 1000 | fieldsummary | eval index=\\"$indexname$\\", sourcetype=\\"$sourcetypename$\\" | fields index, sourcetype, field\"\r\n|dedup index sourcetype field\r\n|eval last_found=(now())\r\n|convert ctime(last_found)\r\n|dedup index sourcetype field\r\n|outputlookup exploremydata.csv
How can you do this without the CVS?
Leave a comment
You must log in to post a comment.