manderso, Author at

Bucket Status Dashboard

Shows status of buckets per indexer host, when they rolled from warm to cold, and cold to frozen. Gives a timechart and table of each, as well as detailed bucket names per index & host.

<form>
  <label>Bucket Status</label>
  <fieldset submitButton="false">
    <input type="time" token="field1">
      <label></label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="multiselect" token="hostpicker" searchWhenChanged="true">
      <label>Which Host</label>
      <choice value="*">All</choice>
      <default>*</default>
      <initialValue>*</initialValue>
      <fieldForLabel>host</fieldForLabel>
      <fieldForValue>host</fieldForValue>
      <search>
        <query>index=_internal sourcetype=splunkd component=BucketMover warm_to_cold idx!="_i*" idx!="_a*" Role=Indexer 
| stats count by host 
|  fields - count</query>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </search>
      <delimiter>  OR host=</delimiter>
    </input>
    <input type="multiselect" token="idx_picker" searchWhenChanged="true">
      <label>Which Index</label>
      <choice value="*">All</choice>
      <default>*</default>
      <initialValue>*</initialValue>
      <fieldForLabel>idx_name</fieldForLabel>
      <fieldForValue>idx_name</fieldForValue>
      <search>
        <query>index=_internal sourcetype=splunkd component=BucketMover "*starting warm_to_cold*" OR "*freeze succeeded*"
| rex field=event_message "indexes\/(?&lt;idx_name&gt;.+?(?=\/))"
| stats count by idx_name 
|  fields - count</query>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </search>
      <delimiter>  OR idx_name=</delimiter>
    </input>
  </fieldset>
  <row>
    <panel>
      <title>Warm to Cold Timechart</title>
      <chart>
        <search>
          <query>index=_internal sourcetype=splunkd component=BucketMover "*starting warm_to_cold*" host=$hostpicker$
| rex field=event_message "indexes\/(?&lt;idx_name&gt;.+?(?=\/))"
| search idx_name=$idx_picker$ NOT idx_name="_*"
| timechart count by  idx_name</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.abbreviation">none</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.abbreviation">none</option>
        <option name="charting.axisY.scale">log</option>
        <option name="charting.axisY2.abbreviation">none</option>
        <option name="charting.axisY2.enabled">0</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">line</option>
        <option name="charting.chart.bubbleMaximumSize">50</option>
        <option name="charting.chart.bubbleMinimumSize">10</option>
        <option name="charting.chart.bubbleSizeBy">area</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.showDataLabels">none</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">none</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.mode">standard</option>
        <option name="charting.legend.placement">right</option>
        <option name="charting.lineWidth">2</option>
        <option name="refresh.display">progressbar</option>
        <option name="trellis.enabled">0</option>
        <option name="trellis.scales.shared">1</option>
        <option name="trellis.size">medium</option>
      </chart>
    </panel>
    <panel>
      <title>Cold to Frozen Timechart</title>
      <chart>
        <search>
          <query>index=_internal sourcetype=splunkd component=BucketMover "*will attempt to freeze*" host=$hostpicker$
| rex field=event_message "indexes\/(?&lt;idx_name&gt;.+?(?=\/))"
| search idx_name=$idx_picker$ NOT idx_name="_*"
| timechart count by  idx_name</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.abbreviation">none</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.abbreviation">none</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.abbreviation">none</option>
        <option name="charting.axisY2.enabled">0</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">line</option>
        <option name="charting.chart.bubbleMaximumSize">50</option>
        <option name="charting.chart.bubbleMinimumSize">10</option>
        <option name="charting.chart.bubbleSizeBy">area</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.showDataLabels">none</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">none</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.mode">standard</option>
        <option name="charting.legend.placement">right</option>
        <option name="charting.lineWidth">2</option>
        <option name="refresh.display">progressbar</option>
        <option name="trellis.enabled">0</option>
        <option name="trellis.scales.shared">1</option>
        <option name="trellis.size">medium</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <title>Buckets moving from warm to cold</title>
      <table>
        <search>
          <query>index=_internal sourcetype=splunkd component=BucketMover warm_to_cold idx!="_i*" idx!="_a*" host=$hostpicker$ 
| rex field=event_message "indexes\/(?&lt;idx_name&gt;.+?(?=\/))"
| search idx_name=$idx_picker$ NOT idx_name="_*"
| stats count by  idx_name
| sort - count</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">10</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">none</option>
        <option name="percentagesRow">false</option>
        <option name="refresh.display">progressbar</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
      </table>
    </panel>
    <panel>
      <title>Buckets rolling to Frozen</title>
      <table>
        <search>
          <query>index=_internal sourcetype=splunkd component=BucketMover candidate!="*_i*" candidate!="*_a*" host=$hostpicker$ "*will attempt to freeze*"
| rex field=event_message "indexes\/(?&lt;idx_name&gt;.+?(?=\/))"
| search idx_name=$idx_picker$ NOT idx_name="_*"
| stats count by  idx_name
| sort - count</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">10</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">none</option>
        <option name="percentagesRow">false</option>
        <option name="refresh.display">progressbar</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <title>Index, Host and Bucket</title>
      <table>
        <title>cold to frozen uses bkt, warm to cold uses bucket</title>
        <search>
          <query>index=_internal sourcetype=splunkd component=BucketMover "*starting warm_to_cold*" OR "*freeze succeeded*" host=$hostpicker$
| rex field=event_message "indexes\/(?&lt;idx_name&gt;.+?(?=\/))"
| fillnull value=NULL
| search idx_name=$idx_picker$ NOT idx_name="_*"
| eval Message=if(like(event_message, "%warm_to_cold%"), "Warm to Cold", "Cold to Frozen")
| stats count by idx_name bucket bkt host Message| fields - count</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">25</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">none</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
      </table>
    </panel>
  </row>
</form>

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

<form>

<label>Bucket Status</label>

</default>

</input>

<label>Which Host</label>

<query>index=_internal sourcetype=splunkd component=BucketMover warm_to_cold idx!="_i*" idx!="_a*" Role=Indexer

| stats count by host

| fields - count</query>

</search>

</input>

<label>Which Index</label>

<query>index=_internal sourcetype=splunkd component=BucketMover "*starting warm_to_cold*" OR "*freeze succeeded*"

| rex field=event_message "indexes\/(?<idx_name>.+?(?=\/))"

| stats count by idx_name

| fields - count</query>

</search>

</input>

</fieldset>

<row>

<panel>

<title>Warm to Cold Timechart</title>

<chart>

<query>index=_internal sourcetype=splunkd component=BucketMover "*starting warm_to_cold*" host=$hostpicker$

| rex field=event_message "indexes\/(?<idx_name>.+?(?=\/))"

| search idx_name=$idx_picker$ NOT idx_name="_*"

| timechart count by idx_name</query>

<earliest>$field1.earliest$</earliest>

<latest>$field1.latest$</latest>

</search>

<option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>

<option name="charting.axisTitleX.visibility">visible</option>

<option name="charting.axisTitleY.visibility">visible</option>

<option name="charting.axisTitleY2.visibility">visible</option>

<option name="charting.axisX.scale">linear</option>

<option name="charting.axisY2.scale">inherit</option>

<option name="charting.chart.stackMode">default</option>

<option name="charting.chart.style">shiny</option>

<option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>

<option name="charting.legend.mode">standard</option>

<option name="charting.legend.placement">right</option>

<option name="refresh.display">progressbar</option>

<option name="trellis.size">medium</option>

</chart>

</panel>

<panel>

<title>Cold to Frozen Timechart</title>

<chart>

<query>index=_internal sourcetype=splunkd component=BucketMover "*will attempt to freeze*" host=$hostpicker$

| rex field=event_message "indexes\/(?<idx_name>.+?(?=\/))"

| search idx_name=$idx_picker$ NOT idx_name="_*"

| timechart count by idx_name</query>

<earliest>$field1.earliest$</earliest>

<latest>$field1.latest$</latest>

</search>

<option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>

<option name="charting.axisTitleX.visibility">visible</option>

<option name="charting.axisTitleY.visibility">visible</option>

<option name="charting.axisTitleY2.visibility">visible</option>

<option name="charting.axisX.scale">linear</option>

<option name="charting.axisY.scale">linear</option>

<option name="charting.axisY2.scale">inherit</option>

<option name="charting.chart.stackMode">default</option>

<option name="charting.chart.style">shiny</option>

<option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>

<option name="charting.legend.mode">standard</option>

<option name="charting.legend.placement">right</option>

<option name="refresh.display">progressbar</option>

<option name="trellis.size">medium</option>

</chart>

</panel>

</row>

<row>

<panel>

<title>Buckets moving from warm to cold</title>

<table>

<query>index=_internal sourcetype=splunkd component=BucketMover warm_to_cold idx!="_i*" idx!="_a*" host=$hostpicker$

| rex field=event_message "indexes\/(?<idx_name>.+?(?=\/))"

| search idx_name=$idx_picker$ NOT idx_name="_*"

| stats count by idx_name

| sort - count</query>

<earliest>$field1.earliest$</earliest>

<latest>$field1.latest$</latest>

</search>

<option name="percentagesRow">false</option>

<option name="refresh.display">progressbar</option>

<option name="rowNumbers">false</option>

<option name="totalsRow">false</option>

</table>

</panel>

<panel>

<title>Buckets rolling to Frozen</title>

<table>

<query>index=_internal sourcetype=splunkd component=BucketMover candidate!="*_i*" candidate!="*_a*" host=$hostpicker$ "*will attempt to freeze*"

| rex field=event_message "indexes\/(?<idx_name>.+?(?=\/))"

| search idx_name=$idx_picker$ NOT idx_name="_*"

| stats count by idx_name

| sort - count</query>

<earliest>$field1.earliest$</earliest>

<latest>$field1.latest$</latest>

</search>

<option name="percentagesRow">false</option>

<option name="refresh.display">progressbar</option>

<option name="rowNumbers">false</option>

<option name="totalsRow">false</option>

</table>

</panel>

</row>

<row>

<panel>

<title>Index, Host and Bucket</title>

<table>

<title>cold to frozen uses bkt, warm to cold uses bucket</title>

<query>index=_internal sourcetype=splunkd component=BucketMover "*starting warm_to_cold*" OR "*freeze succeeded*" host=$hostpicker$

| rex field=event_message "indexes\/(?<idx_name>.+?(?=\/))"

| fillnull value=NULL

| search idx_name=$idx_picker$ NOT idx_name="_*"

| eval Message=if(like(event_message, "%warm_to_cold%"), "Warm to Cold", "Cold to Frozen")

| stats count by idx_name bucket bkt host Message| fields - count</query>

<earliest>$field1.earliest$</earliest>

<latest>$field1.latest$</latest>

</search>

<option name="percentagesRow">false</option>

<option name="rowNumbers">false</option>

<option name="totalsRow">false</option>

</table>

</panel>

</row>

</form>

Continue Reading →

License and Storage Usage Dashboard

This relies on the search posted earlier: This will display storage and license usage broken down by groups, predefined in the chargeback app customers.csv

<form>
<label>License and Storage Usage</label>
<fieldset submitButton="false">
<input type="dropdown" token="grouppicker">
<label>Group</label>
<choice value="Group1">Group1</choice>
<choice value="Group2">Group2</choice>
<choice value="Group3">Group3</choice>
<choice value="Group4">Group4</choice>
<choice value="Group5">Group5</choice>
<choice value="Group6">Group6</choice>
<choice value="*">All Groups</choice>
<default>*</default>
</input>
</fieldset>
<row>
<panel>
<title>All Storage Usage</title>
<single>
<search>
<query>| inputlookup size_license_group_usage.csv 
| search group=$grouppicker$ 
| stats sum(currentDBSizeMB) AS SplunkIndexSizeMB
| eval SplunkIndexSize=case( 
SplunkIndexSizeMB&gt;=(1024*1024),round(SplunkIndexSizeMB/(1024*1024),2)."TB",
SplunkIndexSizeMB&gt;=(1024),round(SplunkIndexSizeMB/(1024),2)."GB",
1=1,SplunkIndexSizeMB."B") | fields SplunkIndexSize</query>
<earliest>-24h@h</earliest>
<latest>now</latest>
<sampleRatio>1</sampleRatio>
</search>
<option name="drilldown">none</option>
<option name="rangeColors">["0x53a051","0x0877a6","0xf8be34","0xf1813f","0xdc4e41"]</option>
<option name="refresh.display">progressbar</option>
</single>
</panel>
<panel>
<title>Yesterday's License Usage</title>
<single>
<search>
<query>| inputlookup size_license_group_usage.csv 
| search group=$grouppicker$ 
| rename sum(Yesterday) AS ydaydata 
| stats sum(ydaydata) as ydaydata 
| eval volume_converted=case(
ydaydata&gt;=(1024),round(ydaydata/(1024),2)."TB",
1=1,round(ydaydata,2)."GB") 
| rename volume_converted AS "Yesterday's License Usage" 
| fields "Yesterday's License Usage"</query>
<earliest>-24h@h</earliest>
<latest>now</latest>
<sampleRatio>1</sampleRatio>
</search>
<option name="drilldown">none</option>
<option name="rangeColors">["0x53a051","0x0877a6","0xf8be34","0xf1813f","0xdc4e41"]</option>
<option name="refresh.display">progressbar</option>
</single>
</panel>
</row>
<row>
<panel>
<title>License and Storage for Selected Group by Index</title>
<table>
<search>
<query>| inputlookup size_license_group_usage.csv 
| search group=$grouppicker$ 
| rename sum(Average) AS "Average GB License Daily" 
| rename sum(Yesterday) AS "Yesterdays GB License Usage" 
| eval "Index Storage Usage MB" = tostring(currentDBSizeMB, "commas") 
| fields group indexname "Average GB License Daily" "Yesterdays GB License Usage" "Index Storage Usage MB"</query>
<earliest>-24h@h</earliest>
<latest>now</latest>
<sampleRatio>1</sampleRatio>
</search>
<option name="count">100</option>
<option name="dataOverlayMode">none</option>
<option name="drilldown">none</option>
<option name="percentagesRow">false</option>
<option name="rowNumbers">false</option>
<option name="totalsRow">false</option>
<option name="wrap">true</option>
<format type="number" field="Average GB License Daily"></format>
<format type="number" field="Yesterdays GB License Usage"></format>
<format type="color" field="group">
<colorPalette type="sharedList"></colorPalette>
<scale type="sharedCategory"></scale>
</format>
</table>
</panel>
</row>
</form>

<form>

<label>License and Storage Usage</label>

<label>Group</label>

<choice value="Group1">Group1</choice>

<choice value="Group2">Group2</choice>

<choice value="Group3">Group3</choice>

<choice value="Group4">Group4</choice>

<choice value="Group5">Group5</choice>

<choice value="Group6">Group6</choice>

<choice value="*">All Groups</choice>

</input>

</fieldset>

<row>

<panel>

<title>All Storage Usage</title>

<query>| inputlookup size_license_group_usage.csv

| search group=$grouppicker$

| stats sum(currentDBSizeMB) AS SplunkIndexSizeMB

| eval SplunkIndexSize=case(

SplunkIndexSizeMB>=(1024*1024),round(SplunkIndexSizeMB/(1024*1024),2)."TB",

SplunkIndexSizeMB>=(1024),round(SplunkIndexSizeMB/(1024),2)."GB",

1=1,SplunkIndexSizeMB."B") | fields SplunkIndexSize</query>

</search>

<option name="refresh.display">progressbar</option>

</single>

</panel>

<panel>

<title>Yesterday's License Usage</title>

<query>| inputlookup size_license_group_usage.csv

| search group=$grouppicker$

| rename sum(Yesterday) AS ydaydata

| stats sum(ydaydata) as ydaydata

| eval volume_converted=case(

ydaydata>=(1024),round(ydaydata/(1024),2)."TB",

1=1,round(ydaydata,2)."GB")

| rename volume_converted AS "Yesterday's License Usage"

| fields "Yesterday's License Usage"</query>

</search>

<option name="refresh.display">progressbar</option>

</single>

</panel>

</row>

<row>

<panel>

<title>License and Storage for Selected Group by Index</title>

<table>

<query>| inputlookup size_license_group_usage.csv

| search group=$grouppicker$

| rename sum(Average) AS "Average GB License Daily"

| rename sum(Yesterday) AS "Yesterdays GB License Usage"

| eval "Index Storage Usage MB" = tostring(currentDBSizeMB, "commas")

| fields group indexname "Average GB License Daily" "Yesterdays GB License Usage" "Index Storage Usage MB"</query>

</search>

<option name="percentagesRow">false</option>

<option name="rowNumbers">false</option>

<option name="totalsRow">false</option>

</format>

</table>

</panel>

</row>

</form>

Continue Reading →

Build License usage by Group

This was cobbled together from multiple searches I found. This search feeds the license and storage dashboard posted here: It relies on the Chargeback app for the customers.csv form.

index=_internal source=*license_usage.log type="Usage" earliest=-30d@d latest=@d 
| eval indexname = if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx) 
| eval sourcetypename = st 
| bin _time span=1d 
| stats values(poolsz) as poolsz sum(b) as b by _time, pool, indexname, sourcetypename 
| eval GB=(b/1024/1024/1024) 
| eval pool=(poolsz/1024/1024/1024) 
| fields _time, indexname, sourcetypename, GB, pool 
| eval day=strftime(_time,"%a") 
| eval yesterday=tostring(strftime(relative_time(now(),"-d@d"),"%Y-%m-%d")) 
| eval time=tostring(strftime(_time,"%Y-%m-%d")) 
| eval isyesterday=if(yesterday==time,"true","is_average") 
| eval Yesterday=if(isyesterday="true",GB,0) 
| stats values(pool) as pool sum(Yesterday) as Yesterday avg(GB) as Average by indexname, sourcetypename 
| stats values(pool) as pool list(sourcetypename) sum(Yesterday) sum(Average) by indexname 
| join indexname 
[| inputlookup customers.csv 
| rename idx AS indexname 
| fields group indexname
] 
| join indexname 
[| rest /services/data/indexes search="totalEventCount!=0" 
| stats sum(currentDBSizeMB) AS currentDBSizeMB by title 
| rename title AS indexname 
| fields indexname currentDBSizeMB] 
| fields group indexname sum(Yesterday) sum(Average) currentDBSizeMB 
| sort - sum(Yesterday) | outputlookup size_license_group_usage.csv

index=_internal source=*license_usage.log type="Usage" earliest=-30d@d latest=@d

| eval indexname = if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx)

| eval sourcetypename = st

| bin _time span=1d

| stats values(poolsz) as poolsz sum(b) as b by _time, pool, indexname, sourcetypename

| eval GB=(b/1024/1024/1024)

| eval pool=(poolsz/1024/1024/1024)

| fields _time, indexname, sourcetypename, GB, pool

| eval day=strftime(_time,"%a")

| eval yesterday=tostring(strftime(relative_time(now(),"-d@d"),"%Y-%m-%d"))

| eval time=tostring(strftime(_time,"%Y-%m-%d"))

| eval isyesterday=if(yesterday==time,"true","is_average")

| eval Yesterday=if(isyesterday="true",GB,0)

| stats values(pool) as pool sum(Yesterday) as Yesterday avg(GB) as Average by indexname, sourcetypename

| stats values(pool) as pool list(sourcetypename) sum(Yesterday) sum(Average) by indexname

| join indexname

[| inputlookup customers.csv

| rename idx AS indexname

| fields group indexname

]

| join indexname

[| rest /services/data/indexes search="totalEventCount!=0"

| stats sum(currentDBSizeMB) AS currentDBSizeMB by title

| rename title AS indexname

| fields indexname currentDBSizeMB]

| fields group indexname sum(Yesterday) sum(Average) currentDBSizeMB

| sort - sum(Yesterday) | outputlookup size_license_group_usage.csv

Continue Reading →

Auditd hosts in all environments

Shows the login activity to our linux environments, sudo commands per host and users. Admin Notes: index=main was changed to index=* due to not everyone using the same index. This dashboard has been tested for code errors, but not for search errors. Please comment if you have any issues!

<form>
  <label>Audit All Hosts</label>
  <fieldset submitButton="false">
    <input type="time" token="field1">
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="dropdown" token="field2" searchWhenChanged="true">
      <label>Environment</label>
      <choice value="*">All</choice>
      <choice value="*dev*">DEV</choice>
      <choice value="*prd*">PROD</choice>
      <choice value="*int*">INTG</choice>
      <choice value="*tst*">TEST</choice>
      <choice value="*inf*">INF</choice>
      <choice value="*qa*">QA</choice>
      <fieldForLabel>env</fieldForLabel>
      <fieldForValue>env</fieldForValue>
      <default>*</default>
      <initialValue>*</initialValue>
    </input>
  </fieldset>
  <row>
    <panel>
      <title>Audit Auth Logs By GeoIp</title>
      <map>
        <title>(ssh originating locations, not updated with Environment dropdown)</title>
        <search>
          <query>index=* "ssh" "audit.res"=success type=USER_LOGIN hostname=*| iplocation addr | geostats latfield=lat longfield=lon count</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="drilldown">none</option>
        <option name="mapping.choroplethLayer.colorBins">5</option>
        <option name="mapping.choroplethLayer.colorMode">auto</option>
        <option name="mapping.choroplethLayer.maximumColor">0xaf575a</option>
        <option name="mapping.choroplethLayer.minimumColor">0x62b3b2</option>
        <option name="mapping.choroplethLayer.neutralPoint">0</option>
        <option name="mapping.choroplethLayer.shapeOpacity">0.75</option>
        <option name="mapping.choroplethLayer.showBorder">1</option>
        <option name="mapping.data.maxClusters">100</option>
        <option name="mapping.legend.placement">bottomright</option>
        <option name="mapping.map.center">(38.41,-108.41)</option>
        <option name="mapping.map.panning">1</option>
        <option name="mapping.map.scrollZoom">0</option>
        <option name="mapping.map.zoom">4</option>
        <option name="mapping.markerLayer.markerMaxSize">50</option>
        <option name="mapping.markerLayer.markerMinSize">10</option>
        <option name="mapping.markerLayer.markerOpacity">0.8</option>
        <option name="mapping.showTiles">1</option>
        <option name="mapping.tileLayer.maxZoom">7</option>
        <option name="mapping.tileLayer.minZoom">0</option>
        <option name="mapping.tileLayer.tileOpacity">1</option>
        <option name="mapping.type">marker</option>
        <option name="refresh.display">progressbar</option>
        <option name="trellis.enabled">0</option>
        <option name="trellis.scales.shared">1</option>
        <option name="trellis.size">medium</option>
      </map>
    </panel>
    <panel>
      <title>Failed Auth by Host</title>
      <chart>
        <search>
          <query>index=* "failed" extracted_source="/var/log/audit/audit.log" "audit.type"=USER_LOGIN hostname=$field2$ | bin size bins=30 |timechart count by hostname</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.abbreviation">none</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.abbreviation">none</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.abbreviation">none</option>
        <option name="charting.axisY2.enabled">0</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">line</option>
        <option name="charting.chart.bubbleMaximumSize">50</option>
        <option name="charting.chart.bubbleMinimumSize">10</option>
        <option name="charting.chart.bubbleSizeBy">area</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.showDataLabels">minmax</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">none</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisEnd</option>
        <option name="charting.legend.mode">standard</option>
        <option name="charting.legend.placement">bottom</option>
        <option name="charting.lineWidth">2</option>
        <option name="refresh.display">progressbar</option>
        <option name="trellis.enabled">0</option>
        <option name="trellis.scales.shared">1</option>
        <option name="trellis.size">medium</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <title>Top Sudoers</title>
      <table>
        <search>
          <query>index=* extracted_source="/var/log/secure" sudoer!=nrpe hostname=$field2$| stats count by sudoer command | sort - count</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">10</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">none</option>
        <option name="percentagesRow">false</option>
        <option name="refresh.display">progressbar</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
      </table>
    </panel>
    <panel>
      <title>Sudoers</title>
      <viz type="simple_xml_examples.tagcloud">
        <search>
          <query>index=* "su" extracted_source="/var/log/secure" hostname=$field2$ | stats count by sudoer</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
        <option name="simple_xml_examples.tagcloud.labelField">sudoer</option>
        <option name="simple_xml_examples.tagcloud.maxFontSize">36</option>
        <option name="simple_xml_examples.tagcloud.minFontSize">8</option>
        <option name="simple_xml_examples.tagcloud.valueField">count</option>
        <option name="trellis.enabled">0</option>
        <option name="trellis.scales.shared">1</option>
        <option name="trellis.size">medium</option>
      </viz>
    </panel>
    <panel>
      <title>Auditd Top Accounts</title>
      <viz type="simple_xml_examples.tagcloud">
        <search>
          <query>index=* "su" audit.log.acct=* extracted_source="/var/log/audit/audit.log" hostname=$field2$ | top audit.log.acct</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
        <option name="simple_xml_examples.tagcloud.labelField">audit.log.acct</option>
        <option name="simple_xml_examples.tagcloud.maxFontSize">48</option>
        <option name="simple_xml_examples.tagcloud.minFontSize">12</option>
        <option name="simple_xml_examples.tagcloud.valueField">count</option>
      </viz>
    </panel>
  </row>
  <row>
    <panel>
      <title>Sudo count by User By Command By Host</title>
      <table>
        <search>
          <query>index=* "su" extracted_source="/var/log/secure" hostname=$field2$| stats count by sudoer command hostname</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">10</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">none</option>
        <option name="percentagesRow">false</option>
        <option name="refresh.display">progressbar</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
      </table>
    </panel>
  </row>
</form>

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

<form>

<label>Audit All Hosts</label>

</default>

</input>

<label>Environment</label>

</input>

</fieldset>

<row>

<panel>

<title>Audit Auth Logs By GeoIp</title>

<map>

<title>(ssh originating locations, not updated with Environment dropdown)</title>

<query>index=* "ssh" "audit.res"=success type=USER_LOGIN hostname=*| iplocation addr | geostats latfield=lat longfield=lon count</query>

<earliest>$field1.earliest$</earliest>

<latest>$field1.latest$</latest>

</search>

<option name="mapping.legend.placement">bottomright</option>

<option name="mapping.type">marker</option>

<option name="refresh.display">progressbar</option>

<option name="trellis.size">medium</option>

</map>

</panel>

<panel>

<title>Failed Auth by Host</title>

<chart>

<query>index=* "failed" extracted_source="/var/log/audit/audit.log" "audit.type"=USER_LOGIN hostname=$field2$ | bin size bins=30 |timechart count by hostname</query>

<earliest>$field1.earliest$</earliest>

<latest>$field1.latest$</latest>

</search>

<option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>

<option name="charting.axisTitleX.visibility">visible</option>

<option name="charting.axisTitleY.visibility">visible</option>

<option name="charting.axisTitleY2.visibility">visible</option>

<option name="charting.axisX.scale">linear</option>

<option name="charting.axisY.scale">linear</option>

<option name="charting.axisY2.scale">inherit</option>

<option name="charting.chart.showDataLabels">minmax</option>

<option name="charting.chart.stackMode">default</option>

<option name="charting.chart.style">shiny</option>

<option name="charting.legend.labelStyle.overflowMode">ellipsisEnd</option>

<option name="charting.legend.mode">standard</option>

<option name="charting.legend.placement">bottom</option>

<option name="refresh.display">progressbar</option>

<option name="trellis.size">medium</option>

</chart>

</panel>

</row>

<row>

<panel>

<title>Top Sudoers</title>

<table>

<query>index=* extracted_source="/var/log/secure" sudoer!=nrpe hostname=$field2$| stats count by sudoer command | sort - count</query>

<earliest>$field1.earliest$</earliest>

<latest>$field1.latest$</latest>

</search>

<option name="percentagesRow">false</option>

<option name="refresh.display">progressbar</option>

<option name="rowNumbers">false</option>

<option name="totalsRow">false</option>

</table>

</panel>

<panel>

<title>Sudoers</title>

<query>index=* "su" extracted_source="/var/log/secure" hostname=$field2$ | stats count by sudoer</query>

<earliest>$field1.earliest$</earliest>

<latest>$field1.latest$</latest>

</search>

<option name="refresh.display">progressbar</option>

<option name="simple_xml_examples.tagcloud.labelField">sudoer</option>

<option name="simple_xml_examples.tagcloud.valueField">count</option>

<option name="trellis.size">medium</option>

</viz>

</panel>

<panel>

<title>Auditd Top Accounts</title>

<query>index=* "su" audit.log.acct=* extracted_source="/var/log/audit/audit.log" hostname=$field2$ | top audit.log.acct</query>

<earliest>$field1.earliest$</earliest>

<latest>$field1.latest$</latest>

</search>

<option name="refresh.display">progressbar</option>

<option name="simple_xml_examples.tagcloud.labelField">audit.log.acct</option>

<option name="simple_xml_examples.tagcloud.valueField">count</option>

</viz>

</panel>

</row>

<row>

<panel>

<title>Sudo count by User By Command By Host</title>

<table>

<query>index=* "su" extracted_source="/var/log/secure" hostname=$field2$| stats count by sudoer command hostname</query>

<earliest>$field1.earliest$</earliest>

<latest>$field1.latest$</latest>

</search>

<option name="percentagesRow">false</option>

<option name="refresh.display">progressbar</option>

<option name="rowNumbers">false</option>

<option name="totalsRow">false</option>

</table>

</panel>

</row>

</form>

Continue Reading →

Top Header cpu & memory status

I didn’t like the CPU input from the Splunk TA Nix app, so I created this small ingest from top. The script takes a snapshot of the top command, and looks directly at the header:

top -b -n 1 | sed -n '1,5p'

1	top -b -n 1 \| sed -n '1,5p'

and comes back with the first 5 lines of Top:

top - 15:20:55 up 26 days, 9:53, 1 user, load average: 0.89, 0.59, 0.48
 Tasks: 125 total, 1 running, 124 sleeping, 0 stopped, 0 zombie
 Cpu(s): 3.5%us, 0.7%sy, 0.0%ni, 95.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
 Mem: 8059752k total, 4138600k used, 3921152k free, 759256k buffers
 Swap: 2097148k total, 96532k used, 2000616k free, 2228520k cached

top - 15:20:55 up 26 days, 9:53, 1 user, load average: 0.89, 0.59, 0.48

Tasks: 125 total, 1 running, 124 sleeping, 0 stopped, 0 zombie

Cpu(s): 3.5%us, 0.7%sy, 0.0%ni, 95.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 8059752k total, 4138600k used, 3921152k free, 759256k buffers

Swap: 2097148k total, 96532k used, 2000616k free, 2228520k cached

and the following query pulls CPU load average data […]

Continue Reading →

Show uptime in Days

The following query shows uptime of all systems over a certain period of time (days_uptime). Replace my indexes w/ yours.

 index=os OR index=idx_appdev sourcetype=Unix:Uptime OR sourcetype="WMI:Uptime" |dedup host |eval DaysUp=round(SystemUpTime/86400,2) |eval Years=round(DaysUp/365,2) |eval Months=round(DaysUp/30,2)|search DaysUp > $days_uptime$ | table host DaysUp Years Months SystemUpTime |sort - SystemUpTime |

1	index=os OR index=idx_appdev sourcetype=Unix:Uptime OR sourcetype="WMI:Uptime" \|dedup host \|eval DaysUp=round(SystemUpTime/86400,2) \|eval Years=round(DaysUp/365,2) \|eval Months=round(DaysUp/30,2)\|search DaysUp > $days_uptime$ \| table host DaysUp Years Months SystemUpTime \|sort - SystemUpTime \|

Looks like: hostname | DaysUP | Years | Months | SystemUpTime and $days_uptime$ is a text box in my case.

Continue Reading →

Search to show what apps are ready to be updated

| rest splunk_server=local /services/apps/local | search update.version=* | table title version update.version

1	\| rest splunk_server=local /services/apps/local \| search update.version=* \| table title version update.version

If that Splunk has internet access, it’ll have the

<i><span style="font-size: 10.0pt; color: #333333;">update.*</span></i>

1	<i><span style="font-size: 10.0pt; color: #333333;">update.*</span></i>

fields filled with the latest version if there is an update available for any app installed on that system. The

<i><span style="font-size: 10.0pt; color: #333333;">splunk_server</span></i>

1	<i><span style="font-size: 10.0pt; color: #333333;">splunk_server</span></i>

filter should be usable for querying search peers as well. Using that scheduled daily or weekly, you could alert yourself of any update. […]

Continue Reading →

Permissions for splunk users

Another view for which splunk user can do what in your splunk environment

| rest /services/authentication/users | mvexpand roles | table realname, title, roles, email | join roles [ rest /services/authorization/roles | rename title as roles | search srchIndexesAllowed=* | table roles srchIndexesAllowed]

1	\| rest /services/authentication/users \| mvexpand roles \| table realname, title, roles, email \| join roles [ rest /services/authorization/roles \| rename title as roles \| search srchIndexesAllowed=* \| table roles srchIndexesAllowed]

Continue Reading →

Author: manderso