_internal Archives -

Deployed application status

Created this dashboard to see when or if an application was deployed successfully. Close to splunkninja’s query, this will also show if the host in question also restarted to apply the new app.

<form>
  <label>Deployed Applications</label>
  <fieldset submitButton="false">
    <input type="checkbox" token="loglevelpicker" searchWhenChanged="true">
      <label>Log Level</label>
      <choice value="INFO">INFO</choice>
      <choice value="WARN*">WARNING</choice>
      <choice value="ERROR">ERROR</choice>
      <default>INFO,WARN*,ERROR</default>
      <valuePrefix>log_level=</valuePrefix>
      <delimiter> OR </delimiter>
    </input>
    <input type="time" token="field1">
      <label></label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="multiselect" token="hostpicker">
      <label>Host</label>
      <choice value="*">All</choice>
      <default>*</default>
      <valuePrefix>host=</valuePrefix>
      <delimiter> OR </delimiter>
      <fieldForLabel>host</fieldForLabel>
      <fieldForValue>host</fieldForValue>
      <search>
        <query>index=_internal sourcetype=splunkd component=DeployedApplication
| stats count by host</query>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </search>
    </input>
    <input type="multiselect" token="apppicker" searchWhenChanged="true">
      <label>Application</label>
      <choice value="*">All</choice>
      <valuePrefix>*</valuePrefix>
      <valueSuffix>*</valueSuffix>
      <delimiter> OR </delimiter>
      <fieldForLabel>applicationx</fieldForLabel>
      <fieldForValue>applicationx</fieldForValue>
      <search>
        <query>index=_internal sourcetype=splunkd component=DeployedApplication
| rex field=file "var(\/|\\\\)run(\/|\\\\)\w+(\/|\\\\)(?&lt;app2&gt;\w+)-" 
| rex field=message "(etc|run)(\/|\\\\)(apps|\w+)(\/|\\\\)(?&lt;app3&gt;\w+)-\d+\.bundle" 
| rex field=message "etc(\/|\\\\)apps(\/|\\\\)(?&lt;app5&gt;[^\/|\\\\|']+)" 
| eval applicationx=coalesce(app,app2,app3,app5,application) 
| stats count by applicationx 
| fields - count</query>
        <earliest>-7d@h</earliest>
        <latest>now</latest>
      </search>
    </input>
  </fieldset>
  <row>
    <panel>
      <table>
        <search>
          <query>index=_internal sourcetype=splunkd component=DeployedApplication $loglevelpicker$ $hostpicker$ $apppicker$
| table _time host app log_level event_message 
| sort - _time</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">50</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">none</option>
        <option name="percentagesRow">false</option>
        <option name="refresh.display">progressbar</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <title>Last restart time</title>
      <event>
        <search>
          <query>index=_internal sourcetype=splunkd log_level=INFO $hostpicker$ component=loader event_message="Splunkd starting*"</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">50</option>
        <option name="list.drilldown">none</option>
        <option name="list.wrap">1</option>
        <option name="maxLines">0</option>
        <option name="raw.drilldown">full</option>
        <option name="refresh.display">progressbar</option>
        <option name="rowNumbers">0</option>
        <option name="table.drilldown">all</option>
        <option name="table.sortDirection">asc</option>
        <option name="table.wrap">1</option>
        <option name="type">list</option>
      </event>
    </panel>
  </row>
</form>

100

101

102

<form>

<label>Deployed Applications</label>

<label>Log Level</label>

<choice value="WARN*">WARNING</choice>

<choice value="ERROR">ERROR</choice>

<default>INFO,WARN*,ERROR</default>

<valuePrefix>log_level=</valuePrefix>

</input>

</default>

</input>

<query>index=_internal sourcetype=splunkd component=DeployedApplication

| stats count by host</query>

</search>

</input>

<label>Application</label>

<fieldForLabel>applicationx</fieldForLabel>

<fieldForValue>applicationx</fieldForValue>

<query>index=_internal sourcetype=splunkd component=DeployedApplication

| rex field=file "var(\/|\\\\)run(\/|\\\\)\w+(\/|\\\\)(?<app2>\w+)-"

| rex field=message "etc(\/|\\\\)apps(\/|\\\\)(?<app5>[^\/|\\\\|']+)"

| eval applicationx=coalesce(app,app2,app3,app5,application)

| stats count by applicationx

| fields - count</query>

</search>

</input>

</fieldset>

<row>

<panel>

<table>

<query>index=_internal sourcetype=splunkd component=DeployedApplication $loglevelpicker$ $hostpicker$ $apppicker$

| table _time host app log_level event_message

| sort - _time</query>

<earliest>$field1.earliest$</earliest>

<latest>$field1.latest$</latest>

</search>

<option name="percentagesRow">false</option>

<option name="refresh.display">progressbar</option>

<option name="rowNumbers">false</option>

<option name="totalsRow">false</option>

</table>

</panel>

</row>

<row>

<panel>

<title>Last restart time</title>

<event>

<query>index=_internal sourcetype=splunkd log_level=INFO $hostpicker$ component=loader event_message="Splunkd starting*"</query>

<earliest>$field1.earliest$</earliest>

<latest>$field1.latest$</latest>

</search>

<option name="refresh.display">progressbar</option>

</event>

</panel>

</row>

</form>

Continue Reading →

REST API: Table all Splunk User Email Addresses

The following simple Splunk query will put all Splunk User accounts with an email address into a panel for copy and paste purposes (such as copying all email addresses to send in an email). I’ve added a semi colon delimiter in order to literally be copy and paste into an application such as Microsoft Outlook. […]

Continue Reading →

Bucket Status Dashboard

Shows status of buckets per indexer host, when they rolled from warm to cold, and cold to frozen. Gives a timechart and table of each, as well as detailed bucket names per index & host.

<form>
  <label>Bucket Status</label>
  <fieldset submitButton="false">
    <input type="time" token="field1">
      <label></label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="multiselect" token="hostpicker" searchWhenChanged="true">
      <label>Which Host</label>
      <choice value="*">All</choice>
      <default>*</default>
      <initialValue>*</initialValue>
      <fieldForLabel>host</fieldForLabel>
      <fieldForValue>host</fieldForValue>
      <search>
        <query>index=_internal sourcetype=splunkd component=BucketMover warm_to_cold idx!="_i*" idx!="_a*" Role=Indexer 
| stats count by host 
|  fields - count</query>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </search>
      <delimiter>  OR host=</delimiter>
    </input>
    <input type="multiselect" token="idx_picker" searchWhenChanged="true">
      <label>Which Index</label>
      <choice value="*">All</choice>
      <default>*</default>
      <initialValue>*</initialValue>
      <fieldForLabel>idx_name</fieldForLabel>
      <fieldForValue>idx_name</fieldForValue>
      <search>
        <query>index=_internal sourcetype=splunkd component=BucketMover "*starting warm_to_cold*" OR "*freeze succeeded*"
| rex field=event_message "indexes\/(?&lt;idx_name&gt;.+?(?=\/))"
| stats count by idx_name 
|  fields - count</query>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </search>
      <delimiter>  OR idx_name=</delimiter>
    </input>
  </fieldset>
  <row>
    <panel>
      <title>Warm to Cold Timechart</title>
      <chart>
        <search>
          <query>index=_internal sourcetype=splunkd component=BucketMover "*starting warm_to_cold*" host=$hostpicker$
| rex field=event_message "indexes\/(?&lt;idx_name&gt;.+?(?=\/))"
| search idx_name=$idx_picker$ NOT idx_name="_*"
| timechart count by  idx_name</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.abbreviation">none</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.abbreviation">none</option>
        <option name="charting.axisY.scale">log</option>
        <option name="charting.axisY2.abbreviation">none</option>
        <option name="charting.axisY2.enabled">0</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">line</option>
        <option name="charting.chart.bubbleMaximumSize">50</option>
        <option name="charting.chart.bubbleMinimumSize">10</option>
        <option name="charting.chart.bubbleSizeBy">area</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.showDataLabels">none</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">none</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.mode">standard</option>
        <option name="charting.legend.placement">right</option>
        <option name="charting.lineWidth">2</option>
        <option name="refresh.display">progressbar</option>
        <option name="trellis.enabled">0</option>
        <option name="trellis.scales.shared">1</option>
        <option name="trellis.size">medium</option>
      </chart>
    </panel>
    <panel>
      <title>Cold to Frozen Timechart</title>
      <chart>
        <search>
          <query>index=_internal sourcetype=splunkd component=BucketMover "*will attempt to freeze*" host=$hostpicker$
| rex field=event_message "indexes\/(?&lt;idx_name&gt;.+?(?=\/))"
| search idx_name=$idx_picker$ NOT idx_name="_*"
| timechart count by  idx_name</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>
        <option name="charting.axisLabelsX.majorLabelStyle.rotation">0</option>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisX.abbreviation">none</option>
        <option name="charting.axisX.scale">linear</option>
        <option name="charting.axisY.abbreviation">none</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.axisY2.abbreviation">none</option>
        <option name="charting.axisY2.enabled">0</option>
        <option name="charting.axisY2.scale">inherit</option>
        <option name="charting.chart">line</option>
        <option name="charting.chart.bubbleMaximumSize">50</option>
        <option name="charting.chart.bubbleMinimumSize">10</option>
        <option name="charting.chart.bubbleSizeBy">area</option>
        <option name="charting.chart.nullValueMode">gaps</option>
        <option name="charting.chart.showDataLabels">none</option>
        <option name="charting.chart.sliceCollapsingThreshold">0.01</option>
        <option name="charting.chart.stackMode">default</option>
        <option name="charting.chart.style">shiny</option>
        <option name="charting.drilldown">none</option>
        <option name="charting.layout.splitSeries">0</option>
        <option name="charting.layout.splitSeries.allowIndependentYRanges">0</option>
        <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>
        <option name="charting.legend.mode">standard</option>
        <option name="charting.legend.placement">right</option>
        <option name="charting.lineWidth">2</option>
        <option name="refresh.display">progressbar</option>
        <option name="trellis.enabled">0</option>
        <option name="trellis.scales.shared">1</option>
        <option name="trellis.size">medium</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <title>Buckets moving from warm to cold</title>
      <table>
        <search>
          <query>index=_internal sourcetype=splunkd component=BucketMover warm_to_cold idx!="_i*" idx!="_a*" host=$hostpicker$ 
| rex field=event_message "indexes\/(?&lt;idx_name&gt;.+?(?=\/))"
| search idx_name=$idx_picker$ NOT idx_name="_*"
| stats count by  idx_name
| sort - count</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">10</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">none</option>
        <option name="percentagesRow">false</option>
        <option name="refresh.display">progressbar</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
      </table>
    </panel>
    <panel>
      <title>Buckets rolling to Frozen</title>
      <table>
        <search>
          <query>index=_internal sourcetype=splunkd component=BucketMover candidate!="*_i*" candidate!="*_a*" host=$hostpicker$ "*will attempt to freeze*"
| rex field=event_message "indexes\/(?&lt;idx_name&gt;.+?(?=\/))"
| search idx_name=$idx_picker$ NOT idx_name="_*"
| stats count by  idx_name
| sort - count</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">10</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">none</option>
        <option name="percentagesRow">false</option>
        <option name="refresh.display">progressbar</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <title>Index, Host and Bucket</title>
      <table>
        <title>cold to frozen uses bkt, warm to cold uses bucket</title>
        <search>
          <query>index=_internal sourcetype=splunkd component=BucketMover "*starting warm_to_cold*" OR "*freeze succeeded*" host=$hostpicker$
| rex field=event_message "indexes\/(?&lt;idx_name&gt;.+?(?=\/))"
| fillnull value=NULL
| search idx_name=$idx_picker$ NOT idx_name="_*"
| eval Message=if(like(event_message, "%warm_to_cold%"), "Warm to Cold", "Cold to Frozen")
| stats count by idx_name bucket bkt host Message| fields - count</query>
          <earliest>$field1.earliest$</earliest>
          <latest>$field1.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">25</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">none</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
      </table>
    </panel>
  </row>
</form>

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

<form>

<label>Bucket Status</label>

</default>

</input>

<label>Which Host</label>

<query>index=_internal sourcetype=splunkd component=BucketMover warm_to_cold idx!="_i*" idx!="_a*" Role=Indexer

| stats count by host

| fields - count</query>

</search>

</input>

<label>Which Index</label>

<query>index=_internal sourcetype=splunkd component=BucketMover "*starting warm_to_cold*" OR "*freeze succeeded*"

| rex field=event_message "indexes\/(?<idx_name>.+?(?=\/))"

| stats count by idx_name

| fields - count</query>

</search>

</input>

</fieldset>

<row>

<panel>

<title>Warm to Cold Timechart</title>

<chart>

<query>index=_internal sourcetype=splunkd component=BucketMover "*starting warm_to_cold*" host=$hostpicker$

| rex field=event_message "indexes\/(?<idx_name>.+?(?=\/))"

| search idx_name=$idx_picker$ NOT idx_name="_*"

| timechart count by idx_name</query>

<earliest>$field1.earliest$</earliest>

<latest>$field1.latest$</latest>

</search>

<option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>

<option name="charting.axisTitleX.visibility">visible</option>

<option name="charting.axisTitleY.visibility">visible</option>

<option name="charting.axisTitleY2.visibility">visible</option>

<option name="charting.axisX.scale">linear</option>

<option name="charting.axisY2.scale">inherit</option>

<option name="charting.chart.stackMode">default</option>

<option name="charting.chart.style">shiny</option>

<option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>

<option name="charting.legend.mode">standard</option>

<option name="charting.legend.placement">right</option>

<option name="refresh.display">progressbar</option>

<option name="trellis.size">medium</option>

</chart>

</panel>

<panel>

<title>Cold to Frozen Timechart</title>

<chart>

<query>index=_internal sourcetype=splunkd component=BucketMover "*will attempt to freeze*" host=$hostpicker$

| rex field=event_message "indexes\/(?<idx_name>.+?(?=\/))"

| search idx_name=$idx_picker$ NOT idx_name="_*"

| timechart count by idx_name</query>

<earliest>$field1.earliest$</earliest>

<latest>$field1.latest$</latest>

</search>

<option name="charting.axisLabelsX.majorLabelStyle.overflowMode">ellipsisNone</option>

<option name="charting.axisTitleX.visibility">visible</option>

<option name="charting.axisTitleY.visibility">visible</option>

<option name="charting.axisTitleY2.visibility">visible</option>

<option name="charting.axisX.scale">linear</option>

<option name="charting.axisY.scale">linear</option>

<option name="charting.axisY2.scale">inherit</option>

<option name="charting.chart.stackMode">default</option>

<option name="charting.chart.style">shiny</option>

<option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option>

<option name="charting.legend.mode">standard</option>

<option name="charting.legend.placement">right</option>

<option name="refresh.display">progressbar</option>

<option name="trellis.size">medium</option>

</chart>

</panel>

</row>

<row>

<panel>

<title>Buckets moving from warm to cold</title>

<table>

<query>index=_internal sourcetype=splunkd component=BucketMover warm_to_cold idx!="_i*" idx!="_a*" host=$hostpicker$

| rex field=event_message "indexes\/(?<idx_name>.+?(?=\/))"

| search idx_name=$idx_picker$ NOT idx_name="_*"

| stats count by idx_name

| sort - count</query>

<earliest>$field1.earliest$</earliest>

<latest>$field1.latest$</latest>

</search>

<option name="percentagesRow">false</option>

<option name="refresh.display">progressbar</option>

<option name="rowNumbers">false</option>

<option name="totalsRow">false</option>

</table>

</panel>

<panel>

<title>Buckets rolling to Frozen</title>

<table>

<query>index=_internal sourcetype=splunkd component=BucketMover candidate!="*_i*" candidate!="*_a*" host=$hostpicker$ "*will attempt to freeze*"

| rex field=event_message "indexes\/(?<idx_name>.+?(?=\/))"

| search idx_name=$idx_picker$ NOT idx_name="_*"

| stats count by idx_name

| sort - count</query>

<earliest>$field1.earliest$</earliest>

<latest>$field1.latest$</latest>

</search>

<option name="percentagesRow">false</option>

<option name="refresh.display">progressbar</option>

<option name="rowNumbers">false</option>

<option name="totalsRow">false</option>

</table>

</panel>

</row>

<row>

<panel>

<title>Index, Host and Bucket</title>

<table>

<title>cold to frozen uses bkt, warm to cold uses bucket</title>

<query>index=_internal sourcetype=splunkd component=BucketMover "*starting warm_to_cold*" OR "*freeze succeeded*" host=$hostpicker$

| rex field=event_message "indexes\/(?<idx_name>.+?(?=\/))"

| fillnull value=NULL

| search idx_name=$idx_picker$ NOT idx_name="_*"

| eval Message=if(like(event_message, "%warm_to_cold%"), "Warm to Cold", "Cold to Frozen")

| stats count by idx_name bucket bkt host Message| fields - count</query>

<earliest>$field1.earliest$</earliest>

<latest>$field1.latest$</latest>

</search>

<option name="percentagesRow">false</option>

<option name="rowNumbers">false</option>

<option name="totalsRow">false</option>

</table>

</panel>

</row>

</form>

Continue Reading →

Apps Deployed from Deployment Server

Want to show what apps have been deployed to forwarders from a deployment server (DS)? Try this Splunk Search:

index=_internal sourcetype=splunkd component=DeployedApplication installing 
| stats count latest(_time) AS latest_time by host app 
| convert ctime(latest_time)

index=_internal sourcetype=splunkd component=DeployedApplication installing

| stats count latest(_time) AS latest_time by host app

| convert ctime(latest_time)

Continue Reading →

List of Forwarders that are Deployment Clients

Need a list of Forwarders that are talking to a Deployment Server? Try this:

index=_internal sourcetype=splunkd component=DC* Handshake | stats count by host

1	index=_internal sourcetype=splunkd component=DC* Handshake \| stats count by host

Additional REST query (performed on the DS) will return desired results (Thanks to Lyx for pointing this out!):

| rest /services/deployment/server/clients splunk_server=local | table hostname applications.*.serverclasses | untable hostname, applications, serverclass | rex field=applications "applications\.(?.+)\.serverclasses" | stats dc(hostname) as hostname by apps

1	\| rest /services/deployment/server/clients splunk_server=local \| table hostname applications.*.serverclasses \| untable hostname, applications, serverclass \| rex field=applications "applications\.(?.+)\.serverclasses" \| stats dc(hostname) as hostname by apps

Continue Reading →

Host not sending logs for x days

This Splunk Query will show hosts that stopped sending logs for at least 48 hours. You’ll want to change the time range to be relevant to your environment, and you may need to tweak the 48 hour range to something that is more appropriate for your environment.

| tstats count as countAtToday latest(_time) as lastTime where index!="*_" by host sourcetype index 
| eval age=now()-lastTime 
| sort age d 
| fieldformat lastTime=strftime(lastTime,"%Y/%m/%d %H:%M:%S") 
| eval age=round((age/60/60),1) 
| search age>=48 
| eval age=age."hour" 
| dedup host

| tstats count as countAtToday latest(_time) as lastTime where index!="*_" by host sourcetype index

| eval age=now()-lastTime

| sort age d

| fieldformat lastTime=strftime(lastTime,"%Y/%m/%d %H:%M:%S")

| eval age=round((age/60/60),1)

| search age>=48

| eval age=age."hour"

| dedup host

Continue Reading →

List All Splunk Users & Associated Roles

The following Splunk query will show a table of all users and their roles:

| rest /services/authentication/users | stats values(roles) as Roles by user

1	\| rest /services/authentication/users \| stats values(roles) as Roles by user

*Admin Notes* I’ve found the following query to work better in my environment:

| rest /services/authentication/users | stats values(roles) as Roles by title

1	\| rest /services/authentication/users \| stats values(roles) as Roles by title

Continue Reading →

Dashboard and App views by user

This Splunk query / search shows historical access to dashboards and apps on a local splunk server.

index=_internal sourcetype=splunk_web_access host=* user=*
| rex field=uri_path ".*/(?<title>[^/]*)$" 
| join title
[| rest /servicesNS/-/-/data/ui/views splunk_server=*
| search isDashboard=1 isVisible=1 
| rename eai:acl.app as app 
| fields title app ]
| rename title as dashboard
| stats count by _time user dashboard app host

index=_internal sourcetype=splunk_web_access host=* user=*

| rex field=uri_path ".*/(?<title>[^/]*)$"

| join title

[| rest /servicesNS/-/-/data/ui/views splunk_server=*

| search isDashboard=1 isVisible=1

| rename eai:acl.app as app

| fields title app ]

| rename title as dashboard

| stats count by _time user dashboard app host

Continue Reading →

Show how much disk space is used by _internal

The following Splunk query will return disk space used by the _internal index.

index=_internal source=*license_usage.log type=Usage | eval gb=b/1024/1024/1024 | timechart span=1d sum(gb) as GB by host useother=false | untable _time host gb | top limit=1 host | join time [search index=_internal source=*license_usage.log type=Usage | eval gb=b/1024/1024/1024 | timechart span=1d sum(gb) as GB by sourcetype useother=false |untable _time sourcteype sum(GB)]

Continue Reading →

Find queues that are nearly full

Easily find Splunk Instances (both forwarders and full installs) that have queues that are over 80% full!

index=_internal group=queue | eval percfull=((current_size_kb/max_size_kb)*100) | search percfull>80 | dedup host, name | table _time host name current_size_kb max_size_kb

1	index=_internal group=queue \| eval percfull=((current_size_kb/max_size_kb)*100) \| search percfull>80 \| dedup host, name \| table _time host name current_size_kb max_size_kb

Continue Reading →

Events Sent to Null Que – Internal Logs

This will show events that have been sent to the null que within the splunk internal logs.

index=_internal component=metrics processor=nullqueue group=pipeline sourcetype=splunkd | table _time log_level name processor cpu_seconds executes, cumulative_hits

1	index=_internal component=metrics processor=nullqueue group=pipeline sourcetype=splunkd \| table _time log_level name processor cpu_seconds executes, cumulative_hits

Continue Reading →

REST Call for Memory & CPU usage on Splunk Servers

This Splunk search will show you use and available CPU and Memory statistics. Depending on your environment you may see multiple Splunk servers:

| rest  /services/server/status/resource-usage/hostwide | eval cpu_count = if(isnull(cpu_count), "N/A", cpu_count) | eval cpu_usage = cpu_system_pct + cpu_user_pct | eval mem_used_pct = round(mem_used / mem * 100 , 2) | eval mem_used = round(mem_used, 0) | eval mem = round(mem, 0) |eval mem=tostring(mem, "commas") | eval mem_used=tostring(mem_used, "commas")| fields splunk_server, cpu_count, cpu_usage, mem, mem_used, mem_used_pct | sort - cpu_usage, -mem_used | rename splunk_server AS Instance, cpu_count AS "CPU Cores", cpu_usage AS "CPU Usage (%)", mem AS "Physical Memory Capacity (MB)", mem_used AS "Physical Memory Usage (MB)", mem_used_pct AS "Physical Memory Usage (%)"

| rest /services/server/status/resource-usage/hostwide | eval cpu_count = if(isnull(cpu_count), "N/A", cpu_count) | eval cpu_usage = cpu_system_pct + cpu_user_pct | eval mem_used_pct = round(mem_used / mem * 100 , 2) | eval mem_used = round(mem_used, 0) | eval mem = round(mem, 0) |eval mem=tostring(mem, "commas") | eval mem_used=tostring(mem_used, "commas")| fields splunk_server, cpu_count, cpu_usage, mem, mem_used, mem_used_pct | sort - cpu_usage, -mem_used | rename splunk_server AS Instance, cpu_count AS "CPU Cores", cpu_usage AS "CPU Usage (%)", mem AS "Physical Memory Capacity (MB)", mem_used AS "Physical Memory Usage (MB)", mem_used_pct AS "Physical Memory Usage (%)"

Slightly modified from: http://www.brainfold.net/2016/03/frequently-used-rest-api-calls-in-splunk.html

Continue Reading →

REST Call for a list of Lookup Files

Use this splunk search to get a list of all lookup files:

| rest /services/data/transforms/lookups | table eai:acl.app eai:appName filename title fields_list updated id

1	\| rest /services/data/transforms/lookups \| table eai:acl.app eai:appName filename title fields_list updated id

Continue Reading →

REST Call for Splunk Server Role Status

This REST Splunk search returns the status of roles on each Splunk server in your environment.

| rest /services/server/introspection | table title splunk_server status updated

1	\| rest /services/server/introspection \| table title splunk_server status updated

Continue Reading →

Splunk Objects With Permissions Granted to Non-existent Roles

Useful search to show a bit of detail on roles and user permissions.

| rest /servicesNS/-/-/admin/directory count=0 splunk_server=local
 | fields eai:acl.app, eai:acl.owner, eai:acl.perms.*, eai:acl.sharing, eai:location, title
 | eval perms=mvappend('eai:acl.perms.read','eai:acl.perms.write')
 | fields - eai:acl.perms.*
 | mvexpand perms
 | where perms!="*" AND NOT
 [
 | rest /servicesNS/-/-/authorization/roles count=0 splunk_server=local
 | fields title
 | rename title as perms
 ]

| rest /servicesNS/-/-/admin/directory count=0 splunk_server=local

| fields eai:acl.app, eai:acl.owner, eai:acl.perms.*, eai:acl.sharing, eai:location, title

| eval perms=mvappend('eai:acl.perms.read','eai:acl.perms.write')

| fields - eai:acl.perms.*

| mvexpand perms

| where perms!="*" AND NOT

[

| rest /servicesNS/-/-/authorization/roles count=0 splunk_server=local

| fields title

| rename title as perms

]

I found this at: https://gist.github.com/acharlieh/3254a7ab13297c760376 Credit goes to acharlieh!

Continue Reading →

Every index explicitly granted to a role

Self explanatory, maps roles to indexes. Useful if you have a lot of indexes!

| rest /servicesNS/-/-/authorization/roles count=0 splunk_server=local
 | fields title,srchIndexesAllowed
 | rename srchIndexesAllowed as index title as role
 | mvexpand index
 | where NOT match(index,".*\*.*")

| rest /servicesNS/-/-/authorization/roles count=0 splunk_server=local

| fields title,srchIndexesAllowed

| rename srchIndexesAllowed as index title as role

| mvexpand index

| where NOT match(index,".*\*.*")

I found this at: https://gist.github.com/acharlieh/3254a7ab13297c760376 Credit goes to acharlieh!

Continue Reading →

Average Splunk Web requests by hour

This query is pretty awesome! It helped enlighten us to exactly when our splunk infrastructure is being hit with users

index=_internal sourcetype=splunk_web_access
 [ rest / splunk_server=local
 | fields splunk_server
 | rename splunk_server as host ]
 | bin _time span=1d
 | stats count by date_hour _time
 | appendpipe [ fields _time
 | dedup _time
 | eval date_hour=mvrange(0,24,1)
 | eval count=0
 | mvexpand date_hour ]
 | stats sum(count) as count by date_hour _time
 | stats avg(count) as avg by date_hour
 | sort date_hour

index=_internal sourcetype=splunk_web_access

[ rest / splunk_server=local

| fields splunk_server

| rename splunk_server as host ]

| bin _time span=1d

| stats count by date_hour _time

| appendpipe [ fields _time

| dedup _time

| eval date_hour=mvrange(0,24,1)

| eval count=0

| mvexpand date_hour ]

| stats sum(count) as count by date_hour _time

| stats avg(count) as avg by date_hour

| sort date_hour

I found this at: https://gist.github.com/acharlieh/3254a7ab13297c760376 Credit goes to acharlieh!

Continue Reading →

All indexes not explicitly granted to a role

| rest /servicesNS/-/-/data/indexes count=0
 | stats max(isInternal) as internal, max(disabled) as disabled, max(isReadOnly) as readonly by title
 | fillnull
 | where internal=0 AND disabled=0 AND readonly=0
 | fields title
 | rename title as index
 | join index type=left
 [ rest /servicesNS/-/-/authorization/roles count=0 splunk_server=local
 | fields title,srchIndexesAllowed
 | rename srchIndexesAllowed as index title as role
 | mvexpand index
 | where NOT match(index,".*\*.*") ]
 | search NOT role=*
 | fields index

| rest /servicesNS/-/-/data/indexes count=0

| stats max(isInternal) as internal, max(disabled) as disabled, max(isReadOnly) as readonly by title

| fillnull

| where internal=0 AND disabled=0 AND readonly=0

| fields title

| rename title as index

| join index type=left

[ rest /servicesNS/-/-/authorization/roles count=0 splunk_server=local

| fields title,srchIndexesAllowed

| rename srchIndexesAllowed as index title as role

| mvexpand index

| where NOT match(index,".*\*.*") ]

| search NOT role=*

| fields index

I found this at: https://gist.github.com/acharlieh/3254a7ab13297c760376 Credit goes to acharlieh!

Continue Reading →

List of Sourcetypes Sent by Forwarder

The following Splunk search will show a list of sourcetypes (or perhaps sources I need to confirm) being sent by each Forwarder:

index="_internal" |where host!=splunk_server | stats values(series) as Sourcetypes by host

1	index="_internal" \|where host!=splunk_server \| stats values(series) as Sourcetypes by host

Continue Reading →

Detailed list of Errors Per Host

The following Splunk search will return a detailed list (by message) of errors associated with hosts running a universal forwarder:

index=_internal sourcetype="splunkd" log_level="ERROR" | stats sparkline count dc(host) as uniqhosts last(message) as message last(_time) as last first(_time) as first by punct  | convert ctime(last) ctime(first) | table message count uniqhosts sparkline first last | sort -count | rename message as "Error Output" count as Count uniqhosts as "Number of Hosts" first as "First Occurance" last as "Most Recent Occurance"

index=_internal sourcetype="splunkd" log_level="ERROR" | stats sparkline count dc(host) as uniqhosts last(message) as message last(_time) as last first(_time) as first by punct | convert ctime(last) ctime(first) | table message count uniqhosts sparkline first last | sort -count | rename message as "Error Output" count as Count uniqhosts as "Number of Hosts" first as "First Occurance" last as "Most Recent Occurance"

Continue Reading →

Tag: _internal