datamodels Archives -

Data model Acceleration Details

This Splunk Search shows you a lot of good information about your data model acceleration and performance.

| rest /services/admin/summarization by_tstats=t splunk_server=local count=0 <br />| eval key=replace(title,(("tstats:DM_" . 'eai:acl.app') . "_"),""), datamodel=replace('summary.id',(("DM_" . 'eai:acl.app') . "_"),"") <br />| join type=left key <br />    [| rest /services/data/models splunk_server=local count=0 <br />    | table title, "acceleration.cron_schedule", "eai:digest" <br />    | rename title as key <br />    | rename "acceleration.cron_schedule" as cron] <br />| table datamodel, "eai:acl.app", "summary.access_time", "summary.is_inprogress", "summary.size", "summary.latest_time", "summary.complete", "summary.buckets_size", "summary.buckets", cron, "summary.last_error", "summary.time_range", "summary.id", "summary.mod_time", "eai:digest", "summary.earliest_time", "summary.last_sid", "summary.access_count" <br />| rename "summary.id" as summary_id, "summary.time_range" as retention, "summary.earliest_time" as earliest, "summary.latest_time" as latest, "eai:digest" as digest <br />| rename "summary.*" as "*", "eai:acl.*" as "*" <br />| sort datamodel <br />| rename access_count as "Datamodel_Acceleration.access_count", access_time as "Datamodel_Acceleration.access_time", app as "Datamodel_Acceleration.app", buckets as "Datamodel_Acceleration.buckets", buckets_size as "Datamodel_Acceleration.buckets_size", cron as "Datamodel_Acceleration.cron", complete as "Datamodel_Acceleration.complete", datamodel as "Datamodel_Acceleration.datamodel", digest as "Datamodel_Acceleration.digest", earliest as "Datamodel_Acceleration.earliest", is_inprogress as "Datamodel_Acceleration.is_inprogress", last_error as "Datamodel_Acceleration.last_error", last_sid as "Datamodel_Acceleration.last_sid", latest as "Datamodel_Acceleration.latest", mod_time as "Datamodel_Acceleration.mod_time", retention as "Datamodel_Acceleration.retention", size as "Datamodel_Acceleration.size", summary_id as "Datamodel_Acceleration.summary_id" <br />| fields + "Datamodel_Acceleration.access_count", "Datamodel_Acceleration.access_time", "Datamodel_Acceleration.app", "Datamodel_Acceleration.buckets", "Datamodel_Acceleration.buckets_size", "Datamodel_Acceleration.cron", "Datamodel_Acceleration.complete", "Datamodel_Acceleration.datamodel", "Datamodel_Acceleration.digest", "Datamodel_Acceleration.earliest", "Datamodel_Acceleration.is_inprogress", "Datamodel_Acceleration.last_error", "Datamodel_Acceleration.last_sid", "Datamodel_Acceleration.latest", "Datamodel_Acceleration.mod_time", "Datamodel_Acceleration.retention", "Datamodel_Acceleration.size", "Datamodel_Acceleration.summary_id" <br />| rename "Datamodel_Acceleration.*" as "*" <br />| join type=outer last_sid <br />    [| rest splunk_server=local count=0 /services/search/jobs reportSearch=summarize* <br />    | rename sid as last_sid <br />    | fields + last_sid, runDuration] <br />| eval "size(MB)"=round((size / 1048576),1), "retention(days)"=if((retention == 0),"unlimited",round((retention / 86400),1)), "complete(%)"=round((complete * 100),1), "runDuration(s)"=round(runDuration,1) <br />| sort 100 + datamodel <br />| table datamodel, app, cron, "retention(days)", earliest, latest, is_inprogress, "complete(%)", "size(MB)", "runDuration(s)", last_error

| rest /services/admin/summarization by_tstats=t splunk_server=local count=0 | eval key=replace(title,(("tstats:DM_" . 'eai:acl.app') . "_"),""), datamodel=replace('summary.id',(("DM_" . 'eai:acl.app') . "_"),"") | join type=left key [| rest /services/data/models splunk_server=local count=0 | table title, "acceleration.cron_schedule", "eai:digest" | rename title as key | rename "acceleration.cron_schedule" as cron] | table datamodel, "eai:acl.app", "summary.access_time", "summary.is_inprogress", "summary.size", "summary.latest_time", "summary.complete", "summary.buckets_size", "summary.buckets", cron, "summary.last_error", "summary.time_range", "summary.id", "summary.mod_time", "eai:digest", "summary.earliest_time", "summary.last_sid", "summary.access_count" | rename "summary.id" as summary_id, "summary.time_range" as retention, "summary.earliest_time" as earliest, "summary.latest_time" as latest, "eai:digest" as digest | rename "summary.*" as "*", "eai:acl.*" as "*" | sort datamodel | rename access_count as "Datamodel_Acceleration.access_count", access_time as "Datamodel_Acceleration.access_time", app as "Datamodel_Acceleration.app", buckets as "Datamodel_Acceleration.buckets", buckets_size as "Datamodel_Acceleration.buckets_size", cron as "Datamodel_Acceleration.cron", complete as "Datamodel_Acceleration.complete", datamodel as "Datamodel_Acceleration.datamodel", digest as "Datamodel_Acceleration.digest", earliest as "Datamodel_Acceleration.earliest", is_inprogress as "Datamodel_Acceleration.is_inprogress", last_error as "Datamodel_Acceleration.last_error", last_sid as "Datamodel_Acceleration.last_sid", latest as "Datamodel_Acceleration.latest", mod_time as "Datamodel_Acceleration.mod_time", retention as "Datamodel_Acceleration.retention", size as "Datamodel_Acceleration.size", summary_id as "Datamodel_Acceleration.summary_id" | fields + "Datamodel_Acceleration.access_count", "Datamodel_Acceleration.access_time", "Datamodel_Acceleration.app", "Datamodel_Acceleration.buckets", "Datamodel_Acceleration.buckets_size", "Datamodel_Acceleration.cron", "Datamodel_Acceleration.complete", "Datamodel_Acceleration.datamodel", "Datamodel_Acceleration.digest", "Datamodel_Acceleration.earliest", "Datamodel_Acceleration.is_inprogress", "Datamodel_Acceleration.last_error", "Datamodel_Acceleration.last_sid", "Datamodel_Acceleration.latest", "Datamodel_Acceleration.mod_time", "Datamodel_Acceleration.retention", "Datamodel_Acceleration.size", "Datamodel_Acceleration.summary_id" | rename "Datamodel_Acceleration.*" as "*" | join type=outer last_sid [| rest splunk_server=local count=0 /services/search/jobs reportSearch=summarize* | rename sid as last_sid | fields + last_sid, runDuration] | eval "size(MB)"=round((size / 1048576),1), "retention(days)"=if((retention == 0),"unlimited",round((retention / 86400),1)), "complete(%)"=round((complete * 100),1), "runDuration(s)"=round(runDuration,1) | sort 100 + datamodel | table datamodel, app, cron, "retention(days)", earliest, latest, is_inprogress, "complete(%)", "size(MB)", "runDuration(s)", last_error

Continue Reading →

Splunk CIM Assist

Got tired of having to go through each data source to determine what indexes should go into the Splunk_SA_CIM search macros, this does the leg work.

index=*
| fields index, tag, user, action, object_category
| eval datamodel = if(tag="alert", index."."."alert", datamodel)
| eval datamodel = if(tag="listening" AND tag="port", index."."."application_state_deprecated"."."."endpoint", datamodel)
| eval datamodel = if(tag="process" AND tag="report", index."."."application_state_deprecated"."."."endpoint", datamodel)
| eval datamodel = if(tag="service" AND tag="report", index."."."application_state_deprecated"."."."endpoint", datamodel)
| eval datamodel = if(tag="authentication" AND action!="success" AND user!="*$", index."."."authentication", datamodel)
| eval datamodel = if(tag="certificate", index."."."certificates", datamodel)
| eval datamodel = if(tag="change" AND NOT (object_category=file OR object_category=directory OR object_category=registry), index."."."change"."."."change_analysis_deprecated", datamodel)
| eval datamodel = if(tag="dlp" AND tag="incident", index."."."data_loss_prevention", datamodel)
| eval datamodel = if(tag="database", index."."."database", datamodel)
| eval datamodel = if(tag="email", index."."."email", datamodel)
| eval datamodel = if(tag="endpoint" AND tag="filesystem", index."."."endpoint", datamodel)
| eval datamodel = if(tag="endpoint" AND tag="registry", index."."."endpoint", datamodel)
| eval datamodel = if(tag="track_event_signatures" AND (signature="*" OR signature_id="*"), index."."."event_signatures", datamodel)
| eval datamodel = if(tag="messaging", index."."."interprocess_messaging", datamodel)
| eval datamodel = if(tag="ids" AND tag="attack", index."."."intrusion_detection", datamodel)
| eval datamodel = if(tag="inventory" AND (tag="cpu" OR tag="memory" OR tag="network" OR tag="storage" OR (tag="system" AND tag="version") OR tag="user" OR tag="virtual"), index."."."inventory", datamodel)
| eval datamodel = if(tag="jvm", index."."."jvm", datamodel)
| eval datamodel = if(tag="malware" AND tag="attack", index."."."malware", datamodel)
| eval datamodel = if(tag="network" AND tag="resolution" AND tag="dns", index."."."network_resolution_dns", datamodel)
| eval datamodel = if(tag="network" AND tag="session", index."."."network_sessions", datamodel)
| eval datamodel = if(tag="network" AND tag="communicate", index."."."network_traffic", datamodel)
| eval datamodel = if(tag="performance" AND (tag="cpu" OR tag="facilities" OR tag="memory" OR tag="storage" OR tag="network" OR (tag="os" AND ((tag="time" AND tag="synchronize") OR tag="uptime"))), index."."."performance", datamodel)
| eval datamodel = if(tag="ticketing", index."."."ticket_managment", datamodel)
| eval datamodel = if(tag="update" AND tag="status", index."."."updates", datamodel)
| eval datamodel = if(tag="vulnerability" AND tag="report", index."."."vulnerabilities", datamodel)
| eval datamodel = if(tag="web", index."."."web", datamodel)
| rex field=datamodel "(?<index>[^\\.]+)\.(?<datamodel>.*)"
| makemv delim="." datamodel
| stats values(index) as index by datamodel

index=*

| fields index, tag, user, action, object_category

| eval datamodel = if(tag="alert", index."."."alert", datamodel)

| eval datamodel = if(tag="listening" AND tag="port", index."."."application_state_deprecated"."."."endpoint", datamodel)

| eval datamodel = if(tag="process" AND tag="report", index."."."application_state_deprecated"."."."endpoint", datamodel)

| eval datamodel = if(tag="service" AND tag="report", index."."."application_state_deprecated"."."."endpoint", datamodel)

| eval datamodel = if(tag="authentication" AND action!="success" AND user!="*$", index."."."authentication", datamodel)

| eval datamodel = if(tag="certificate", index."."."certificates", datamodel)

| eval datamodel = if(tag="change" AND NOT (object_category=file OR object_category=directory OR object_category=registry), index."."."change"."."."change_analysis_deprecated", datamodel)

| eval datamodel = if(tag="dlp" AND tag="incident", index."."."data_loss_prevention", datamodel)

| eval datamodel = if(tag="database", index."."."database", datamodel)

| eval datamodel = if(tag="email", index."."."email", datamodel)

| eval datamodel = if(tag="endpoint" AND tag="filesystem", index."."."endpoint", datamodel)

| eval datamodel = if(tag="endpoint" AND tag="registry", index."."."endpoint", datamodel)

| eval datamodel = if(tag="track_event_signatures" AND (signature="*" OR signature_id="*"), index."."."event_signatures", datamodel)

| eval datamodel = if(tag="messaging", index."."."interprocess_messaging", datamodel)

| eval datamodel = if(tag="ids" AND tag="attack", index."."."intrusion_detection", datamodel)

| eval datamodel = if(tag="inventory" AND (tag="cpu" OR tag="memory" OR tag="network" OR tag="storage" OR (tag="system" AND tag="version") OR tag="user" OR tag="virtual"), index."."."inventory", datamodel)

| eval datamodel = if(tag="jvm", index."."."jvm", datamodel)

| eval datamodel = if(tag="malware" AND tag="attack", index."."."malware", datamodel)

| eval datamodel = if(tag="network" AND tag="resolution" AND tag="dns", index."."."network_resolution_dns", datamodel)

| eval datamodel = if(tag="network" AND tag="session", index."."."network_sessions", datamodel)

| eval datamodel = if(tag="network" AND tag="communicate", index."."."network_traffic", datamodel)

| eval datamodel = if(tag="performance" AND (tag="cpu" OR tag="facilities" OR tag="memory" OR tag="storage" OR tag="network" OR (tag="os" AND ((tag="time" AND tag="synchronize") OR tag="uptime"))), index."."."performance", datamodel)

| eval datamodel = if(tag="ticketing", index."."."ticket_managment", datamodel)

| eval datamodel = if(tag="update" AND tag="status", index."."."updates", datamodel)

| eval datamodel = if(tag="vulnerability" AND tag="report", index."."."vulnerabilities", datamodel)

| eval datamodel = if(tag="web", index."."."web", datamodel)

| rex field=datamodel "(?<index>[^\\.]+)\.(?<datamodel>.*)"

| makemv delim="." datamodel

| stats values(index) as index by datamodel

Continue Reading →

Listing Data models

|datamodel

|rex field=_raw "\"description\":\"(?<Description>\w+|\w+\s+\w+|\w+\s+\w+\s+\w+|\w+\s+\w+\s+\w+\s+\w+\s+\w+|\w+\s+\w+\s+\w+\s+\w+\s+\w+\s+\w+|\w+\s+\w+\s+\w+\s+\w+\s+\w+\s+\w+\s+\w+)\"\,"

|rex field=_raw "\"modelName\":\"(?<DataSetName>\w+|\w+\s+\w+|\w+\s+\w+\s+\w+)\"\,"

|rex field=_raw "\"parentName\":\"(?<ParentName>\w+|\w+\s+\w+|\w+\s+\w+\s+\w+)\"\,"

|rex field=_raw "\"autoextractSearch\":(?<SearchDetails>.*\")\,\"previewSearch.*"

|table Description DataSetName SearchDetails

|eval SearchDetails=replace(SearchDetails,",\"previewSearch.*","")

|fillnull Description value="Description not available"

|datamodel

|rex field=_raw "\"modelName\":\"(?<DataSetName>\w+|\w+\s+\w+|\w+\s+\w+\s+\w+)\"\,"

|rex field=_raw "\"parentName\":\"(?<ParentName>\w+|\w+\s+\w+|\w+\s+\w+\s+\w+)\"\,"

|rex field=_raw "\"autoextractSearch\":(?<SearchDetails>.*\")\,\"previewSearch.*"

|table Description DataSetName SearchDetails

|eval SearchDetails=replace(SearchDetails,",\"previewSearch.*","")

|fillnull Description value="Description not available"

Continue Reading →

Find success login after 10 failures with streamstats

If you have the Authentication data model configured you can use the following search to quickly find successful logins after 10 failed attempts!

| from datamodel:"Authentication"."Authentication"
| search action=failure or action=success
| reverse
| streamstats window=0 current=true reset_after="(action=\"success\")" count as failure_count by src
| where action="success" and failure_count > 10

| from datamodel:"Authentication"."Authentication"

| search action=failure or action=success

| reverse

| streamstats window=0 current=true reset_after="(action=\"success\")" count as failure_count by src

| where action="success" and failure_count > 10

Continue Reading →

Search Traffic by Source IP

GoSplunk Admin Notes: If you have a data model enabled that matches the search below, this might work for you!

| datamodel Network_Traffic All_Traffic search | search All_Traffic.src_ip=10.x.x.x | stats count by All_Traffic.src_ip, All_Traffic.dest,All_Traffic.action, dstcountry | dedup All_Traffic.dest

1	\| datamodel Network_Traffic All_Traffic search \| search All_Traffic.src_ip=10.x.x.x \| stats count by All_Traffic.src_ip, All_Traffic.dest,All_Traffic.action, dstcountry \| dedup All_Traffic.dest

Continue Reading →

Sourcetype: datamodels

Data model Acceleration Details

Splunk CIM Assist

Listing Data models

Find success login after 10 failures with streamstats

Search Traffic by Source IP