SPL
List forwarders generating socket errors due to unkown SSL protocol
Description
If you are using SSL on port 9997 or 9998 (or other port) to send logs from your forwarders to your indexers, you know that a misconfigured forwarder or one with incorrect certs can generate these errors. This search gives you a succinct summary of those hosts. Use Today in the timepicker.
Note: Add an IP address and a space before the closing dquote on line one to check one host, then create a dashboard with this search and an input field of IP so that installers can check for these logs, or you can if you have your indexer in the cloud (so you have no shell access. Remove the last line (sort) for the dashboard. In Dashboard, < and > need to be < and >
0 0
index=_internal source="/opt/splunk/var/log/splunk/splunkd.log" HttpListener "Socket error from " | rex "(?<errorLog>WARN\s+HttpListener\s.*?Socket error.*)" | rex field=errorLog "WARN\s+HttpListener\s.*?Socket error from\s+(?<ip>[^ ]+)" | lookup dnsLookup ip OUTPUT host as ForwarderInDNS | rename ip as ForwarderIP | stats earliest(_time) AS EarliestSeen latest(_time) as LatestSeen count by ForwarderIP ForwarderInDNS errorLog | convert timeformat="%Y/%m/%d - %H:%M:%S" ctime(EarliestSeen) | convert timeformat="%Y/%m/%d - %H:%M:%S" ctime(LatestSeen) | Eval Comment="This error indicates a problem with certificate installation or path to certs configuration" | fields EarliestSeen LatestSeen ForwarderIP ForwarderInDNS errorLog Comment count | sort ForwarderInDNS
Comments
6 total
Thanks for sharing! I do not have a default \"dnsLookup\" in my environment. Did you build this with an app or acquire it from your networking team and upload it as a lookup?\r\n\r\nIf the query works the way I think it works, this is pretty neat!
Take a look at https://gosplunk.com/list-forwarders-generating-socket-errors-due-to-unkown-ssl-protocol/#comments\r\n\r\nIf you have this stanza, you should be able to use the dnslookup (and if not, you can just remove that line and references to it from the search):\r\n\r\n[dnslookup]\r\nexternal_cmd = external_lookup.py clienthost clientip\r\nfields_list = clienthost,clientip\r\n\r\nHowever, I've added a local version to my account and it looks like this:\r\n\r\n[dnslookup]\r\ndisabled = 0\r\nexternal_cmd = /opt/splunk/etc/apps/search/lookups/external_lookup.py host ip\r\nfields_list = host, ip\r\n\r\nSo in my search I'm using 'host' and 'ip' instead of 'clienthost' and 'clientip'\r\n\r\nYou can do that or modify my search to use 'clienthost' and 'clientip' instead.\r\n\r\nWhen I do this in the search:\r\n\r\n| lookup dnsLookup ip OUTPUT host as ForwarderInDNS\r\n\r\nI'm passing ip in place of clientip because of my transform, and when it returns host I'm using 'as' to rename it.
Also, it is mentioned here in Splunk Answers:\r\nhttps://answers.splunk.com/answers/105246/dns-resolution-in-a-search.html
Thanks wrangler2x!
I have a bad paste in my first comment back to <a class='bp-suggestions-mention' href='https://gosplunk.com/members/john117/' rel='nofollow'>@john117</a>, where I said \"Take a look at...\" The paste should have been $SPLUNK_HOME/etc/system/default/transforms.conf
index=_internal source=\"/opt/splunk/var/log/splunk/splunkd.log\" HttpListener \"Socket error from \"\r\n| rex \"(?WARN\s+HttpListener\s.*?Socket error.*)\"\r\n| rex field=errorLog \"WARN\s+HttpListener\s.*?Socket error from\s+(?[^ ]+):\"\r\n| lookup dnslookup clientip AS ip OUTPUT clienthost as ForwarderInDNS\r\n| rename ip as ForwarderIP\r\n| stats earliest(_time) AS EarliestSeen latest(_time) as LatestSeen count by ForwarderIP ForwarderInDNS errorLog\r\n| convert timeformat=\"%Y/%m/%d - %H:%M:%S\" ctime(EarliestSeen)\r\n| convert timeformat=\"%Y/%m/%d - %H:%M:%S\" ctime(LatestSeen)\r\n| Eval Comment=\"This error indicates a problem with certificate installation or path to certs configuration\"\r\n| fields EarliestSeen LatestSeen ForwarderIP ForwarderInDNS errorLog Comment count\r\n| sort ForwarderInDNS
Leave a comment
You must log in to post a comment.