List forwarders generating socket errors due to unkown SSL protocol

If you are using SSL on port 9997 or 9998 (or other port) to send logs from your forwarders to your indexers, you know that a misconfigured forwarder or one with incorrect certs can generate these errors. This search gives you a succinct summary of those hosts. Use Today in the timepicker.

Note: Add an IP address and a space before the closing dquote on line one to check one host, then create a dashboard with this search and an input field of IP so that installers can check for these logs, or you can if you have your indexer in the cloud (so you have no shell access. Remove the last line (sort) for the dashboard. In Dashboard, < and > need to be &lt; and &gt;

index=_internal source="/opt/splunk/var/log/splunk/splunkd.log" HttpListener "Socket error from "
| rex "(?<errorLog>WARN\s+HttpListener\s.*?Socket error.*)"
| rex field=errorLog "WARN\s+HttpListener\s.*?Socket error from\s+(?<ip>[^ ]+)"
| lookup dnsLookup ip OUTPUT host as ForwarderInDNS
| rename ip as ForwarderIP
| stats earliest(_time) AS EarliestSeen latest(_time) as LatestSeen count by ForwarderIP ForwarderInDNS errorLog
| convert timeformat="%Y/%m/%d - %H:%M:%S" ctime(EarliestSeen)
| convert timeformat="%Y/%m/%d - %H:%M:%S" ctime(LatestSeen)
| Eval Comment="This error indicates a problem with certificate installation or path to certs configuration"
| fields EarliestSeen LatestSeen ForwarderIP ForwarderInDNS errorLog Comment count
| sort ForwarderInDNS
Share This:
Tagged:

Comments

  1. john117

    Thanks for sharing! I do not have a default “dnsLookup” in my environment. Did you build this with an app or acquire it from your networking team and upload it as a lookup?

    If the query works the way I think it works, this is pretty neat!

    Reply
  2. wrangler2x

    Take a look at https://gosplunk.com/list-forwarders-generating-socket-errors-due-to-unkown-ssl-protocol/#comments

    If you have this stanza, you should be able to use the dnslookup (and if not, you can just remove that line and references to it from the search):

    [dnslookup]
    external_cmd = external_lookup.py clienthost clientip
    fields_list = clienthost,clientip

    However, I’ve added a local version to my account and it looks like this:

    [dnslookup]
    disabled = 0
    external_cmd = /opt/splunk/etc/apps/search/lookups/external_lookup.py host ip
    fields_list = host, ip

    So in my search I’m using ‘host’ and ‘ip’ instead of ‘clienthost’ and ‘clientip’

    You can do that or modify my search to use ‘clienthost’ and ‘clientip’ instead.

    When I do this in the search:

    | lookup dnsLookup ip OUTPUT host as ForwarderInDNS

    I’m passing ip in place of clientip because of my transform, and when it returns host I’m using ‘as’ to rename it.

    Reply
  6. JMS

    index=_internal source=”/opt/splunk/var/log/splunk/splunkd.log” HttpListener “Socket error from ”
    | rex “(?WARN\s+HttpListener\s.*?Socket error.*)”
    | rex field=errorLog “WARN\s+HttpListener\s.*?Socket error from\s+(?[^ ]+):”
    | lookup dnslookup clientip AS ip OUTPUT clienthost as ForwarderInDNS
    | rename ip as ForwarderIP
    | stats earliest(_time) AS EarliestSeen latest(_time) as LatestSeen count by ForwarderIP ForwarderInDNS errorLog
    | convert timeformat=”%Y/%m/%d – %H:%M:%S” ctime(EarliestSeen)
    | convert timeformat=”%Y/%m/%d – %H:%M:%S” ctime(LatestSeen)
    | Eval Comment=”This error indicates a problem with certificate installation or path to certs configuration”
    | fields EarliestSeen LatestSeen ForwarderIP ForwarderInDNS errorLog Comment count
    | sort ForwarderInDNS

    Reply

Leave A Comment?