topic Re: Regex, and extracting the IP + hostname from _internal in Splunk Search

Regex, and extracting the IP + hostname from _internal

oliverj — Tue, 12 Mar 2019 20:43:11 GMT

One of my ongoing gripes with splunk is that there is no way to see the IP and the hostname -- either my forwarder sends a hostname, or an IP. Not both. I know the information is there, as I can see it in the _internal splunkd log.
So, I was trying to make a list of all IP+hostnames out of this list:

index=_internal uri=* component=HttpPubSubConnection

Result:

HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_ipaddress_8089_instancename_hostname_FAB4D96E-5A4A-4593-8914-635506217E40

In the URI, it has the ip and a hostname. So, I want to extract this!
In the splunk field extractor, I type my own regex,

 ^(?:[^_]+_){4}(?P<internal_ip>[^_ ]+)

This works great! It shows all my matches, I save it, run my search, all is well.

Now if I do the same without the props.conf:

index=_internal uri=* component=HttpPubSubConnection | regex  "^(?:[^_]+_){1}(?P<internal_ip>[^_ ]+)" | table internal_ip

Nothing.
I could do it all via props.conf, but there are so many situations where that regex matches outside the specific search I want to run that I figured it was much more efficient to just do the regex in the search string. But, no extraction seems to be happening.
Any suggestions?

Re: Regex, and extracting the IP + hostname from _internal

richgalloway — Tue, 12 Mar 2019 21:02:25 GMT

The regex command does not do field extractions. Use rex, instead.

... | rex field=uri "[^_]+_(?P<internal_ip>[^_ ]+)_\d+_[^_]+_(?<hostname>[^_]+)" | ...

Re: Regex, and extracting the IP + hostname from _internal

oliverj — Wed, 13 Mar 2019 13:26:55 GMT

That was exactly what I needed. Thank you.

Re: Regex, and extracting the IP + hostname from _internal

oliverj — Wed, 13 Mar 2019 13:56:25 GMT

To anyone who finds this post later -- this is a terrible search. I just realized it only pulls data from hosts that use deployment servers, not all UniversalForwarders. So anything that sends without being a part of the deployment server (for us, hundreds of devices we don't own) will not show up. Sigh.

But the search/extraction still works great! Just...back to the drawing board.

Re: Regex, and extracting the IP + hostname from _internal

richgalloway — Thu, 14 Mar 2019 12:23:40 GMT

Try component=tcpinconnection.