topic Re: How do I find all unique IP addresses in a file? in Splunk Search

How do I find all unique IP addresses in a file?

bfaber — Tue, 25 May 2010 22:27:42 GMT

If I have data that looks like

(date) srcip=x.x.x.x dstip=y.y.y.y

How can I create a single list of all unique IPs regardless of src/dst?

I imagine this is some sort of funky stats option...

Re: How do I find all unique IP addresses in a file?

Lowell — Tue, 25 May 2010 23:09:00 GMT

You could use a search like this:

source=my_file | rex max_match=100 "\b(?<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b" | stats values(ip) as ip_list

That should make a multivalued field called ip and populates it with any IP-like values found in the event's raw text. Then the stats command will build a single list of unique values of your ip addresses.

Regex hint: Note that the regex "\b" is for boundary matching. It should match an "=" or a space before the IP address, and should also allow for a comma after the IP address; all of which may be common values before/after an ip address. Also, \b also matches the very beginning and very end of an event.

If you already have your ip address fields defined and you have different names for different sourcetype (which tends to happen), you can use the eval command to combine them. (You can also setup a field alias, but sometimes that may not always be preferable.) For example, say you had fields called dst, DST, dest, and dstip , you could pull them into a single field using a command like so:

| eval new_destip=coalesce(dst,DST,dest,dstip)

So if you want to look at both a source ip address and a dest ip address and then combine them, you could use the same approach for both fields, then use some ugly tricks to convert that into a single multi-value field, and then you can use the stats command to get your list of unique IPs....

| eval d=coalesce(dst,DST,dest,dstip,"") | eval s=coalesce(src,SRC,srcip,"") | eval ips=s.";".d | eval ips=split(ips, ";") | stats values(ips) as ip

Note: the eval split() function is new in Splunk 4.1.

Re: How do I find all unique IP addresses in a file?

erydberg — Tue, 25 May 2010 23:12:57 GMT

I think an easy way to do it is to do a field extraction of the ip addresses, and then do a

"... | dedup ip | fields ip | fields - _*"

to remove dupes and get only the ip address field.

Re: How do I find all unique IP addresses in a file?

Lowell — Tue, 25 May 2010 23:22:20 GMT

Keep in mind that using dedup will probably not work as you would expect when dealing with multi-valued fields. The stats command will multi-valued fields properly. So | stats values(ip) is probably preferable.

Re: How do I find all unique IP addresses in a file?

Simeon — Tue, 25 May 2010 23:22:54 GMT

In theory, Splunk should have automatically extracted the srcip and dstip as fields. The basic commands to get a list of unique values is to use the chart and dedup command. However, you want to list those individual fields as the same field which could require some eval and case statements. For just a single field, you could probably do this:

source=/your/log/file.txt | dedup srcip | chart count by srcip

Re: How do I find all unique IP addresses in a file?

Lowell — Tue, 25 May 2010 23:41:12 GMT

Hmm. I think you either want just dedup to get a single list of values. Or use the chart command. Using both, as shown, the count will always be 1 for each value of scrip

Re: How do I find all unique IP addresses in a file?

gkanapathy — Wed, 26 May 2010 20:35:55 GMT

eval split is new in 4.1, but older versions can use makemv to do the same thing.