Looking for network traffic from a list of domain names in firewall traffic that only has IP addresses.

[jon_d_irish_ctr]

I have a csv file with domain names. I need to search my firewall traffic to see if any traffic matches to/from any of these domains. The firewall logs only contain IP addresses, so I have to convert the domain names to IP addresses before searching for matches against the logs. I was hoping I could do this with the built-in "dnslookup" function. However, I am not having any luck getting this to work.

I would appreciate any help!

Thanks,
Jon

[woodcock]

Just make sure that your Indexers and Search Head can talk to a DNS server and use the built-in capability like this:

index=YouShouldAlwaysSpecifyAnIndex sourcetype=YourFirewallSourcetypeHere [
| inputlookup flash_mi-000085-mw
| lookup dnsLookup clienthost AS host OUTPOUT clientip AS host | table host ]

[jon_d_irish_ctr]

I am getting the inputlookup and the dnsLookup to work properly now. The problem is trying to get my output to display only matches between the IP addresses resolved from the dnsLookup function and the IP addresses in my firewall logs. I made a couple of changes:

I changed the transform and the lookup tables. I changed "host" to "fqdn" as "host" is also defined in the firewall logs so it was making the results rather odd. Anyway, after making that change, I tried this search:

| inputlookup flash_mi-000085-mw | lookup dnsLookup host as fqdn | appendcols [search sourcetype=cisco:asa dest_ip="xxx.xxx.0.0/16"] | eval sans=("https://isc.sans.edu/ipinfo.html?ip="+ip) | eval cymon=("https://cymon.io/"+ip) | where fqdn!="" | table fqdn ip dest_ip sans cymon | rename cymon as "cymon(RIPE)"

[somesoni2]

What issue you face with build-in dnslookup function?

[jon_d_irish_ctr]

My problem is that I am not exactly sure how to utilize it. I have read the "Splunk Enterprise Knowledge Manager Manual" and I have tried to match up with the instructions in it. So far, I have tried this:

I have a csv file (/opt/splunk/etc/system/lookups/000085.csv that looks like this:
host, ip
axxxxxxxxx.com
gxxxxxxxxx.com
hxxxxxxxxx.com
kxxxxxxxxx.com
oxxxxxxxxx.com
pxxxxxxxxx.com
pdxxxxxxxx.com
prxxxxxxxx.com
rxxxxxxxxx.com
txxxxxxxxx.com

The transform, located at: /opt/splunk/etc/system/local has this stanza:
[dnsLookup]
external_cmd = external_lookup.py host ip
fields_list = host, ip

My search string looks like:
sourcetype=cisco:asa | lookup 000085.csv as src_host OUTPUT src_host as host | lookup dnsLookup host as host_match | table host_match

I have changed everything so many times that I am sure that I have messed something up.

Thanks,
Jon

[alemarzu]

Try like this.

You should change/rename the DNS column from your CSV, from host to something else to avoid collisions against metadata fields. Lets say you have host_dns and ip fields in your CSV.

sourcetype=cisco:asa | fields ip | lookup 000085.csv ip OUTPUT host_dns | table host_dns ip 

[jon_d_irish_ctr]

That doesn't quite seem to work. I am trying to break this down into two different stages. First, I just want to verify that I am getting the contents of
000085.csv read into splunk. Once I am sure that is working right, I will then try to feed those results into the dnslookup function.

Couldn't I verify that the table has been read in correctly with this string:
| lookup 000085.csv host OUTPUT host | table host

[alemarzu]

Oh I see, try like this.

| inputlookup 000085.csv 

[jon_d_irish_ctr]

I had looked at that earlier, but I was not sure I was implementing it properly. I created a copy of the csv file with only the field host on the first line and then renamed it to 000085.txt:

host
axxxxxxxxx.com
gxxxxxxxxx.com
hxxxxxxxxx.com
kxxxxxxxxx.com
oxxxxxxxxx.com
pxxxxxxxxx.com
pdxxxxxxxx.com
prxxxxxxxx.com
rxxxxxxxxx.com
txxxxxxxxx.com

I then issue this search:
| inputlookup flash_mi-000085-mw-dns.txt | table host

However, I don't get any results. Shouldn't it list the vlaues of 000085.txt?

[alemarzu]

Thats because of the extension, is not CSV. But you can fix that by addint this to your transforms.conf

[flash_mi-000085-mw-dns]
filename = flash_mi-000085-mw-dns.txt

And search by | inputlookup flash_mi-000085-mw-dns

[jon_d_irish_ctr]

so I got the first part of this working by making a couple of changes. First I changed the extension of the file from .txt to .csv. I also changed the header from host to src_fqdn. Here is what the new .csv file looks like:
src_fqdn
axxxxxxxxx.com
gxxxxxxxxx.com
hxxxxxxxxx.com
kxxxxxxxxx.com
oxxxxxxxxx.com
pxxxxxxxxx.com
pdxxxxxxxx.com
prxxxxxxxx.com
rxxxxxxxxx.com
txxxxxxxxx.com

And here is what the transform stanza looks like:
[000085]
filename = 000085.csv
index_fields_list = src_fqdn
outputlookup_check_permission
check_permission

I have verified that the inputlookup is working with this:
| inputlookup flash_mi-000085-mw

which results in:
src_fqdn

axxxxxxxxx.com
gxxxxxxxxx.com
hxxxxxxxxx.com
kxxxxxxxxx.com
oxxxxxxxxx.com
pxxxxxxxxx.com
pdxxxxxxxx.com
prxxxxxxxx.com
rxxxxxxxxx.com
txxxxxxxxx.com

Now, I have all my values "read-into" splunk as src_fqdn. Next, I need to convert them to IP addresses with the built-in "dnslookup" function. However, I cannot figure out how to get this to work.

[jon_d_irish_ctr]

I think my problem with the dnsLookup part is that there is a transform stanza already defined for it:

[dnsLookup]
external_cmd = external_lookup.py host ip
fields_list = host, ip

but the fields do not match the field I have in my lookup transform stanza even though I have fqdn's in my list, should I change the header from src_fqdn to ip, so that the dnsLookup stanza will match?

[alemarzu]

Try using the one thats it is shipped by default in Splunk.

Those fields are clienthost and clientip and lets assume that your IP field on your log its called orig_ip

Query should be this one to reverse dnslookup.

 ... | table orig_ip | lookup dnslookup clientip as orig_ip

[jon_d_irish_ctr]

So I actually got it to work! I changed the inputlookup file headers one more time to:
host, ip (I did this to match the fields defined in the dnsLookup transform stanza. After doing that, and restarting splunk, I then issued this search:

| inputlookup flash_mi-000085-mw | lookup dnsLookup host | table host ip

It actually returns a table with both the fqdn and it's corresponding IP address. It's great that this works, but I am not sure why it works.

[alemarzu]

If you were editing transforms.conf a restart was needed for sure.

[jon_d_irish_ctr]

So this is my final search string for just reviewing the data in the csv file:

| inputlookup flash_mi-000085-mw | lookup dnsLookup host | where ip!= " " | eval sans=("https://isc.sans.edu/ipinfo.html?ip="+ip) | eval cymon=("https://cymon.io/"+ip) | table host ip sans cymon | rename cymon as "cymon(RIPE)"

This provides all the "output" data that I need. Now, I need to add the firewall data and search for matches to the csv list. I know adding soutcetype=cisco:asa will pull all the firewall data in, I am just not sure yet how I will build this to search for matches. I'll have to think about this some more.

[alemarzu]

Assuming that your IP field in your firewall is called "orig_ip", try this.

| inputlookup flash_mi-000085-mw | lookup dnsLookup host | where ip!= " " | eval sans=("https://isc.sans.edu/ipinfo.html?ip="+ip) | eval cymon=("https://cymon.io/"+ip) | table host ip sans cymon | rename cymon as "cymon(RIPE)"  | search [search sourcetype=firewall orig_ip AS ip | fields ip]

[jon_d_irish_ctr]

So I am REALLY close now! I did have to make one change in the transform and the lookup table. I changed "host" to "fqdn" as "host" is also defined in the firewall logs so it was making the results rather odd. Anyway, after making that change, I tried this search:

| inputlookup flash_mi-000085-mw | lookup dnsLookup host as fqdn | appendcols [search sourcetype=cisco:asa dest_ip="xxx.xxx.0.0/16"] | eval sans=("https://isc.sans.edu/ipinfo.html?ip="+ip) | eval cymon=("https://cymon.io/"+ip) | where fqdn!="" | table fqdn ip dest_ip sans cymon | rename cymon as "cymon(RIPE)"

I just don't think it is matching correctly though. My intent is to display only the firewall traffic that matches an IP address resolved from the dnsLookup function.

[jon_d_irish_ctr]

After a little more research, I have determined that I don't want to be using the "appendcols" command as I really don't want to be adding the information to the results. I simply want to compare all inbound/outbound traffic from our LAN to the results of "fqdn" and then display any matches. Thus, I came up with this search:

| inputlookup flash_mi-000085-mw | lookup dnsLookup host as fqdn | search sourcetype=cisco:asa dest_ip="X.X.0.0/16" OR src_ip = "X.X.0.0/16" AND fqdn | eval sans=("https://isc.sans.edu/ipinfo.html?ip="+ip) | eval cymon=("https://cymon.io/"+ip) | where fqdn!="" | table fqdn ip dest_ip sans cymon | rename cymon as "cymon(RIPE)"

However, this is still not working.

[jon_d_irish_ctr]

I think what is really confusing me is how I can read in the csv file full of domain names BEFORE I then run them through the dnslookup function.

[alemarzu]

Hi @jon.d.irish.ctr

Have you read this ?

https://docs.splunk.com/Documentation/Splunk/6.6.3/Knowledge/Configureexternallookups#External_looku...