bro_dns shows the results of a dns query as what I presume is an array, for example:
answers: [ [-]
mt-ingestion-service-mr22.itunes.apple.com
mt-ingestion-service-mr22.itunes-apple.com.akadns.net
17.110.234.28
17.110.232.46
17.110.232.45
17.110.234.27
]
query: xp.itunes-apple.com.akadns.net
I have IP addresses from other data sources that I want to search against "answers" so that I can get "query" from the above, but I cannot figure out how to search within this array (or set, or whatever it actually is). Please help!
If answers & query are fields in your bro_dns data:
baseSearchWithYourIPAddressField ipAddressField=*
| join ipAddressFIeld
[
search index=bro_dns_index
| makemv answers
| mvexpand answers
| table answers query
| rename answers AS ipAddressField
]
| table *
If answers & query are not fields in your bro_dns data:
baseSearchWithYourIPAddressField ipAddressField=*
| join ipAddressFIeld
[
search index=bro_dns_index
| rex "answers.+\[(?<answers>.*)\]"
| rex "query:\s+(?<query>.+)"
| makemv answers
| mvexpand answers
| table answers query
| rename answers AS ipAddressField
]
| table *
Thank you, this is getting me started, but I am still not quite there. I had to write "answers" as "answers{}" for this to work, and also I had to add the word "search" to the beginning of the statement in join, like this:
[search sourcetype=bro_dns | makemv answers{} | mvexpand answers{} | rename answers{} as ipAddressField]
This returns the results I want, but only a subset. The output of the base search (from bro_conn) includes some fields that are in common with the subsearch results from bro_dns, namely proto and port. This join command only returns results where all of these fields match, rather than just where answers matches my IPaddressfield. In bro_dns, all of the results have proto=udp and port=53, so basically I am only getting "query" results returned when the base search (bro_conn) also has proto=udp and port=53. The results of an http transaction on port 80, for example, does not return the query information despite this info being present in the bro_dns data. I also tried adding type=outer to the join command, but this doesn't help... 😞
Appreciate any further suggestions.
This interface isn't very welcoming of copy/pastes, so I am hand-formatting in my example:
This is an event in bro_dns that retuns a result from the join command when id.resp_h=17.249.105.246:
query:
something.apple.com
answers{}:
17.249.105.246
This is an event in bro_dns that does not return a result on the join when id.resp_h=17.248.143.90
query:
something.icloud.com
answers{}:
something.apple-dns.net
17.248.143.90
I have checked a bunch of results, and as long as answers{} contains more than one entry, it will not match the join search.
This is my search:
sourcetype=bro_conn | spath conn_state
| search conn_state!=S0 OR conn_state!=REJ
| lookup bro_conn_state.csv conn_state OUTPUT conn_state_meaning
| join id.resp_h
[ search sourcetype=bro_dns earliest=-1@hr
| makemv answers{}
| mvexpand answers{}
| table query,answers{} | rename answers{} AS id.resp_h]
| table _time,id.orig_h,dhcp_host_name,id.resp_h,id.resp_p,proto,query,answers{},conn_state_meaning
| sort -_time
When I remove the join from this search and compare the results, that's when I see all of the missing events.
It's the table command that was limiting your fields. I have adjusted my answer. Forgot to put the {} but you get the idea. Give the revised answer a shot
I've actually already tried that and that doesn't seem to be the problem.
I think I may have figured out what the issue is, but don't know how to solve it. The answers{} array typically contains multiple results (IPs and names). The join command is only returning results if the IP address I am searching for is an exact match to answers{}. In other words, only if answers{} contains a single entry which is the IP address in question. If answers contains multiple IPs, one of which is the one I am looking for, it returns nothing. So it seems I need a way to search for a partial match within the answers array.
Example:
query answers{}
p70-buy.itunes-apple.com.akadns.net 17.154.66.73
p70-buy.itunes.apple.com p70-buy.itunes-apple.com.akadns.net 17.154.66.73
I can match "17.154.66.73" on the first row above, but not the second, because answers also contains "p70-buy.itunes-apple.com.akadns.net"
That's what the makemv and mvexpand are there for. Can you give a sample of the data? (sensitive details anonymized)