How is the index=threat_activity filled up with data in splunk Enterprise Security (ES)? How can I add an additional field to it?

[btiggemann]

We have got squid proxy logs that are compared with the threat lists in splunk ES.
It works fine, but on the list on splunk ES Advanced Threat - Threatlist Activity - Threat Activity Details we only see ip addresses in the dest field.

In the log events of squid I also have the URL, which is much more human readable.
What I want is to add the field uri_host also to my data in the index=threat_activity.

It looks like the index is filled by the a saved search: Threat - Source And Destination Matches - Threat Gen
The data looks like:

11/27/2015 14:15:00 +0100, search_name="Threat - Source And Destination Matches - Threat Gen", search_now=1448630100.000, info_min_time=1448622000.000, info_max_time=1448630100.000, info_search_time=1448630114.038, dest="xxx.xxx.xx.xxx", orig_sourcetype="cisco:asa", src="yyy.yyy.yyy.yyy", threat_collection=ip_intel, threat_collection_key="emerging_threats_ip_blocklist|43.229.52.0/22", threat_key=emerging_threats_ip_blocklist, threat_match_field=src, threat_match_value="43.229.53.53"

The search looks like this:

| src_dest_tstats("allowed") | truncate_domain_dedup(src) | truncate_domain_dedup(dest) | threatintel_multilookup(src) | threatintel_multilookup(dest) | search threat_collection_key=* | fields - count | zipexpand_threat_matches | fields sourcetype,src,dest,threat*

I tried to add just | fields sourcetype,src,dest, uri_host, threat* but this is not working.

Does anybody have a description of this macros? Or where can I find them to adjust them?

[saravanan90]

Paste the query in search box and press Ctrl+shift+E. You will get to know the full query, modify the query based on your requirement.
"Threat - Source And Destination Matches - Threat Gen" check logs from datamodels of Network_Traffic, Web & IDS.

[renjujacob88]

If you want to capture the domain field values in the threat activity dashboard, you need toa create a saved search ( say for example "Threat - URL squid Matches - Threat Gen".

It would be good if you have a datamodel for squid or you can go with normal index command. Please find the query below.

The query will look for the squid domains, followed by comparing with the lookup to see if there's a hit.

| tstats prestats=true local=false values(sourcetype) as sourcetype,values(squid.src),values(squid.dest) from datamodel=squid by squid.domain | eval url='squid.domain' | eval threat_match_field="squid.domain" | eval url=if(isnull(url),'squid.domain',url) | eval threat_match_field=if(isnull(threat_match_field),"url",threat_match_field) | stats values(sourcetype) as sourcetype,values(squid.src) as src,values(squid.dest) as dest by url,threat_match_field | lookup update=true ip_intel domain as url OUTPUTNEW

Hope this works

[ssuresh · Accepted solution]

Hello,

You need to understand how this search works. This correlation rule is running against all the data source and matching the fields against the Source types. In squid proxy sourcetype, extract the field called dest for the URL. then you threat list activity will match the dest filed to URL. Hope this should work.

[btiggemann]

Yes, that's it. I already fixed it last year.

[esix_splunk]

You can find the macro in the GUI under Settings -> Advanced Search -> Search Macros. You can dissect them from there.

[renjujacob88]

If you want to capture the domain field values in the threat activity dashboard, you need to write a search driven lookup ( say for example "Threat - URL squid Matches - Threat Gen".

It would be good if you have a datamodel for squid or you can go with normal index command. Please find the query below.

The query will look for the squid domains, followed by comparing with the lookup to see if hit.

| tstats prestats=true local=false values(sourcetype) as sourcetype,values(squid.src),values(squid.dest) from datamodel=squid by squid.domain | eval url='squid.domain' | eval threat_match_field="squid.domain" | eval url=if(isnull(url),'squid.domain',url) | eval threat_match_field=if(isnull(threat_match_field),"url",threat_match_field) | stats values(sourcetype) as sourcetype,values(squid.src) as src,values(squid.dest) as dest by url,threat_match_field | lookup update=true ip_intel domain as url OUTPUTNEW

Hope this works