topic Re: Join 2 large tstats data sets in Splunk Search

Join 2 large tstats data sets

btorresgil — Tue, 10 Sep 2013 19:22:02 GMT

I need to join two large tstats namespaces on multiple fields. For example, I have these two tstats:

| tstats count(dst_ip) AS cdip FROM bad_traffic groupby protocol dst_port dst_ip

and

| tstats count(dst_ip) AS cdipt FROM all_traffic groupby protocol dst_port dst_ip src_ip

I need all src_ip fields from all_traffic namespace where the dst_ip, dst_port, and protocol of the all_traffic entry match a dst_ip, dst_port, protocol combination in the bad_traffic namespace. Effectively this gives a list of all the source ip for traffic that matches bad traffic.

I read through the stats, tstats, and eval manuals, but I'm stuck on how to do this efficiently. I can do it with a join on the two tstats commands above, but the datasets are so large it takes forever. I considered doing a prestat and append on the tstats, but I can't seem to get the desired results this way.

Thanks for any help!

Re: Join 2 large tstats data sets

sanjay_shrestha — Mon, 16 Feb 2015 19:10:02 GMT

How did you solve this issue?

Re: Join 2 large tstats data sets

dart — Fri, 27 Feb 2015 10:52:03 GMT

I'd do it like this:

| tstats count(dst_ip) AS cdip FROM bad_traffic groupby protocol dst_port dst_ip | tstats append=true count(dst_ip) AS cdipt FROM all_traffic groupby protocol dst_port dst_ip src_ip | stats sum(cdip) as cdip sum(cdipt) as cdipt values(src_ip) as src_ip by protocol dst_port dst_ip | search cdip>0 cdipt>0

does that give you what you want?

Re: Join 2 large tstats data sets

David — Wed, 09 Dec 2015 15:28:15 GMT

I'm not able to replicate this.. You can't do append=true without prestats, and it doesn't seem like you can sum() a count() prestats clause (e.g., sum(cdip) where cdip = prestats count(dst_ip)). Does this work?

Re: Join 2 large tstats data sets

woodcock — Tue, 05 Dec 2017 05:10:43 GMT

Why is an answer that cannot even run accepted?

Re: Join 2 large tstats data sets

David — Tue, 05 Dec 2017 05:19:17 GMT

@woodcock because some hero has yet to propose a better answer, I can only assume 😉

Re: Join 2 large tstats data sets

dturnbull_splun — Wed, 06 Dec 2017 00:53:01 GMT

I'm not sure if my previous answer used to work or was simply incorrect. Tstats does work with union:

| tstats summariesonly=false values(Authentication.tag) as tag, values(Authentication.app) as app, count(eval('Authentication.action'=="failure")) as failure, count(eval('Authentication.action'=="success")) as success from datamodel=Authentication by Authentication.src |union [tstats count from datamodel=Authentication BY Authentication.src ]

So to rework the example above:

| tstats count(dst_ip) AS cdip FROM bad_traffic groupby protocol dst_port dst_ip | union [tstats  count(dst_ip) AS cdipt FROM all_traffic groupby protocol dst_port dst_ip src_ip] | stats sum(cdip) as cdip sum(cdipt) as cdipt values(src_ip) as src_ip by protocol dst_port dst_ip | search cdip>0 cdipt>0

You might need to make the source datamodel names explicit and rename, but that should be straightforward, either before the union, after it or inside the subsearch.

Re: Join 2 large tstats data sets

woodcock — Wed, 13 Dec 2017 02:49:30 GMT

Sadly, I am not he, but I will followup here if I find him!

Re: Join 2 large tstats data sets

David — Wed, 13 Dec 2017 03:25:06 GMT

Duncan is though! Scroll down but a little and you’ll see the new and improved answer.

Re: Join 2 large tstats data sets

Sanjai676 — Thu, 21 Feb 2019 16:45:57 GMT

Perfect..!! worked like a charm. good logic.

Re: Join 2 large tstats data sets

landen99 — Mon, 02 Dec 2019 21:25:20 GMT

I downvoted this post because doesn't work