topic Re: How to combine results from several data models to accelerate multiple objects within a single data model? in Splunk Search

How to combine results from several data models to accelerate multiple objects within a single data model?

PrinceOfEval — Tue, 29 Sep 2020 06:36:24 GMT

I'm using Splunk 6.1.4, which is unable to accelerate multiple objects within a single data model. Because of this, I've created 4 data models and accelerated each.

I try to combine the results like this:

| tstats prestats=TRUE append=TRUE summariesonly=TRUE count FROM datamodel=Thing1 by sourcetype Object1.src_ip Object1.src_port Object1.dest_ip Object1.dest_port Object1.sc_filter_result  
| tstats prestats=TRUE append=TRUE summariesonly=TRUE count FROM datamodel=Thing2 by sourcetype Object2.src_ip Object2.src_port Object2.dest_ip Object2.dest_port Object2.signature Object2.act  
| tstats prestats=TRUE append=TRUE summariesonly=TRUE count FROM datamodel=Thing3 by sourcetype Object3.src_ip Object3.src_port Object3.dest_ip Object3.dest_port Object3.action  
| tstats prestats=TRUE append=TRUE summariesonly=TRUE count FROM datamodel=Thing4 by sourcetype Object4.src_ip Object4.dest_ip Object4.signature Object4.src_port Object4.dest_port 

| eval src_ip=coalesce(Object1.src_ip, Object2.src_ip, Object3.src_ip, Object4.src_ip) 
| eval dest_ip=coalesce(Object1.dest_ip, Object2.dest_ip, Object3.dest_ip, Object4.dest_ip) 
| eval src_port=coalesce(Object1.src_port, Object2.src_port, Object3.src_port, Object4.src_port) 
| eval dest_port=coalesce(Object1.dest_port, Object2.dest_port, Object3.dest_port, Object4.dest_port) 
| eval signature=coalesce(Object2.signature, Object4.src_ip) 
| eval action=coalesce(Object1.sc_filter_result, Object2.act, Object3.action, Object4.src_ip)

| table sourcetype signature action src_ip src_port dest_ip dest_port

It doesn't work at all, and I think this is because the eval command only applies within the namespace of each datamodel object. That is, src_ip doesn't get created by the eval statement, only Object(n).src_ip. How can I combine data from multiple data models (while taking advantage of acceleration)?

Thanks!

Re: How to combine results from several data models to accelerate multiple objects within a single data model?

jacobwilkins — Mon, 06 Jul 2015 14:34:04 GMT

I don't quite understand what you mean by "unable to accelerate multiple objects"...

What you are doing here looks upside-down from how you should use a DM.

You need to start with a DM that has a root event that brings in all the data you are looking for. You might have to OR several base searches together:

(index=foo sourcetype=bar) OR (index=baz sourcetype=qux)

Accelerate the DM that includes all your criteria.

Re: How to combine results from several data models to accelerate multiple objects within a single data model?

PrinceOfEval — Mon, 06 Jul 2015 17:39:44 GMT

From http://docs.splunk.com/Documentation/Splunk/6.1.4/Knowledge/Acceleratedatamodels under the heading Data model acceleration caveats: "Data model acceleration only affects the first event object hierarchy in a data model" -- that's the part about being unable to accelerate multiple objects.

I was hoping to segment the data model by sourcetype for several reasons. The most important is so I wouldn't have to rebuild the entire huge summary if there was an issue with one of the data sources. I think I got this from the docs somewhere, which is why I initially tried creating a DM with 4 root events.

I'll give it a shot with a single root event I suppose, but it seems like it should be pretty straitforward to combine data from multiple DMs in a search.

Re: How to combine results from several data models to accelerate multiple objects within a single data model?

bmacias84 — Mon, 06 Jul 2015 21:40:15 GMT

Try the multisearch command.

Re: How to combine results from several data models to accelerate multiple objects within a single data model?

PrinceOfEval — Tue, 07 Jul 2015 16:59:21 GMT

tstats doesn't seem to be a streaming command, which means it won't work in a multisearch subsearch.

Re: How to combine results from several data models to accelerate multiple objects within a single data model?

proletariat99 — Thu, 03 Dec 2015 21:08:34 GMT

I'm having the same problem, and I don't think there's an answer that will satisfy our needs (even in 6.3.1). I think the solution is to build a new data model that combines the constraints of the other data models and then accelerate that. That's the only thing I've found that works. Also, remember that only the first tier of the data model is accelerated. Child constraints aren't accelerated.