Knowledge Management

How to combine two accelerated saved searches to access the summary indexes of both searches?

TiagoTLD1
Communicator

Hello,

I have two searches, A and B, and they are accelerated.

When I run A or B separately, job inspector shows they are accessing their specific summary indexes to get the data I need. However, when I do search A and appencols search B, it only accesses the summary index of search A, making the part of appendcols to take too long.

Is there any way to make Splunk be able to access both summary indexes of saved searches?

Thank you

0 Karma
1 Solution

lguinn2
Legend

You don't actually have "summary indexes." For accelerated saved searches, Splunk automatically creates "data summaries" (also called "acceleration summaries"). These are quite different things in Splunk!

There are a few ways to do what you want. Splunk does not use an acceleration summary in a subsearch, although I can't find any documentation that says why it can't. So you might be able to trick Splunk by using the savedsearch command. I doubt that this will work, but it would be relatively easy to try.

Another way would be to actually use a summary index, and have both searches run individually to populate the summary index. Whenever you want to see the results, you then need a third search that simply retrieves the output of the two populating searches and combines them. This third search would be extremely fast. The downside is that the creation and maintenance of a summary index is a somewhat manual process - it is not automatic like report acceleration.

A final way would be to try the loadjob command. To use this technique, you would set up the second search as a scheduled search. You would need to have the scheduled search run relatively frequently, probably every 5 minutes or more frequently. Then, the overall search would include the first search and the second search (subsearch) would consist of the loadjob command. However, you would not be able to use appendcols - you would have use a join instead, to make sure that the 5-minute intervals aligned properly.

View solution in original post

rabitoblanco
Path Finder

What about restructuring your search so it's something like this format?

index =indexA sourcetype=sourcetypeA index=indexB sourcetype=sourcetypeB | transforming commands |appendcols| putitalltogetheretc

If that's accelerated, all of the data is being pulled before the first transforming command so it should be fast.

0 Karma

TiagoTLD1
Communicator

I tried that but since the subsearch uses mvzip, expand and rex, it is not worth it

lguinn2
Legend

You don't actually have "summary indexes." For accelerated saved searches, Splunk automatically creates "data summaries" (also called "acceleration summaries"). These are quite different things in Splunk!

There are a few ways to do what you want. Splunk does not use an acceleration summary in a subsearch, although I can't find any documentation that says why it can't. So you might be able to trick Splunk by using the savedsearch command. I doubt that this will work, but it would be relatively easy to try.

Another way would be to actually use a summary index, and have both searches run individually to populate the summary index. Whenever you want to see the results, you then need a third search that simply retrieves the output of the two populating searches and combines them. This third search would be extremely fast. The downside is that the creation and maintenance of a summary index is a somewhat manual process - it is not automatic like report acceleration.

A final way would be to try the loadjob command. To use this technique, you would set up the second search as a scheduled search. You would need to have the scheduled search run relatively frequently, probably every 5 minutes or more frequently. Then, the overall search would include the first search and the second search (subsearch) would consist of the loadjob command. However, you would not be able to use appendcols - you would have use a join instead, to make sure that the 5-minute intervals aligned properly.

TiagoTLD1
Communicator

Thank you for your suggestions!

1 - The savedsearch replacing the code did not bring any better results, only the part before the subsearch used acceleration summaries.

2 - Will try that, despite the fact that automatic maintenance won't be available with that scenario.

Thanks

0 Karma

TiagoTLD1
Communicator
index=KK (A=1 B!="x" C!="y"  D=1) OR info OR (E=*) OR F=* OR G=* 
| timechart avg(xxx) as RT, sum(eval(if(r1=1,1,0))) as CL, sum(eval(if(isnotnull('B') OR 'AX'!="0", 1, 0))) as ERR, sum(qq) as M, sum(g) as GT, avg(eval(100*cur/max)) as T, sum(CT) as cc, avg(dp) as DD
| fields _time, CL, RT, ERR, M, GT, T, CC, D
| appendcols [ search index=KK W=* 
| bucket _time span=5m
| eval ff= mvzip('WA','WB') 
| eval fff= mvzip(ff, 'WC') 
| mvexpand fff
| rex field=fff "(?<CB>\d+),(?<CM>\d+),(?<P>.+)"
| stats sum(CB) AS cbs , sum(CM) AS cmx by _time 
| eval rate=100*(cbs/cmx)
| timechart cont=false max(rate) AS Pool%  partial=false ]
0 Karma

somesoni2
SplunkTrust
SplunkTrust

A nice explanation is provided by @martin_mueller in the following post where above syntax will not use the accelerated result for subsearch.

https://answers.splunk.com/answers/234242/can-searches-with-joins-be-accelerated.html#answer-234251

0 Karma

TiagoTLD1
Communicator

Hi, thank you for your time.

I checked the link you posted and in the end, it shows exactly the same behaviour I am experiencing. The fact that only the first transforming command is used might be the "problem", weird though.

Thanks anyway

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Can you provide your search? How are you calling those accelerated saved searches (assuming) in join subsearch?

0 Karma

woodcock
Esteemed Legend

Yes, show us the search.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...