Conditional append

[patrycja]

Hello,
I don't know if it possible, but I want to make a conditional append in my search query.
I'm using saved search to get some data from index1 and save it in index2 to speed up loading a dashboard. Saved search is scheduled to run every hour, so I cannot display most recent data on dashboard (from the last scheduled search). So I wanted to append my query (which gets data from index2) by data from index1 from the last hour. The beginning of my query looks that way:

index="index2" 
| append [search index="index1" earliest=-60m@m | ...some other searches...] ...

It very slows down my search, because I need to search through two indexes and merge them. My idea is to make a filter which you can set to "true" if you want to include the last data from second index or "false" if you don't care about most recent logs and you want your query to execute fast.
I wanted something like this:

index="index2" 
| IF $input$=true THEN (append [search index="index1" earliest=-60m@m | ... ])

Any idea how to achieve that?

[skoelpin]

Don't do this.. This will not scale and will slow your search down. You should look into refactroring your query, accelerating the reports, pushing the data to a summary index, or creating accelerated data models.

[patrycja]

Why? I thought that Splunk is searching by time very fast. I just need an hour of logs from the big index and the rest from my prepared index. What's the difference between pushing the data to a summary index or any other "normal" index? And what do you mean by refactoring? How can I achieve what I want by accelerating reports or data models?

[gcusello]

Hi patrycja,
you could schedule your search and write output in a summary index (using collect command), then run a search on it.
Bye.
Giuseppe

[patrycja]

This is what I do. But I can collect logs only once per hour. It mean that when your query executes at 10:00 and you are looking at dashboard at 10:50 you won't see events which occurred between 10:00 and 10:50. You will see only events which occurred BEFORE 10:00. That's why I want to add an hour from my base index.

[gcusello]

Yes, iyou could run your summary search more frequently, but anyway you have this limitation.
Bye.
Giuseppe