How can I create a subsearch with multiple time ranges?

[aguthrie1190]

I'm trying to write a subsearch that searches multiple sections of time. What I have works until I wrap it in brackets for the subsearch. Here is what I have so far:

| gentimes start=-3
| eval earliest=starttime
| eval latest=starttime+20
| eval search="My search here"
| fields search earliest latest
| format "index=notAnIndex earliest=-1m | append [search" "" "" "" "] | append [search" "]"
| rex field=search mode=sed "s/\"//g"

This returns this search string:

 index=notAnIndex earliest=-1m 
| append [search earliest=1521504000 latest=1521504020 My search here ] 
| append [search earliest=1521590400 latest=1521590420 My search here ] 
| append [search earliest=1521676800 latest=1521676820 My search here ]

I have to use append instead of OR's because using OR will search the whole time frame between the earliest and latest time specified, which defeats the purpose of this. If I copy and paste the returned search string into a new search, it works great. If I wrap the original search in brackets, I get this error:

Error in 'append' command: The last argument must be a subsearch.

Looking through the job inspector, I can see that my earliest and latest times are being stripped out of the search. And then the error is probably from that _time that is tacked on to the end of the subsearch

litsearch index=notAnIndex 
| append [search My search here ] 
| append [search My search here ] 
| append [search My search here ] _time>=1521676800.000 _time<1521676820.000 
| fields keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server" 
| remotetl nb=300 et=1521676800.000000 lt=1521676820.000000 remove=true max_count=1000 max_prefetch=100

This seems like a bug, but maybe I just don't understand how the subsearch is supposed to work here. Has anyone found a way to make something like this work?

[dlee360]

I’m going to be honest, i didn’t read your question throughly and I’m on mobile, but it seems like you could do this via multisearch instead of all the sub searches.

| multisearch [search index=test earliest=-3d latest=-2d] [search index=test earliest=-2d latest=-1d] | stats blah

[aguthrie1190]

Unfortunately that doesn't do the trick. But I'm definitely going to start using multisearch, that seems like a lot simpler solution than append.

Still, when I format that string into a subsearch, it pulls the times outside of each of the individual searches.

[| gentimes start=-3
| eval earliest=starttime
| eval latest=starttime+30
| eval search="*"
| fields search earliest latest
| format "| multisearch [search" "" "" "" "] [search" "]"
| rex field=search mode=sed "s/\"//g"
| rex field=search mode=sed "s/'/\"/g"]

leads to this error:
Error in 'multisearch' command: Invalid argument: '_time>=1522022400.000'

And this is from logs for what it's actually searching in the subsearch:

litsearch | multisearch [search ] [search ] [search ] _time>=1522022400.000 _time<1522022430.000 
| fields keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server" 
| remotetl nb=300 et=1522022400.000000 lt=1522022430.000000 remove=true max_count=1000 max_prefetch=100

So that _time>=1522022400.000 _time<1522022430.000 bit is messing up the subsearch.

[p_gurav]

Hi,

Can you try:

| append [search index=anyindex earliest=1521504000 latest=1521504020 | remaining search ] 

[aguthrie1190]

Can you expand a little? I don't know where you meant for me to put that. I added it to the beginning of the format section, so that the subsearch would output the append at the beginning, but unfortunately that did not work.

[p_gurav]

Can you tell e what search you are running in append command, so i can explain it better.

[Richfez]

Agreed, I think the actual search you are running would be useful - there may be something in there that's "upsetting" the closing bracket or causing this issue.

Could you post the actual search please?

[aguthrie1190]

[ | gentimes start=-3
| eval earliest=starttime
| eval latest=starttime+20
| eval search="*"
| fields search earliest latest
| format "index=notAnIndex earliest=-1m | append [search" "" "" "" "] | append [search" "]"
| rex field=search mode=sed "s/\"//g"]

[aguthrie1190]

That should get you every log for the 20 seconds between midnight and 00:00:20 for the last three days. If you want to limit that to just a specific string:

[ | gentimes start=-3
| eval earliest=starttime
| eval latest=starttime+20
| eval search="call_start"
| fields search earliest latest
| format "index=notAnIndex earliest=-1m | append [search" "" "" "" "] | append [search" "]"
| rex field=search mode=sed "s/\"//g"]

If you remove the brackets, you can see that the search is formatted properly. If you look at the job inspector it appears that the earliest/latest times are stripped from the subsearch

[aguthrie1190]

This seems like it is a bug. I've made a simpler search to show the behavior.

This works:

[| gentimes start=-1
| eval search="* earliest=-5m"]

This doesn't work:

[| gentimes start=-1
| eval search="* earliest=-5m | append [ search * earliest=-20m latest=-15m ]"]

[aguthrie1190]

You can replace "My search here" with the search "*" to see the behavior. I'm using "call_start" as that's a frequently seen log in my system.

The idea is to take the output of gentimes, get a lot of 30 second slices going back many days, and then append each of those 30 second slices together to get a picture of what happens historically.

I'm using Splunk 6.4.0 in production, I tried this in a lab in 7, and it didn't error out with the same error message, but the same behavior was seen in the job inspector.