Manually including the output of a subsearch in a search returns events, but why do I get no results using the subsearch directly in the search?

[pankaj_vohra]

As part of our index, we log events for every request we make to our downstream systems. Each system which receives a request appends a TraceContext (GUID) to the incoming TraceContext. Idea is to have a way to get the chain of events.

Here are sample set of event messages:

SourceName=QueryAPI
EventCode=11
Payload={
"TraceContext":"91e30cbc-5bf7-43cb-b615-ce83e3abad36|662deb50-22bc-4211-bf19-a49acc2a790d",
"EventName":"Start"
}

SourceName=QueryAPI
EventCode=10
Payload={
"TraceContext":"91e30cbc-5bf7-43cb-b615-ce83e3abad36|662deb50-22bc-4211-bf19-a49acc2a790d|5a595ffe-9a5d-4abd-93fb-d57c3f427af0",
"EventName":"Receive"
}

SourceName=QueryAPI
EventCode=9
Payload={
"TraceContext":"91e30cbc-5bf7-43cb-b615-ce83e3abad36|662deb50-22bc-4211-bf19-a49acc2a790d|5a595ffe-9a5d-4abd-93fb-d57c3f427af0",
"EventName":"Send"
}

SourceName=QueryAPI
EventCode=12
Payload={
"TraceContext":"91e30cbc-5bf7-43cb-b615-ce83e3abad36|662deb50-22bc-4211-bf19-a49acc2a790d",
"EventName":"Stop"
}

I am running the search below to identify the starting event TraceContext and using that in subsearch, hoping to see all the matching events, but my search does not produce any events:

index=IndexName SourceName=SomeName [search index=IndexName SourceName=SomeName EventCode=11 | table TraceContext]

If I just run the subsearch search separately and use the TraceContext string and use that for a new search, I can find all the matching events.

index=IndexName SourceName=SomeName EventCode=11 | table TraceContext 

-> returns "91e30cbc-5bf7-43cb-b615-ce83e3abad36|662deb50-22bc-4211-bf19-a49acc2a790d"

and running

index=IndexName SourceName=SomeName "91e30cbc-5bf7-43cb-b615-ce83e3abad36|662deb50-22bc-4211-bf19-a49acc2a790d"

returns all records. I am not sure what am I doing wrong.

[miront]

I have had the same thing happen to me in the past. I opened a ticket with Splunk and they didn't know what to make of it either. The funny thing is, it worked fine on one Search Head and did not work on another.

[woodcock · Accepted solution]

Try this:

 index=IndexName SourceName=SomeName [search index=IndexName SourceName=SomeName EventCode=11 | return $TraceContext]

[pankaj_vohra]

TraceContext is not an indexed field. It is part of a JSON string, which is indexed as Payload field.

[pankaj_vohra]

by default return statement only returns the first matching event. However we can specify the count also. I modified the query as below and I am getting required results now:

index=IndexName SourceName=SomeName [search index=IndexName SourceName=SomeName EventCode=11 | return 100 $TraceContext]

[pankaj_vohra]

This works fine but it only returns events for one TraceContext. I would like to to get all events matching all traceContext available in the index.

[woodcock]

Do all events have a field named TraceContext? If so, then you can do this:

index=IndexName SourceName=SomeName [search index=IndexName SourceName=SomeName EventCode=11 | fields TraceContext]