Splunk Search

Manually including the output of a subsearch in a search returns events, but why do I get no results using the subsearch directly in the search?

pankaj_vohra
Engager

As part of our index, we log events for every request we make to our downstream systems. Each system which receives a request appends a TraceContext (GUID) to the incoming TraceContext. Idea is to have a way to get the chain of events.

Here are sample set of event messages:

SourceName=QueryAPI
EventCode=11
Payload={
"TraceContext":"91e30cbc-5bf7-43cb-b615-ce83e3abad36|662deb50-22bc-4211-bf19-a49acc2a790d",
"EventName":"Start"
}

SourceName=QueryAPI
EventCode=10
Payload={
"TraceContext":"91e30cbc-5bf7-43cb-b615-ce83e3abad36|662deb50-22bc-4211-bf19-a49acc2a790d|5a595ffe-9a5d-4abd-93fb-d57c3f427af0",
"EventName":"Receive"
}

SourceName=QueryAPI
EventCode=9
Payload={
"TraceContext":"91e30cbc-5bf7-43cb-b615-ce83e3abad36|662deb50-22bc-4211-bf19-a49acc2a790d|5a595ffe-9a5d-4abd-93fb-d57c3f427af0",
"EventName":"Send"
}

SourceName=QueryAPI
EventCode=12
Payload={
"TraceContext":"91e30cbc-5bf7-43cb-b615-ce83e3abad36|662deb50-22bc-4211-bf19-a49acc2a790d",
"EventName":"Stop"
}

I am running the search below to identify the starting event TraceContext and using that in subsearch, hoping to see all the matching events, but my search does not produce any events:

index=IndexName SourceName=SomeName [search index=IndexName SourceName=SomeName EventCode=11 | table TraceContext]

If I just run the subsearch search separately and use the TraceContext string and use that for a new search, I can find all the matching events.

index=IndexName SourceName=SomeName EventCode=11 | table TraceContext 

-> returns "91e30cbc-5bf7-43cb-b615-ce83e3abad36|662deb50-22bc-4211-bf19-a49acc2a790d"

and running

index=IndexName SourceName=SomeName "91e30cbc-5bf7-43cb-b615-ce83e3abad36|662deb50-22bc-4211-bf19-a49acc2a790d"

returns all records. I am not sure what am I doing wrong.

0 Karma
1 Solution

woodcock
Esteemed Legend

Try this:

 index=IndexName SourceName=SomeName [search index=IndexName SourceName=SomeName EventCode=11 | return $TraceContext]

View solution in original post

miront
Explorer

I have had the same thing happen to me in the past. I opened a ticket with Splunk and they didn't know what to make of it either. The funny thing is, it worked fine on one Search Head and did not work on another.

0 Karma

woodcock
Esteemed Legend

Try this:

 index=IndexName SourceName=SomeName [search index=IndexName SourceName=SomeName EventCode=11 | return $TraceContext]

pankaj_vohra
Engager

TraceContext is not an indexed field. It is part of a JSON string, which is indexed as Payload field.

0 Karma

pankaj_vohra
Engager

by default return statement only returns the first matching event. However we can specify the count also. I modified the query as below and I am getting required results now:

index=IndexName SourceName=SomeName [search index=IndexName SourceName=SomeName EventCode=11 | return 100 $TraceContext]

0 Karma

pankaj_vohra
Engager

This works fine but it only returns events for one TraceContext. I would like to to get all events matching all traceContext available in the index.

0 Karma

woodcock
Esteemed Legend

Do all events have a field named TraceContext? If so, then you can do this:

index=IndexName SourceName=SomeName [search index=IndexName SourceName=SomeName EventCode=11 | fields TraceContext]
0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July Tech Talks, Office Hours, and Webinars!

What are Community Office Hours?Community Office Hours is an interactive 60-minute Zoom series where ...

Updated Data Type Articles, Anniversary Celebrations, and More on Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

A Prelude to .conf25: Your Guide to Splunk University

Heading to Boston this September for .conf25? Get a jumpstart by arriving a few days early for Splunk ...