How to build a search with queries depending on each other?

[moguai]

I have a scenario where typical HTTP requests are logged in Splunk.
Every request has an unique identifier which is saved in a "request_id" field.
Between request and response the server generates a set of logs/events and each one has  this "request_id" added.

So far so good. Now it is possible to find the appropriate server logs for a client HTTP issue via the "request_id".

Now I often also have the requirement to find all errors for a particular device.

In this scenario the device identifier is part of the first log/event of the request, but the following logs/events do not have this information any more.

So basically something like:
1. req.begin "Started... deviceId=12345", request_id="1"
2. .... "request_id="1" deviceId=???
3. .... "request_id="1" deviceId=???
4. .... "request_id="1" deviceId=???
5. req.end ... "request_id="1" deviceId=???

The search would look like:
- Search for all "request.begin" events with device identifier "xyz"
- Get all "request_id" of those events
- and finally get all events containing one of the above "request_id"

Not sure how to build the query for this. 🤔

Would be very grateful for some tips!

Best regards
Tore

[jacobpevans]

Greetings @moguai ,

Take a look at this search. It should get you close to what you're after. Note that the eventstats and the stats will work by themselves without the other. I included both so you will have more options with manipulating your dataset.

           | makeresults | eval _raw = "req.begin \"Started... deviceId=\"12345\", request_id=\"1\""
| append [ | makeresults | eval _raw = ".... \"request_id=\"1\" deviceId=???" ]
| append [ | makeresults | eval _raw = ".... \"request_id=\"1\" deviceId=???" ]
| append [ | makeresults | eval _raw = ".... \"request_id=\"1\" deviceId=???" ]
| append [ | makeresults | eval _raw = "req.end ... \"request_id=\"1\" deviceId=???" ]
| reverse
| rex "deviceId=\"?(?<deviceId>\d+)\"?"
| rex "request_id=\"?(?<request_id>\d+)\"?"

| eventstats values(deviceId) as deviceId by request_id
| stats list(_raw) as events by request_id

[moguai]

Hi @jacobpevans ,

thanks for the fast response! I think I am still a bit confused. 😅

Maybe a bit more practical:

sourcetype="requests" device_id="XYZ"

As result I get e.g. 10 events with 10 different "request_id" (separate field) from that XYZ device.

Than I need to query all events matching the 10 "request_id" I got from the 10 events, because the "device_id" is only available for the first log.

sourcetype="requests" request_id="1" OR request_id="2" OR request_id="3" ... 

Would still the approach you mentioned the way to go?

BR,
Tore

[jacobpevans · Accepted solution]

I misunderstood your question. Try this:

sourcetype=requests
| eventstats values(device_id) as device_id by request_id
| search device_id="XYZ"

Side note: you should start every search with "index=ABC" for much more optimized searching.

[moguai]

Thanks a lot!