Splunk Search

How do I search for correlating events from Source log?

roderickjones
Engager

So we log API calls and response errors. However I'm having issues searching for the corrilating event from a log. 

 

Example:

Im searching for event related to and endpoint https://api.lyft.com/v1/paydisputes/oi/orders? and I'm about to get the log back of what was sent. 

Nov 24 16:20:17 ip-11-222-111-100 order-service: [2020-11-24 16:20:17.399] [INFO ] [417863580]: [reactor-http-server-epoll-8] c.v.o.o.m.s.HostedApiRestClient [] - Sending Request - Method: 'GET', Path: 'https://api.test.com/v1/paydisputes/oi/orders?arn=1363473846864745645&authorizationCode=039319&settlementAmount=12.17&authorizationAmount=12.17&settlementDate=2020-11-16T00:00:00Z&authorizationDate=2020-11-16T00:00:00Z&settlementCurrency=USD&authorizationCurrency=USD&creditCardBin=473702&creditCard4=1788&orderId=14738858272569688&transactionId=234235423523543&sellerId=42201', Headers: '{x-test-api-version=[1.4], Accept=[application/json], Authorization=[Bearer ******], Content-Type=[application/json]}' 

* host = ip-11-222-111-100
* index = syslog
* source = /var/log/syslog
* sourcetype = syslog

When I go to "Show Source" there is a associated event :

Nov 24 16:20:17 ip-11-222-111-100 order-service: [2020-11-24 16:20:17.512] [ERROR] [417863693]: [reactor-http-client-epoll-11] c.v.o.o.OrderGlobalWebExceptionHandler [] - Error occurred while processing web requestorg.springframework.web.server.ResponseStatusException: 404 NOT_FOUND "Failed to receive order from merchant hosted api with statusCode=404, Not Found"#012#011at com.verifi.orderservice.order.merchanthosted.service.HostedApiRestClient.handleResponse(HostedApiRestClient.java:68)#012#011at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:153)#012#011at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)#012#011at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)#012#011at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)#012#011at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)#012#011at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76)#012#011at .....

Nov 24 16:20:17 ip-11-222-111-100 #011org.springframework.web.server.ResponseStatusException: 404 NOT_FOUND "Failed to receive order from merchant hosted api with statusCode=404, Not Found"

 

I can't seem to make a search that would allow me to pull any errors associated with the GET call based on the next event that shows an error code. I think it's because they are not correlated in any way other than time possibly?

to4kawa
Ultra Champion

No example is not enough.

If reactor-http-server-epoll adds up, it is possible, I guess.

index=_internal | head 1 | fields _raw | eval _raw="Nov 24 16:20:17 ip-11-222-111-100 order-service: [2020-11-24 16:20:17.399] [INFO] [417863580]: [reactor-http-server-epoll-8] c.v.o.o.m.s.HostedApiRestClient [] - Sending Request - Method: 'GET', Path: 'https://api.test.com/v1/paydisputes/oi/orders?arn=1363473846864745645&authorizationCode=039319&settlementAmount=12.17&authorizationAmount=12.17&settlementDate=2020-11-16T00:00:00Z&authorizationDate=2020-11-16T00:00:00Z&settlementCurrency=USD&authorizationCurrency=USD&creditCardBin=473702&creditCard4=1788&orderId=14738858272569688&transactionId=234235423523543&sellerId=42201', Headers: '{x-test-api-version=[1.4], Accept=[application/json], Authorization=[Bearer ******], Content-Type=[application/json]}' "
| appendpipe [ eval _raw="Nov 24 16:20:17 ip-11-222-111-100 order-service: [2020-11-24 16:20:17.512] [ERROR] [417863693]: [reactor-http-client-epoll-11] c.v.o.o.OrderGlobalWebExceptionHandler [] - Error occurred while processing web requestorg.springframework.web.server.ResponseStatusException: 404 NOT_FOUND \"Failed to receive order from merchant hosted api with statusCode=404, Not Found\"#012#011at com.verifi.orderservice.order.merchanthosted.service.HostedApiRestClient.handleResponse(HostedApiRestClient.java:68)#012#011at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:153)#012#011at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)#012#011at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)#012#011at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)#012#011at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)#012#011at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76)#012#011a" 
| appendpipe [ eval _raw="Nov 24 16:20:17 ip-11-222-111-100 #011org.springframework.web.server.ResponseStatusException: 404 NOT_FOUND \"Failed to receive order from merchant hosted api with statusCode=404, Not Found\""]]
| rex "(?<time>\w{3} \d\d \S+) (?<ip>\S+)"
| rex "\[(?<status>ERROR|INFO)\]"
| rex "Method: (?<method>\S+),"
| rex "statusCode=(?<status_code>\d+)"
| stats values(method) as method values(status_code) as status_code values(status) as status by ip time

rush job



Get Updates on the Splunk Community!

.conf25 Community Recap

Hello Splunkers, And just like that, .conf25 is in the books! What an incredible few days — full of learning, ...

Splunk App Developers | .conf25 Recap & What’s Next

If you stopped by the Builder Bar at .conf25 this year, thank you! The retro tech beer garden vibes were ...

Congratulations to the 2025-2026 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...