Splunk Search

How do I search for correlating events from Source log?

roderickjones
Engager

So we log API calls and response errors. However I'm having issues searching for the corrilating event from a log. 

 

Example:

Im searching for event related to and endpoint https://api.lyft.com/v1/paydisputes/oi/orders? and I'm about to get the log back of what was sent. 

Nov 24 16:20:17 ip-11-222-111-100 order-service: [2020-11-24 16:20:17.399] [INFO ] [417863580]: [reactor-http-server-epoll-8] c.v.o.o.m.s.HostedApiRestClient [] - Sending Request - Method: 'GET', Path: 'https://api.test.com/v1/paydisputes/oi/orders?arn=1363473846864745645&authorizationCode=039319&settlementAmount=12.17&authorizationAmount=12.17&settlementDate=2020-11-16T00:00:00Z&authorizationDate=2020-11-16T00:00:00Z&settlementCurrency=USD&authorizationCurrency=USD&creditCardBin=473702&creditCard4=1788&orderId=14738858272569688&transactionId=234235423523543&sellerId=42201', Headers: '{x-test-api-version=[1.4], Accept=[application/json], Authorization=[Bearer ******], Content-Type=[application/json]}' 

* host = ip-11-222-111-100
* index = syslog
* source = /var/log/syslog
* sourcetype = syslog

When I go to "Show Source" there is a associated event :

Nov 24 16:20:17 ip-11-222-111-100 order-service: [2020-11-24 16:20:17.512] [ERROR] [417863693]: [reactor-http-client-epoll-11] c.v.o.o.OrderGlobalWebExceptionHandler [] - Error occurred while processing web requestorg.springframework.web.server.ResponseStatusException: 404 NOT_FOUND "Failed to receive order from merchant hosted api with statusCode=404, Not Found"#012#011at com.verifi.orderservice.order.merchanthosted.service.HostedApiRestClient.handleResponse(HostedApiRestClient.java:68)#012#011at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:153)#012#011at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)#012#011at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)#012#011at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)#012#011at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)#012#011at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76)#012#011at .....

Nov 24 16:20:17 ip-11-222-111-100 #011org.springframework.web.server.ResponseStatusException: 404 NOT_FOUND "Failed to receive order from merchant hosted api with statusCode=404, Not Found"

 

I can't seem to make a search that would allow me to pull any errors associated with the GET call based on the next event that shows an error code. I think it's because they are not correlated in any way other than time possibly?

to4kawa
Ultra Champion

No example is not enough.

If reactor-http-server-epoll adds up, it is possible, I guess.

index=_internal | head 1 | fields _raw | eval _raw="Nov 24 16:20:17 ip-11-222-111-100 order-service: [2020-11-24 16:20:17.399] [INFO] [417863580]: [reactor-http-server-epoll-8] c.v.o.o.m.s.HostedApiRestClient [] - Sending Request - Method: 'GET', Path: 'https://api.test.com/v1/paydisputes/oi/orders?arn=1363473846864745645&authorizationCode=039319&settlementAmount=12.17&authorizationAmount=12.17&settlementDate=2020-11-16T00:00:00Z&authorizationDate=2020-11-16T00:00:00Z&settlementCurrency=USD&authorizationCurrency=USD&creditCardBin=473702&creditCard4=1788&orderId=14738858272569688&transactionId=234235423523543&sellerId=42201', Headers: '{x-test-api-version=[1.4], Accept=[application/json], Authorization=[Bearer ******], Content-Type=[application/json]}' "
| appendpipe [ eval _raw="Nov 24 16:20:17 ip-11-222-111-100 order-service: [2020-11-24 16:20:17.512] [ERROR] [417863693]: [reactor-http-client-epoll-11] c.v.o.o.OrderGlobalWebExceptionHandler [] - Error occurred while processing web requestorg.springframework.web.server.ResponseStatusException: 404 NOT_FOUND \"Failed to receive order from merchant hosted api with statusCode=404, Not Found\"#012#011at com.verifi.orderservice.order.merchanthosted.service.HostedApiRestClient.handleResponse(HostedApiRestClient.java:68)#012#011at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:153)#012#011at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)#012#011at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)#012#011at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)#012#011at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)#012#011at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76)#012#011a" 
| appendpipe [ eval _raw="Nov 24 16:20:17 ip-11-222-111-100 #011org.springframework.web.server.ResponseStatusException: 404 NOT_FOUND \"Failed to receive order from merchant hosted api with statusCode=404, Not Found\""]]
| rex "(?<time>\w{3} \d\d \S+) (?<ip>\S+)"
| rex "\[(?<status>ERROR|INFO)\]"
| rex "Method: (?<method>\S+),"
| rex "statusCode=(?<status_code>\d+)"
| stats values(method) as method values(status_code) as status_code values(status) as status by ip time

rush job



Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...