Splunk Search

How do I search for correlating events from Source log?

roderickjones
Engager

So we log API calls and response errors. However I'm having issues searching for the corrilating event from a log. 

 

Example:

Im searching for event related to and endpoint https://api.lyft.com/v1/paydisputes/oi/orders? and I'm about to get the log back of what was sent. 

Nov 24 16:20:17 ip-11-222-111-100 order-service: [2020-11-24 16:20:17.399] [INFO ] [417863580]: [reactor-http-server-epoll-8] c.v.o.o.m.s.HostedApiRestClient [] - Sending Request - Method: 'GET', Path: 'https://api.test.com/v1/paydisputes/oi/orders?arn=1363473846864745645&authorizationCode=039319&settlementAmount=12.17&authorizationAmount=12.17&settlementDate=2020-11-16T00:00:00Z&authorizationDate=2020-11-16T00:00:00Z&settlementCurrency=USD&authorizationCurrency=USD&creditCardBin=473702&creditCard4=1788&orderId=14738858272569688&transactionId=234235423523543&sellerId=42201', Headers: '{x-test-api-version=[1.4], Accept=[application/json], Authorization=[Bearer ******], Content-Type=[application/json]}' 

* host = ip-11-222-111-100
* index = syslog
* source = /var/log/syslog
* sourcetype = syslog

When I go to "Show Source" there is a associated event :

Nov 24 16:20:17 ip-11-222-111-100 order-service: [2020-11-24 16:20:17.512] [ERROR] [417863693]: [reactor-http-client-epoll-11] c.v.o.o.OrderGlobalWebExceptionHandler [] - Error occurred while processing web requestorg.springframework.web.server.ResponseStatusException: 404 NOT_FOUND "Failed to receive order from merchant hosted api with statusCode=404, Not Found"#012#011at com.verifi.orderservice.order.merchanthosted.service.HostedApiRestClient.handleResponse(HostedApiRestClient.java:68)#012#011at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:153)#012#011at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)#012#011at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)#012#011at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)#012#011at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)#012#011at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76)#012#011at .....

Nov 24 16:20:17 ip-11-222-111-100 #011org.springframework.web.server.ResponseStatusException: 404 NOT_FOUND "Failed to receive order from merchant hosted api with statusCode=404, Not Found"

 

I can't seem to make a search that would allow me to pull any errors associated with the GET call based on the next event that shows an error code. I think it's because they are not correlated in any way other than time possibly?

to4kawa
Ultra Champion

No example is not enough.

If reactor-http-server-epoll adds up, it is possible, I guess.

index=_internal | head 1 | fields _raw | eval _raw="Nov 24 16:20:17 ip-11-222-111-100 order-service: [2020-11-24 16:20:17.399] [INFO] [417863580]: [reactor-http-server-epoll-8] c.v.o.o.m.s.HostedApiRestClient [] - Sending Request - Method: 'GET', Path: 'https://api.test.com/v1/paydisputes/oi/orders?arn=1363473846864745645&authorizationCode=039319&settlementAmount=12.17&authorizationAmount=12.17&settlementDate=2020-11-16T00:00:00Z&authorizationDate=2020-11-16T00:00:00Z&settlementCurrency=USD&authorizationCurrency=USD&creditCardBin=473702&creditCard4=1788&orderId=14738858272569688&transactionId=234235423523543&sellerId=42201', Headers: '{x-test-api-version=[1.4], Accept=[application/json], Authorization=[Bearer ******], Content-Type=[application/json]}' "
| appendpipe [ eval _raw="Nov 24 16:20:17 ip-11-222-111-100 order-service: [2020-11-24 16:20:17.512] [ERROR] [417863693]: [reactor-http-client-epoll-11] c.v.o.o.OrderGlobalWebExceptionHandler [] - Error occurred while processing web requestorg.springframework.web.server.ResponseStatusException: 404 NOT_FOUND \"Failed to receive order from merchant hosted api with statusCode=404, Not Found\"#012#011at com.verifi.orderservice.order.merchanthosted.service.HostedApiRestClient.handleResponse(HostedApiRestClient.java:68)#012#011at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:153)#012#011at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)#012#011at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)#012#011at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)#012#011at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)#012#011at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76)#012#011a" 
| appendpipe [ eval _raw="Nov 24 16:20:17 ip-11-222-111-100 #011org.springframework.web.server.ResponseStatusException: 404 NOT_FOUND \"Failed to receive order from merchant hosted api with statusCode=404, Not Found\""]]
| rex "(?<time>\w{3} \d\d \S+) (?<ip>\S+)"
| rex "\[(?<status>ERROR|INFO)\]"
| rex "Method: (?<method>\S+),"
| rex "statusCode=(?<status_code>\d+)"
| stats values(method) as method values(status_code) as status_code values(status) as status by ip time

rush job



Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...