Splunk Search

Why my subsearch is not working?

user9025
Path Finder

I am parsing logs using splunk and there are two types of logs :

1. API endpoint info and user ID

2. Logs which contains specific error that I am interested in.(Lets say error is ERROR_FAIL)

 

I need all logs for a particular user hitting endpoint and getting ERROR_FAIL.

Both the logs have same request id for one instance of api call.

So firstly I want to filter the request ID from point 1, which will give me request id for the api and user I am interested in, and based on that request id ,I wana see all the logs that have failed because of error(ERROR_FAIL).

Now If i use following query ,I get all the request ids for user and API:

index=app-Prod sourcetype=prod-app-logs "api/rest/v1/entity" " 123" | table xrid

 

Now if I add this in sub-search. it does not work:Final query

 

index=app-Prod sourcetype=prod-app-logs  [search index=app-Prod sourcetype=prod-app-logs "api/rest/v1/entity" "123" | table xrid]  "ERROR_FAIL"  |  table xrid

 

This does not return anything.

There are logs where 123 user hits "api/rest/v1/entity" and gets "ERROR_FAIL".How can i make my query correct?

Labels (2)
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

How many events does the subquery process? Do you have any messages in job inspector about the subsearch being truncated?

0 Karma

user9025
Path Finder

Subquery should return lot of event may be 500Kapprox.I dont see data getting truncated.I see success message once query completes.I reduced time stamp, now subquery returns 70k, still not working,

For eg:

index=app-Prod sourcetype=prod-app-logs   "*api/rest/v1/entity*" "987edf3s"

I see following result:

INFO [2022-04-05T05:30:44,457] [app-b2.in.abc.com:ajp-nio-0.0.0.0-8009-exec-56:220405053043920.696] [apid=1234567] [xrid=987edf3s] (ApiLoggingFilter.logTheApiAnalyticsData:421) REST API Usage Tracking Data. REST EndPoint : GET /entity  ; ApiUser : User[12345] ; UserAgent : curl/7.66.0 ; RemoteHost : 123.234.567.89 ; RequestURL : https://server/api/rest/v1/entity

 

Now i try to move this to inner query and I have following query:

 

index=app-Prod sourcetype=prod-app-logs[search iindex=app-Prod sourcetype=prod-app-logs ""*api/rest/v1/entity*" "12345" | table xrid]

 

 

I expect that it should show me logs with xrid=987edf3s But it is not showing.

Time to run both above was last 24 hours. What am I missing

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

If the subquery has too many events (and 500K definitely sounds like too many), the subquery doesn't return the events it is supposed to so the primary query doesn't get filtered.

0 Karma

user9025
Path Finder

As i told in comment, i ran for last 1 day with inly 70k records. Still its not working.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Even 70k is too many - I think the limit might be 50k - try with a smaller set

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...