topic Re: Why my subsearch is not working? in Splunk Search

Why my subsearch is not working?

user9025 — Tue, 05 Apr 2022 13:51:41 GMT

I am parsing logs using splunk and there are two types of logs :

1. API endpoint info and user ID

2. Logs which contains specific error that I am interested in.(Lets say error is ERROR_FAIL)

I need all logs for a particular user hitting endpoint and getting ERROR_FAIL.

Both the logs have same request id for one instance of api call.

So firstly I want to filter the request ID from point 1, which will give me request id for the api and user I am interested in, and based on that request id ,I wana see all the logs that have failed because of error(ERROR_FAIL).

Now If i use following query ,I get all the request ids for user and API:

index=app-Prod sourcetype=prod-app-logs "api/rest/v1/entity" " 123" | table xrid

Now if I add this in sub-search. it does not work:Final query

index=app-Prod sourcetype=prod-app-logs [search index=app-Prod sourcetype=prod-app-logs "api/rest/v1/entity" "123" | table xrid] "ERROR_FAIL" | table xrid

This does not return anything.

There are logs where 123 user hits "api/rest/v1/entity" and gets "ERROR_FAIL".How can i make my query correct?

Re: Why my subsearch is not working?

ITWhisperer — Tue, 05 Apr 2022 13:59:52 GMT

How many events does the subquery process? Do you have any messages in job inspector about the subsearch being truncated?

Re: Why my subsearch is not working?

user9025 — Tue, 05 Apr 2022 15:27:13 GMT

Subquery should return lot of event may be 500Kapprox.I dont see data getting truncated.I see success message once query completes.I reduced time stamp, now subquery returns 70k, still not working,

For eg:

index=app-Prod sourcetype=prod-app-logs "*api/rest/v1/entity*" "987edf3s"

I see following result:

INFO [2022-04-05T05:30:44,457] [app-b2.in.abc.com:ajp-nio-0.0.0.0-8009-exec-56:220405053043920.696] [apid=1234567] [xrid=987edf3s] (ApiLoggingFilter.logTheApiAnalyticsData:421) REST API Usage Tracking Data. REST EndPoint : GET /entity ; ApiUser : User[12345] ; UserAgent : curl/7.66.0 ; RemoteHost : 123.234.567.89 ; RequestURL : https://server/api/rest/v1/entity

Now i try to move this to inner query and I have following query:

index=app-Prod sourcetype=prod-app-logs[search iindex=app-Prod sourcetype=prod-app-logs ""*api/rest/v1/entity*" "12345" | table xrid]

I expect that it should show me logs with xrid=987edf3s But it is not showing.

Time to run both above was last 24 hours. What am I missing

Re: Why my subsearch is not working?

ITWhisperer — Tue, 05 Apr 2022 15:45:35 GMT

If the subquery has too many events (and 500K definitely sounds like too many), the subquery doesn't return the events it is supposed to so the primary query doesn't get filtered.

Re: Why my subsearch is not working?

user9025 — Tue, 05 Apr 2022 15:46:42 GMT

As i told in comment, i ran for last 1 day with inly 70k records. Still its not working.

Re: Why my subsearch is not working?

ITWhisperer — Tue, 05 Apr 2022 15:50:42 GMT

Even 70k is too many - I think the limit might be 50k - try with a smaller set