I am parsing logs using splunk and there are two types of logs :
1. API endpoint info and user ID
2. Logs which contains specific error that I am interested in.(Lets say error is ERROR_FAIL)
I need all logs for a particular user hitting endpoint and getting ERROR_FAIL.
Both the logs have same request id for one instance of api call.
So firstly I want to filter the request ID from point 1, which will give me request id for the api and user I am interested in, and based on that request id ,I wana see all the logs that have failed because of error(ERROR_FAIL).
Now If i use following query ,I get all the request ids for user and API:
index=app-Prod sourcetype=prod-app-logs "api/rest/v1/entity" " 123" | table xrid
Now if I add this in sub-search. it does not work:Final query
index=app-Prod sourcetype=prod-app-logs [search index=app-Prod sourcetype=prod-app-logs "api/rest/v1/entity" "123" | table xrid] "ERROR_FAIL" | table xrid
This does not return anything.
There are logs where 123 user hits "api/rest/v1/entity" and gets "ERROR_FAIL".How can i make my query correct?
How many events does the subquery process? Do you have any messages in job inspector about the subsearch being truncated?
Subquery should return lot of event may be 500Kapprox.I dont see data getting truncated.I see success message once query completes.I reduced time stamp, now subquery returns 70k, still not working,
For eg:
index=app-Prod sourcetype=prod-app-logs "*api/rest/v1/entity*" "987edf3s"
I see following result:
Now i try to move this to inner query and I have following query:
index=app-Prod sourcetype=prod-app-logs[search iindex=app-Prod sourcetype=prod-app-logs ""*api/rest/v1/entity*" "12345" | table xrid]
I expect that it should show me logs with xrid=987edf3s But it is not showing.
Time to run both above was last 24 hours. What am I missing
If the subquery has too many events (and 500K definitely sounds like too many), the subquery doesn't return the events it is supposed to so the primary query doesn't get filtered.
As i told in comment, i ran for last 1 day with inly 70k records. Still its not working.
Even 70k is too many - I think the limit might be 50k - try with a smaller set