Extend search results data by correlation-id (and exclude on other messages)

[sscholl]

Hello, I have 500 HTTP messages in my access log. Also I have corresponding events from other log sources with the same correlation-id. Now I want to join the information to enhance the results.

Access Log Events:

2024-04-25T11:00:26+00:00 [info] type=access  status=500 xCorrelationId=90e2a321-f522-466f-9ffa-72cbdaa1a576 ....

2024-04-25T10:15:25+00:00 [info] type=access status=500 xCorrelationId=9b1833f5-776b-44c3-92d7-d603abdfecf8 ...

Other Events:

2024-04-25T10:15:24+00:00 xCorrelationId=9b1833f5-776b-44c3-92d7-d603abdfecf8  NoHandlerFoundException: No endpoint GET

My actual intention is, to exclude the results from main search, if there is another event with the same correlation-id but containing specific exceptions like "NoHandlerFoundException". That means, i need a search per result from the main search.

Do you know a solution for this?

Thanks!

[ITWhisperer]

<main index> NOT [search <other source> NoHandlerFoundException | stats count by xCorrelationId | fields xCorrelationId | format]

However, depending on how may exceptions you have, you may run into limitations as the sub-search with the format command will essentially return a long string which might be too large to be parsed in the main search.

Another way to do it is to search both sources, and correlate by xCorrelationId and exclude those xCorrelationId's which have the exception, but this still means you are retrieving both full sets of events and correlating them before you can filter out any.

[bowesmana]

The number of values in the subsearch cannot be too large as it will perform really badly, but one slight change to @ITWhisperer subsearch is to do

<main index> NOT [
    search <other source> NoHandlerFoundException 
    | stats values(xCorrelationId) as search
    | format]

which will perform faster. It will change the outer search from

<main index> NOT ( ( ( xCorrelationId=A OR xCorrelationId=B OR... ) ) )

to

<main index> NOT ( ( ( A OR B OR C OR D ... ) ) )

where A, B etc are the values or xCorrelationId

The key point is having a field name 'search' in the output rather than xCorrelationId, which changes the effect of the format command.