Splunk Search

Missing transaction caused by multisearch / subsearch

p4085f9
Engager

Hi all

I have a riddle. Query A and query B does not collect the same events and I don’t understand why.

Query A) results 2 events as transaction

 

| multisearch  
  [search (11111111 OR 22222222) host=x index=y level=z (logger=a "text_a") ]
  [search (11111111 OR 22222222) host=x index=y level=z (logger=b message="text_b") ]
| rex field=_raw "<sg: ID>(?<ID>.*?)<"
| transaction ID keepevicted=false startswith="text_a" endswith=message="text_b"

 

 

Query B) results 1 event as transaction

 

| multisearch  
  [search (11111111 OR 22222222) host=x index=y level=z (logger=a "text_a") ]
  [search                        host=x index=y level=z (logger=b message="text_b") ]
| rex field=_raw "<sg: ID>(?<ID>.*?)<"
| transaction ID keepevicted=false startswith="text_a" endswith=message="text_b"

 

 

11111111 and 22222222 is used as an ID to test the query and to confirm the correctness.

But if I remove these IDs from second search like in query B) than I get only one result, the other is missing.

 

I thought at the first time, it is because of the enormous amount of records. I used a time filter to reduce the records, at the end with 19.351 events. Unfortunately it didn’t help.

Of course, if I replace the multisearch to OR, it works.

Query C) If I move the ID filter in second search, booth events are there.

 

| multisearch  
  [search                        host=x index=y level=z (logger=a "text_a") ]
  [search (11111111 OR 22222222) host=x index=y level=z (logger=b message="text_b") ]
| rex field=_raw "<sg: ID>(?<ID>.*?)<"
| transaction ID keepevicted=false startswith="text_a" endswith=message="text_b"

 

 

Query D) Just to be sure, if I remove the "text_a" and message="text_b" from search the event is still missing.

 

| multisearch  
  [search (11111111 OR 22222222) host=x index=y level=z logger=a ]
  [search                        host=x index=y level=z logger=b ]
| rex field=_raw "<sg:ID>(?<ID>.*?)<"
| transaction ID keepevicted=false startswith="text_a" endswith=message="text_b"

 

 

Maybe someone of you already had similar issues in transaction with multi-search and know, what could cause this problem.

Thank you for your answers.

Best regards,
Robert

Labels (4)
0 Karma
1 Solution

PickleRick
Ultra Champion

Don't use multisearch for something that relies on the events order (like transaction). Since multisearch spawns and runs two searches in parallel and returns interleaved results from both of them as soon as it receives them there is no guarantee that your results will be in any particular order, that they will not get reordered if you change subsearch parameters or even that you always get the results in the same order.

That means that in your case adding/removing parameters causes the subsearch to execute with different timing so your resulting event set is ordered so it can or cannot find a transaction. For example - one run returns events with order text_a, text_b, text_a, text_b. But the other one orders them text_a, text_a, text_b, text_b. So you still have all events but from one ordering you can find two transactions but from the other one - just one.

View solution in original post

p4085f9
Engager

Dear PickleRick,
Thank you very much for your answer.
Best regards,
Robert

0 Karma

PickleRick
Ultra Champion

Don't use multisearch for something that relies on the events order (like transaction). Since multisearch spawns and runs two searches in parallel and returns interleaved results from both of them as soon as it receives them there is no guarantee that your results will be in any particular order, that they will not get reordered if you change subsearch parameters or even that you always get the results in the same order.

That means that in your case adding/removing parameters causes the subsearch to execute with different timing so your resulting event set is ordered so it can or cannot find a transaction. For example - one run returns events with order text_a, text_b, text_a, text_b. But the other one orders them text_a, text_a, text_b, text_b. So you still have all events but from one ordering you can find two transactions but from the other one - just one.

Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...