Splunk Search

Missing transaction caused by multisearch / subsearch

p4085f9
Engager

Hi all

I have a riddle. Query A and query B does not collect the same events and I don’t understand why.

Query A) results 2 events as transaction

 

| multisearch  
  [search (11111111 OR 22222222) host=x index=y level=z (logger=a "text_a") ]
  [search (11111111 OR 22222222) host=x index=y level=z (logger=b message="text_b") ]
| rex field=_raw "<sg: ID>(?<ID>.*?)<"
| transaction ID keepevicted=false startswith="text_a" endswith=message="text_b"

 

 

Query B) results 1 event as transaction

 

| multisearch  
  [search (11111111 OR 22222222) host=x index=y level=z (logger=a "text_a") ]
  [search                        host=x index=y level=z (logger=b message="text_b") ]
| rex field=_raw "<sg: ID>(?<ID>.*?)<"
| transaction ID keepevicted=false startswith="text_a" endswith=message="text_b"

 

 

11111111 and 22222222 is used as an ID to test the query and to confirm the correctness.

But if I remove these IDs from second search like in query B) than I get only one result, the other is missing.

 

I thought at the first time, it is because of the enormous amount of records. I used a time filter to reduce the records, at the end with 19.351 events. Unfortunately it didn’t help.

Of course, if I replace the multisearch to OR, it works.

Query C) If I move the ID filter in second search, booth events are there.

 

| multisearch  
  [search                        host=x index=y level=z (logger=a "text_a") ]
  [search (11111111 OR 22222222) host=x index=y level=z (logger=b message="text_b") ]
| rex field=_raw "<sg: ID>(?<ID>.*?)<"
| transaction ID keepevicted=false startswith="text_a" endswith=message="text_b"

 

 

Query D) Just to be sure, if I remove the "text_a" and message="text_b" from search the event is still missing.

 

| multisearch  
  [search (11111111 OR 22222222) host=x index=y level=z logger=a ]
  [search                        host=x index=y level=z logger=b ]
| rex field=_raw "<sg:ID>(?<ID>.*?)<"
| transaction ID keepevicted=false startswith="text_a" endswith=message="text_b"

 

 

Maybe someone of you already had similar issues in transaction with multi-search and know, what could cause this problem.

Thank you for your answers.

Best regards,
Robert

Labels (4)
0 Karma
1 Solution

PickleRick
SplunkTrust
SplunkTrust

Don't use multisearch for something that relies on the events order (like transaction). Since multisearch spawns and runs two searches in parallel and returns interleaved results from both of them as soon as it receives them there is no guarantee that your results will be in any particular order, that they will not get reordered if you change subsearch parameters or even that you always get the results in the same order.

That means that in your case adding/removing parameters causes the subsearch to execute with different timing so your resulting event set is ordered so it can or cannot find a transaction. For example - one run returns events with order text_a, text_b, text_a, text_b. But the other one orders them text_a, text_a, text_b, text_b. So you still have all events but from one ordering you can find two transactions but from the other one - just one.

View solution in original post

p4085f9
Engager

Dear PickleRick,
Thank you very much for your answer.
Best regards,
Robert

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Don't use multisearch for something that relies on the events order (like transaction). Since multisearch spawns and runs two searches in parallel and returns interleaved results from both of them as soon as it receives them there is no guarantee that your results will be in any particular order, that they will not get reordered if you change subsearch parameters or even that you always get the results in the same order.

That means that in your case adding/removing parameters causes the subsearch to execute with different timing so your resulting event set is ordered so it can or cannot find a transaction. For example - one run returns events with order text_a, text_b, text_a, text_b. But the other one orders them text_a, text_a, text_b, text_b. So you still have all events but from one ordering you can find two transactions but from the other one - just one.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...