where in subsearch

[bowa]

sourcetype="log4j" source="*server*"
| rex field=_raw "nonce created : (?<nonce>[0-9a-z-]*)"
| transaction thread startswith="startTx" endswith="closeTx"

This first search gives transactions that have a field 'nonce' ... now i want to filter out only the transactions with a specific nonce ... i can find the nonces i am interested in with this query:

sourcetype="log4j" failed nonce
| rex field=_raw "Failed with nonce (?<nonce>[0-9a-z-]*)"
| fields + nonce

So i would like to do some sort of

| where nonce in [search {search2}] 

What is the correct syntax to do such a thing.

Do mind that this loglines that are in search2 are not part of the transaction in the first search, so i cant just filter the transactions more based on their own contence.

[David · Accepted solution]

The right way to do it is to first have the nonce extracted in your props.conf (this simplifies the rest), such as:

[YourSourcetype]
EXTRACT-nonce-failure = Failed with nonce (?<nonce>[0-9a-z-]*)
EXTRACT-nonce-created = nonce created : (?<nonce>[0-9a-z-]*)

You can then do a subsearch first for the failure nonces, and send that to the main search:

sourcetype="log4j" source="*server*" 
   | transaction thread startswith="startTx" endswith="closeTx"
   | search [search sourcetype="log4j" failed nonce | fields nonce]

I believe that should work, at least. What is more efficient, though, assuming that the nonce failure messages also reference the thread field, is to first isolate the failure threads, and then search for the transactions where they're there:

[search sourcetype="log4j" failed nonce | fields thread] 
sourcetype="log4j" source="*server*" 
   | transaction thread startswith="startTx" endswith="closeTx"

That should give you a shortcut to a transaction that includes only the threads that have failed nonces. One bit of minutia is that depending on the number of threads that are re-used, you might need to re-run the limiting search at the end:

[search sourcetype="log4j" failed nonce | fields thread] 
sourcetype="log4j" source="*server*" 
   | transaction thread startswith="startTx" endswith="closeTx"
   | search "Failed with nonce"

The reason why it's more efficient to limit what hits the transaction command is that transaction is very slow.

Let me know if that works for you.

[Stephen_Sorkin · Accepted solution]

The output of a subsearch is a valid search expression that will match an event when it matches all the fields of any of the rows of the subsearch. So, if your subsearch only emits a single field, nonce, then it will yield a search expression like: nonce=row_1_nonce OR nonce=row_2_nonce OR ....

With this you can compose your search like:

sourcetype="log4j" source="*server*"
| rex field=_raw "nonce created : (?<nonce>[0-9a-z-]*)"
| transaction thread startswith="startTx" endswith="closeTx"
| search [search sourcetype="log4j" failed nonce
         | rex field=_raw "Failed with nonce (?<nonce>[0-9a-z-]*)"
         | dedup nonce
         | fields nonce]

[bowa]

Perfect ! exactly what i needed. Thanks for explaining how this subsearch works like the OR matches.

[David · Accepted solution]

The right way to do it is to first have the nonce extracted in your props.conf (this simplifies the rest), such as:

[YourSourcetype]
EXTRACT-nonce-failure = Failed with nonce (?<nonce>[0-9a-z-]*)
EXTRACT-nonce-created = nonce created : (?<nonce>[0-9a-z-]*)

You can then do a subsearch first for the failure nonces, and send that to the main search:

sourcetype="log4j" source="*server*" 
   | transaction thread startswith="startTx" endswith="closeTx"
   | search [search sourcetype="log4j" failed nonce | fields nonce]

I believe that should work, at least. What is more efficient, though, assuming that the nonce failure messages also reference the thread field, is to first isolate the failure threads, and then search for the transactions where they're there:

[search sourcetype="log4j" failed nonce | fields thread] 
sourcetype="log4j" source="*server*" 
   | transaction thread startswith="startTx" endswith="closeTx"

That should give you a shortcut to a transaction that includes only the threads that have failed nonces. One bit of minutia is that depending on the number of threads that are re-used, you might need to re-run the limiting search at the end:

[search sourcetype="log4j" failed nonce | fields thread] 
sourcetype="log4j" source="*server*" 
   | transaction thread startswith="startTx" endswith="closeTx"
   | search "Failed with nonce"

The reason why it's more efficient to limit what hits the transaction command is that transaction is very slow.

Let me know if that works for you.

[bowa]

no the thread variable is different for the failed nonce and the creation of it.

i know it won't be the most performant search, but in this case the result is more important than the timings

[stanwin]

Will the EXTRACT strategy work for multi row output like top command which necessitates multikv?

Specifically there is custom multikv parsing logic , but extracted fields need to be referenced in search query where the EXTRACT will help..

index=main   sourcetype=JOBI    [ search index=main   sourcetype=JOBI |  tail 1 |   multikv conf=JOBI_mkv
| eval JOB  =   Subs.".".User.".".Number | return JOB ]  
| head 1
|   multikv conf=JOBI_mkv
| eval JOB  =   Subs.".".User.".".Number 

eval JOB = Subs.".".User.".".Number

concatenates 3 fields to create a primary key.