transaction commmand with huge event base

[lwalhoefer]

Hi,

I've the following _raw event base:

• line1 field1=field1Value field2=field2Value sometext: a_string

• line2 field1=field1Value field2=field2Value sometext2 val=400

• line3 field1=field1Value field2=field2Value sometext2 val=600

... and like to have the table that only contains events where val reaches a limit. When this limit is reached, I like to see the value behind "sometext: " (=a_string) from the event above with same field1Value and field2Value.

The resulting table should have the cols:

• field1 | field2 | val | msg

A row should have the values:

• field1Value | field1Value | 600 | a_string

Here's my try with the transaction command:

index=myindex "sometext" OR ("sometext2" AND val>500) 
| transaction field1 field2 
| rex field=_raw "sometext: (?<msg>.*)" 
| table field1 field2 val msg

The 2 issues are:

• the msg field is always emtpy and seem to not extracted correctly
• The first part of the query (up to the first pipe symbol) is returning a huge number of events (~200k) and thus the transaction seem takes an unacceptable time. Is the transaction a good way to accomplish such a resulting table? I suppose a "join" is not an option?

Any ideas?

Thanks!

[gkanapathy]

I don't know about your extraction problem, but I might use:

index=myindex "sometext" OR ("sometext2" AND val>500) 
| rex "sometext: (?<msg>.*)"
| stats val,msg by field1,field2

This will run a lot faster if you have the data over multiple indexers, since stats will map-reduce much better than transaction.

You could do this:

index=myindex ("sometext" OR ("sometext2" AND val>500))
[ search index=myindex "sometext2" AND val>500 
  | fields + field1 field2 
  | format maxresults=10000 ] )
| rex "sometext: (?<msg>.*)"
| stats val,msg by field1,field2

but whether that is actually faster depends on your data.