topic How can I search event based on other event value and combine the results? in Splunk Search

How can I search event based on other event value and combine the results?

noammeir — Wed, 19 Oct 2022 14:41:13 GMT

our system logs test runs as single events. in some cases we would have a re-run of a test. both events are logically related but are separate for each run (the original run and the re-run).

I wish to extract data from both events and present it together, have tried several approaches but none worked so far.

step 1:

identifying the re-run event and getting a unique identifier for the original run using some textual parsing on the workarea path:

index=my_index aa_data_source="my_info" is_rerun=True | eval orig_workarea=workarea_path | rex field=orig_workarea mode=sed "s@/rerun?$@@"

step 2:
now, I would like to find and match the original run event for each of the results.

tried map:

| map search="search index=my_index aa_data_source=my_info workarea_path=$orig_workarea$ " maxsearches=100000

this is probably wrong because it is both resource expensive and after I found the original event per result, I could only use the data of the original event (result of map) - didnt find how to combine it with the re-run event data I searched upon.

I also tried subsearch in various ways, the main problem is that the subsearch cannot use the "orig_workarea" I extract from the primary search because it runs first.

step 3 would be present the results from both events together. meaning - take field_from_eventA, field_from_eventB and place them in the same raw (note that renaming might be required for the fields since both events have the same fields)

kind of in a dead end here, could use ideas on how to implement this search. any ideas are welcome

thanks,

noam

Re: searching event based on other event value and combining the results

noammeir — Wed, 19 Oct 2022 14:44:59 GMT

forgot to mention - I might be able to use transaction or stats based on the orig_workarea value since it will be the same path with addition. however - at the point I evaluated orig_workarea, I can no longer pipe to transaction since I already eliminated some of the events in the process - the one I need included.

moreover - even if I was able to use transaction - not sure how I extract the data from the fields of the two events I'd have in it once they are grouped (again, assume renaming might be required since both events have the same fields)

Re: How can I search event based on other event value and combine the results?

ITWhisperer — Wed, 19 Oct 2022 14:48:01 GMT

Try something like this to gather the field values from related events

index=my_index aa_data_source="my_info" is_rerun=True | eval orig_workarea=workarea_path | rex field=orig_workarea mode=sed "s@/rerun?$@@" | stats list(*) as * by orig_workarea

Re: How can I search event based on other event value and combine the results?

noammeir — Wed, 19 Oct 2022 18:59:28 GMT

thanks for the prompt reply!

almost there...

removing some of the conditions (is_rerun=True in this simplified case) indeed grouped the relevant events together.

so now I have some cases where there is a single event (no re-run exists) and some where there are two (re-run exists)

1) how do you suggest to filter out only the ones with two events? moreover - filter according to some conditions on the fields of one of the events (for example field_A_in_rerun != field_B_in_rerun)

2) once I filtered the relevant events, how can I access the specific event fields in each of them?

use case example:

I want to show a table of rerun_event_field_A, original_run_field_A, some_more_fields - in cases where field_A != field_B in rerun event

thanks,

Noam

Re: How can I search event based on other event value and combine the results?

ITWhisperer — Thu, 20 Oct 2022 10:55:00 GMT

workarea_path will be a multi-value field with values from all the events so you could find just those with only two events by doing this

| where mvcount(workarea_path) = 2

By using list() (instead of values()) in the stats command you maintain all the values and they maintain correlation with each other i.e. the first value in all the lists are from the same event, as are all the second values, etc. You can reference these with mvindex

| eval first_path=mvindex(workarea_path,0) | eval second_path=mvindex(workarea_path,1)

Re: How can I search event based on other event value and combine the results?

noammeir — Sun, 23 Oct 2022 10:16:57 GMT

thanks for the suggestion!

it works however I chose a different approach in this specific case - filtering the data I need before reaching the stats command:

index=my_index aa_data_source="my_info" is_rerun=True
| where fieldA!=fieldB ```filtering out cases```
| eval orig_some_field= if(is_rerun=="False", some_field, NULL) ```saving data to show only in original```

| eval rerun_some_field= if(is_rerun=="True", some_field, NULL) ```saving data to show only in re-run```

| eval orig_workarea=workarea_path

| rex field=orig_workarea mode=sed "s@/rerun?$@@"

| stats count(aa_data_source) as event_count list(*) as * by orig_workarea ```using stats to group the events together```
| where event_count==2 ```additional filtering after stats```
|table orig_some_field rerun_some_field orig_workarea