How to tag or group events by multiple possible values of a field without repeated value?

[yshen]

Here is the data for illustration:

(To facilitate experiment, I provide below the query snippet to recreate the data in Splunk query.)

| makeresults
| eval _raw=
"Date	Time	DEVICE	ATTRIBUTE	STATE	TagExpected
2021-03-19	11:56:22.449	K30	SOR_B_STATUS.STATE	SOR_FAILED	1
2021-03-19	12:16:17.564	K30	SOR_A_STATUS.STATE	SOR_FAILED	1
2021-03-19	12:17:55.191	K30	SOR_RESTRICT_STATUS.STATE	SOR_SPEED_RESTRICT_ON	1
2021-03-19	12:21:16.659	K30	SOR_A_STATUS.STATE	SOR_FAILED	2
2021-03-19	12:32:42.247	K30	SOR_B_STATUS.STATE	SOR_FAILED	2
2021-03-19	12:51:21.456	A60	SOR_RESTRICT_STATUS.STATE	SOR_SPEED_RESTRICT_ON	2
2021-03-19	12:51:52.949	A60	SOR_A_STATUS.STATE	SOR_FAILED	1
2021-03-19	12:54:01.077	A60	SOR_B_STATUS.STATE	SOR_FAILED	1
2021-03-19	13:01:26.367	A60	SOR_RESTRICT_STATUS.STATE	SOR_SPEED_RESTRICT_ON	1
2021-03-19	13:01:26.818	K30	SOR_A_STATUS.STATE	SOR_FAILED	3
2021-03-19	13:02:41.142	K30	SOR_B_STATUS.STATE	SOR_FAILED	3
2021-03-19	13:08:19.694	A60	SOR_RESTRICT_STATUS.STATE	SOR_SPEED_RESTRICT_ON	2
2021-03-19	13:09:14.433	K30	SOR_B_STATUS.STATE	SOR_FAILED	4
2021-03-19	13:10:19.149	W34	SOR_RESTRICT_STATUS.STATE	SOR_SPEED_RESTRICT_OFF	1
2021-03-19	13:16:12.847	A60	SOR_B_STATUS.STATE	SOR_FAILED	3
2021-03-19	13:24:59.420	A60	SOR_A_STATUS.STATE	SOR_FAILED	3
2021-03-19	13:24:59.870	A60	SOR_RESTRICT_STATUS.STATE	SOR_SPEED_RESTRICT_ON	3
2021-03-19	13:25:48.068	A60	SOR_RESTRICT_STATUS.STATE	SOR_SPEED_RESTRICT_OFF
2021-03-19	13:35:47.614	A60	SOR_A_STATUS.STATE	SOR_FAILED	4
2021-03-19	13:38:19.632	A90	SOR_B_STATUS.STATE	SOR_FAILED	1
2021-03-19	13:46:10.118	R20	SOR_B_STATUS.STATE	SOR_FAILED	1
2021-03-19	13:50:30.328	R50	SOR_A_STATUS.STATE	SOR_FAILED	1
2021-03-19	13:54:58.831	W20	SOR_RESTRICT_STATUS.STATE	SOR_SPEED_RESTRICT_ON	1
2021-03-19	13:55:30.622	W20	SOR_RESTRICT_STATUS.STATE	SOR_SPEED_RESTRICT_OFF
2021-03-19	13:56:38.060	A60	SOR_A_STATUS.STATE	SOR_FAILED	5
2021-03-19	14:02:19.102	K30	SOR_B_STATUS.STATE	SOR_FAILED	5
2021-03-19	14:08:51.212	R50	SOR_A_STATUS.STATE	SOR_FAILED	2
2021-03-19	14:09:47.657	R20	SOR_B_STATUS.STATE	SOR_FAILED	2
2021-03-19	14:11:10.387	C30	SOR_B_STATUS.STATE	SOR_FAILED	1
2021-03-19	15:01:15.315	C30	SOR_B_STATUS.STATE	SOR_FAILED	2
2021-03-19	15:02:33.670	R65	SOR_A_STATUS.STATE	SOR_FAILED	1
2021-03-19	15:06:56.258	C50	SOR_B_STATUS.STATE	SOR_FAILED	1
2021-03-19	15:09:32.583	R50	SOR_A_STATUS.STATE	SOR_FAILED	3
2021-03-19	15:09:33.484	R50	SOR_RESTRICT_STATUS.STATE	SOR_SPEED_RESTRICT_ON	3
2021-03-19	15:09:40.240	R50	SOR_RESTRICT_STATUS.STATE	SOR_SPEED_RESTRICT_OFF
2021-03-19	15:36:17.104	A90	SOR_B_STATUS.STATE	SOR_FAILED	1" 
| multikv forceheader=1 
| eval combined=Date." ".Time 
| eval _time=strptime('combined', "%Y-%m-%d %H:%M:%S.%Q") 
| fields _time DEVICE ATTRIBUTE STATE TagExpected

In the above records, I’d like to tag or group the events by the following rules: For a device value, e.g. K30, I’d like to tag the events with DEVICE value K30 with first occurrences of SOR_B_STATUS.STATE or SOR_A_STATUS.STATE (their occurrences do not matter) and SOR_RESTRICT_STATUS.STATE after with the same tag value, to be one group.

For the same DEVICE value, next occurrences of SOR_B_STATUS.STATE or SOR_A_STATUS.STATE (their occurrences do not matter) and SOR_RESTRICT_STATUS.STATE after I’d group them to be tag 2, to be a different group, etc.

Subset of such event group will also be considered to be separate group.

The events in a group does not need to be adjacent in time.

I don’t insist on the tag value, any mechanism to group the events would be fine.

I feel that using streamstats or transaction in some fashion might solve the problem. But I need help to wrap around my brain to work it help.

Really appreciate that you could give me some help or hint.

Thanks in advance, and have nice weekends!

[gcusello]

Hi @yshen,

your requirement is very complex and I hope to have full understood.

Anyway, you have to extract fields and then use the transaction command, something like this:

index=your_index
| rex "^\d+-\d+-\d+\s+\d+:\d+:\d+\.\d+\s+(?<DEVICE>\w+)\s+(?<ATTRIBUTE>[^ ]+)\s+(?<STATE>\w+)\s"
| transaction DEVICE startswith="SOR_A_STATUS.STATE OR SOR_B_STATUS.STATE" endswith="SOR_RESTRICT_STATUS.STATE"
| ...

You can check the regex at https://regex101.com/r/8shToG/1

Ciao.

Giuseppe

[yshen]

@gcusello Thanks for the pointer. 

My problem is a real one that I need for some field troubleshooting. Maybe, my framing of my problem could be simplified. But it's what I can do at the moment. Maybe, once I know the solution, then the problem statement might be simplified. 

The extraction of the fields of DEVICE, ATTRIBUTE, etc. has already been done in the respective sourcetype.

Your suggestion of using transaction with startwith and endwith might work sometimes. 

But I eventually need to identify those event groups that have subsets of the expected events. For example, 

Certain group may only has event of 

"SOR_RESTRICT_STATUS.STATE"

 so the transaciton may need to startwith 

"SOR_RESTRICT_STATUS.STATE"

as well. 

Likewise, some other subset may only have 

SOR_A_STATUS.STATE OR SOR_B_STATUS.STATE

so the transaction may need also to endwith 

SOR_A_STATUS.STATE OR SOR_B_STATUS.STATE

by this school of thought, the eventual transaction definition would be like:

 transaction DEVICE startswith="SOR_A_STATUS.STATE OR SOR_B_STATUS.STATE OR SOR_RESTRICT_STATUS.STATE" endswith="SOR_A_STATUS.STATE OR SOR_B_STATUS.STATE OR SOR_RESTRICT_STATUS.STATE"

I have tried how it would up with the same condition for startwith and endwith

I am concerned that it would lose the expected differentiation for the transactions. 

Maybe a question to experts.