topic Re: How to merge consecutive event values from a single field? in Splunk Search

How to merge consecutive event values from a single field?

sureshmurgan — Thu, 20 Jun 2019 11:48:16 GMT

This is the requirement.
We are collecting a log file that has the following events (along with others)in the same field named ADSite_Membership, these two lines occurs consecutively in the same order as I have given below:

Initiating task ContentDownload for CI ScopeId_B4419992-1C14-4FC6-AB4E-D7730CD14853/DeploymentType_4a3a8593-e333-4bb4-a3f9-bb1ce19cf0eb.6 (BlueDom - User Interactive) for target: , consumer: {719C1F1E-63E2-46C6-9469-E38B881EEA5A}
Unable to get locations, no need to continue with download

I am trying to extract two separate fields from the above values but its not working, Here's the rex command I am using:

|rex field=ADSite_Membership "(?ms)Initiating task ContentDownload\s\w+\s\w+\s(?P<Deployment>ScopeId.*)\sfor.*(?P<Status>Initiating content download.*)"

(Note: Willard value 'star' is not showing up in some sections of the rex command above)

I am intending to extract :

ScopeId_B4419992-1C14-4FC6-AB4E-D7730CD14853/DeploymentType_4a3a8593-e333-4bb4-a3f9-bb1ce19cf0eb.6 (BlueDom - User Interactive) as Deployment

Unable to get locations, no need to continue with a download as Status.

How can I do that?

Re: How to merge consecutive event values from a single field?

gcusello — Thu, 20 Jun 2019 12:27:06 GMT

Hi sureshmurgan,
try this regex

(?ms)Initiating task ContentDownload\s\w+\s\w+\s(?P<Deployment>ScopeId.*)\sfor.*\}\s*(?P<Status>.*)

or using the rex command

| rex field=ADSite_Membership "(?ms)Initiating task ContentDownload\s\w+\s\w+\s(?P<Deployment>ScopeId.*)\sfor.*\}\s*(?P<Status>.*)"

You can test it at https://regex101.com/r/AUKmB6/1

Bye.
Giuseppe

Re: How to merge consecutive event values from a single field?

sureshmurgan — Thu, 20 Jun 2019 13:04:33 GMT

Hi Giuseppe,
I get to see Deployment field now but Status is still blank. I have tried both rex commands you had shared.

Please note there could be many events in the field and we need to stop extracting only the two consecutive lines and should not go beyond that.

Re: How to merge consecutive event values from a single field?

gcusello — Thu, 20 Jun 2019 13:13:52 GMT

Hi sureshmurgan,
Let me understand: Status is always blank or sometimes?
If it's always blank, please share other examples to tune your regex.

If instead, it's sometimes blank and you want to extract fields only when both the fields contain values, you could put a filter after the rex command:

| search Deployment=* Status=*

Bye.
Giuseppe

Re: How to merge consecutive event values from a single field?

sureshmurgan — Wed, 30 Sep 2020 00:58:50 GMT

Hi Giuseppe, Status is not showing any value at all. I have given explained it further with a sample. Please have a look and let me know if you need more information. Thanks for your help with this!

I am searching events with the following values in the field ADSite_Membership,

ADSite_Membership="Initiating task ContentDownload for CI*" OR ADSite_Membership="Initiating Content Download*" OR ADSite_Membership="Unable to get locations*"

If I pull the events as is, I get the below results,

2019-06-19 12:36:15.000 UK922061 Initiating task ContentDownload for CI ScopeId_B4419992-1C14-4FC6-AB4E-D7730CD14853/DeploymentType_4a3a8593-e333-4bb4-a3f9-bb1ce19cf0eb.6 (GP4.1.11 BlueCoatDecom - User Interactive) for target: , consumer: {CA8EDD4A-6D29-4CE0-90AD-0EB22011B165}

2019-06-19 12:36:15.000 UK922061 Initiating task ContentDownload for CI ScopeId_B4419992-1C14-4FC6-AB4E-D7730CD14853/DeploymentType_28ce7e7a-307a-417b-baf1-70c6bf0889d9.10 (Intel GFX driver 6th 7th 8th Apollo Gemini (EUDM - ALL)) for target: , consumer: {FE0C1DDA-725A-4FF2-9E20-D78968FAC529}

2019-06-20 12:58:02.414 UK922061 Unable to get locations, no need to continue with download

2019-06-20 13:07:52.000 UK922061 Initiating task ContentDownload for CI ScopeId_B4419992-1C14-4FC6-AB4E-D7730CD14853/DeploymentType_28ce7e7a-307a-417b-baf1-70c6bf0889d9.10 (Intel GFX driver 6th 7th 8th Apollo Gemini (EUDM - ALL)) for target: , consumer: {546E275F-FE56-4EA8-A37B-41508E57148D}

Requirement : If you look at the third row result, I have "Unable to get locations*" . When we find this result, I want to extract it as Status and want to extract the ScopeId_* from the previous event as Deployment.

Re: How to merge consecutive event values from a single field?

sureshmurgan — Wed, 30 Sep 2020 00:58:52 GMT

Here's little more information about the query and the result and what I want.

Search :
ADSite_Membership="Initiating task ContentDownload for CI*" OR ADSite_Membership="Initiating Content Download*" OR ADSite_Membership="Unable to get locations*"
|table _time ADSite_Membership

Search Result:
2019-06-19 12:36:15.000 Initiating task ContentDownload for CI ScopeId_B4419992-1C14-4FC6-AB4E-D7730CD14853/DeploymentType_4a3a8593-e333-4bb4-a3f9-bb1ce19cf0eb.6 (GP4.1.11 BlueCoatDecom - User Interactive) for target: , consumer: {CA8EDD4A-6D29-4CE0-90AD-0EB22011B165}

2019-06-19 12:36:15.000 Initiating task ContentDownload for CI ScopeId_B4419992-1C14-4FC6-AB4E-D7730CD14853/DeploymentType_28ce7e7a-307a-417b-baf1-70c6bf0889d9.10 (Intel GFX driver 6th 7th 8th Apollo Gemini (EUDM - ALL)) for target: , consumer: {FE0C1DDA-725A-4FF2-9E20-D78968FAC529}

2019-06-20 12:58:02.414 Unable to get locations, no need to continue with download

2019-06-20 13:07:52.000 Initiating task ContentDownload for CI ScopeId_B4419992-1C14-4FC6-AB4E-D7730CD14853/DeploymentType_28ce7e7a-307a-417b-baf1-70c6bf0889d9.10 (Intel GFX driver 6th 7th 8th Apollo Gemini (EUDM - ALL)) for target: , consumer: {546E275F-FE56-4EA8-A37B-41508E57148D}

Requirement : If you look at the 3rd row, it has "Unable to get locations*". When we get it in results I want to extract that as Status and also I want to extract the value from previous event "ScopeId*" as Deployment

Re: How to merge consecutive event values from a single field?

gcusello — Fri, 21 Jun 2019 12:24:24 GMT

Hi sureshmurgan,
Let me understand: you have fields in different logs: in some logs you have Deployment and in other logs you have Status, you never have both the fields in the same log, is it correct?

In this case, try to use two different regexes, one for each field:

(?ms)Initiating task ContentDownload\s\w+\s\w+\s(?P<Deployment>ScopeId[^}]*)
^\d*-\d*-\d*\s*\d*:\d*:\d*\.\d*\s\w*\s*(?P<Status>Unable.*)

If you have more Status Messages, you could put in the regex the initial words in OR.

Bye.
Giuseppe

Re: How to merge consecutive event values from a single field?

gcusello — Fri, 21 Jun 2019 12:32:41 GMT

Hi sureshmurgan,
try this one

Initiating task ContentDownload\s\w+\s\w+\s(?P<Deployment>ScopeId[^}]*)\}\s+\d*-\d*-\d*\s*\d*:\d*:\d*\.\d*\s\w*\s*(?P<Status>Unable.*)

You can test it at https://regex101.com/r/m7xR0g/1

Bye.
Giuseppe

Re: How to merge consecutive event values from a single field?

wenthold — Fri, 28 Jun 2019 23:07:21 GMT

The only way I can think of to pull this off is to use streamstats to merge to events (since you said they are consecutive) then stats to remove duplicate events. I noticed the example you provided was in time ascending order, but this was built assuming the results are in time descending order:

 ADSite_Membership="Initiating task ContentDownload for CI*" OR ADSite_Membership="Initiating Content Download*" OR ADSite_Membership="Unable to get locations*"
| table _time ADSite_Membership
| streamstats current=f window=1 last(ADSite_Membership) as prev_ADSite_Membership `comment("Use the ADSite_Membership field from the previous event")`
| eval status=case(like(prev_ADSite_Membership, "%Unable to get locations%"),prev_ADSite_Membership,1==1,null) `comment("Identify the status events and set the status field")`
| rex field=ADSite_Membership "Initiating task ContentDownload for CI (?<Deployment>ScopeId_.*?) for target" `comment("Extract the Deployment field")`
| search Deployment=*  `comment("Drop the source status events which will not have a Deployment field defined")`
| fields - prev_ADSite_Membership  `comment("Drop prev_ADSite_Membership from the fields list")`