topic Re: Dynamically Extract Fields at Search-Time in Getting Data In

Dynamically Extract Fields at Search-Time

nateloepker — Mon, 20 May 2024 20:42:47 GMT

Hello,

I'm trying to dynamically set some extractions to save myself time and effort from writing hundreds of extractions.

In my orgs IdAM solution, we have hundreds of various user claims.

ie)

Data={"Claims":{"http://wso2.org/claims/user":"username","http://wso2.org/claims/role":"user_role",...etc}

I would like to set up a single extraction that will extract all of these claims.

My idea was the following

props.conf EXTRACT-nrl_test = MatchAllClaims transforms.conf [MatchAllClaims] FORMAT = user_$1::$2 REGEX = \"http:\/\/wso2.org\/claims\/(\w+)\":\"([^\"]+) MV_ADD = true

I was hoping this would extract the field dynamically, but it did not work.

is there a way to accomplish this with one extraction?

Thank you

Re: Dynamically Extract Fields at Search-Time

richgalloway — Tue, 21 May 2024 00:23:37 GMT

EXTRACT props do not invoke a transform. Use REPORT, instead.

Re: Dynamically Extract Fields at Search-Time

gcusello — Tue, 21 May 2024 05:36:49 GMT

Hi @nateloepker ,

your data seems to have a json format, did you tried using INDEXED_EXTRACTIONS = json in your sourcetype definition oer the spath command (https://docs.splunk.com/Documentation/Splunk/9.2.1/SearchReference/Spath)?

Ciao.

Giuseppe

Re: Dynamically Extract Fields at Search-Time

nateloepker — Tue, 21 May 2024 17:46:06 GMT

hey @gcusello ,

JSON extractions will not work for this. The full event is not in JSON, only the data portion

example event (without all of the claims)...

TID: [-1234] [2024-05-21 17:40:35,777] [asdf-asdf-asdf-asdf-asdf ] INFO {AUDIT_LOG} - Initiator=initiator Action=action Target=target Data= {"Claims":{ "http://wso2.org/claims/username":"user", "http://wso2.org/claims/role":"role"}}

Re: Dynamically Extract Fields at Search-Time

nateloepker — Tue, 21 May 2024 17:47:11 GMT

Thank you, good catch. Not sure how I messed that up.

I have corrected that error now, but I still do not get multiple fields extracted from this single extraction.