Dynamically Extract Fields at Search-Time

[nateloepker]

Hello,

I'm trying to dynamically set some extractions to save myself time and effort from writing hundreds of extractions.

In my orgs IdAM solution, we have hundreds of various user claims.

ie)

 Data={"Claims":{"http://wso2.org/claims/user":"username","http://wso2.org/claims/role":"user_role",...etc}

I would like to set up a single extraction that will extract all of these claims.

My idea was the following

props.conf

EXTRACT-nrl_test = MatchAllClaims

transforms.conf

[MatchAllClaims]
FORMAT = user_$1::$2
REGEX = \"http:\/\/wso2.org\/claims\/(\w+)\":\"([^\"]+)
MV_ADD = true

I was hoping this would extract the field dynamically, but it did not work.

is there a way to accomplish this with one extraction?

Thank you

[gcusello]

Hi @nateloepker ,

your data seems to have a json format, did you tried using INDEXED_EXTRACTIONS = json in your sourcetype definition oer the spath command (https://docs.splunk.com/Documentation/Splunk/9.2.1/SearchReference/Spath)?

Ciao.

Giuseppe

[nateloepker]

hey @gcusello ,

JSON extractions will not work for this. The full event is not in JSON, only the data portion

example event (without all of the claims)...

TID: [-1234]  [2024-05-21 17:40:35,777] [asdf-asdf-asdf-asdf-asdf ] INFO {AUDIT_LOG} - Initiator=initiator Action=action Target=target Data= {"Claims":{ "http://wso2.org/claims/username":"user", "http://wso2.org/claims/role":"role"}}

[richgalloway]

EXTRACT props do not invoke a transform.  Use REPORT, instead.

[nateloepker]

Thank you, good catch. Not sure how I messed that up.

I have corrected that error now, but I still do not get multiple fields extracted from this single extraction.