topic Re: Need some help with FIELD_HEADER_REGEX and json data in Getting Data In

Need some help with FIELD_HEADER_REGEX and json data

korstiaans — Tue, 19 Jul 2022 09:42:06 GMT

Hi Splunkers,

I have a question related to a json file that I'm trying to parse.I want to remove the first part of it until {"kind"), see sample file is added below.

I tried using the FIELD_REGEX_HEADER in props.conf which I think is supposed to that so far I've tried an failed with the following:

FIELD_HEADER_REGEX={"activities":\s\[(.)
FIELD_HEADER_REGEX={"activities":\s\[
FIELD_HEADER_REGEX={"activities":
FIELD_HEADER_REGEX=\{\"activities\"\:

Some of the above work on regexr.com with the sample data.

{"activities": [{"kind": "admin#reports#activity", "id": {"time": "2022-07-18T14:04:19.866Z", "uniqueQualifier": "-2451221827967636314", "applicationName": "redacted", "customerId": "redacted"}, "etag": "\"dng2uCItaXPqmMj2MG4RUqVkRjnE_4kf0VvQ0_WkiTg/6j3Reg7FneLgLDfjE-lZuZUOrdc\"", "actor": {"callerType": "USER", "email": "redacted", "profileId": "redacted"}, "ipAddress": "redacted", "events": [{"type": "SECURITY_INVESTIGATION", "name": "SECURITY_INVESTIGATION_QUERY", "parameters": [{"name": "INVESTIGATION_DATA_SOURCE", "value": "USER LOG EVENTS"}, {"name": "INVESTIGATION_QUERY", "value": "(empty)"}]}]},

Any help is appreciated thank you!

Re: Need some help with FIELD_HEADER_REGEX and json data

gcusello — Tue, 19 Jul 2022 09:48:05 GMT

Hi @korstiaans,

I had a problem like your and I solved with SEDCMD:

SEDCMD-remove_header = s/\{\"activities\": \[\{\"kind\".*/\[\{\"kind\".*/flags

But before this, did you tried the spath command?

I think that should solve your need..

Ciao.

Giuseppe

Re: Need some help with FIELD_HEADER_REGEX and json data

korstiaans — Tue, 19 Jul 2022 09:59:14 GMT

Hi Giuseppe,

Thanks for the quick response, so it's only a problem for the first line I managed to split the other events and they are indexes as json so no need for spath.

I'll try your SEDCMD suggestion and get back with the results.

Re: Need some help with FIELD_HEADER_REGEX and json data

korstiaans — Tue, 19 Jul 2022 10:14:22 GMT

Unfortunately it didn't work for completeness sake I've added the props.conf I used.

[gws:test]
KV_MODE= json
LINE_BREAKER = }]},(.)
SEDCMD-remove_header =SEDCMD-remove_header = s/\{\"activities\": \[\{\"kind\".*/\[\{\"kind\".*/flags
disabled=false
pulldown_type=true
SHOULD_LINEMERGE = false