topic Re: How to remove json key value pairs from events log data in Splunk Search

How to remove json key value pairs from events log data

ayushram — Mon, 27 Feb 2023 08:07:40 GMT

Splunk search events returns json format log data. I want to remove a particular key:value pair since the value of this key is huge (in terms of length) and unnecessary. How can I do so.

sample log data:

{
"abcd1": "asd",
"abcd2": [],
"abcd3": true,
"toBeRemoved": [{
"abcd8": 234,
"abcd9": [{
"abcd10": "asd234"
}],
"abcd11": "asdasd"
}],
"abcd4": 324.234,
"abcd5": "dfsad dfsdf",
"abcd6": 0,
"abcd7": "asfsdf"
}

The key:value pair to be removed has been marked in bold.

! NOTE THIS IS FORMATTED DATA, FIELDS CAN HAVE STRINGS, NUMBERS, BOTH, LISTS, ETC !

Re: How to remove json key value pairs from events log data

gcusello — Mon, 27 Feb 2023 08:20:02 GMT

Hi @ayushram,

if you want to remove the highlighted data from the logs before indexing you have to add to your props.conf:

[your_sourcetype] SEDCMD = s/(?ms)\"toBeRemoved\":.*\}\],//g

remember that this props.conf must be added on your Indexers or (if present) on your Heavy Forwarders.

Ciao.

Giuseppe

Re: How to remove json key value pairs from events log data

ayushram — Mon, 27 Feb 2023 08:47:49 GMT

I do not have access to pros.conf

Is there any way to do this from search itself?
I want my final data in " | table ", but it's not loading wherever this highlighted field appears (since it has too many characters)

Re: How to remove json key value pairs from events log data

ITWhisperer — Mon, 27 Feb 2023 10:36:47 GMT

Try something like this - this assumes "toBeRemove" is not the first element i.e. is is preceded by a comma (which needs to be removed).

| rex mode=sed "s/(?ms),\s*\"toBeRemoved\":\s*\[([^\[\]]+|\[[^\]]*\])*\]//g"

Re: How to remove json key value pairs from events log data

gcusello — Tue, 28 Feb 2023 07:37:52 GMT

Hi @ayushram,

you can avoid to display a part of your logs in your searches, but accessing the raw log it's all visible:

| rex mode=sed "s/(?ms)\"toBeRemoved\":.*\}\],//g"

Ciao.

Giuseppe