topic Re: Help with props.conf configuration to remove outer curly bracket before ingesting JSON file to get event ID in Getting Data In

Help with props.conf configuration to remove outer curly bracket before ingesting JSON file to get event ID

lim2 — Tue, 29 Sep 2020 16:12:09 GMT

props.conf to remove outer curly bracket before ingesting json file from
{
"filters": [
{
"id": "94960710-78a8-139d-6e52-5845eba8ebc9",
"name": "admin",
},
{
"id": "ec6402e0-3874-bc4d-715b-1c1f1b7bd695",
"name": "test",
}
]
}

to
"filters": [
{
"id": "94960710-78a8-139d-6e52-5845eba8ebc9",
"name": "admin",
},
{
"id": "ec6402e0-3874-bc4d-715b-1c1f1b7bd695",
"name": "test",
}
]
so that I can get id/name under different events
I tried with JSON_TRIM_BRACES_IN_ARRAY_NAMES=true and mount_point=filters with no result
Please advise.
Thanks

Re: Help with props.conf configuration to remove outer curly bracket before ingesting JSON file to get event ID

gjanders — Sun, 08 Oct 2017 13:02:29 GMT

 [your_sourcetype_goes_here]
 SEDCMD-removebracket = s/^{//g

That is assuming the { is the very first line of the event, if not the regular expression might need some tweaking. If you wanted to work over multiline mode you can as per this answer

Re: Help with props.conf configuration to remove outer curly bracket before ingesting JSON file to get event ID

lim2 — Tue, 10 Oct 2017 12:07:46 GMT

Thanks for the idea, I will also try with SEDCMD-removebracket = 1d;$d (from sed -e -i '1d;$d' jsonfile)

Re: Help with props.conf configuration to remove outer curly bracket before ingesting JSON file to get event ID

gjanders — Tue, 10 Oct 2017 12:29:56 GMT

From the props.conf documentation you will need to use a s/... version, it appears to be slightly different to sed on Unix but I think you get the idea of the answer 🙂

* A sed script is a space-separated list of sed commands. Currently the
  following subset of sed commands is supported:
    * replace (s) and character substitution (y).
* Syntax:
    * replace - s/regex/replacement/flags
      * regex is a perl regular expression (optionally containing capturing
        groups).
      * replacement is a string to replace the regex match. Use \n for back
        references, where "n" is a single digit.
      * flags can be either: g to replace all matches, or a number to
        replace a specified match.
    * substitute - y/string1/string2/
      * substitutes the string1[i] with string2[i]

Re: Help with props.conf configuration to remove outer curly bracket before ingesting JSON file to get event ID

lim2 — Tue, 10 Oct 2017 14:26:36 GMT

Added following line in props.conf seemed to have worked for removing the outer curly brackets.
SEDCMD-removebracket = '1d;$d'

Re: Help with props.conf configuration to remove outer curly bracket before ingesting JSON file to get event ID

gjanders — Tue, 10 Oct 2017 19:44:18 GMT

Interesting, that doesn't quite match the documentation but glad it works 🙂
Perhaps you can accept your own answer here?

Re: Help with props.conf configuration to remove outer curly bracket before ingesting JSON file to get event ID

MuS — Tue, 10 Oct 2017 19:59:50 GMT

I messaged the docs team to verify with dev team this works this way and if so update the docs 😉

cheers, MuS

Re: Help with props.conf configuration to remove outer curly bracket before ingesting JSON file to get event ID

MuS — Thu, 19 Oct 2017 22:14:16 GMT

Little update here: This is under investigation now

Re: Help with props.conf configuration to remove outer curly bracket before ingesting JSON file to get event ID

malmoore — Mon, 23 Oct 2017 21:51:22 GMT

Hi,

I confirmed with an engineer that this option will work in the way it's been depicted here.