topic Re: Can you help me split a JSON event with regex in props.conf? in Getting Data In

Can you help me split a JSON event with regex in props.conf?

keiran_harris — Tue, 29 Sep 2020 22:33:10 GMT

Hi guru's:

i have JSON data that looks like the below.

{
    "BOMxmlDlTime": 0.6584670543670654,
    "TODAY-PLUS-0": {
        "BOMPREDdate": "2018-12-29",
        "BOMPREDdescBrief": "Sunny.",
        "BOMPREDfireDanger": "Very High",
        "BOMPREDiconCode": 1,
        "BOMPREDrainChance": 0,
        "BOMPREDtempMax": 30
    },
    "TODAY-PLUS-1": {
        "BOMPREDdate": "2018-12-30",
        "BOMPREDdescBrief": "Mostly sunny.",
        "BOMPREDiconCode": 3,
        "BOMPREDrainChance": 5,
        "BOMPREDtempMax": 31,
        "BOMPREDtempMin": 22
    },
    "TODAY-PLUS-2": {
        "BOMPREDdate": "2018-12-31",
        "BOMPREDdescBrief": "Possible shower.",
        "BOMPREDiconCode": 17,
        "BOMPREDrainChance": 40,
        "BOMPREDtempMax": 31,
        "BOMPREDtempMin": 22
    },
    "kCryptoDictType": "BOMpredictions"
}

I want to split each chunk of "TODAY-PLUS-X: { xxxxxx }," into its own event. I've been reading and attempting various things for the last few hours and nothing I can seem to do allows me to do this. The obvious place to split it would be at the line with:

},

... so ive been playing with various combinations of MUST_BREAK_AFTER with regex in props.conf to do this, but nothing seems to make the event split. Here's my current props.conf (in the /etc/system/local/ directory, but have also tried in the /etc/app/xxxxx/local). And, I've been doing a full Splunk restart each time I edit props.conf if anyone was wondering. The last 5 lines i think are bits I've been manually editing (the first bits are auto created from the GUI when o created my new _jsonFUTURE sourcetype

[_jsonFUTURE]
INDEXED_EXTRACTIONS = json
KV_MODE = none
NO_BINARY_CHECK = true
category = Structured
description = K - required by BOM predictions data (needs special event splitting and date extraction logic)
disabled = false
pulldown_type = 1
BREAK_ONLY_BEFORE_DATE = false
MUST_BREAK_AFTER = \},
TIME_PREFIX = \"BOMPREDdate\":\s+\"
TIME_FORMAT = %y-%m-%d
MAX_DAYS_HENCE = 7

Can anyone please tell me what I'm doing wrong? I'm pulling my hair out !!!!

Thanks in advance guys,

Re: Can you help me split a JSON event with regex in props.conf?

gcusello — Sat, 29 Dec 2018 08:45:06 GMT

Hi keiran_harris,
try something like this

NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=\"TODAY-PLUS-\d+\":
CHARSET=AUTO
disabled=false
pulldown_type=true
TIME_FORMAT=%Y-%m-%d
TIME_PREFIX=\"BOMPREDdate\":\ "

Bye.
Giuseppe

Re: Can you help me split a JSON event with regex in props.conf?

keiran_harris — Tue, 29 Sep 2020 22:37:36 GMT

Ill give it a go. So is the main difference with your suggestion is use BREAK_ONLY_BEFORE rather than MUST_BREAK_AFTER i guess. Whats the logic there? Ill let you know as soon as i can retest thanks, keiran.

Re: Can you help me split a JSON event with regex in props.conf?

keiran_harris — Sat, 05 Jan 2019 06:13:43 GMT

Just circling back an updating here. Couldn't get the above to work.
So i fixed it by re-writing my code to structure the JSON in a way that splunk would inherently split it, without any tweaking of props.conf.

heres what it spits out now, and splu7nk is parsing, just fine:

{
    "BOMPREDapiPollTime": 1546140373.072095, 
    "BOMPREDdate": "2018-12-30T11:44:43+11:00", 
    "BOMPREDdescBrief": "Mostly sunny.", 
    "BOMPREDdescDetail": "Hot and mostly sunny. Winds west to northwesterly 15 to 20 km/h shifting east to northeasterly 15 to 25 km/h in the late morning and early afternoon then becoming light in the late evening.", 
    "BOMPREDfireDanger": "Very High", 
    "BOMPREDiconCode": 3, 
    "BOMPREDrainChance": 10, 
    "BOMPREDtempMax": 32, 
    "BOMPREDuvAlert": "Sun protection 8:30am to 5:20pm, UV Index predicted to reach 13 [Extreme]"
}
{
    "BOMPREDapiPollTime": 1546140373.072095, 
    "BOMPREDdate": "2018-12-31T00:00:00+11:00", 
    "BOMPREDdescBrief": "Shower or two.", 
    "BOMPREDdescDetail": "Partly cloudy. Medium (50%) chance of showers, most likely in the evening. The chance of a thunderstorm in the afternoon and evening. Light winds becoming northeasterly 15 to 20 km/h in the evening then becoming light in the late evening.", 
    "BOMPREDiconCode": 11, 
    "BOMPREDrainChance": 50, 
    "BOMPREDrainMM": "0 to 1 mm", 
    "BOMPREDtempMax": 29, 
    "BOMPREDtempMin": 22
}
{
    "BOMPREDapiPollTime": 1546140373.072095, 
    "BOMPREDdate": "2019-01-01T00:00:00+11:00", 
    "BOMPREDdescBrief": "Mostly sunny.", 
    "BOMPREDdescDetail": "Partly cloudy. Slight (20%) chance of a shower. The chance of a thunderstorm in the morning. Light winds becoming northeasterly 15 to 20 km/h during the day then becoming light during the afternoon.", 
    "BOMPREDiconCode": 3, 
    "BOMPREDrainChance": 20, 
    "BOMPREDtempMax": 30, 
    "BOMPREDtempMin": 23
}

Re: Can you help me split a JSON event with regex in props.conf?

woodcock — Fri, 08 Feb 2019 22:23:51 GMT

You need to click Accept to close this question.

Re: Can you help me split a JSON event with regex in props.conf?

gcusello — Wed, 31 Jul 2019 13:03:54 GMT

Hi keiran_harris,
if you're satisfied by this answer, please accept and/or upvote it.

Bye, see next time.
Giuseppe