How to extract a field that returns after a line_breaker

[jorambokma]

Hello,

We are trying to split a nested json message into seperated events.
As we not wish to use the spath function it would be nice to split the events;
Our biggest challange right now is that we want to add the kId field in every event.

Example of the json;

{
    "kId": 47,
    "mId": 96,
    "resultStatus": "OK",
    "results": [{
        "creationDate": 1560509581000,
        "name": "test",
        "resultStatus": "OK",
        "duration": 0.858
    }, {
        "creationDate": 1560509581000,
        "name": "test2",
        "resultStatus": "OK",
        "duration": 0.858
    }]
}

we managed to get seperated events by using the following configurations:

props.conf
[measurements]
TRANSFORMS-kId = extract_kId
LINE_BREAKER = (},){|([){|(])}
SHOULD_LINEMERGE = false

transforms.conf
[extract_kId]
REGEX = kId[\"]:[,]
FORMAT = kId::$1
WRITE_META = true

events now look like this;

{"creationDate":1560509581000,"name":"test","resultStatus":"OK","duration":0.858}
{"creationDate":1560509581000,"name":"test2","resultStatus":"OK","duration":0.858}

How can we add the kId into this events?

Thanks in advance!

[FrankVl · Accepted solution]

I don't think you can with Splunk. Event breaking is one of the first things that happens, so even if you could come up with a transforms or so to copy that kld to all the 'subsections', that all happens after line breaking, so the information is no longer available since Splunk processes each event on its own without knowledge of previous events.

You'll probably have to look at writing a script that pre-processes the logs before the are read by Splunk.

[jorambokma]

Thanks for your answer. Would sedcmd an option? Is it possible to use sedcmd with variables?

[FrankVl]

sedcmd also happens after event breaking. But anyway that is in essence just a search&replace and while you can use the /g flag to repeat the sedcmd, you cannot carry that kId found in the first part forward to later matches.

I think the only solution (other than preprocessing) would be to ingest the event as a whole and process it using SPL (using a combination of spath and multivalue tricks).

[jorambokma]

Thanks. We will investigate our preprocessing options.