Splunk Search

How to extract a field that returns after a line_breaker

Explorer

Hello,

We are trying to split a nested json message into seperated events.
As we not wish to use the spath function it would be nice to split the events;
Our biggest challange right now is that we want to add the kId field in every event.

Example of the json;

{
    "kId": 47,
    "mId": 96,
    "resultStatus": "OK",
    "results": [{
        "creationDate": 1560509581000,
        "name": "test",
        "resultStatus": "OK",
        "duration": 0.858
    }, {
        "creationDate": 1560509581000,
        "name": "test2",
        "resultStatus": "OK",
        "duration": 0.858
    }]
}

we managed to get seperated events by using the following configurations:

props.conf
[measurements]
TRANSFORMS-kId = extractkId
LINE
BREAKER = (},){|([){|(])}
SHOULD_LINEMERGE = false

transforms.conf
[extractkId]
REGEX = kId[\"]:[,]
FORMAT = kId::$1
WRITE
META = true

events now look like this;

{"creationDate":1560509581000,"name":"test","resultStatus":"OK","duration":0.858}
{"creationDate":1560509581000,"name":"test2","resultStatus":"OK","duration":0.858}

How can we add the kId into this events?

Thanks in advance!

0 Karma
1 Solution

Ultra Champion

I don't think you can with Splunk. Event breaking is one of the first things that happens, so even if you could come up with a transforms or so to copy that kld to all the 'subsections', that all happens after line breaking, so the information is no longer available since Splunk processes each event on its own without knowledge of previous events.

You'll probably have to look at writing a script that pre-processes the logs before the are read by Splunk.

View solution in original post

Ultra Champion

I don't think you can with Splunk. Event breaking is one of the first things that happens, so even if you could come up with a transforms or so to copy that kld to all the 'subsections', that all happens after line breaking, so the information is no longer available since Splunk processes each event on its own without knowledge of previous events.

You'll probably have to look at writing a script that pre-processes the logs before the are read by Splunk.

View solution in original post

Explorer

Thanks for your answer. Would sedcmd an option? Is it possible to use sedcmd with variables?

0 Karma

Ultra Champion

sedcmd also happens after event breaking. But anyway that is in essence just a search&replace and while you can use the /g flag to repeat the sedcmd, you cannot carry that kId found in the first part forward to later matches.

I think the only solution (other than preprocessing) would be to ingest the event as a whole and process it using SPL (using a combination of spath and multivalue tricks).

0 Karma

Explorer

Thanks. We will investigate our preprocessing options.

0 Karma