Getting Data In

Can you help me with a line breaking issue in a Multi-line JSON file input with an array of records?

reed_kelly
Contributor

I have input files from MS Graph with pretty-printed JSON that looks something like the following (ellipses used liberally...). I am unable to find the right LINE_BREAKER value or BREAK_ONLY_BEFORE or BREAK_ONLY_AFTER to split the records on the comma between the }, and the {. Note that this sample has had the indentation with extra spaces removed.

    [
        {
            "@odata.type":  "#Microsoft.graph....",
            "id":  "...",
            "...":  "...",
            "foobar":  { 
                                  "foo1":  "bar1",
                                  "foo2":  "bar2",
                            },
            "...":  "...",
            "barfoo":  {
                                  "bar1":  "foo1",
                                  "bar2":  "foo2",
                            }
        },
        {
            "@odata.type":  "#Microsoft.graph....",
            "id":  "...",
            "...":  "...",
        },
        {
            "...":  "...",
            "...":  "...",
        }
    ]

This props.conf fails, because there are other }, strings within each record (see end of "foobar"):

[json]
TRUNCATE = 0
KV_MODE = json
TIME_PREFIX = \"xxxEventDateTime\":\"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N
MUST_BREAK_AFTER = \s*}\,

This is for a 6.5.x Splunk hwf feeding a 6.5.x indexer cluster.

0 Karma
1 Solution

reed_kelly
Contributor

I found the following to work:

TRUNCATE = 0
SHOULD_LINEMERGE = false
PREAMBLE_REGEX = ^\s*\[\s*$
LINE_BREAKER = }(,\s*[\r\n]*\s*){

View solution in original post

0 Karma

reed_kelly
Contributor

I found the following to work:

TRUNCATE = 0
SHOULD_LINEMERGE = false
PREAMBLE_REGEX = ^\s*\[\s*$
LINE_BREAKER = }(,\s*[\r\n]*\s*){
0 Karma
Get Updates on the Splunk Community!

Celebrate CX Day with Splunk: Take our interactive quiz, join our LinkedIn Live ...

Today and every day, Splunk celebrates the importance of customer experience throughout our product, ...

How to Get Started with Splunk Data Management Pipeline Builders (Edge Processor & ...

If you want to gain full control over your growing data volumes, check out Splunk’s Data Management pipeline ...

Out of the Box to Up And Running - Streamlined Observability for Your Cloud ...

  Tech Talk Streamlined Observability for Your Cloud Environment Register    Out of the Box to Up And Running ...