Why is my JSON log appearing as one event?

[ew09]

I have a 300KB JSON file (I have checked using jsonlint that it is valid format) that I am having troubles with.

When I go to upload it, the preview shows it as completely one event. I know that by default it truncates at 10,000 bytes, so I set TRUNCATE equal to 0. I am still having the same issue. There are many nested arrays in the log and after uploading, it can only access the first element of the the first array. For example:

spath output=barNames path=bar{}.barName | table barNames 

only returns the first barName, even though it has a second object in that path. It seems like Splunk is not even uploading the whole JSON file?

Here is an overview of the file

{
    "version" : "1.0",
    "bar" : [
           {
                 "barName" : "name1",
                 "moreRecords" : [ ... ]
           },
           {
                 "barName" : "name2",
                  "moreRecords" : [ .... ]
            }
       ]
}

In this case, that entire file is a single JSON event. After loading, I can only access the values under bar[0], nothing in bar[1].

I have tried changing KV_MODE to JSON, line merge settings, etc. The data preview never seperates into separate events. Any ideas?

Thanks!

[ryanoconnor]

Alright I'm going to give you a sample here. Working with JSON can be a little bit tricky but it's usually very possible if it's valid JSON. Since I don't have your exact events I'll show you what I used as a test case.

This was my sample.json file

{
    "version": "1.0",
    "bar": [{
        "barName": "name1",
        "moreRecords": [{
            "test": "1"
        }]
    }, {
        "barName": "name2",
        "moreRecords": [{
            "test": "2"
        }]
    }]
}

If I use this as test data and imported it into Splunk using the following props.conf

[test_json_sourcetype]
KV_MODE = json
DATETIME_CONFIG = CURRENT

I do get data extracting correctly. I'm then able to run the following search:

index=test   | spath path=bar{} output=x  | mvexpand x | spath input=x | table barName moreRecords{}.test | rename moreRecords{}.* as * | search barName=name1

And that gets me the following output

[ryanoconnor]

What does the format of the log file look like? Is it possible Splunk doesn't know how to break up the individual events? Are all of the events separated by new lines?

[ew09]

The posted code is the unexpanded version of the log file. Inside the moreRecords[...] there are more arrays of objects...with even more nested arrays. There does not seem to be newlines after all events, I will need to check with the person who is outputting these log files in the first place. It is over 17,000 lines and seems pretty messy

[ryanoconnor]

Ya it's hard to say without having the actual event. I don't see any reference to bar[0] or bar[1]. Do any of the fields appear to be extracting in Splunk or is it coming in as raw data?

[ew09]

Yes, all fields are being extracted in the first 'bar'. It populates bar{}.moreRecords{}.metrics.anotherarray{}.field for example. What it doesn't do, is get anything from the second object in the 'bar' array. Why can't I do something like bar{1}.foo but I can access bar{0}?