breaking large JSON array from REST input into event

[spacemanaki]

I have a REST API which returns a very large, but valid, JSON payload. The structure of this JSON is a single array of many objects. Last I checked the response is around 1.2mb or roughly 1million chars. Here is a sample of the JSON, pretty printed (the actual response contains no newlines):

[
    {
        "barcode": "10010208",
        "comment": null,
        "flagged": 1,
        "fromCode": "war_rep",
        "fromStation": "Warehouse Repair",
        "lastTrackScan": "12/10/2015 12:31:48 AM",
        "muted": true,
        "priority": "RED",
        "reservations": 1,
        "sku": "TB44_10",
        "toCode": "war_rep",
        "toStation": "Warehouse Repair"
    },
    {
        "barcode": "10011135",
        "comment": null,
        "flagged": 1,
        "fromCode": "cus_rec",
        "fromStation": "Customer Receiving",
        "lastTrackScan": "12/09/2015 10:17:12 AM",
        "muted": true,
        "priority": "RED",
        "reservations": 2,
        "sku": "RR52_8",
        "toCode": "ins",
        "toStation": "Pre-Inspection"
    },
... many more
]

After adding a REST data input that made an HTTP GET req once every 60s in Splunk, we were able to successfully have this JSON broken into events, with one event per object. But following an upgrade, this stopped working. Now, the payload isn't parsed as JSON but appears to be treated as a single event, and is truncated at 10,000 chars. We're still using sourcetype="_json" but somehow this isn't working.

We're using Splunk Enterprise 6.3.1

[Damien_Dallimor]

In your REST stanza , declare a custom response handler for splitting the JSON events.Conveniently , one ships with the release as splitting out events from a large JSON array is quite a common use case.

[mhlesourd]

Hi,

You should customised the props.conf for the appropriate source type.
For my part, I have put "TRUNCATE = 0" and it is now working like a charm.

Kind regards

[jkat54]

What if you add TRUNCATE = 90999999999 or some arbitrarily large number to your props?