All Apps and Add-ons

breaking large JSON array from REST input into event

spacemanaki
Engager

I have a REST API which returns a very large, but valid, JSON payload. The structure of this JSON is a single array of many objects. Last I checked the response is around 1.2mb or roughly 1million chars. Here is a sample of the JSON, pretty printed (the actual response contains no newlines):

[
    {
        "barcode": "10010208",
        "comment": null,
        "flagged": 1,
        "fromCode": "war_rep",
        "fromStation": "Warehouse Repair",
        "lastTrackScan": "12/10/2015 12:31:48 AM",
        "muted": true,
        "priority": "RED",
        "reservations": 1,
        "sku": "TB44_10",
        "toCode": "war_rep",
        "toStation": "Warehouse Repair"
    },
    {
        "barcode": "10011135",
        "comment": null,
        "flagged": 1,
        "fromCode": "cus_rec",
        "fromStation": "Customer Receiving",
        "lastTrackScan": "12/09/2015 10:17:12 AM",
        "muted": true,
        "priority": "RED",
        "reservations": 2,
        "sku": "RR52_8",
        "toCode": "ins",
        "toStation": "Pre-Inspection"
    },
... many more
]

After adding a REST data input that made an HTTP GET req once every 60s in Splunk, we were able to successfully have this JSON broken into events, with one event per object. But following an upgrade, this stopped working. Now, the payload isn't parsed as JSON but appears to be treated as a single event, and is truncated at 10,000 chars. We're still using sourcetype="_json" but somehow this isn't working.

We're using Splunk Enterprise 6.3.1

Damien_Dallimor
Ultra Champion

In your REST stanza , declare a custom response handler for splitting the JSON events.Conveniently , one ships with the release as splitting out events from a large JSON array is quite a common use case.

alt text

mhlesourd
New Member

Hi,

You should customised the props.conf for the appropriate source type.
For my part, I have put "TRUNCATE = 0" and it is now working like a charm.

Kind regards

0 Karma

jkat54
SplunkTrust
SplunkTrust

What if you add TRUNCATE = 90999999999 or some arbitrarily large number to your props?

0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!

Review:





Or Learn More in Our Blog >>