REST API Modular Input: Is there a limit on JSON size before parsing?

[DanielFordWA]

Hi,

Is there any limit on the size of the JSON the REST API Modular Input add-on can handle before the responsehandlers.py goes to work?

I have written a custom response handler to parse a large JSON file into separate events before indexing in Splunk.

In UAT and NFR environments everything works fine. The JSON is around 16.5k lines and 400K characters.

In PROD the data fails to index. The JSON is around 100k lines and 3M characters.

The sourcetype is set to the below.

[RESTMAR]
TRUNCATE = 0
TIME_PREFIX = lastModifiedDate

I can hit the endpoint with POSTMAN for both UAT /NFR and PROD, with the separate consumer secrets and keys, and get a successful response for each Environment.

I have edited the rest.py line 42 to put the logger on debug

#set up logging
logging.root
logging.root.setLevel(logging.DEBUG)

I will continue to investigate but has anyone else had this issue?

Thanks,

Dan

[Damien_Dallimor · Accepted solution]

There is no imposed limits in the code.
Any error messages ?

Search : index=_internal ExecProcessor error rest.py

[DanielFordWA]

Hi Damien,

I think the issue was caused by moving the responsehandlers.py file from a windows box to a Linux box, or editing a previously created input rather than creating a new one.

The solution.

• Test on NFR Linux box
• Copy files from NFR box
• Reinstall REST Modular input app in PROD
• Add python files copied from NFR Linux box
• Restart Splunk
• Create New REST Modular input (editing a previously created input did not work)
• Test data

Thanks for the response!

Cheers,

Dan