JSON payloads not getting indexed into Splunk

nareshinsvu — Mon, 24 Aug 2020 06:57:15 GMT

Hi,

Below is my props.conf on my Heavy Forwarder. I have recently found that there are few JSON messages completely missed getting indexed into Splunk. It's a high transaction system.

When I actually check my source json logs, eg: out of 10 json payloads, 1-2 doesn't get indexed. But all the 10 json payloads are having similar content and same number of lines

[dp_json]
SEDCMD-strip_prefix = s/^[^{]+//g
SEDCMD-dumpxml = s/(\<|\>\\r\\n).*//g
SEDCMD-remove = s/\"(shippingAddress)\"\s+\:\s+{[\s\S]*?(?=\n.*?{)//g
INDEXED_EXTRACTIONS=JSON
NO_BINARY_CHECK = true
category = Custom
description = dp_json_custom
disabled = false
pulldown_type = true
DATETIME_CONFIG = CURRENT
TRUNCATE = 100000
MAX_EVENTS = 10000

I couldn't troubleshoot the splunkd.log on forwarder because I continuously get below messages in it. I can't ask the source application system to change the json payload message to rectify this error. So, I am living with this error.

08-24-2020 13:19:52.474 +1000 ERROR JsonLineBreaker - JSON StreamId:10360380474397151566 had parsing error:Unexpected character while looking for value: 'a' - data_source="D:\Logs\myjson.log", data_host="myjsonhost", data_sourcetype="dp_json"

Is there a way to get notifications if any events are missed indexing?

Hope someone would ve faced same issue. Need urgent resolution as we don't want to miss any data in Splunk.

Thanks,

Naresh

Re: JSON payloads not getting indexed into Splunk

to4kawa — Mon, 24 Aug 2020 08:30:26 GMT

INDEXED_EXTRACTIONS=json can work before SEDCMD.
INDEXED_EXTRACTIONS works for valid JSON.

I'm surprised your setup is working properly.

Re: JSON payloads not getting indexed into Splunk

nareshinsvu — Mon, 24 Aug 2020 13:54:45 GMT

Hi @to4kawa

Do you reckon it resolves my issue ? By moving SEDCMDs after INDEXED_EXTRACTIONS=json

I am not sure if my conf is wrong as I picked it from other Splunk answers blog. It works for me except the issue I raised in this ticket.

topic Re: JSON payloads not getting indexed into Splunk in Getting Data In

JSON payloads not getting indexed into Splunk

Re: JSON payloads not getting indexed into Splunk

Re: JSON payloads not getting indexed into Splunk