What is causing json messages to not always be indexed?

[dhuynh]

Hi everyone,

For one of our client we are sending in json log data via log4j2 to the splunk cloud HEC token.

we are using the /event/collector/raw endpoint.

What I notice is that the fields are not extracted consistently. We do not see any pattern in our process so we cannot pinpoint the exact location of the issue. 

dhuynh_0-1686606054908.png

I am using the following source type with its configs:

dhuynh_2-1686606398965.png

Hopefully can someone see what might cause this issue.

Thankyou in advanced.

Duy

[nyc_jason]

Hello dhuynh, there are a few possible reasons this could be happening. First, please check for payload character set issues (such as non UTF-8 characters, which can cause JSON to break. Also, check the splunk logs for errors. You can find HEC parsing errors in the _introspection index. 

[dhuynh]

@nyc_jason  thankyou for your fast reply.

when checking the _introspection index I dont see any parsing error. Everything gets parsed correctly. what is weird is that it happens randomly. so when I rerun the process again then the data might be parsed correctly.