Splunk Dev

HEC random issue

smalonso
Explorer

Hello,

I'm using HEC to send data to our splunk cloud instance in _json format. Currently I have 2 pods - 1 HEC - 2 indexes architecture.
I do a "transaction" and I have 3 splunk JSON events going one to indexA and the 2 others to indexB.

Randomly I'm getting an error
{"type":"INTERNAL","description":"error during post to splunk for tenantId=monitoring err=StatusCodeError: 400 - {\\"text\\":\\"Incorrect index\\",\\"code\\":7,\\"invalid-event-number\\":1}"},

However indexes are correct as I see events on them. For example in the log sample above I have only one event in indexB (Where I should have 2) and one in indexA. Sometimes is one event failing, sometimes 2, and sometimes is one index and sometimes is the other.

In introspection index I see parse error is 1, however I don't know where I can see a more detailed explanation.
I've already check HTTP collector troubleshooting page, again I'm in splunk cloud with no access to config files, so I'm kind of block.

Can someone help me to either to know how to get more information or share if you've faced a similar issue?

0 Karma

smalonso
Explorer

Hello, for anyone interested and update on this issue:

The problem happens when using 1 server - 2 HEC - 4 indexes (2 for each HEC)
I made a few tests such as

  • 2 pod - 1 HEC - 4 indexes
  • 2 pods - 2 HEC - 2 indexes (Only one index per HEC)

In both cases the issue is not present. I couldn't find the reason yet.

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...