Getting Data In

Why are source/host/sourcetype fields dropping through HEC?

Mels
Engager

I'm posting a json struct such as 

 

 

 

{
  "index": "test_metrics",
  "time": 1679920906.0,
  "event": "metric",
  "host": "myhostname",
  "source": "build.mybuildplan",
  "sourcetype": "trace_profile",
  "fields": {
    "metric_name:metric1": 1234,
    "metric_name:metric2": 1234,
    "metric_name:metric3": 1234,
    ...
    "metric_name:metricN": 1234
  }
}

 

 

 

I noticed that on our splunk enterprise server, I can successfully post it, but the source, host, and sourcetype fields are not visible in Splunk (version 9.0.1).

After some debugging on a local Splunk install I found that when I reduce N enough, these fields suddenly come through. Moreover, when I find the largest N for which these fields are shown properly and then make the name of the last metric longer (e.g. "metric_name:metricN_lorem_ipsum_etc"), it also starts to drop these fields. So it looks like it's related to the length of all metric names in the json combined?

My questions:
- Has anyone else experienced this?
- What's the magic limit I'm hitting here?
- Most importantly: Why can't I see any error message anywhere? It seems to be silently dropping some info. Is this a bug that could be fixed?

Labels (5)
Get Updates on the Splunk Community!

Splunk Lantern | Spotlight on Security: Adoption Motions, War Stories, and More

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Cloud | Empowering Splunk Administrators with Admin Config Service (ACS)

Greetings, Splunk Cloud Admins and Splunk enthusiasts! The Admin Configuration Service (ACS) team is excited ...

Tech Talk | One Log to Rule Them All

One log to rule them all: how you can centralize your troubleshooting with Splunk logs We know how important ...