Sending splunkmetrics via HEC from telegraf: Why does Splunk show bytes received but no bytes indexed?

[thomasyung]

From splunks logs (and _introspection) I can see the data coming in, but not being indexed. I have indexes created and working with other data sources, but I can't seem to see any events from this telegraf source.

Please see the relevant part of my telegraf config, using the [[outputs.http]] plugin.

[global_tags]
  # dc = "us-east-1" # will tag all metrics with dc=us-east-1
  # rack = "1a"
  ## Environment variables can be used as tags, and throughout the config file
  #user = "telegraf"
  index = "main"

[agent]
  interval = "30s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  debug = false
  quiet = false
  logtarget = "file"
  logfile = "/var/log/telegraf/telegraf.log"
  logfile_rotation_interval = "0d"
  logfile_rotation_max_size = "1MB"
  logfile_rotation_max_archives = 5
  hostname = ""
  omit_hostname = false


[[outputs.http]]
   ## URL is the address to send metrics to
   url = "http://my-splunk-instance:8088/services/collector"


   ## HTTP method, one of: "POST" or "PUT"
   method = "POST"
  
   # DEV ONLY
   insecure_skip_verify = false

   data_format = "splunkmetric"
   splunkmetric_hec_routing = true

   ## Additional HTTP headers
   [outputs.http.headers]
      Content-Type = "application/json"
      Authorization = "Splunk my-splunk-token"
      X-Splunk-Request-Channel = "my-splunk-token"

 
Do I need to create a specific index and list this in the hec token config? Is there a source type I'm somehow discarding?

[sistemistiposta]

Hello @thomasyung , did you solve this problem? I have a similar problem. I have many Telegraf agent hosts. Intermittently, some of them are indicized  and the others are not indicized.

I see no errors in splunkd.log. The index queues are empties. I have already applied the suggestions described here in order to improve Splunk performances.

Are there some practices about HEC indexing in a single Splunk host? When I had one token all was working fine.

Then I added a second token with other seven Telegraf agent hosts, and the indexes start to miss data from some host.

No errors in splunkd.log or monitoring HEC console. Frustrating...

Thank you

Kind Regards

Marco

[PickleRick]

Check your HEC input parameters. If you're not providing index field within the event, you should have the destination index set within inputs.conf for this particular HEC token.

Also if you're sending to HEC without TLS it should be relatively easy to do a tcpdump of the network traffic to make sure if the events are really accepted by the input and get "lost" somewhere along the way.

Check your input with btool to see what index is effectively set in your resulting config on the indexer/forwarder where you have the HEC input configured.

If you didn't set any specific index for that input and the source is not sending an index field, the data is most probably trying to be ingested into your default index (usually the main index).

If the source is setting the index field, make sure that your HEC input allows receiving events for this index.

There are many things that can go wrong 😉

[sistemistiposta]

@PickleRick you are right. I didn't explain in detail my setup.

My problem was the know issue SPL-212284. If you don't set  

batch_search_max_pipeline = 1

even if allow_batch_mode = 0 mstats will randomly fail.

I don't know why _introspection shows 0 as data_indexed yet. Really my data is fully indexed.

Kind Regards

Marco