topic Re: Sending splunkmetrics via HEC from telegraf, Splunk shows bytes received but no bytes indexed. in Getting Data In

Sending splunkmetrics via HEC from telegraf: Why does Splunk show bytes received but no bytes indexed?

thomasyung — Tue, 29 Mar 2022 20:57:25 GMT

From splunks logs (and _introspection) I can see the data coming in, but not being indexed. I have indexes created and working with other data sources, but I can't seem to see any events from this telegraf source.

Please see the relevant part of my telegraf config, using the [[outputs.http]] plugin.

[global_tags] # dc = "us-east-1" # will tag all metrics with dc=us-east-1 # rack = "1a" ## Environment variables can be used as tags, and throughout the config file #user = "telegraf" index = "main" [agent] interval = "30s" round_interval = true metric_batch_size = 1000 metric_buffer_limit = 10000 collection_jitter = "0s" flush_interval = "10s" flush_jitter = "0s" precision = "" debug = false quiet = false logtarget = "file" logfile = "/var/log/telegraf/telegraf.log" logfile_rotation_interval = "0d" logfile_rotation_max_size = "1MB" logfile_rotation_max_archives = 5 hostname = "" omit_hostname = false [[outputs.http]] ## URL is the address to send metrics to url = "http://my-splunk-instance:8088/services/collector" ## HTTP method, one of: "POST" or "PUT" method = "POST" # DEV ONLY insecure_skip_verify = false data_format = "splunkmetric" splunkmetric_hec_routing = true ## Additional HTTP headers [outputs.http.headers] Content-Type = "application/json" Authorization = "Splunk my-splunk-token" X-Splunk-Request-Channel = "my-splunk-token"

Do I need to create a specific index and list this in the hec token config? Is there a source type I'm somehow discarding?

Re: Sending splunkmetrics via HEC from telegraf, Splunk shows bytes received but no bytes indexed.

sistemistiposta — Fri, 18 Mar 2022 09:56:39 GMT

Hello @thomasyung , did you solve this problem? I have a similar problem. I have many Telegraf agent hosts. Intermittently, some of them are indicized and the others are not indicized.

I see no errors in splunkd.log. The index queues are empties. I have already applied the suggestions described here in order to improve Splunk performances.

Are there some practices about HEC indexing in a single Splunk host? When I had one token all was working fine.

Then I added a second token with other seven Telegraf agent hosts, and the indexes start to miss data from some host.

No errors in splunkd.log or monitoring HEC console. Frustrating...

Thank you

Kind Regards

Marco

Re: Sending splunkmetrics via HEC from telegraf, Splunk shows bytes received but no bytes indexed.

PickleRick — Fri, 18 Mar 2022 10:45:00 GMT

Check your HEC input parameters. If you're not providing index field within the event, you should have the destination index set within inputs.conf for this particular HEC token.

Also if you're sending to HEC without TLS it should be relatively easy to do a tcpdump of the network traffic to make sure if the events are really accepted by the input and get "lost" somewhere along the way.

Check your input with btool to see what index is effectively set in your resulting config on the indexer/forwarder where you have the HEC input configured.

If you didn't set any specific index for that input and the source is not sending an index field, the data is most probably trying to be ingested into your default index (usually the main index).

If the source is setting the index field, make sure that your HEC input allows receiving events for this index.

There are many things that can go wrong 😉

Re: Sending splunkmetrics via HEC from telegraf, Splunk shows bytes received but no bytes indexed.

sistemistiposta — Tue, 29 Mar 2022 09:13:59 GMT

@PickleRick you are right. I didn't explain in detail my setup.

My problem was the know issue SPL-212284. If you don't set

batch_search_max_pipeline = 1

even if allow_batch_mode = 0 mstats will randomly fail.

I don't know why _introspection shows 0 as data_indexed yet. Really my data is fully indexed.

Kind Regards

Marco