Getting Data In

Cloudflare logs to Heavy Forwarder - Pipeline data does not have indexKey?

jcrosby21
Path Finder

I am trying to send my cloudflare HTTP logs to my externally exposed splunk heavy forwarder (on prem).

I have installed the Cloudflare App on the heavy forwarder and the searchead:
https://splunkbase.splunk.com/app/4501/#/details

I know the data is making it to my heavy forwarder that has the application installed. However, the data isn't being correctly ingested. I am finding this type of log on my _internal index on my forwarder, and it appears to be for each event that cloudflare has sent to my forwarder.  I have rebooted the forwarder since adding the application:
09-15-2022 10:16:22.804 -0400 WARN TcpOutputProc [5288 indexerPipe] - Pipeline data does not have indexKey. [_hecTeleVersionKey] = default\n[_hecTeleAppKey] = default\n[_raw] = \n[_meta] = punct::\n[MetaData:Source] = source::http:Cloudflare5xx\n[MetaData:Host] = host::readactedhost.com\n[MetaData:Sourcetype] = sourcetype::cloudflare:json\n[_done] = _done\n[_linebreaker] = _linebreaker\n[_time] = 1663251382\n[_conf] = source::http:Cloudflare5xx|host::readactedhost.com|cloudflare:json|\n

My HEC token is configured as:
[http://Cloudflare5xx]
description = Used to get cloudflare logs into splunk for server 5xx errors
disabled = 0
indexes = cloudflare
token = 7xxxxxxxx

I am stumped what "Pipeline data does not have indexKey" means and cannot find a next step.  If the logs are being sent, and making it to the forwarder, are there more steps than having the application there to interpret the information sent to the services/collector/raw endpoint?  I have never ingested on the /raw endpoint before so I wonder if something is missing.

Labels (1)
0 Karma
1 Solution

jcrosby21
Path Finder

This error was because I was sending information to the /raw endpoint on my HTTP Event Collector.  With this endpoint the HEC inputs.conf must be specifying the particular index to load the raw events into.  I mistakenly thought that the Cloudflare app would do this for me with props.conf, it has an index defined within the app, but this was incorrect.  With the other HEC endpoint the event specifies the index ITSELF so the learning was the raw endpoint requires more information in the HEC inputs.conf.  I also needed to tweak the cloudflare app's TZ (UTC) , INDEXED_EXTRACTIONS (json), and KV_MODE (none) in the applications props.conf to properly ingest once they were being placed on the index.

View solution in original post

0 Karma

jcrosby21
Path Finder

This error was because I was sending information to the /raw endpoint on my HTTP Event Collector.  With this endpoint the HEC inputs.conf must be specifying the particular index to load the raw events into.  I mistakenly thought that the Cloudflare app would do this for me with props.conf, it has an index defined within the app, but this was incorrect.  With the other HEC endpoint the event specifies the index ITSELF so the learning was the raw endpoint requires more information in the HEC inputs.conf.  I also needed to tweak the cloudflare app's TZ (UTC) , INDEXED_EXTRACTIONS (json), and KV_MODE (none) in the applications props.conf to properly ingest once they were being placed on the index.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...