Getting Data In

Cloudflare logs to Heavy Forwarder - Pipeline data does not have indexKey?

jcrosby21
Path Finder

I am trying to send my cloudflare HTTP logs to my externally exposed splunk heavy forwarder (on prem).

I have installed the Cloudflare App on the heavy forwarder and the searchead:
https://splunkbase.splunk.com/app/4501/#/details

I know the data is making it to my heavy forwarder that has the application installed. However, the data isn't being correctly ingested. I am finding this type of log on my _internal index on my forwarder, and it appears to be for each event that cloudflare has sent to my forwarder.  I have rebooted the forwarder since adding the application:
09-15-2022 10:16:22.804 -0400 WARN TcpOutputProc [5288 indexerPipe] - Pipeline data does not have indexKey. [_hecTeleVersionKey] = default\n[_hecTeleAppKey] = default\n[_raw] = \n[_meta] = punct::\n[MetaData:Source] = source::http:Cloudflare5xx\n[MetaData:Host] = host::readactedhost.com\n[MetaData:Sourcetype] = sourcetype::cloudflare:json\n[_done] = _done\n[_linebreaker] = _linebreaker\n[_time] = 1663251382\n[_conf] = source::http:Cloudflare5xx|host::readactedhost.com|cloudflare:json|\n

My HEC token is configured as:
[http://Cloudflare5xx]
description = Used to get cloudflare logs into splunk for server 5xx errors
disabled = 0
indexes = cloudflare
token = 7xxxxxxxx

I am stumped what "Pipeline data does not have indexKey" means and cannot find a next step.  If the logs are being sent, and making it to the forwarder, are there more steps than having the application there to interpret the information sent to the services/collector/raw endpoint?  I have never ingested on the /raw endpoint before so I wonder if something is missing.

Labels (1)
0 Karma
1 Solution

jcrosby21
Path Finder

This error was because I was sending information to the /raw endpoint on my HTTP Event Collector.  With this endpoint the HEC inputs.conf must be specifying the particular index to load the raw events into.  I mistakenly thought that the Cloudflare app would do this for me with props.conf, it has an index defined within the app, but this was incorrect.  With the other HEC endpoint the event specifies the index ITSELF so the learning was the raw endpoint requires more information in the HEC inputs.conf.  I also needed to tweak the cloudflare app's TZ (UTC) , INDEXED_EXTRACTIONS (json), and KV_MODE (none) in the applications props.conf to properly ingest once they were being placed on the index.

View solution in original post

0 Karma

jcrosby21
Path Finder

This error was because I was sending information to the /raw endpoint on my HTTP Event Collector.  With this endpoint the HEC inputs.conf must be specifying the particular index to load the raw events into.  I mistakenly thought that the Cloudflare app would do this for me with props.conf, it has an index defined within the app, but this was incorrect.  With the other HEC endpoint the event specifies the index ITSELF so the learning was the raw endpoint requires more information in the HEC inputs.conf.  I also needed to tweak the cloudflare app's TZ (UTC) , INDEXED_EXTRACTIONS (json), and KV_MODE (none) in the applications props.conf to properly ingest once they were being placed on the index.

0 Karma
Get Updates on the Splunk Community!

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer at Splunk .conf24 ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...

Share Your Ideas & Meet the Lantern team at .Conf! Plus All of This Month’s New ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...