All Apps and Add-ons

S3 bucket with CSV files not extracting fields at index time


We have a S3 bucket containing many csv files, each with different header fields that need to be extracted at index time. The current configs in place for this is:

KV_MODE = auto
TRUNCATE = 999999

This is on the heavy forwarder server that has the AWS add-on installed (latest version) in addition to being on the indexers. I have downloaded a sample csv file from S3 and imported it into Splunk via the UI and it parses correctly, yet it does not when setting this up via the Splunk_TA_aws app (UI or file) to use S3.

It seems that the AWS add on is causing it to ignore the HEADER_FIELD_LINE_NUMBER = 1 and INDEXED_EXTRACTIONS = csv setting entirely. Is anyone else seeing this, does anyone have a solution? Search time extractions are not an option here due to the fields changing frequently.


@ShaneNewman .. were you able to resolve this issue ? I am also getting same issues ? Please let me know..




0 Karma


you uploaded the CSV using the UI , right? Can you compare the stanzas in the .conf files for the UI input vis a vis the AWS input? there might be some differences.
Several users have reported changing the sourcetype name [aws:s3:csv] sometimes cause an issue, once some of them reverted back to using just [aws:s3] thngs started wokring
can you try the compare and tinker with the sourcetype

0 Karma


@ShaneNewman Did you get a resolution to this? I am seeing the same thing myself when I run a "Generic S3" input for a custom input for CSV files in an S3 bucket.

The header lines keep getting indexed and the fields are not extracted when I search the data.

0 Karma


I know it has been a while but did anyone ever get this issue resolved? On the newest version of the AWS Add-On and still unable to figure out reading in data from CSV files with field extractions.

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Build Scalable Security While Moving to Cloud - Guide From Clayton Homes

 Clayton Homes faced the increased challenge of strengthening their security posture as they went through ...

Mission Control | Explore the latest release of Splunk Mission Control (2.3)

We’re happy to announce the release of Mission Control 2.3 which includes several new and exciting features ...

Cloud Platform | Migrating your Splunk Cloud deployment to Python 3.7

Python 2.7, the last release of Python 2, reached End of Life back on January 1, 2020. As part of our larger ...