All Apps and Add-ons

Splunk Add-on for Amazon Web Services: How to get a CSV file stored in Amazon S3 to properly split at index-time?

Path Finder

I'm having trouble getting a CSV file that I've stored in Amazon S3 to properly split at index-time.

I'm using the Splunk Add-on for AWS, which allows me to define an S3 bucket to monitor. It pulls the data down just fine when a new CSV is uploaded:

disabled = false
aws_account = Splunk Reader
bucket_name = mybucket
index = jm
initial_scan_datetime = default
interval = 30
max_items = 100000
max_retries = 10
recursion_depth = 3
sourcetype = s3_autoruns
whitelist = .*/autoruns.txt$
blacklist = .*
character_set = UTF-16LE

I have in my props.conf a working transform (which changes the Host field to part of the S3 url), so I know this stanza is hitting for this data.

TRANSFORMS-s3host = transform-s3-integhost

With this, I get an event per line of the file.

I think I should be able to add to my props.conf:


But when I do that, it does not change anything. I still get one event per line, and no EntryLocation field to search on.

Any thoughts?


Splunk Employee
Splunk Employee

I have run into this similar issue when streaming data via scripted input into Splunk. In the interim, please use the DELIMS option for search time field extractions:

Path Finder

If I mirror the S3 bucket to a local directory and monitor it, it splits nicely:
disabled = 0
crcSalt = <SOURCE>
index = jm
sourcetype = s3_autoruns
whitelist = .*/autoruns.txt$


0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!