All Apps and Add-ons
Highlighted

Splunk Add-on for Amazon Web Services: Exporting an S3 bucket and specifying the sourcetype as CSV, why is the header not parsed correctly?

Explorer

Hello all,

I am exporting an S3 bucket with only CSV and when I specified the sourcetype as CSV, I have all my events, but no fields about that event (aka the header is like an event). However, when I am importing my CSV in Splunk Web, the header is correctly inserted.

Any idea about how fix that?

Thanks !

Highlighted

Re: Splunk Add-on for Amazon Web Services: Exporting an S3 bucket and specifying the sourcetype as CSV, why is the header not parsed correctly?

Contributor

Can you post your props config for this input?

0 Karma
Highlighted

Re: Splunk Add-on for Amazon Web Services: Exporting an S3 bucket and specifying the sourcetype as CSV, why is the header not parsed correctly?

Explorer

my inputs.conf is :

[aws_s3://zx]
aws_account = yx
bucket_name = xy
character_set = auto
ct_blacklist = ^(?:Describe|List|Get)
host_name = s3.amazonaws.com
initial_scan_datetime = 2016-04-10T16:58:20+0200
key_name = devops/
max_items = 100000
max_retries = 3
polling_interval = 60
recursion_depth = -1
sourcetype = csv
ct_excluded_events_index =
index = data
disabled = 0
0 Karma
Highlighted

Re: Splunk Add-on for Amazon Web Services: Exporting an S3 bucket and specifying the sourcetype as CSV, why is the header not parsed correctly?

Explorer

Anyone has the same problem ?

Highlighted

Re: Splunk Add-on for Amazon Web Services: Exporting an S3 bucket and specifying the sourcetype as CSV, why is the header not parsed correctly?

Explorer

Did you make any progress I am observing the same behaviour.

0 Karma
Highlighted

Re: Splunk Add-on for Amazon Web Services: Exporting an S3 bucket and specifying the sourcetype as CSV, why is the header not parsed correctly?

SplunkTrust
SplunkTrust

Hi Menaham, I believe this issue will be resolved by creating a props.conf with a [csv] stanza (this input's sourcetype, can be anything you want), and then setting the "INDEXED_EXTRACTIONS = CSV" config at that stanza. More info can be found in the Structured Data section here : http://docs.splunk.com/Documentation/Splunk/latest/Admin/propsconf

Please let me know if this answers your question!

0 Karma
Highlighted

Re: Splunk Add-on for Amazon Web Services: Exporting an S3 bucket and specifying the sourcetype as CSV, why is the header not parsed correctly?

Explorer

It's not working, seems to be a bug with the add-on

0 Karma
Highlighted

Re: Splunk Add-on for Amazon Web Services: Exporting an S3 bucket and specifying the sourcetype as CSV, why is the header not parsed correctly?

Splunk Employee
Splunk Employee

Hello Menahem, I am trying to understand your expectation better. Could you explain further about your meaning behind "no fields about that event (aka the header is like an event)"

It is my understanding that the header of the file is used for field extractions. It would then be processed as a separate event and the fields that were extracted from the header will appear to the left under "Interesting Fields."

Thanks

0 Karma
Highlighted

Re: Splunk Add-on for Amazon Web Services: Exporting an S3 bucket and specifying the sourcetype as CSV, why is the header not parsed correctly?

Explorer

Hello Phadnett,
The problem is very simple, i don't have the different field's (even in "Interesting Fields") that is in the header of the file

0 Karma
Highlighted

Re: Splunk Add-on for Amazon Web Services: Exporting an S3 bucket and specifying the sourcetype as CSV, why is the header not parsed correctly?

Path Finder

I've added some information as a reply to the main question as I'm seeing the same issue but to clarify the problem is that when reading the file from S3 the header is not used for field extraction and is treated as if it were a separate event. So for a file like:

Header1,Header2,Header3
Row1Value1,Row1Value2,Row1Value3
Row2Value1,Row2Value2,Row2Value3

Using AWS S3 you get events like this:
Event 1 Header1,Header2,Header3
Event 2 Row1Value1,Row1Value2,Row1Value3
Event 3 Row2Value1,Row2Value2,Row2Value3
And no searchtime field extraction occurs

but if you add the file locally (just using the Add Data dialogue):
Event 1 Row1Value1,Row1Value2,Row1Value3
Event 2 Row2Value1,Row2Value2,Row2Value3
and the header field is used for field extraction at searchtime.

0 Karma