topic Re: Why is my configuration not extracting CSV at index-time? in Getting Data In

Why is my configuration not extracting CSV at index-time?

gschmitz — Thu, 04 Sep 2014 06:39:43 GMT

Hi all,
I'm trying to use INDEXED_EXTRACTIONS = CSV but for some reason it's just not working. My input looks as follows

***SPLUNK*** sourcetype=csv source=index/host/query.sql

"SESSION_ID","LOGON_TIME","SCHEMA_NAME","TOTAL_SESSION_MEMORY"

"119","2014-08-22 11:04:03","SYS","813704"

and my props.conf

[csv]

DATETIME_CONFIG=NONE

INDEXED_EXTRACTIONS=CSV

TRANSFORMS-index=index-as-first-folder

None of the four fields are extracted, but the TRANSFORMS as well as the DATETIME_CONFIG take effect. Can anybody spot a mistake?

Re: Why is my configuration not extracting CSV at index-time?

richgalloway — Thu, 04 Sep 2014 13:07:21 GMT

Would you please share the relevant transform?

Re: Why is my configuration not extracting CSV at index-time?

jrodman — Thu, 04 Sep 2014 22:08:43 GMT

By default, the triple-splat ***SPLUNK*** magic cookie is not enabled for logfiles, and is probably breaking the default csv handling. It's enabled for the first line for scripted inputs by default, if I recall correctly. I'm not aware if if INDEXED_EXTRACTIONS can work for scripted inputs. The design requires a certain amount of seeking around which makes it hard to do fully generic stream processing.

Re: Why is my configuration not extracting CSV at index-time?

gschmitz — Fri, 05 Sep 2014 00:39:24 GMT

It's got nothing to do with the CSV, but sets the index to the name of the first folder in my source 🙂

Re: Why is my configuration not extracting CSV at index-time?

gschmitz — Fri, 05 Sep 2014 00:42:14 GMT

Hi,

the output is actually from a script which changes the sourcetype repeatedly through the run of the script. So it's respected since the DATETIME_CONFIG is applied (the event was earlier in sent to August, so I'm pretty sure).

Lastly, I have another scripted input which uses TSV, but in this case I don't use ***SPLUNK***. Either INDEXED_TRANSACTIONS is really determined based on the first line, or CSV is broken. I may just change the delimiters to pipes or tabs and check it out later.

Best Regards.

Re: Why is my configuration not extracting CSV at index-time?

gschmitz — Fri, 05 Sep 2014 02:31:36 GMT

Ok. I tried PSV and TSV as well. No success. Opened case 187571.

Re: Why is my configuration not extracting CSV at index-time?

woodcock — Mon, 28 Sep 2020 19:47:34 GMT

Because the file has no header, you should use INDEXED_EXTRACTIONS = csv with the FIELD_NAMES option:
http://docs.splunk.com/Documentation/Splunk/6.2.2/Data/Extractfieldsfromfileheadersatindextime

Perhaps the magic cookie will be honored later down the line or perhaps it will be treated as an event (and you can strip it with a transform to null queue).