topic Re: how to extract the csv fields at index-time? in Knowledge Management

How to extract the csv fields at index-time?

rajasekhar14 — Fri, 20 May 2022 16:06:27 GMT

hi all,
i'm trying extract the fields from the csv files and my csv file is looks like this,

just want to extract all fields at index-time only.

field1,filed2,-,-,-,etc

and my props.conf is
[sourcetype]
INDEXED_EXTRACTIONS = CSV
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = ,

but this is not successful, am i missing something ?

Re: how to extract the csv fields at index-time?

chrisyounger — Mon, 18 Feb 2019 20:04:02 GMT

Hi @rajasekhar14

Your config looks correct. Just make sure this props.conf file Is on the universal forwarder and not the indexer.

All the best

Re: how to extract the csv fields at index-time?

rajasekhar14 — Mon, 18 Feb 2019 20:37:07 GMT

Hi Chris,
I haven’t deployed to UF, because we have HF in place between Indexers and UFs. So I deployed to HF.

Re: how to extract the csv fields at index-time?

chrisyounger — Mon, 18 Feb 2019 21:29:30 GMT

Confusingly, CSV indexed extractions actually happen on the universal forwarder. It needs to be done here becuase it needs to use the header of the file regularly so it knows the column names.

Re: how to extract the csv fields at index-time?

ddrillic — Tue, 19 Feb 2019 02:03:32 GMT

Absolutely - a bit more at Do we need props.conf on the indexer when indexing a csv file?

Re: how to extract the csv fields at index-time?

chrisyounger — Tue, 19 Feb 2019 02:43:45 GMT

No I don't think so. However it doesn't harm to have it there.

Re: how to extract the csv fields at index-time?

woodcock — Tue, 19 Feb 2019 05:12:09 GMT

The INDEXED_EXTRACTIONS feature, unlike most index-time-related features, actually happens on the UF. So your props.conf must be sent to your UF and Splunk restarted there.

Re: how to extract the csv fields at index-time?

ashajambagi — Tue, 19 Feb 2019 10:19:07 GMT

@rajasekhar14 Where have you placed your props.conf? can you show the stanza in inputs.conf.

Refer this link - https://answers.splunk.com/answers/719666/data-not-getting-extracted-correctly-as-per-csv.html

Re: how to extract the csv fields at index-time?

rajasekhar14 — Tue, 19 Feb 2019 12:37:53 GMT

Chris, we are parsing at HF level, so I deployed tob HF.

Re: how to extract the csv fields at index-time?

rajasekhar14 — Tue, 19 Feb 2019 12:42:35 GMT

@ashajambagi here is the my inputs.conf
[monitor:/D:\mytest/splunk.csv]
sourcetype=test
index=myindex
crcSalt =
initCrcLength = 256

Re: how to extract the csv fields at index-time?

rajasekhar14 — Tue, 19 Feb 2019 12:43:44 GMT

@woodcock, as per all your suggestions i placed these settings in UF and restarted it, but now no luck.

Re: how to extract the csv fields at index-time?

ashajambagi — Tue, 29 Sep 2020 23:22:43 GMT

Check the format for this : [monitor:/D:\mytest/splunk.csv]

[sourcetype] #have you mentioned test here instead of sourcetype?
INDEXED_EXTRACTIONS = CSV
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = ,

Re: how to extract the csv fields at index-time?

woodcock — Tue, 19 Feb 2019 13:58:03 GMT

What do you mean "no luck" exactly?

Is your data coming in to splunk? If so, then it definitely is working.
With INDEXED_EXTRACTIONS it is ALL or NONE.
I suspect that you expect this change to fix data that is already in wrong. It will NOT do that. You have to send NEW data in, and then it should work. If data is not coming in, then the only thing that might be causing you a problem is that your sourcetype does not match or your Timestamping is wrong so the events are ending up in a timeframe that you did not expect. Try a timepicker with All time to check for the latter.

Re: how to extract the csv fields at index-time?

rajasekhar14 — Tue, 26 Feb 2019 19:05:21 GMT

once i deployed these settings to UF its working

Re: how to extract the csv fields at index-time?

rajasekhar14 — Tue, 26 Feb 2019 19:08:33 GMT

@woodcock now its working. previously it didn't deployed to UF. i have a question that why we need to deployed to UF only? in my case UF is forwarding to HF, and HF is forwarding to Indexers. So my all parsing is happening in HF level to avoid load on Indexers.

Re: how to extract the csv fields at index-time?

woodcock — Tue, 26 Feb 2019 22:06:33 GMT

Parsing on HF swaps CPU load for port I/O load and a different CPU load in that payload per event is much fatter and is very inefficient. See here:

https://www.splunk.com/blog/2016/12/12/universal-or-heavy-that-is-the-question.html

Re: how to extract the csv fields at index-time?

biko — Fri, 20 May 2022 16:02:10 GMT

thanks woodcok, this saved my day, at least what was left of it after struggling for hours.

This behaviour seems very counter-intuitive; I am used to the concept of UFs beeing dumb and having no notion of events