topic Re: Data Not Getting Extracted Correctly as per CSV in Splunk Enterprise

Data Not Getting Extracted Correctly as per CSV

anandhalagarasa — Wed, 30 Jan 2019 04:53:26 GMT

We got an requirement to ingest a CSV file from a client machine. And in that CSV file we have headers in place as well. Headers are as mentioned something like that below:

Received SenderAddress RecipientAddress Subject Status FromIP Size MessageId
1/30/2019 4:29 xxxx@gmail.com yyyy@gmail.com Test Message Delivered 1.x.x.x 1234 xxx.gmail.com

So I have written the inputs.conf as below:

[monitor://X:\Test*.csv]
index = test
sourcetype = test_logs
crcSalt =
initCrcLength = 4999
disabled = 0

And have ingested the same into Splunk but the logs are getting extracted as in excel. So should we need to place any props and transforms if yes what would be the props and transforms.conf and where should i need to place the props and transforms as well.

Also the log file is not upated delay in Splunk as well. Actually new logs are already there in client machine but still its not reached Splunk as well.

So kindly help on this request.

Re: Data Not Getting Extracted Correctly as per CSV

vishaltaneja070 — Wed, 30 Jan 2019 06:25:37 GMT

@anandhalagarasan

Didn't get you? Do you want to extract that files as it shows in csv like command delimiter?

Re: Data Not Getting Extracted Correctly as per CSV

DMohn — Wed, 30 Jan 2019 09:06:45 GMT

You have to place a props.conf file on the indexing side, defining your sourcetype test_logs and assigning the CSV indexed extractions here. Otherwise Splunk won't be able to identify a header and assign the field extractions correctly.

See Docs for reference: https://docs.splunk.com/Documentation/Splunk/7.2.3/Data/Extractfieldsfromfileswithstructureddata

If you don't want to use indexed extractions, you can still place a props.conf on the search head and assign a search-time extraction there.

In regards of the logs not being updated - try to set a correct timestamp extraction stanza in your props. This should help Splunk to identify the timestamps correctly. Also, I do not see the need to use the crcSalt setting here. This should not be neccessary in this case.

Re: Data Not Getting Extracted Correctly as per CSV

ashajambagi — Tue, 29 Sep 2020 23:02:05 GMT

@anandhalagarasan
Try using sourcetype = csv in inputs.conf , splunk will take first line as header and automatically extract fields.

If you want to use custom sourcetype,define it in props.conf and place it in the $SPLUNK_HOME/etc/apps//local/ or $SPLUNK_HOME/etc/system/local

props.conf

[test_logs]
INDEXED_EXTRACTIONS = CSV
HEADER_FIELD_LINE_NUMBER=1
FIELD_DELIMITER=,

inputs.conf

[monitor:///opt/tsti.csv]
index = new
sourcetype = test_logs
disabled = false

Can you explain what does it mean when you say "the logs are getting extracted as in excel."?