topic Ingesting csv files via inputs.conf in Getting Data In

Ingesting csv files via inputs.conf

Dayalss — Wed, 28 Aug 2024 03:22:48 GMT

Hi,

Im currently working on ingesting 8 csv files from a path using inputs.conf on a UF.

And the data is getting ingested . The issue is these 8 csv files are overwritten daily by new data by a automation script so the data inside the csv file is changed daily.

I want to ingest the complete csv data daily into Splunk , but what I can see is only a small set of data is getting ingested but not the complete csv file data.

My inputs.conf is

[monitor://C:\file.csv]
disabled = false
sourcetype = xyz
index = abcd
crcSalt = <DATETIME>

Can someone please help me , whether Im using the correct input or not?

The ultimate requirement is to ingest the complete csv data from the 8 csv files daily into Splunk.

Thank you.

Re: Ingesting csv files via inputs.conf

PickleRick — Wed, 28 Aug 2024 05:21:50 GMT

1. You obviously can't read data from 8 files if you have input set for just one of them 😉

2. Leave the crcSalt setting alone. It is very very rarely needed. Usually you should rather set initCrcLength if the files have common header/preamble

3. What do you mean by "small set of data is being ingested"?

4. Did you check

splunk list monitor

and

splunk list inputstatus

Re: Ingesting csv files via inputs.conf

Dayalss — Wed, 28 Aug 2024 05:42:13 GMT

Hi,

Sorry for the confusion , I just pasted a single input stanza , however I have 8 different monitoring stanza's in my inputs.conf and they are all working and ingesting the data.

crcSalt = <DATETIME>
What It Does: This setting includes the file's last modification time in the checksum calculation.
Use Case: It's useful when you want Splunk to reindex the file if the file's last modified timestamp changes, even if the content stays the same.

So for my usecase I need to ingest the complete csv file data daily , so used crcSalt = <DATETIME>. (Im doing right or wrong , please correct)

Small set of data means only getting few rows data from the csv file and not the complete csv data.

Can you please help.

Thank you

Re: Ingesting csv files via inputs.conf

PickleRick — Wed, 28 Aug 2024 11:26:38 GMT

About the number of files - yes, I figured as much. It was suppose to be a little joke to lighten the mood a bit. Maybe a missed one. Nevermind.

"What It Does: This setting includes the file's last modification time in the checksum calculation." - No, it does not. It includes literal "DATETIME" string in CRC calculation (which doesn't change the situation much). The only possible "dynamic" setting specified in the spec file for inputs.conf is the <SOURCE> setting which is substituted with each file's path. Other than that, the strings are constant literals.

Are the files updated or fully rewritten?

As usual with any problems with ingesting files, the first debugging steps are to run

splunk list monitor

and

splunk list inputstatus

and see if there's something unusual about those files

Re: Ingesting csv files via inputs.conf

Dayalss — Wed, 28 Aug 2024 11:30:53 GMT

Yes the files are getting fully overwritten and checked the input status and no issues found.