Getting Data In

Ingesting csv files via inputs.conf

Dayalss
Engager

Hi,

Im currently working on ingesting 8 csv files from a path using inputs.conf on a UF.

And the data is getting ingested . The issue is these 8 csv files are overwritten daily by new data by a automation script so the data inside the csv file is changed daily.

 

I want to ingest the complete csv data daily into Splunk , but what I can see is only a small set of data is getting ingested but not the complete csv file data.

 

My inputs.conf is 

[monitor://C:\file.csv]
disabled = false
sourcetype = xyz
index = abcd
crcSalt = <DATETIME>

 

Can someone please help me , whether Im using the correct input or not?

 

The ultimate requirement is to ingest the complete csv data from the 8 csv files daily into Splunk.

 

Thank you.

Labels (1)
0 Karma

Dayalss
Engager

Yes the files are getting fully overwritten and checked the input status and no issues found.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

1. You obviously can't read data from 8 files if you have input set for just one of them 😉

2. Leave the crcSalt setting alone. It is very very rarely needed. Usually you should rather set initCrcLength if the files have common header/preamble

3. What do you mean by "small set of data is being ingested"?

4. Did you check

splunk list monitor

and

splunk list inputstatus
0 Karma

Dayalss
Engager

Hi,

 

Sorry for the confusion , I just pasted a single input stanza , however I have 8 different monitoring stanza's in my inputs.conf and they are all working and ingesting the data.

  • crcSalt = <DATETIME>
    What It Does: This setting includes the file's last modification time in the checksum calculation.

    Use Case: It's useful when you want Splunk to reindex the file if the file's last modified timestamp changes, even if the content stays the same.

So for my usecase I need to ingest the complete csv file data daily , so used crcSalt = <DATETIME>. (Im doing right or wrong , please correct)

Small set of data means only getting few rows data from the csv file and not the complete csv data.

Can you please help.

 

Thank you

 

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

About the number of files - yes, I figured as much. It was suppose to be a little joke to lighten the mood a bit. Maybe a missed one. Nevermind.

"What It Does: This setting includes the file's last modification time in the checksum calculation." - No, it does not. It includes literal "DATETIME" string in CRC calculation (which doesn't change the situation much). The only possible "dynamic" setting specified in the spec file for inputs.conf is the <SOURCE> setting which is substituted with each file's path. Other than that, the strings are constant literals.

Are the files updated or fully rewritten?

As usual with any problems with ingesting files, the first debugging steps are to run

splunk list monitor

and

splunk list inputstatus

and see if there's something unusual about those files

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...