Getting Data In

New CSV data not being picked up by Splunk

ncorby
New Member

I have set up a Data input in Splunk which allows me to search a series of CSV files conatined within this folder.
Each file is identical in structures and each file represents a particular months worth of data (ie Jan-2014.csv, Feb-2014 etc).

the Data for these files originates from a SQL server Databae and are generated as part of a nightly extract where by the data is returned from the Start of the Month to the current date. Results are then saved in a file that looks something like MyFilename_[CurrentMonth]_[CurrentYear].CSV.

I initially created a series of monthly files that went allthe way back to Jan 2013 and put them into the Input folder. Once this was doen I could search this data effectively...

However....
Subsequent loads where the latest months data is being recreated with updated data are not appearing in my Splunk Searches!!!
I've taken a look at the Latest Months Data File and (using Excel) observed that there is recent data in it...but for some reason Splunk is not picking up any new data since the time of that inital load...
Please help!!!

0 Karma
1 Solution

dmaislin_splunk
Splunk Employee
Splunk Employee

Might be related to the files looking identical at the top of the file. Add this to your inputs.conf. http://docs.splunk.com/Documentation/Splunk/6.0.2/admin/inputsconf

crcSalt = <SOURCE>

crcSalt = <string>
* Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only 
  performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same 
  file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the 
  CRC is based on only the first few lines of the file, it is possible for legitimately different files to have 
  matching CRCs, particularly if they have identical headers.)
* If set, <string> is added to the CRC.
* If set to the literal string <SOURCE> (including the angle brackets), the full directory path to the source file 
  is added to the CRC. This ensures that each file being monitored has a unique CRC.   When crcSalt is invoked, 
  it is usually set to <SOURCE>.
* Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed 
  after it has rolled. 
* Defaults to empty. 

View solution in original post

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

Might be related to the files looking identical at the top of the file. Add this to your inputs.conf. http://docs.splunk.com/Documentation/Splunk/6.0.2/admin/inputsconf

crcSalt = <SOURCE>

crcSalt = <string>
* Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only 
  performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same 
  file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the 
  CRC is based on only the first few lines of the file, it is possible for legitimately different files to have 
  matching CRCs, particularly if they have identical headers.)
* If set, <string> is added to the CRC.
* If set to the literal string <SOURCE> (including the angle brackets), the full directory path to the source file 
  is added to the CRC. This ensures that each file being monitored has a unique CRC.   When crcSalt is invoked, 
  it is usually set to <SOURCE>.
* Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed 
  after it has rolled. 
* Defaults to empty. 
0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

And just add this to the inputs.conf

crcSalt = <SOURCE>

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

Please select the checkbox next to my answer to accept if this solves the issue.

0 Karma

ncorby
New Member

Great Thanks..New to this

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...