Getting Data In

New CSV data not being picked up by Splunk

ncorby
New Member

I have set up a Data input in Splunk which allows me to search a series of CSV files conatined within this folder.
Each file is identical in structures and each file represents a particular months worth of data (ie Jan-2014.csv, Feb-2014 etc).

the Data for these files originates from a SQL server Databae and are generated as part of a nightly extract where by the data is returned from the Start of the Month to the current date. Results are then saved in a file that looks something like MyFilename_[CurrentMonth]_[CurrentYear].CSV.

I initially created a series of monthly files that went allthe way back to Jan 2013 and put them into the Input folder. Once this was doen I could search this data effectively...

However....
Subsequent loads where the latest months data is being recreated with updated data are not appearing in my Splunk Searches!!!
I've taken a look at the Latest Months Data File and (using Excel) observed that there is recent data in it...but for some reason Splunk is not picking up any new data since the time of that inital load...
Please help!!!

0 Karma
1 Solution

dmaislin_splunk
Splunk Employee
Splunk Employee

Might be related to the files looking identical at the top of the file. Add this to your inputs.conf. http://docs.splunk.com/Documentation/Splunk/6.0.2/admin/inputsconf

crcSalt = <SOURCE>

crcSalt = <string>
* Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only 
  performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same 
  file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the 
  CRC is based on only the first few lines of the file, it is possible for legitimately different files to have 
  matching CRCs, particularly if they have identical headers.)
* If set, <string> is added to the CRC.
* If set to the literal string <SOURCE> (including the angle brackets), the full directory path to the source file 
  is added to the CRC. This ensures that each file being monitored has a unique CRC.   When crcSalt is invoked, 
  it is usually set to <SOURCE>.
* Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed 
  after it has rolled. 
* Defaults to empty. 

View solution in original post

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

Might be related to the files looking identical at the top of the file. Add this to your inputs.conf. http://docs.splunk.com/Documentation/Splunk/6.0.2/admin/inputsconf

crcSalt = <SOURCE>

crcSalt = <string>
* Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only 
  performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same 
  file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the 
  CRC is based on only the first few lines of the file, it is possible for legitimately different files to have 
  matching CRCs, particularly if they have identical headers.)
* If set, <string> is added to the CRC.
* If set to the literal string <SOURCE> (including the angle brackets), the full directory path to the source file 
  is added to the CRC. This ensures that each file being monitored has a unique CRC.   When crcSalt is invoked, 
  it is usually set to <SOURCE>.
* Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed 
  after it has rolled. 
* Defaults to empty. 
0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

And just add this to the inputs.conf

crcSalt = <SOURCE>

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

Please select the checkbox next to my answer to accept if this solves the issue.

0 Karma

ncorby
New Member

Great Thanks..New to this

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...