Getting Data In

Why is Splunk not capturing new content in a CSV file input and indexing it?

bosch_softtec
Path Finder

Hi,

Splunk 6.5.0

I have the scenario that I have to import every hour a csv (File A) file from a system which has no network access/connection to the Splunk environment. I've created a csv (File B) file on the Splunk indexer which is defined as data input.

Every hour there is running a process which cuts the contend from File A and writes it to File B. Mostly Splunk is indexing the content but sometimes it happens, that File B is filled with the new content but Splunk doesn't index it.

Has somebody an idea why that happens?

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Does the file have a large header that is the same in every instance? Splunk will look at the first 256 bytes to determine if it's already indexed the file and refuse to ingest it again. Consider changing the crcSalt or initCrcLength settings in inputs.conf.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

Does the file have a large header that is the same in every instance? Splunk will look at the first 256 bytes to determine if it's already indexed the file and refuse to ingest it again. Consider changing the crcSalt or initCrcLength settings in inputs.conf.

---
If this reply helps you, Karma would be appreciated.

bosch_softtec
Path Finder

Hi, it could happen that there is no change in the header or first 256 bytes.
Regarding your hint, i'll change the size of the initCrcLength to a higher value and check if my problem has gone.

0 Karma
Get Updates on the Splunk Community!

Fall Into Learning with New Splunk Education Courses

Every month, Splunk Education releases new courses to help you branch out, strengthen your data science roots, ...

Super Optimize your Splunk Stats Searches: Unlocking the Power of tstats, TERM, and ...

By Martin Hettervik, Senior Consultant and Team Leader at Accelerate at Iver, Splunk MVPThe stats command is ...

How Splunk Observability Cloud Prevented a Major Payment Crisis in Minutes

Your bank's payment processing system is humming along during a busy afternoon, handling millions in hourly ...