Getting Data In

Is the file CRC on a Forwarder unique to the input? Can I change input method through partial ingestion?

mcrawford44
Communicator

We have some customers indexing recovery data from a data outage. These files are 15-30 minutes of logging each. Up to several GB.

Thus far they have been using a standard monitor. But have been pulling files out of the monitor folder. They were "guessing" when Splunk was finished indexing instead of validating with event counts. I have checked, and some of the files were partially ingested.

I want to move them to a batch monitor, but I have questions;

  • Will these files be re-indexed fully, or will they resume based on CRC?
  • If a file has already been fully indexed with the standard monitor, will it be skipped if moved to the batch folder?
  • Is the CRC unique to each input, or can it be used for all inputs at any time?
  • If they will not resume, how would you suggest we remediate the issue without duplicate events?

Thanks in advance!

0 Karma
1 Solution

mcrawford44
Communicator

The answer is;

CRC appear to be unique to a monitor. Moving the files in anyway to a new monitor path will result in the re-indexing of that file. No resumes.

View solution in original post

0 Karma

mcrawford44
Communicator

The answer is;

CRC appear to be unique to a monitor. Moving the files in anyway to a new monitor path will result in the re-indexing of that file. No resumes.

0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!

Review:





Or Learn More in Our Blog >>