Getting Data In

Is the file CRC on a Forwarder unique to the input? Can I change input method through partial ingestion?

mcrawford44
Communicator

We have some customers indexing recovery data from a data outage. These files are 15-30 minutes of logging each. Up to several GB.

Thus far they have been using a standard monitor. But have been pulling files out of the monitor folder. They were "guessing" when Splunk was finished indexing instead of validating with event counts. I have checked, and some of the files were partially ingested.

I want to move them to a batch monitor, but I have questions;

  • Will these files be re-indexed fully, or will they resume based on CRC?
  • If a file has already been fully indexed with the standard monitor, will it be skipped if moved to the batch folder?
  • Is the CRC unique to each input, or can it be used for all inputs at any time?
  • If they will not resume, how would you suggest we remediate the issue without duplicate events?

Thanks in advance!

0 Karma
1 Solution

mcrawford44
Communicator

The answer is;

CRC appear to be unique to a monitor. Moving the files in anyway to a new monitor path will result in the re-indexing of that file. No resumes.

View solution in original post

0 Karma

mcrawford44
Communicator

The answer is;

CRC appear to be unique to a monitor. Moving the files in anyway to a new monitor path will result in the re-indexing of that file. No resumes.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In January, the Splunk Threat Research Team had one release of new security content via the Splunk ES Content ...

Expert Tips from Splunk Professional Services, Ensuring Compliance, and More New ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Observability Release Update: AI Assistant, AppD + Observability Cloud Integrations & ...

This month’s releases across the Splunk Observability portfolio deliver earlier detection and faster ...