Getting Data In

Is the file CRC on a Forwarder unique to the input? Can I change input method through partial ingestion?

mcrawford44
Communicator

We have some customers indexing recovery data from a data outage. These files are 15-30 minutes of logging each. Up to several GB.

Thus far they have been using a standard monitor. But have been pulling files out of the monitor folder. They were "guessing" when Splunk was finished indexing instead of validating with event counts. I have checked, and some of the files were partially ingested.

I want to move them to a batch monitor, but I have questions;

  • Will these files be re-indexed fully, or will they resume based on CRC?
  • If a file has already been fully indexed with the standard monitor, will it be skipped if moved to the batch folder?
  • Is the CRC unique to each input, or can it be used for all inputs at any time?
  • If they will not resume, how would you suggest we remediate the issue without duplicate events?

Thanks in advance!

0 Karma
1 Solution

mcrawford44
Communicator

The answer is;

CRC appear to be unique to a monitor. Moving the files in anyway to a new monitor path will result in the re-indexing of that file. No resumes.

View solution in original post

0 Karma

mcrawford44
Communicator

The answer is;

CRC appear to be unique to a monitor. Moving the files in anyway to a new monitor path will result in the re-indexing of that file. No resumes.

0 Karma
Get Updates on the Splunk Community!

Observability | How to Think About Instrumentation Overhead (White Paper)

Novice observability practitioners are often overly obsessed with performance. They might approach ...

Cloud Platform | Get Resiliency in the Cloud Event (Register Now!)

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud  Today many enterprises ...

The Great Resilience Quest: 10th Leaderboard Update

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out >> As our brave ...