Getting Data In

Do not index file based on content

Random_Walk
Path Finder

Greetings All,

I'm indexing a bunch of metrics files written every 10 minutes. Just after midnight I get a file containing the same format metrics, but each value is the sum for the previous day. This totals file I want to ignore (It messes up all sorts of use cases of the metric data). The only way to reliably identify a totals file is that the third line holds a timestamp, and this will be all zero. Any other file will have a normal ISO timestamp in this point

REGEX = ^TimeStamp\s+:\s+0000-00-00\s00.00.00.000

Is there a way to block that file's ingestion based on the content of a single line? 

 

Thanks,

R.

Labels (4)
0 Karma

Vardhan
Contributor

Hi @Random_Walk ,

Then use a script to write those kinds of events in a separate file. And if your are ingesting these files through UF then use Blacklist option to ignore the files without reading.

0 Karma

Vardhan
Contributor

Hi,

you can drop the events before indexing with the help of below settings.

props.conf

TRANSFORMS-information = eventsDrop

Transforms.conf

[eventsDrop]
REGEX =^TimeStamp\s+:\s+0000-00-00\s00.00.00.000
DEST_KEY = queue
FORMAT = nullQueue

0 Karma

Random_Walk
Path Finder

Hi Vardhan,

Thanks for the hint, but unfortunately this only drops the line with the Timstamp. I need to discard the entire file in the case where the file contains this 'flag' timestamp.

I'm thinking it may need to be scripted input, but I'm wondering if there are any other clever tricks.

Thanks,

R.

0 Karma
Get Updates on the Splunk Community!

Leveraging Detections from the Splunk Threat Research Team & Cisco Talos

  Now On Demand  Stay ahead of today’s evolving threats with the combined power of the Splunk Threat Research ...

New in Splunk Observability Cloud: Automated Archiving for Unused Metrics

Automated Archival is a new capability within Metrics Management; which is a robust usage & cost optimization ...

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...