Greetings All,
I'm indexing a bunch of metrics files written every 10 minutes. Just after midnight I get a file containing the same format metrics, but each value is the sum for the previous day. This totals file I want to ignore (It messes up all sorts of use cases of the metric data). The only way to reliably identify a totals file is that the third line holds a timestamp, and this will be all zero. Any other file will have a normal ISO timestamp in this point
REGEX = ^TimeStamp\s+:\s+0000-00-00\s00.00.00.000
Is there a way to block that file's ingestion based on the content of a single line?
Thanks,
R.
Hi @Random_Walk ,
Then use a script to write those kinds of events in a separate file. And if your are ingesting these files through UF then use Blacklist option to ignore the files without reading.
Hi,
you can drop the events before indexing with the help of below settings.
props.conf
TRANSFORMS-information = eventsDrop
Transforms.conf
[eventsDrop]
REGEX =^TimeStamp\s+:\s+0000-00-00\s00.00.00.000
DEST_KEY = queue
FORMAT = nullQueue
Hi Vardhan,
Thanks for the hint, but unfortunately this only drops the line with the Timstamp. I need to discard the entire file in the case where the file contains this 'flag' timestamp.
I'm thinking it may need to be scripted input, but I'm wondering if there are any other clever tricks.
Thanks,
R.