We are monitoring through HF a directory through 'batch' input.
The directory contains 100s of zip files, each zip contains 1000s of log files.
We have a requirement to write down each log file separately, after it was being forwarded.
1. We tried to use splunkd.log in the HF
The component "ArchiveProcessor" does not write each log file within the zip.
The component "Metrics" holds stastics of top 10 every 30 seconds, so it does not write every file.
Can we change the 'Metrics' paremeters, to write top X and not top 10?
Is there any other component we can use, even in DEBUG mode, to get that information?
2. We tried to approach this through the index itself in the indexer.
The log files timestamp is not close to 'now', but is spreaded across many years, so it is very slow and not efficient to search them through the index.
For example: (| tstats count where earliest=0 latest=now index=myindex by source)
