Hello everybody.
I have a problem with monitoring multiple files in a Heavy Forwarder. I mounted a folder with sshfs and I monitoring transactions files but the software always create a file per transaction, and I receive almost 3000 files per day. The HF does not have problem monitoring files from 1 day (3000 files) but when I start receiving files from the second day (6000 files) the HF cannot handle it, I executed a "ls" command and I never get a response from the server, when something like this happens, the customer always rename the folder and the HF start working again so I decided to split those files in different directories per day or maybe per hour but I'm not sure if the HF forwarder can handle it without problems.
Does anyone can tell me, based on their experience, what is the best way to monitoring those files?
Is it a good idea to split the data in different directories? Is Splunk not going to have any problem monitoring different directories?
I read this thread and it helped me to monitoring multiples files but I did not see something about monitoring too many files.
https://answers.splunk.com/answers/220025/what-is-recommended-to-monitor-multiple-files-in-t.html?utm_source=typeahead&utm_medium=newquestion&utm_campaign=no_votes_sort_relev
Any help will be appreciate.
This is an example of my inputs.conf file:
## ECC Events
monitor:///home/splunk/sshfslogs/XXX.XXX.XXX.XXX/ecc/splunk]
disabled = 0
sourcetype = mysourcetype_ecc
index = my_index
host_segment = 4
## PO Events
[monitor:///home/splunk/sshfslogs/XXX.XXX.XXX.XXX/po/splunk]
disabled = 0
sourcetype = mysourcetype_po
index = my_index
host_segment = 4
... View more