Monitoring Splunk

how can I improve indexing performance of Splunk monitoring huge number of fies?

Takajian
Builder

My application generates around 100,000 files per day. Although I tested to index them by monitoring files, it took almost a week. Anybody know cause of the issue and solution? I think it will be improved if number of active log files were not so many, but I have to do so in our environment.

Tags (1)
0 Karma

sbrant_splunk
Splunk Employee
Splunk Employee

It is possible to run multiple instances of Splunk forwarders. This is particularly easy on *nix systems. With this number of files, you may want to investigate this solution.

0 Karma

sbrant_splunk
Splunk Employee
Splunk Employee

There is no hard limit and it will require some testing to discover the breaking point. This question, looking for the same answer, basically says the same:

http://splunk-base.splunk.com/answers/57806/is-there-a-limit-on-the-number-of-files-a-forwarder-can-...

How are the files being written now? Are they in separate directories? Is there a naming convention?

0 Karma

macnica
Engager

Is there any sizing guideline of number of files that a forwarder can monitor? I would like to make plan how many forwarders is required for my case.

0 Karma