How can I troubleshoot why suddenly 8 of 10 subfolders with proxy logs have stopped being indexed?

[daniel_augustyn]

I've been sending proxy logs to the FTP server and from there I installed an universal forwarder to send the logs to the Splunk indexers. They are all in a gz format. Everything was working fine until a day when I've noticed that proxy logs stopped getting indexed. There are about 10 subfolders and only 2 of them are still getting indexed, and the rest of the proxy logs had stopped getting indexed on the same day. How should I troubleshoot this?

Not sure why some of the subfolders with gz files (proxy logs from each site) has stopped getting indexed and the rest is still going.

[daniel_augustyn · Accepted solution]

The issue was because I added to much to my stanza for monitoring too many files at once and Splunk basically filled up the buffer. Since proxy logs are in gz format, it took long time for Splunk to catch up. Splunk will need to finish a single gz file before it could move to the next one. I also increased thruput in limints.conf.