How to troubleshoot why my heavy forwarder is unable to keep up with processing log data?

[ronj_clark]

I have a heavy forwarder running on a RHEL 6 server that has 16 processors and 16GB. This heavy forwarder has usually kept up with all of the logs that were sent to it, but a few months ago, I am pretty sure I overwhelmed it. Now, I have moved all of the extra logs off of this server to another server and I am back to the original set of logs that I started with. However, it will not keep up.

The logs are Cisco ASA firewall logs, WSA logs, and some other small volume syslogs. In the inputs.conf file, the WSA logs are set to "batch" mode and all of the rest are in monitor mode.

inputs.conf:

####  WSA Logs  ####
[batch:///logs/sawmill]
disabled = 0
# followTail = 0
index = wsa
initCrcLength = 1024
sourcetype = cisco:wsa:squid
whitelist = aclog.*
move_policy = sinkhole
crcSalt = <SOURCE>

Any one having this same issue? Any help is appreciated. I can post snippets of conf files if it will help.

Thank you in advance,
Ron

[esix_splunk]

Your box is still receiving the other logs? Its a bit hard to trouble shoot this without more information. Here are some typical things to start looking at though:

1) System resource utilization [ CPU / Memory / HD (I/o ] - Any of these constrained?
2) Look for errors in _internal, turn up debuging level -- https://wiki.splunk.com/Community:Troubleshooting_Monitor_Inputs
3) Have you enable input parallelization?

You should be able to open a support ticket also and send them a diag, this might provide more information to Support..

[ronj_clark]

esix,

It is version 6.1.1. Yes the box is still receiving other logs.

1. system resource utilization is what I would call normal. 2 of the 16 procs are over 50%, but not above 75%. all other procs are under 40%. Less than half of the 16GB RAM is being used.
2. The only errors in the logs are some time parsing errors, but they are intermittent. Most of the logs are just entries of this box connecting to my index cluster to deliver logs.
3. No, I have not enabled input parallelization. I will look this up.

I cannot open a ticket with splunk as we do not have support contract.

Thanks!