Can you answer some questions about maxKBps involving replacing a heavy forwarder with a universal forwarder?

[TobiasBoone]

I replaced a very old heavy forwarder today with a universal forwarder that some of our network gear was pointing syslogs too. The flip went smooth but we quickly noticed that the number of logs we were indexing prior to the replacement was well over twice what we were getting afterwards. We found maxKBps and set it to 0 on that universal forwarder which fixed the issue however>

1. How can I tell if other forwarders may be hitting this upper limit? Should it not show up somewhere in _internal?
2. How can I use the deployment server to push multiple limits.confs to our many forwarders, or does limits.conf need to live exclusively in /etc/system/local and be edited individually on each? If we find multiple more servers with this limit, I would like to push the fix in bulk.

[chrisyounger · Accepted solution]

HI @TobiasBoone

Here is the first query which I got out of the "Alerts for Admins" splunkbase app https://splunkbase.splunk.com/app/3796/#/details :

(index=_internal sourcetype=splunkd (source=*splunkd.log OR source=*splunkd.log) "has reached maxKBps. As a result, data forwarding may be throttled") 
| bin _time span=1h 
| stats count as countPerHost by host, _time 
| where (countPerHost > 1)

1. Yes you can push limits.conf from the deployment server. Just create a folder in etc/deployment-apps containing a local folder that has a limits.conf in it.

All the best

[TobiasBoone]

so treat "local" like an app and assign the clients to it that I want to receive it? Do I run the risk of overwriting legitimate etc/system/local stuff or will it just merge?

[chrisyounger]

it will go into the /etc/apps/ folder of the universal forwarders so it effectively merge. Look at other apps you have in the deployment-apps` folder to see the exact structure.

https://docs.splunk.com/Documentation/Splunk/7.2.3/Updating/Createdeploymentapps

[TobiasBoone]

We have dozens of deployment apps in our environment, I was just not clear if limits.conf pushed to an app effectively rolled up to what is documented as a system-wide setting in /local or if it would simply be ignored in an app.

[chrisyounger]

Yep its a system wide setting even if its in an app 🙂