Getting Data In

Can you answer some questions about maxKBps involving replacing a heavy forwarder with a universal forwarder?

TobiasBoone
Communicator

I replaced a very old heavy forwarder today with a universal forwarder that some of our network gear was pointing syslogs too. The flip went smooth but we quickly noticed that the number of logs we were indexing prior to the replacement was well over twice what we were getting afterwards. We found maxKBps and set it to 0 on that universal forwarder which fixed the issue however>

  1. How can I tell if other forwarders may be hitting this upper limit? Should it not show up somewhere in _internal?
  2. How can I use the deployment server to push multiple limits.confs to our many forwarders, or does limits.conf need to live exclusively in /etc/system/local and be edited individually on each? If we find multiple more servers with this limit, I would like to push the fix in bulk.
0 Karma
1 Solution

chrisyounger
SplunkTrust
SplunkTrust

HI @TobiasBoone

Here is the first query which I got out of the "Alerts for Admins" splunkbase app https://splunkbase.splunk.com/app/3796/#/details :

(index=_internal sourcetype=splunkd (source=*splunkd.log OR source=*splunkd.log) "has reached maxKBps. As a result, data forwarding may be throttled") 
| bin _time span=1h 
| stats count as countPerHost by host, _time 
| where (countPerHost > 1)
  1. Yes you can push limits.conf from the deployment server. Just create a folder in etc/deployment-apps containing a local folder that has a limits.conf in it.

All the best

View solution in original post

chrisyounger
SplunkTrust
SplunkTrust

HI @TobiasBoone

Here is the first query which I got out of the "Alerts for Admins" splunkbase app https://splunkbase.splunk.com/app/3796/#/details :

(index=_internal sourcetype=splunkd (source=*splunkd.log OR source=*splunkd.log) "has reached maxKBps. As a result, data forwarding may be throttled") 
| bin _time span=1h 
| stats count as countPerHost by host, _time 
| where (countPerHost > 1)
  1. Yes you can push limits.conf from the deployment server. Just create a folder in etc/deployment-apps containing a local folder that has a limits.conf in it.

All the best

TobiasBoone
Communicator

so treat "local" like an app and assign the clients to it that I want to receive it? Do I run the risk of overwriting legitimate etc/system/local stuff or will it just merge?

0 Karma

chrisyounger
SplunkTrust
SplunkTrust

it will go into the /etc/apps/ folder of the universal forwarders so it effectively merge. Look at other apps you have in the deployment-apps` folder to see the exact structure.

https://docs.splunk.com/Documentation/Splunk/7.2.3/Updating/Createdeploymentapps

0 Karma

TobiasBoone
Communicator

We have dozens of deployment apps in our environment, I was just not clear if limits.conf pushed to an app effectively rolled up to what is documented as a system-wide setting in /local or if it would simply be ignored in an app.

0 Karma

chrisyounger
SplunkTrust
SplunkTrust

Yep its a system wide setting even if its in an app 🙂

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...