Getting Data In

Can you answer some questions about maxKBps involving replacing a heavy forwarder with a universal forwarder?

TobiasBoone
Communicator

I replaced a very old heavy forwarder today with a universal forwarder that some of our network gear was pointing syslogs too. The flip went smooth but we quickly noticed that the number of logs we were indexing prior to the replacement was well over twice what we were getting afterwards. We found maxKBps and set it to 0 on that universal forwarder which fixed the issue however>

  1. How can I tell if other forwarders may be hitting this upper limit? Should it not show up somewhere in _internal?
  2. How can I use the deployment server to push multiple limits.confs to our many forwarders, or does limits.conf need to live exclusively in /etc/system/local and be edited individually on each? If we find multiple more servers with this limit, I would like to push the fix in bulk.
0 Karma
1 Solution

chrisyounger
SplunkTrust
SplunkTrust

HI @TobiasBoone

Here is the first query which I got out of the "Alerts for Admins" splunkbase app https://splunkbase.splunk.com/app/3796/#/details :

(index=_internal sourcetype=splunkd (source=*splunkd.log OR source=*splunkd.log) "has reached maxKBps. As a result, data forwarding may be throttled") 
| bin _time span=1h 
| stats count as countPerHost by host, _time 
| where (countPerHost > 1)
  1. Yes you can push limits.conf from the deployment server. Just create a folder in etc/deployment-apps containing a local folder that has a limits.conf in it.

All the best

View solution in original post

chrisyounger
SplunkTrust
SplunkTrust

HI @TobiasBoone

Here is the first query which I got out of the "Alerts for Admins" splunkbase app https://splunkbase.splunk.com/app/3796/#/details :

(index=_internal sourcetype=splunkd (source=*splunkd.log OR source=*splunkd.log) "has reached maxKBps. As a result, data forwarding may be throttled") 
| bin _time span=1h 
| stats count as countPerHost by host, _time 
| where (countPerHost > 1)
  1. Yes you can push limits.conf from the deployment server. Just create a folder in etc/deployment-apps containing a local folder that has a limits.conf in it.

All the best

TobiasBoone
Communicator

so treat "local" like an app and assign the clients to it that I want to receive it? Do I run the risk of overwriting legitimate etc/system/local stuff or will it just merge?

0 Karma

chrisyounger
SplunkTrust
SplunkTrust

it will go into the /etc/apps/ folder of the universal forwarders so it effectively merge. Look at other apps you have in the deployment-apps` folder to see the exact structure.

https://docs.splunk.com/Documentation/Splunk/7.2.3/Updating/Createdeploymentapps

0 Karma

TobiasBoone
Communicator

We have dozens of deployment apps in our environment, I was just not clear if limits.conf pushed to an app effectively rolled up to what is documented as a system-wide setting in /local or if it would simply be ignored in an app.

0 Karma

chrisyounger
SplunkTrust
SplunkTrust

Yep its a system wide setting even if its in an app 🙂

0 Karma
Get Updates on the Splunk Community!

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...

Thank You for Celebrating CX Day with Splunk!

Yesterday the entire team at Splunk + Cisco joined the global celebration of CX Day - celebrating our ...