Deployment Architecture

Splunk server swapping - why ?

konradwawryn
Explorer

Hi,

I have a problem with my server. Since few weeks we have high load on the server. Today I saw for the first time SWAP on the server. The funny thing is that I still have free RAM memory on the server.

alt text

Do You know why Splunk swapping ? Would be great if somebody could support me and tell me how to reduce swap.

Tags (1)
0 Karma

konradwawryn
Explorer

Last days I increased thruput parameter on 2 splunkforwarders:

[thruput]
maxKBps = 4096

From 256kbps to 4096kbps. Do You think that could be a reason of swapping ?

0 Karma

MuS
Legend

have a look at this answer http://splunk-base.splunk.com/answers/78921/splunkd-using-too-much-ram and keep analyzing what splunkd is using swap

0 Karma

konradwawryn
Explorer

I have connected 30 machines with application servers to Forwarder. We had a delay in Splunk indexers. Thats why I extended that parameter.

This swap is strange.

Could it be that Splunk storing data to index in swap memory ?

0 Karma

MuS
Legend

This setting has an impact to the forwarder only, since it tells the forwarder how much data it can send a once. Does the forwarder really send so much data? You can check that in metrics.log or in the deployment App or the S.o.S. App.

Other thing you can do if possible, revert back the change on the forwarder and see what happens.

0 Karma

MuS
Legend

Hi konradwawryn

since swapping is controlled by the OS, you should troubleshot/check your OS settings like swapiness for example. You also have the possibility to set some memory related limits in splunks limits.conf - but be warned, that you also can break stuff very easily there.

I can recall that in some older 4.2.x version there was a bug, causing the OS to use huge amount of swap (like 20 or 30Gb) but this was fixed later.

cheers, MuS

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...