Splunk server swapping - why ?

[konradwawryn]

Hi,

I have a problem with my server. Since few weeks we have high load on the server. Today I saw for the first time SWAP on the server. The funny thing is that I still have free RAM memory on the server.

Do You know why Splunk swapping ? Would be great if somebody could support me and tell me how to reduce swap.

[konradwawryn]

Last days I increased thruput parameter on 2 splunkforwarders:

[thruput]
maxKBps = 4096

From 256kbps to 4096kbps. Do You think that could be a reason of swapping ?

[MuS]

have a look at this answer http://splunk-base.splunk.com/answers/78921/splunkd-using-too-much-ram and keep analyzing what splunkd is using swap

[konradwawryn]

I have connected 30 machines with application servers to Forwarder. We had a delay in Splunk indexers. Thats why I extended that parameter.

This swap is strange.

Could it be that Splunk storing data to index in swap memory ?

[MuS]

This setting has an impact to the forwarder only, since it tells the forwarder how much data it can send a once. Does the forwarder really send so much data? You can check that in metrics.log or in the deployment App or the S.o.S. App.

Other thing you can do if possible, revert back the change on the forwarder and see what happens.

[MuS]

Hi konradwawryn

since swapping is controlled by the OS, you should troubleshot/check your OS settings like swapiness for example. You also have the possibility to set some memory related limits in splunks limits.conf - but be warned, that you also can break stuff very easily there.

I can recall that in some older 4.2.x version there was a bug, causing the OS to use huge amount of swap (like 20 or 30Gb) but this was fixed later.

cheers, MuS