Splunk Enterprise

Splunk Forwarder Performance

smahtha
Engager

We have a "complicated" directory structure for logging. I wanted to understand the impact on performance for Splunk Forwarder for the below case.

Lets say our logs have the below structure /foo/bar*/*/abc/def/ghi*.log and /foo/bar*/*/abc/def/jkl*.log

Now if we use the above wildcard structure for monitoring purpose, Splunk will translate it as
[monitor://foo/]
whitelist: <appropriately set for ghi*.log and jkl*.log files]

This will effectively monitor the directory and all the subdirectories inside /foo/. And as per my understanding, whenever a new files is added or updated anywhere inside /foo/ it will be compared against the whitelist by Splunk forwarder to decide whether to index it or not.

The number of files eligible for logging is not a lot (hardly 3-4). But the number of files that can be created inside /foo/ (all of which are eligible for whitelist comparison) can be in thousands.

Can anyone give a rough estimate of [if understood and known] the impact of this kind of directory structure will have on Splunk forwarder performance.

Tags (1)
0 Karma
1 Solution

smahtha
Engager

We did a performance profling to understand the impact of the "directory structure and number of files eligible for whitelist comparison" on Splunk forwarder performance. The overall idea is this:
a) If a higher level directory is monitored instead of 'n' lower level directories [eg. /foo/ vs (/foo/bar1/123/abc/def/ and /foo/bar2/123/abc/def/ ... 10-times ... /foo/bar10/123/abc/def/)] CPU usage of the forwarder is higher.
b) If the number of files eligible for whitelist comparison is more then also CPU usage is greater.

Now some hard numbers.
We compared monitoring /foo/ vs ~50 /foo/bar1/123/abc/def/. In the case of /foo/ the number of files eligible for monitoring were around ~1000 whereas the number of files eligible for monitoring in the case of 50 /foo/bar1/123/abc/def/ was ~200.

The CPU usage in the first case rose up to around 10% whereas in the second case it was around 2%. The memory footprint remained roughly the same arounf 120MB.

View solution in original post

smahtha
Engager

We did a performance profling to understand the impact of the "directory structure and number of files eligible for whitelist comparison" on Splunk forwarder performance. The overall idea is this:
a) If a higher level directory is monitored instead of 'n' lower level directories [eg. /foo/ vs (/foo/bar1/123/abc/def/ and /foo/bar2/123/abc/def/ ... 10-times ... /foo/bar10/123/abc/def/)] CPU usage of the forwarder is higher.
b) If the number of files eligible for whitelist comparison is more then also CPU usage is greater.

Now some hard numbers.
We compared monitoring /foo/ vs ~50 /foo/bar1/123/abc/def/. In the case of /foo/ the number of files eligible for monitoring were around ~1000 whereas the number of files eligible for monitoring in the case of 50 /foo/bar1/123/abc/def/ was ~200.

The CPU usage in the first case rose up to around 10% whereas in the second case it was around 2%. The memory footprint remained roughly the same arounf 120MB.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...