High CPU Usage from Splunk UF Linux Add-On Scripts on VM – lshw Impact

[Keigo]

We are running Splunk Universal Forwarder on a virtual machine and using the Splunk Add-on for Unix and Linux.
The VM is configured with 2 vCPUs and 4GB of RAM.

During metric collection, it appears that the hardware.sh script executes the lshw command, which causes a temporary CPU spike of around 20–40%.
Since these scripts run periodically, this behavior may impact performance, especially on resource-constrained VMs.

I would appreciate any insights or experiences regarding the following:

・Recommended VM specifications for running the Linux Add-On
・Ways to reduce CPU load caused by lshw or other scripts
・Is this kind of CPU spike expected behavior for the Add-On?
・Any operational tips or configuration examples to mitigate the impact
Thanks in advance for your help!

[Keigo · Accepted solution]

In the end, we decided not to collect metrics directly from the servers, but instead to send the data already being collected by Cacti to Splunk using HEC. Many thanks to everyone who provided their input.

[thahir]

Hi @Keigo,

You’re using Splunk Universal Forwarder with the Linux Add-on on a 2 vCPU / 4GB RAM VM.
A script (hardware.sh) that runs lshw causes 20–40% CPU spikes, which may impact performance. lshw is not light weight and is overkill most of the use cases, and thhis behavior is expected because lshw is resource-heavy, especially on low-spec machines.

Below are the recommendations

1. Check if you actually need hardware data. 

2. If needed, reduce the frequency to minimize impact

3. Alternative: Run the script via cron during off-peak hours and monitor its output file with Splunk.

4. Use lightweight tools like CollectD for performance metrics instead of heavy scripts.

5. Recommended specs (if you keep such scripts):

    Use 4 vCPUs and 6–8 GB RAM for better performance.

[PickleRick]

The addon for *nix contains several inputs. Some of them are more useful, some less...

The question is why would you run this input in the first place. Is this your only source of HW inventory? And even then - this is something that doesn't change often so the interval between subsequent runs can be quite big without any significant impact to the usefulness of the output data.

[livehybrid]

Hi @Keigo 

The hardware specs for a Splunk UF are Dual-core 1.5GHz+ processor, 1GB+ RAM which you are sufficiently covering here, and there arent specific requirements for higher hardware specs when using the Linux Add-on.

In relation to your other 3 questions, lshw collects deep hardware information which is inherently compute-heavy and thus will cause a bit of a spike on lower resourced systems which might go un-noticed on higher spec'd servers.

My main question is, are you using the information that this provides, and if so does it need to be run at a regular interval?

Like @PrewinThomas said, you could reduce the frequency but you will ultimately still see the spike when it does run, but I would double check that the data is actually being used (often I see users enable a bunch of Linux TA inputs which go unused!). If you do use it then reducing the frequency is the only option. 

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[PrewinThomas]

@Keigo 

Normally production environments use 2–4 vCPUs and 4–8GB RAM to provide enough headroom, but it depends on your server roles/usage...

Ways to Reduce CPU Load
-You can modify inputs.conf to increase the interval for hardware.sh or disable it entirely if hardware inventory isn’t critical
-Run/Schedule scripts during off-peak hours
-Exclude unnecessary scripts (set disabled = 1 for any stanza you don’t need) in inputs.conf
-If you need continuous performance data, CollectD is more efficient i would say and integrates well with Splunk.
Ref#https://docs.splunk.com/Documentation/AddOns/released/Linux/Hardwareandsoftwarerequirements

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!