topic Re: Help me understand the error logs. in Monitoring Splunk

Help me understand the error logs.

KishoreSrini — Thu, 19 Jun 2025 07:01:51 GMT

I am newbie to this env and I'm trying to understand some logs regrading a linux server troubleshoot. A server stopped sending metrics to Splunk (eventlogs are fine). To troubleshoot, I searched the error logs on that time stamp. These are the logs I got,

15:02:44.000: collectd[909]: processmon plugin: Error reading /proc/3605381/stat

15:12:53.000: runsvc.sh[968]: Error reported in diagnostic logs. Please examine the log for more details.

15:12:53.000: runsvc.sh[968]: 2025-06-13 19:12:53Z: Agent connect error: The HTTP request timed out after 00:01:00.. Retrying until reconnected.

15:31:07.000: splunk[3844643]: ERROR - Failed opening "/opt/splunkforwarder/var/log/splunk/splunkd.log": No such file or directory

Please help to understand the issue and troubleshooting steps for the issue(If possible)

Thank you in advance.

Re: Help me understand the error logs.

PrewinThomas — Thu, 19 Jun 2025 07:45:07 GMT

@KishoreSrini
Can you check if there is any permission issue?

collectd: processmon plugin: Error reading /proc/3605381/stat
collectd failed to read process stats, likely because the process with PID 3605381 ended or permissions were insufficient

"/opt/splunkforwarder/var/log/splunk/splunkd.log": No such file or directory - Splunk couldn't access it's main splunkd.log file this also indicates about file unavailablity or permission issue

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

Re: Help me understand the error logs.

livehybrid — Thu, 19 Jun 2025 07:56:41 GMT

Hi @KishoreSrini

I think the collectd and runsvc.sh logs are not Splunk related, these look like they might be associated with VstsAgentService - Is this a VM running on Azure / Azure Pipelines?

Regarding the Splunk error failed to open file - Can you confirm if the file actually exists in the filesystem? And if so, what events are in the splunkd.log? Are there any warnings/errors?

Please could you confirm the ownership on /opt/splunkforwarder/var/log/splunk/splunkd.log and also confirm the user which Splunk is running as:

ps -a | grep -i splunk

🌟 Did this answer help you? If so, please consider:

Adding karma to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

Re: Help me understand the error logs.

KishoreSrini — Thu, 19 Jun 2025 10:21:45 GMT

Hi @livehybrid/@PrewinThomas ,

Yes, The linux server is a VM running on azure. I am checking the access and availability of the file as mentioned. Will let you know once I'm done.

The Splunkd event,
06-13-2025 19:30:53.923 +0000 ERROR AggregatorMiningProcessor [3844932 structuredparsing] - Uncaught exception in Aggregator, skipping an event: Can't open DateParser XML configuration file "/opt/splunkforwarder/etc/datetime.xml": No such file or directory - data_source="/opt/splunkforwarder/var/spool/splunk/tracker.log", data_host="-----", data_sourcetype="splunkd_latency_tracker"

06-13-2025 19:28:30.171 +0000 ERROR ExecProcessor [3844925 ExecProcessor] - message from "/opt/splunkforwarder/etc/apps/pwc_west_ghs_uf_nix_v2/bin/package.sh" /bin/sh: 1: /opt/splunkforwarder/etc/apps/pwc_west_ghs_uf_nix_v2/bin/package.sh: not found

06-13-2025 18:28:29.084 +0000 ERROR ExecProcessor [3844925 ExecProcessor] - message from "/opt/splunkforwarder/etc/apps/pwc_west_ghs_uf_nix_v2/bin/hardware.sh" /bin/sh: 1: /opt/splunkforwarder/etc/apps/pwc_west_ghs_uf_nix_v2/bin/hardware.sh: not found

Is possible to narrow down the issue with these events?

Thank you.

Re: Help me understand the error logs.

livehybrid — Thu, 19 Jun 2025 11:52:29 GMT

Thanks, Im wondering if its a permissions issue. The details on what the process is running as and the ownership of the files in /opt/splunkforwarder should help rule it in/out either way! Let me know if you can get hold of this information.

Thanks