Help me understand the error logs.

[KishoreSrini]

I am newbie to this env and I'm trying to understand some logs regrading a linux server troubleshoot. A server stopped sending metrics to Splunk (eventlogs are fine). To troubleshoot, I searched the error logs on that time stamp. These are the logs I got,

15:02:44.000: collectd[909]: processmon plugin: Error reading /proc/3605381/stat

15:12:53.000: runsvc.sh[968]: Error reported in diagnostic logs. Please examine the log for more details.

15:12:53.000: runsvc.sh[968]: 2025-06-13 19:12:53Z: Agent connect error: The HTTP request timed out after 00:01:00.. Retrying until reconnected.

15:31:07.000: splunk[3844643]: ERROR - Failed opening "/opt/splunkforwarder/var/log/splunk/splunkd.log": No such file or directory

Please help to understand the issue and troubleshooting steps for the issue(If possible)

Thank you in advance.

[livehybrid]

Hi @KishoreSrini 

I think the collectd and runsvc.sh logs are not Splunk related, these look like they might be associated with VstsAgentService - Is this a VM running on Azure / Azure Pipelines?

Regarding the Splunk error failed to open file - Can you confirm if the file actually exists in the filesystem? And if so, what events are in the splunkd.log? Are there any warnings/errors?

Please could you confirm the ownership on /opt/splunkforwarder/var/log/splunk/splunkd.log and also confirm the user which Splunk is running as:

ps -a | grep -i splunk

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[KishoreSrini]

Hi @livehybrid/@PrewinThomas  ,

Yes, The linux server is a VM running on azure. I am checking the access and availability of the file as mentioned. Will let you know once I'm done. 

The Splunkd event,
06-13-2025 19:30:53.923 +0000 ERROR AggregatorMiningProcessor [3844932 structuredparsing] - Uncaught exception in Aggregator, skipping an event: Can't open DateParser XML configuration file "/opt/splunkforwarder/etc/datetime.xml": No such file or directory - data_source="/opt/splunkforwarder/var/spool/splunk/tracker.log", data_host="-----", data_sourcetype="splunkd_latency_tracker"

06-13-2025 19:28:30.171 +0000 ERROR ExecProcessor [3844925 ExecProcessor] - message from "/opt/splunkforwarder/etc/apps/pwc_west_ghs_uf_nix_v2/bin/package.sh" /bin/sh: 1: /opt/splunkforwarder/etc/apps/pwc_west_ghs_uf_nix_v2/bin/package.sh: not found

06-13-2025 18:28:29.084 +0000 ERROR ExecProcessor [3844925 ExecProcessor] - message from "/opt/splunkforwarder/etc/apps/pwc_west_ghs_uf_nix_v2/bin/hardware.sh" /bin/sh: 1: /opt/splunkforwarder/etc/apps/pwc_west_ghs_uf_nix_v2/bin/hardware.sh: not found

Is possible to narrow down the issue with these events?

Thank you.

[livehybrid]

Thanks, Im wondering if its a permissions issue. The details on what the process is running as and the ownership of the files in /opt/splunkforwarder should help rule it in/out either way! Let me know if you can get hold of this information.

Thanks

[PrewinThomas · Accepted solution]

@KishoreSrini 
Can you check if there is any permission issue? 

collectd: processmon plugin: Error reading /proc/3605381/stat
collectd failed to read process stats, likely because the process with PID 3605381 ended or permissions were insufficient

"/opt/splunkforwarder/var/log/splunk/splunkd.log": No such file or directory - Splunk couldn't access it's main splunkd.log file this also indicates about file unavailablity or permission issue

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!