topic How do I resolve *nix devices show status "unstable"/no data shown? in Splunk ITSI

How do I resolve *nix devices show status "unstable"/no data shown?

bigll — Mon, 27 Mar 2023 17:52:48 GMT

Hi.

I see rather strange (for me, newbie) issue with number of *nix devices.
After the UF agent install devices reported data for couple of days, but showed status "unstable".
A day later devices stopped updating in Splunk.
On devices I found an error message.

splunk.service - SYSV: Splunk indexer service

Loaded: loaded (/etc/rc.d/init.d/splunk; bad; vendor preset: disabled)

Active: inactive (dead)

Docs: man:systemd-sysv-generator(8)

Warning: splunk.service changed on disk. Run 'systemctl daemon-reload' to reload
--------------
I found that some people experienced similar issue and fixed with update of init.d script
-------------

splunk_start() {

echo Starting Splunk...

ulimit -Hn 20240

ulimit -Sn 10240
----------------
I implemented the proposed change and it did help for few days.
Now I see devices being updated in Splunk on regular bases, but reported as "unstable" and no CPU/MEMORY/DISK data being reported.

Thank you in advance

Re: How do I resolve *nix devices show status "unstable"/no data shown?

srauhala_splunk — Tue, 28 Mar 2023 11:07:34 GMT

Hi!

Is the problem that the affected nodes stops forwarding data? or that they are flagged as unsatable?

Check if the affected nodes have the splunk UF running or not. To exclude whether the issue is due to the Splunk process not running or whether there is a data forwarding issue (firewall / network / ports etc).

If the Splunk process is not started when a server is rebooted have a look here on how to enable splunk boot start https://docs.splunk.com/Documentation/Splunk/9.0.4/Admin/ConfigureSplunktostartatboottime.

Also if these are Linux hosts note that there is a slightly different setup if the boot on start should be enabled with systemd: See https://docs.splunk.com/Documentation/Splunk/9.0.4/Admin/RunSplunkassystemdservice#Configure_systemd_using_enable_boot-start

/Seb

Re: How do I resolve *nix devices show status "unstable"/no data shown?

bigll — Tue, 28 Mar 2023 12:44:12 GMT

Thank you for the reply.

UF agents starts after reboot with no issues
It runs for a while (day, day and a half)
During that time utilization of all monitored resources are normal.
When it stopped I see messages in logs [242543 TcpOutEloop] - Cooked connection to ip=x.x.x.x:9997 timed out.

source = /opt/splunkforwarder/var/log/splunk/splunkd.log

Sounds like something preventing connection.
I wonder if it size of the queue.

Re: How do I resolve *nix devices show status "unstable"/no data shown?

srauhala_splunk — Tue, 28 Mar 2023 14:00:33 GMT

Hi!

Cooked connection to ip=x.x.x.x:9997 timed out. That would be the connection to the indexer the UF is forwarding too. Is it not attempting to connect to the next indexer in the cluster?

Yes you might be onto something if indexers are at their limit the ingestion queues will fill up and eventually data from UFs will be blocked or at least delayed until the indexer can work. through the queues.

Use the monitoring console to check if the indexing queues are getting blocked could be a good start.

/Seb

Re: How do I resolve *nix devices show status "unstable"/no data shown?

bigll — Wed, 29 Mar 2023 12:03:20 GMT

Actually both, Not getting any data is bigger issue. but else it's important to understand the nature (and fix) an issue with client being "unstable"

topic How do I resolve *nix devices show status &quot;unstable&quot;/no data shown? in Splunk ITSI

How do I resolve *nix devices show status "unstable"/no data shown?

Re: How do I resolve *nix devices show status "unstable"/no data shown?

Re: How do I resolve *nix devices show status "unstable"/no data shown?

Re: How do I resolve *nix devices show status "unstable"/no data shown?

Re: How do I resolve *nix devices show status "unstable"/no data shown?

topic How do I resolve *nix devices show status "unstable"/no data shown? in Splunk ITSI