topic Re: Strange UF behaviour - NO _internal forwarded!!! in Getting Data In

Strange UF behaviour - NO _internal forwarded!!!

verbal_666 — Mon, 19 Jul 2021 07:37:28 GMT

Hi.

I have a strange behaviour from about 48h by an UF, a single one.

1) On UF both metrics and splunkd logs events, NO ERRORS! Connections to outputs is OK!
2) UF has not been touched in last 48h, same conf / same addons / same ALL
3) UF has been updated to clean 7.2.0, but problem permains 🤔 rolled back to previous version...
4) All inputs are sent, _internal (metrics.log/splunkd.log) NOT from 48h!!!
5) I still clean log dir on UF from rotated *.? and online metrics and splunkd, and restarted!!! No way!!!
6) Deleted addons, and redeployed. No way!!!

_internal are missing!!!

Any idea?
Thanks.

Re: Strange UF behaviour - NO _internal forwarded!!!

verbal_666 — Mon, 19 Jul 2021 09:23:52 GMT

Got the problem. And it's even stranger.

An addon, with many many file monitor inputs, is blocking these inputs and also _internal... 🙄

Others inputs in other addons work.

Strange, very strange!

Re: Strange UF behaviour - NO _internal forwarded!!!

gcusello — Mon, 19 Jul 2021 09:39:23 GMT

Hi @verbal_666,

I found this behavior when the forwarder and/or the network and/or the Indexers are overloaded, so Splunk internal logs are skipped because they have a lower priority.

Check if you have (or had) one of the above problems.

Then check the volume of logs sent by that Forwarder and see if reducing those logs you continue to have the problem.

Ciao.

Giuseppe

Re: Strange UF behaviour - NO _internal forwarded!!!

verbal_666 — Mon, 19 Jul 2021 10:23:09 GMT

Done! First action... 🙄

Network is ok, Indexers are ok, i'm receiving datas from other UFs, no problem...

All inputs logs in the issued-UF are less then 50MB...

Also limits thruput is set to 0... no way!!! 48h UF got down, and now has issues... queues are empty!!! 🤔

Tried all workarounds... the only way is deleting inputs that generates the issue...

Maybe server is locked, maybe some log is locked, and loops sending data...

I'll get an eye on this host, and see next days...
Thanks 👍

Re: Strange UF behaviour - NO _internal forwarded!!!

verbal_666 — Mon, 19 Jul 2021 19:59:10 GMT

Gotcha!!!

Maybe server is locked, maybe some log is locked, and loops sending data...

For some reason, server has some fs locked, also an ls locks the terminal sessions!!! 🙄

So SPL UF locks on those paths... blocking all the rest of inputs.conf!!!