Getting Data In

Strange UF behaviour - NO _internal forwarded!!!

verbal_666
Communicator

Hi.

I have a strange behaviour from about 48h by an UF, a single one.

1) On UF both metrics and splunkd logs events, NO ERRORS! Connections to outputs is OK!
2) UF has not been touched in last 48h, same conf / same addons / same ALL
3) UF has been updated to clean 7.2.0, but problem permains 🤔 rolled back to previous version...
4) All inputs are sent, _internal (metrics.log/splunkd.log) NOT from 48h!!!
5) I still clean log dir on UF from rotated *.? and online metrics and splunkd, and restarted!!! No way!!!
6) Deleted addons, and redeployed. No way!!!

_internal are missing!!!

Any idea?
Thanks.

Labels (1)
0 Karma
1 Solution

verbal_666
Communicator

Gotcha!!!


Maybe server is locked, maybe some log is locked, and loops sending data...


For some reason, server has some fs locked, also an ls locks the terminal sessions!!! 🙄

So SPL UF locks on those paths... blocking all the rest of inputs.conf!!!

View solution in original post

0 Karma

verbal_666
Communicator

Got the problem. And it's even stranger.

An addon, with many many file monitor inputs, is blocking these inputs and also _internal... 🙄

Others inputs in other addons work.

Strange, very strange!

0 Karma

gcusello
Legend

Hi @verbal_666,

I found this behavior when the forwarder and/or the network and/or the Indexers are overloaded, so Splunk internal logs are skipped because they have a lower priority.

Check if you have (or had) one of the above problems.

Then check the volume of logs sent by that Forwarder and see if reducing those logs you continue to have the problem.

Ciao.

Giuseppe

0 Karma

verbal_666
Communicator

Done! First action... 🙄

Network is ok, Indexers are ok, i'm receiving datas from other UFs, no problem...

All inputs logs in the issued-UF are less then 50MB...

Also limits thruput is set to 0... no way!!! 48h UF got down, and now has issues... queues are empty!!! 🤔

Tried all workarounds... the only way is deleting inputs that generates the issue...

Maybe server is locked, maybe some log is locked, and loops sending data...

 

I'll get an eye on this host, and see next days...
Thanks 👍

0 Karma

verbal_666
Communicator

Gotcha!!!


Maybe server is locked, maybe some log is locked, and loops sending data...


For some reason, server has some fs locked, also an ls locks the terminal sessions!!! 🙄

So SPL UF locks on those paths... blocking all the rest of inputs.conf!!!

View solution in original post

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!