Getting Data In

Strange UF behaviour - NO _internal forwarded!!!

verbal_666
Contributor

Hi.

I have a strange behaviour from about 48h by an UF, a single one.

1) On UF both metrics and splunkd logs events, NO ERRORS! Connections to outputs is OK!
2) UF has not been touched in last 48h, same conf / same addons / same ALL
3) UF has been updated to clean 7.2.0, but problem permains 🤔 rolled back to previous version...
4) All inputs are sent, _internal (metrics.log/splunkd.log) NOT from 48h!!!
5) I still clean log dir on UF from rotated *.? and online metrics and splunkd, and restarted!!! No way!!!
6) Deleted addons, and redeployed. No way!!!

_internal are missing!!!

Any idea?
Thanks.

Labels (1)
0 Karma
1 Solution

verbal_666
Contributor

Gotcha!!!


Maybe server is locked, maybe some log is locked, and loops sending data...


For some reason, server has some fs locked, also an ls locks the terminal sessions!!! 🙄

So SPL UF locks on those paths... blocking all the rest of inputs.conf!!!

View solution in original post

0 Karma

verbal_666
Contributor

Got the problem. And it's even stranger.

An addon, with many many file monitor inputs, is blocking these inputs and also _internal... 🙄

Others inputs in other addons work.

Strange, very strange!

0 Karma

gcusello
Esteemed Legend

Hi @verbal_666,

I found this behavior when the forwarder and/or the network and/or the Indexers are overloaded, so Splunk internal logs are skipped because they have a lower priority.

Check if you have (or had) one of the above problems.

Then check the volume of logs sent by that Forwarder and see if reducing those logs you continue to have the problem.

Ciao.

Giuseppe

0 Karma

verbal_666
Contributor

Done! First action... 🙄

Network is ok, Indexers are ok, i'm receiving datas from other UFs, no problem...

All inputs logs in the issued-UF are less then 50MB...

Also limits thruput is set to 0... no way!!! 48h UF got down, and now has issues... queues are empty!!! 🤔

Tried all workarounds... the only way is deleting inputs that generates the issue...

Maybe server is locked, maybe some log is locked, and loops sending data...

 

I'll get an eye on this host, and see next days...
Thanks 👍

0 Karma

verbal_666
Contributor

Gotcha!!!


Maybe server is locked, maybe some log is locked, and loops sending data...


For some reason, server has some fs locked, also an ls locks the terminal sessions!!! 🙄

So SPL UF locks on those paths... blocking all the rest of inputs.conf!!!

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Updates (ESCU) - New Releases

In the last month, the Splunk Threat Research Team (STRT) has had 3 releases of new content via the Enterprise ...

Thought Leaders are Validating Your Hard Work and Training Rigor

As a Splunk enthusiast and member of the Splunk Community, you are one of thousands who recognize the value of ...

.conf23 Registration is Now Open!

Time to toss the .conf-etti 🎉 —  .conf23 registration is open!   Join us in Las Vegas July 17-20 for ...