Getting Data In

Strange UF behaviour - NO _internal forwarded!!!

verbal_666
Contributor

Hi.

I have a strange behaviour from about 48h by an UF, a single one.

1) On UF both metrics and splunkd logs events, NO ERRORS! Connections to outputs is OK!
2) UF has not been touched in last 48h, same conf / same addons / same ALL
3) UF has been updated to clean 7.2.0, but problem permains 🤔 rolled back to previous version...
4) All inputs are sent, _internal (metrics.log/splunkd.log) NOT from 48h!!!
5) I still clean log dir on UF from rotated *.? and online metrics and splunkd, and restarted!!! No way!!!
6) Deleted addons, and redeployed. No way!!!

_internal are missing!!!

Any idea?
Thanks.

Labels (1)
0 Karma
1 Solution

verbal_666
Contributor

Gotcha!!!


Maybe server is locked, maybe some log is locked, and loops sending data...


For some reason, server has some fs locked, also an ls locks the terminal sessions!!! 🙄

So SPL UF locks on those paths... blocking all the rest of inputs.conf!!!

View solution in original post

0 Karma

verbal_666
Contributor

Got the problem. And it's even stranger.

An addon, with many many file monitor inputs, is blocking these inputs and also _internal... 🙄

Others inputs in other addons work.

Strange, very strange!

0 Karma

gcusello
Esteemed Legend

Hi @verbal_666,

I found this behavior when the forwarder and/or the network and/or the Indexers are overloaded, so Splunk internal logs are skipped because they have a lower priority.

Check if you have (or had) one of the above problems.

Then check the volume of logs sent by that Forwarder and see if reducing those logs you continue to have the problem.

Ciao.

Giuseppe

0 Karma

verbal_666
Contributor

Done! First action... 🙄

Network is ok, Indexers are ok, i'm receiving datas from other UFs, no problem...

All inputs logs in the issued-UF are less then 50MB...

Also limits thruput is set to 0... no way!!! 48h UF got down, and now has issues... queues are empty!!! 🤔

Tried all workarounds... the only way is deleting inputs that generates the issue...

Maybe server is locked, maybe some log is locked, and loops sending data...

 

I'll get an eye on this host, and see next days...
Thanks 👍

0 Karma

verbal_666
Contributor

Gotcha!!!


Maybe server is locked, maybe some log is locked, and loops sending data...


For some reason, server has some fs locked, also an ls locks the terminal sessions!!! 🙄

So SPL UF locks on those paths... blocking all the rest of inputs.conf!!!

0 Karma
Get Updates on the Splunk Community!

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...