Join the Conversation
- Find Answers
- :
- Splunk Administration
- :
- Getting Data In
- :
- Strange UF behaviour - NO _internal forwarded!!!
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi.
I have a strange behaviour from about 48h by an UF, a single one.
1) On UF both metrics and splunkd logs events, NO ERRORS! Connections to outputs is OK!
2) UF has not been touched in last 48h, same conf / same addons / same ALL
3) UF has been updated to clean 7.2.0, but problem permains 🤔 rolled back to previous version...
4) All inputs are sent, _internal (metrics.log/splunkd.log) NOT from 48h!!!
5) I still clean log dir on UF from rotated *.? and online metrics and splunkd, and restarted!!! No way!!!
6) Deleted addons, and redeployed. No way!!!
_internal are missing!!!
Any idea?
Thanks.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Gotcha!!!
Maybe server is locked, maybe some log is locked, and loops sending data...
For some reason, server has some fs locked, also an ls locks the terminal sessions!!! 🙄
So SPL UF locks on those paths... blocking all the rest of inputs.conf!!!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Got the problem. And it's even stranger.
An addon, with many many file monitor inputs, is blocking these inputs and also _internal... 🙄
Others inputs in other addons work.
Strange, very strange!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @verbal_666,
I found this behavior when the forwarder and/or the network and/or the Indexers are overloaded, so Splunk internal logs are skipped because they have a lower priority.
Check if you have (or had) one of the above problems.
Then check the volume of logs sent by that Forwarder and see if reducing those logs you continue to have the problem.
Ciao.
Giuseppe
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Done! First action... 🙄
Network is ok, Indexers are ok, i'm receiving datas from other UFs, no problem...
All inputs logs in the issued-UF are less then 50MB...
Also limits thruput is set to 0... no way!!! 48h UF got down, and now has issues... queues are empty!!! 🤔
Tried all workarounds... the only way is deleting inputs that generates the issue...
Maybe server is locked, maybe some log is locked, and loops sending data...
I'll get an eye on this host, and see next days...
Thanks 👍
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Gotcha!!!
Maybe server is locked, maybe some log is locked, and loops sending data...
For some reason, server has some fs locked, also an ls locks the terminal sessions!!! 🙄
So SPL UF locks on those paths... blocking all the rest of inputs.conf!!!
Unlock Database Monitoring with Splunk Observability Cloud
Purpose in Action: How Splunk Is Helping Power an Inclusive Future for All
[Upcoming Webinar] Demo Day: Transforming IT Operations with Splunk
