Are you a member of the Splunk Community?
- Find Answers
- :
- Splunk Administration
- :
- Getting Data In
- :
- Re: UF config works on CentOS/RedHat, but not on U...
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Looking for suggestions for the obvious that I might have overlooked as to why a UF config distributed by Deployment Server (and known to reach all endpoints) works on CentOS and RedHat, but despite the fact the fact the UF on Ubuntu is definitely communicating with the target indexer (as evidenced by tcpdump) it only seems send occasional keep-alives, and no logs.
The config includes a monitor for /var/log, so despite being different platforms, there should be some activity on all of them.
I have been scratching my head over this for two weeks, now.
Later edit -
I omitted to mention that in the splunkd log I see (only once):
TailingProcessor - Could not send data to output queue (parsingQueue), retrying...
TcpOutputProc - Connected to idx=x.x.x.x:yyyy
The indexer address and port are correct.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The difference between the machines is that the Ubuntu ones have - in most cases - had the default SplunkUniversalForwarder app bundle deleted. It seems that the handful of newer ones (which have not) are working fine. So the obvious conclusion is that there is something critical missing from the tailored configurations which is present in the UF config installed by default.
Now to find out what, and determine how to fix it without being reliant on the default configuration.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The difference between the machines is that the Ubuntu ones have - in most cases - had the default SplunkUniversalForwarder app bundle deleted. It seems that the handful of newer ones (which have not) are working fine. So the obvious conclusion is that there is something critical missing from the tailored configurations which is present in the UF config installed by default.
Now to find out what, and determine how to fix it without being reliant on the default configuration.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Are you seeing any data forwarded into _internal on your indexers?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Something has come to light. Some more machines I added yesterday ARE reporting in.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Is the splunkd process for the UF running as the "root" user on all systems or is it running as a limited "splunk" or other user on some or all? If running as a limited user, are there permission differences on /var/log and the files within? (this may manifest and you may see "Permission denied" messages in the _internal index. ($SPLUNK_HOME/var/log/splunk/splunkd.log)
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This is actually one of my major bug-bears about Splunk - that it assumes root privelege by default, rather than spawning a root capable agent for parsing local system files, and privelege-dropping everything else.
Nope. Running as root across the board.
Tech Talk Recap | Mastering Threat Hunting
Observability for AI Applications: Troubleshooting Latency
Splunk AI Assistant for SPL vs. ChatGPT: Which One is Better?
