Not getting data from universal forwarder (ubuntu)

[kate]

Not getting data from universal forwarder (ubuntu).

1) Installed Splunk UF version 9.2.0  and credential package from splunk cloud as it should be reporting to splunk cloud. 
2)There are no error logs in splunkd.log and no metric log in internal splunk index in splunk cloud.
3) Port connectivity 9997 is working fine.

The only logs received in splunkd.log is 

02-16-2024 15:53:30.843 +0000 INFO WatchedFile [156345 tailreader0] - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/search_messages.log'.
02-16-2024 15:53:30.852 +0000 INFO WatchedFile [156345 tailreader0] - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log'.
02-16-2024 15:53:30.859 +0000 INFO WatchedFile [156345 tailreader0] - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/btool.log'.
02-16-2024 15:53:30.876 +0000 INFO WatchedFile [156345 tailreader0] - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/mergebuckets.log'.
02-16-2024 15:53:30.885 +0000 INFO WatchedFile [156345 tailreader0] - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/wlm_monitor.log'.
02-16-2024 15:53:30.891 +0000 INFO WatchedFile [156345 tailreader0] - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage_summary.log'.
02-16-2024 15:53:30.898 +0000 INFO WatchedFile [156345 tailreader0] - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/searchhistory.log'.
02-16-2024 15:53:30.907 +0000 INFO WatchedFile [156345 tailreader0] - Will begin reading at offset=2859 for file='/opt/splunkforwarder/var/log/watchdog/watchdog.log'.
02-16-2024 15:53:31.112 +0000 INFO AutoLoadBalancedConnectionStrategy [156338 TcpOutEloop] - Connected to idx=1.2.3.4:9997:2, pset=0, reuse=0. autoBatch=1
02-16-2024 15:53:31.112 +0000 WARN AutoLoadBalancedConnectionStrategy [156338 TcpOutEloop] - Current dest host connection 1.2.3.4:9997, oneTimeClient=0, _events.size()=0, _refCount=1, _waitingAckQ.size()=0, _supportsACK=0, _lastHBRecvTime=Fri Feb 16 15:53:31 2024 is using 18446604251980134224 bytes. Total tcpout queue size is 512000. Warningcount=1
02-16-2024 15:54:00.446 +0000 INFO ScheduledViewsReaper [156309 DispatchReaper] - Scheduled views reaper run complete. Reaped count=0 scheduled views
02-16-2024 15:54:00.446 +0000 INFO CascadingReplicationManager [156309 DispatchReaper] - Using value for property max_replication_threads=2.
02-16-2024 15:54:00.446 +0000 INFO CascadingReplicationManager [156309 DispatchReaper] - Using value for property max_replication_jobs=5.
02-16-2024 15:54:00.447 +0000 WARN AutoLoadBalancedConnectionStrategy [156338 TcpOutEloop] - Current dest host connection 1.2.3.4:9997, oneTimeClient=0, _events.size()=0, _refCount=1, _waitingAckQ.size()=0, _supportsACK=0, _lastHBRecvTime=Fri Feb 16 15:53:31 2024 is using 18446604251980134224 bytes. Total tcpout queue size is 512000. Warningcount=21
02-16-2024 15:54:03.379 +0000 INFO TailReader [156345 tailreader0] - Batch input finished reading file='/opt/splunkforwarder/var/spool/splunk/tracker.log'
02-16-2024 15:54:05.447 +0000 INFO BackgroundJobRestarter [156309 DispatchReaper] - inspect_count=0, restart_count=0

[Richfez]

So "index=_internal" on your cloud instance doesn't see any data where host=X?

Your connections look like they're by IP, but I thought Splunk cloud's little "connect my forwarder up" app did it by DNS entries? Can you confirm one or the other of those things is right?

Maybe the forwarder's time is off or TZ is incorrectly specified, have you checked over a longer period like 24 or 48 hours?

Also a second point to the TZ issue - if the times are in the future, it can be more difficult to find it in Splunk. Try an *all time* search. I know, it sucks, but one does what one must sometimes. 

index=_internal host=*myhost* | stats count by host

Try a wildcard ilke I suggested, using "myhost" as any string that should be reasonably unique in the hostname for the host sending in data.

You can also confirm what hostname it's sending in as by looking in etc/system/local/server.conf on the UF, there's a "hostname" field. If that's picked something "wrong" then guess what?  Your data will show up as whatever it's picked!