Logdata not arriving from all Universal Forwarders

[erikwie]

Enviroment:
Distributed Splunk Enterprise 9.4.x running on Linux 
UFs sends data to 2 HFs S2S loadbalansing
HF sends logs to a single indexer

Problem case:
2 servers on a vlan, lets say 10.1.1.2 and 10.1.1.3, one windows and one linux
Windows server sende logs and data in ingested and indexed, all is fine.
Linux server has nothing in indexes, and nothing in _internal index either

I can see traffic in the firewall from UF to HF beeing allowed.
I can see _internal logs from metrics.log on both HFs that they du get data from both linux and windows servers

I can not find any errors in any splunk logs 

Can anyone please help me to figure out why I do not get any logs from the linux server while i du get from the Windows server?

[PickleRick]

Hmm... that sounds a bit strange. Typically if you don't get internal logs that would mean that you have conmectivity problems somewhere along the way. But you're saying that:

1. Metrics show that data _is_ actually being delivered to HFs (which metrics did you check?)

2. The data is not properly forwarded and indexed on the indexer.

That would mean that either your HFs (or the indexer itself) do perform some sort of selective routing/filtering (you'd have to check your config for that but I find it unlikely) or the data _is_ actually delivered and indexed but you cannot find it (wrong metadata? Time/timezone problems? Search-time filters?).

[erikwie]

1.  group=tcpin_connection sourceIp=10.1.1.2

2. The intermediat HF is set to just recive and send, not doing any routing or filtering.

Indexer is neither set to 

[gcusello]

Hi @erikwie ,

are you using the Splunk_TA_nix add-on to ingest these logs?

did you enabled inputs on this add-on (by default they are all disabled)?

Ciao.

Giuseppe

[erikwie]

Not actually relevent to the problem, but yes

I do not get the splunk internal logs to the _internal index (nor any other data)
The HF logs tells me there is about 430kb of data pr hour pr HF that comes from the 10.1.1.2 UF (linux) and is forwarded.

This means that the outputs.conf is present and correct on the UF.
Conf is deplyed with deployment manager, so same on all linux hosts, problem is not on all linux hosts, but atleast 2 so far.