I think I've missed something but I have a lab environment set up using 4.2
I have a CentOS universal forwarder forwarding data to a CentOS based indexer with the intention to use the *nix app.
The universal forwarder is forwarding to the indexer. According to the indexer's Deployment app, the indexer seeing the data, but that remote host data is not showing up in the *nix App.
Perhaps your events aren't being timestamped properly? Have you attempted to do an all-time(real time) search? Also, you could look at metrics.log to see if the data from these hosts is actually making it in with a search like:
index=_internal source=metrics.log group=* | timechart span=30 sum(kb) by series
It might give you a clue as to if data is coming in from the hosts that are forwarding.
1) I am not sure about the deployment server handshake issues, I would need more context to comment, but in all likelihood they aren't the reason you are seeing this problem.
2) What kind of nonsense data are you seeing? Are events being indexed with the proper timestamps? It would be helpful if you would post some of the data your seeing here.
From doing a real time all time search it looks like data is hitting the indexer, but it looks like it may be nonsense (or rather, not expected) data.
I found info was going to a mirrored port and I'm showing that it's constantly trying to reach a non-existent deployment server. I've cleared out the mirrored port, but I'm still getting the non-existent deployment server handshake woes.
What action needs to be taken beyond setting the forwarder in the remote host and the receiver in the indexer to insure that the proper *nix expected data is going to the right place?