Trying to send logs to Splunk server using forwarder installed on Amazon Linux instances. I am not seeing any data on Splunk server. On forwarder side, I am seeing interfaces.sh related error in /var/log/splunkd.log.
09-10-2019 16:04:52.161 +0000 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/xxx_splunk_ta_nix/bin/interfaces.sh" cat: /sys/class/net/eth0/duplex: Invalid argument
09-10-2019 16:04:52.171 +0000 ERROR ExecProcessor - message from
"/opt/splunkforwarder/etc/apps/xxx_splunk_ta_nix/bin/interfaces.sh" cat: /sys/class/net/eth0/speed: Invalid argument
09-10-2019 16:04:52.280 +0000 INFO TailReader - ...continuing.
09-10-2019 16:05:03.723 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_xxxxxx_3D3B3E31-6E53-4D7A-AB7E-0FAF1FC62062
09-10-2019 16:05:10.007 +0000 WARN FileClassifierManager - The file '/var/log/btmp' is invalid. Reason: binary.
09-10-2019 16:05:10.007 +0000 INFO TailReader - Ignoring file '/var/log/btmp' due to: binary
09-10-2019 16:05:16.196 +0000 WARN TailReader - Could not send data to output queue (parsingQueue), retrying...
09-10-2019 16:05:22.340 +0000 INFO TailReader - ...continuing.
I'm experiencing this issue with all amazon Linux servers.
If data is being terminated before it even reaches the parsing queue indicates a connection problem as data is being dropped before it enters the first pipeline. Please check your firewalls, ports, IP TABLES etc to root out connectivity issues between the indexers and UFs/HFs.
It also worth checking the security groups of the Indexers and the UFs if they are the same or have similar permissions/rules.