Deployment Architecture

Splunk forwarder failed to send logs from amazon linux instance

meet_vadaria
Engager

Trying to send logs to Splunk server using forwarder installed on Amazon Linux instances. I am not seeing any data on Splunk server. On forwarder side, I am seeing interfaces.sh related error in /var/log/splunkd.log.

09-10-2019 16:04:52.161 +0000 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/xxx_splunk_ta_nix/bin/interfaces.sh" cat: /sys/class/net/eth0/duplex: Invalid argument
09-10-2019 16:04:52.171 +0000 ERROR ExecProcessor - message from 
"/opt/splunkforwarder/etc/apps/xxx_splunk_ta_nix/bin/interfaces.sh" cat: /sys/class/net/eth0/speed: Invalid argument
09-10-2019 16:04:52.280 +0000 INFO  TailReader -   ...continuing.
09-10-2019 16:05:03.723 +0000 INFO  HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_xxxxxx_3D3B3E31-6E53-4D7A-AB7E-0FAF1FC62062
09-10-2019 16:05:10.007 +0000 WARN  FileClassifierManager - The file '/var/log/btmp' is invalid. Reason: binary.
09-10-2019 16:05:10.007 +0000 INFO  TailReader - Ignoring file '/var/log/btmp' due to: binary
09-10-2019 16:05:16.196 +0000 WARN  TailReader - Could not send data to output queue (parsingQueue), retrying...
09-10-2019 16:05:22.340 +0000 INFO  TailReader -   ...continuing.

I'm experiencing this issue with all amazon Linux servers.

0 Karma

mguhad
Communicator

Hi,
If data is being terminated before it even reaches the parsing queue indicates a connection problem as data is being dropped before it enters the first pipeline. Please check your firewalls, ports, IP TABLES etc to root out connectivity issues between the indexers and UFs/HFs.
It also worth checking the security groups of the Indexers and the UFs if they are the same or have similar permissions/rules.

Hope this helps.

0 Karma

meet_vadaria
Engager

I don't think it's a port issue. as it's able to connect to indexer on 9997 port as per logs. I also verified with telnet.

09-10-2019 20:04:21.807 +0000 INFO TcpOutputProc - Connected to idx=100.117.33.54:9997, pset=0, reuse=0. using ACK.
09-10-2019 20:04:24.586 +0000 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wss_splunk_ta_nix/bin/interfaces.sh" cat: /sys/class/net/eth0/duplex: Invalid argument
09-10-2019 20:04:24.603 +0000 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wss_splunk_ta_nix/bin/interfaces.sh" cat: /sys/class/net/eth0/speed: Invalid argument
09-10-2019 20:05:18.036 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_100.117.8.197_8089_100.117.8.197_mongop0-i-09e03c274a86ef49b-p1-ugw1.wss.symfedcloud.com_8323AF5D-B129-41AB-8B7B-8A9E95A9C7D0
09-10-2019 20:05:24.418 +0000 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wss_splunk_ta_nix/bin/interfaces.sh" cat: /sys/class/net/eth0/duplex: Invalid argument
09-10-2019 20:05:24.419 +0000 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wss_splunk_ta_nix/bin/interfaces.sh" cat: /sys/class/net/eth0/speed: Invalid argument
09-10-2019 20:05:31.624 +0000 INFO TcpOutputProc - Closing stream for idx=100.117.33.54:9997

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...