topic Re: How can I further troubleshoot why I am unable to send data from a forwarder to receiver? in Getting Data In

How can I further troubleshoot why I am unable to send data from a forwarder to receiver?

boopaljothi — Tue, 29 Dec 2015 16:47:51 GMT

I have installed a universal forwarder in one laptop and Splunk Enterprise in other laptop in my home. Both are connected via ethernet LAN. I am able to share files and folders between those laptops, but Splunk forwarding is not working. I have verified the set up going through the Splunk Answers and it is correct. When I do a netstat on the receiver for the port 9997, it shows that it is listening on that port, but the output is like

netstat -an | find "9997"
  TCP    0.0.0.0:9997        0.0.0.0:9997        LISTENING

Is this correct? Also, I am able to do a telnet to the receiver from forward through this port, but other few ports that I have are working.

telnet command used:

telnet <ethernet ip of receiver>:9997

Can someone help me on how to resolve this? Been struggling to find the answer for quite a while.

Re: How can I further troubleshoot why I am unable to send data from a forwarder to receiver?

jplumsdaine22 — Tue, 29 Dec 2015 17:16:34 GMT

On the forwarder, are there any errors in the splunkd.log ? Also how have you configured your outputs.conf?

Re: How can I further troubleshoot why I am unable to send data from a forwarder to receiver?

jkat54 — Tue, 29 Dec 2015 17:23:59 GMT

Did you open port 9997 in the windows firewall or linux iptables?

Just because you are listening on the port doesnt mean the port is "open".

Re: How can I further troubleshoot why I am unable to send data from a forwarder to receiver?

boopaljothi — Tue, 29 Dec 2015 17:26:42 GMT

in the forwarder i see connection timeout error message in the splunkd logs. i have configured the forwarder when i was installing to send the data to receiver ip address in the port 9997. i will send you the complete data in the file in sometime

Re: How can I further troubleshoot why I am unable to send data from a forwarder to receiver?

jplumsdaine22 — Tue, 29 Dec 2015 17:59:15 GMT

When you say "Also, I am able to do a telnet to the receiver from forward through this port, but other few ports
that I have are working."

Do you mean you CANNOT telnet on 9997 from the forwarder to the indexer but you can for other ports or that you CAN but NOT for other ports??

If you can't establish a connection from the forwarder to the indexer on that port I would rule out network issues first!

Re: How can I further troubleshoot why I am unable to send data from a forwarder to receiver?

boopaljothi — Tue, 29 Dec 2015 18:01:57 GMT

yes i cant establish a telnet to the port 9997 only and few other ports that i see in the netstat output i can establish a connection from forwarder

Re: How can I further troubleshoot why I am unable to send data from a forwarder to receiver?

jplumsdaine22 — Wed, 30 Dec 2015 11:18:07 GMT

If you can establish a connection to 9997 locally on the indexer (try telnet localhost 9997) but not from the forwarder then my guess is you have a firewall blocking you somewhere.

Re: How can I further troubleshoot why I am unable to send data from a forwarder to receiver?

boopaljothi — Thu, 31 Dec 2015 02:24:25 GMT

opening up the firewall port helped resolve the problem

Re: How can I further troubleshoot why I am unable to send data from a forwarder to receiver?

boopaljothi — Tue, 29 Sep 2020 08:15:45 GMT

After enabling the firewall telnet is working and forwarder is able to connect to the indexer. but seeing the error below in forwarder log

01-04-2016 22:06:25.163 -0600 INFO TcpOutputProc - Connected to idx=10.0.0.35:9997
01-04-2016 22:06:29.607 -0600 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd_access.log'.
01-04-2016 22:06:29.616 -0600 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd_ui_access.log'.
01-04-2016 22:06:32.921 -0600 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe"" splunk-winevtlog - EvtDC::bind: Failed to get domain controller name with DsGetDcName: (1355)
01-04-2016 22:06:32.921 -0600 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe"" splunk-winevtlog - EvtDC::connectToDC: DsBind failed: (1355)
01-04-2016 22:06:32.921 -0600 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe"" splunk-winevtlog - WinEventLogChannel::init: Failed to bind to DC, dc_bind_time=25 msec
01-04-2016 22:15:02.025 -0600 INFO TcpOutputProc - Connection to 10.0.0.35:9997 closed. Connection closed by server.
01-04-2016 22:15:22.397 -0600 WARN TcpOutputProc - Cooked connection to ip=10.0.0.35:9997 timed out
01-04-2016 22:15:43.884 -0600 WARN TcpOutputProc - Cooked connection to ip=10.0.0.35:9997 timed out
01-04-2016 22:15:53.944 -0600 INFO TcpOutputProc - Connected to idx=10.0.0.35:9997
01-04-2016 22:30:33.323 -0600 INFO TcpOutputProc - Connection to 10.0.0.35:9997 closed. Connection closed by server.
01-04-2016 22:30:53.389 -0600 WARN TcpOutputProc - Cooked connection to ip=10.0.0.35:9997 timed out
01-04-2016 22:31:03.128 -0600 INFO TcpOutputProc - Connected to idx=10.0.0.35:9997

seeing the below message in receiver splunk web

Received event for unconfigured/disabled/deleted index=wineventlog with source="source::WinEventLog:System" host="host::xxxx" sourcetype="sourcetype::WinEventLog:System". So far received events from 1 missingindex(es).

i am trying to forward the windows event log from forwarder and below is the inputs.conf file from forwarder

[default]
host = xxxx

[WinEventLog://Application]
disabled = 0
index = xxxx
sourcetype = security

what could be the issue. i have created a new index as well in the receiver with that index name