How can I further troubleshoot why I am unable to send data from a forwarder to receiver?

[boopaljothi]

I have installed a universal forwarder in one laptop and Splunk Enterprise in other laptop in my home. Both are connected via ethernet LAN. I am able to share files and folders between those laptops, but Splunk forwarding is not working. I have verified the set up going through the Splunk Answers and it is correct. When I do a netstat on the receiver for the port 9997, it shows that it is listening on that port, but the output is like

netstat -an | find "9997"
  TCP    0.0.0.0:9997        0.0.0.0:9997        LISTENING

Is this correct? Also, I am able to do a telnet to the receiver from forward through this port, but other few ports that I have are working.

telnet command used:

telnet <ethernet ip of receiver>:9997

Can someone help me on how to resolve this? Been struggling to find the answer for quite a while.

[boopaljothi]

After enabling the firewall telnet is working and forwarder is able to connect to the indexer. but seeing the error below in forwarder log

01-04-2016 22:06:25.163 -0600 INFO TcpOutputProc - Connected to idx=10.0.0.35:9997
01-04-2016 22:06:29.607 -0600 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd_access.log'.
01-04-2016 22:06:29.616 -0600 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd_ui_access.log'.
01-04-2016 22:06:32.921 -0600 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe"" splunk-winevtlog - EvtDC::bind: Failed to get domain controller name with DsGetDcName: (1355)
01-04-2016 22:06:32.921 -0600 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe"" splunk-winevtlog - EvtDC::connectToDC: DsBind failed: (1355)
01-04-2016 22:06:32.921 -0600 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe"" splunk-winevtlog - WinEventLogChannel::init: Failed to bind to DC, dc_bind_time=25 msec
01-04-2016 22:15:02.025 -0600 INFO TcpOutputProc - Connection to 10.0.0.35:9997 closed. Connection closed by server.
01-04-2016 22:15:22.397 -0600 WARN TcpOutputProc - Cooked connection to ip=10.0.0.35:9997 timed out
01-04-2016 22:15:43.884 -0600 WARN TcpOutputProc - Cooked connection to ip=10.0.0.35:9997 timed out
01-04-2016 22:15:53.944 -0600 INFO TcpOutputProc - Connected to idx=10.0.0.35:9997
01-04-2016 22:30:33.323 -0600 INFO TcpOutputProc - Connection to 10.0.0.35:9997 closed. Connection closed by server.
01-04-2016 22:30:53.389 -0600 WARN TcpOutputProc - Cooked connection to ip=10.0.0.35:9997 timed out
01-04-2016 22:31:03.128 -0600 INFO TcpOutputProc - Connected to idx=10.0.0.35:9997

seeing the below message in receiver splunk web

Received event for unconfigured/disabled/deleted index=wineventlog with source="source::WinEventLog:System" host="host::xxxx" sourcetype="sourcetype::WinEventLog:System". So far received events from 1 missingindex(es).

i am trying to forward the windows event log from forwarder and below is the inputs.conf file from forwarder

[default]
host = xxxx

[WinEventLog://Application]
disabled = 0
index = xxxx
sourcetype = security

what could be the issue. i have created a new index as well in the receiver with that index name

[jkat54 · Accepted solution]

Did you open port 9997 in the windows firewall or linux iptables?

Just because you are listening on the port doesnt mean the port is "open".

[boopaljothi]

opening up the firewall port helped resolve the problem

[jplumsdaine22]

On the forwarder, are there any errors in the splunkd.log ? Also how have you configured your outputs.conf?

[jplumsdaine22]

When you say "Also, I am able to do a telnet to the receiver from forward through this port, but other few ports
that I have are working."

Do you mean you CANNOT telnet on 9997 from the forwarder to the indexer but you can for other ports or that you CAN but NOT for other ports??

If you can't establish a connection from the forwarder to the indexer on that port I would rule out network issues first!

[boopaljothi]

yes i cant establish a telnet to the port 9997 only and few other ports that i see in the netstat output i can establish a connection from forwarder

[jplumsdaine22]

If you can establish a connection to 9997 locally on the indexer (try telnet localhost 9997) but not from the forwarder then my guess is you have a firewall blocking you somewhere.

[boopaljothi]

in the forwarder i see connection timeout error message in the splunkd logs. i have configured the forwarder when i was installing to send the data to receiver ip address in the port 9997. i will send you the complete data in the file in sometime