Splunk Forwarder can't send data to enterprise

[Cyner__]

I am newbie to splunk. Any help is appreciated

So I have an splunk enterprise in my windows computer. and splunk forwarder in a ubuntu VPS server with a cowrie honeypot built in. So my problem is when i try to ping test my local computer with VPS server , i have %100 packet loss.

Also splunkd log file is full of "cooked connection to "my-local-ip" timed out and

... blocked nfor blocked_seconds=3000. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

errors

Thanks for helping. I am waiting for your response

[gcusello]

Hi @Cyner__ ,

at first did you followed the instructions at https://docs.splunk.com/Documentation/Splunk/9.2.1/Data/Usingforwardingagents ?

In other words:

• did you checked the open route between UF and Splunk on port 9997 (default)? you can do this using telnet.
• did you enabled receiving in Splunk Enterprise ? [Settings > Forwardring and Receiving > Receiving]
• did you enabled forwarding in Universal Forwarder?

When you did the above steps, you can check the connection using the following search

index=_internal host=your_client_host)

Ciao.

Giuseppe

[Cyner__]

Thanks for the help @gcusello 

But my problem is still occurs.

When i use telnet with 9997 port to my computer (tried both private and public ip) telnet runs "connection timed out" error.

i already enabled receiving.

I don't know if i enabled forwarder or not bu i Start'ed it with command and configured output and input file

This is inputs.conf:

[monitor:///home/cowrie/cowrie/var/log/cowrie/cowrie.json]
index = cowrie
sourcetype = json
disabled = false

this is output.conf:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
disabled = false
server = my-private-ip:9997

sorry if i missed something as i said im both new to linux and splunk

[gcusello]

Hi @Cyner__ ,

you should run the telnet from the client, not from the Server:

telnet my-private-ip 9997

If it doesn't answer there's something in the middle (e.g. personal firewalls) that block the connection.

Ciao.

Giuseppe

[Cyner__]

Hi @gcusello 

i think i found my problem. I don't have open 9997 port on my forwarder server i guess.

this is the screenshot. How can i open the 9997 port

[gcusello]

Hi @Cyner__ ,

port 9997 must be opened on the Spunk Enterprise, not on the client, you can open the port in [Settings > Forwarding and Receiving > Receiving].

Infact the telnet test must be done on the client not from the Splunk Server.

Did you completed al the steps described in the document or in my previous answer?

Ciao.

Giuseppe

[Cyner__]

Hi. @gcusello 

yes i did all. what do you mean by client do you mean the server with forwarder or splunk enterprise ?

and when i try to telnet the splunk server via forwarder server "i think its client" connection always times out.

i saw my splunk server (my computer i guess) doesn't have any inputs.conf at directory C:\Program Files\Splunk\etc\system\local path. 

what should i do? 

best regards

[gcusello]

Hi @Cyner__,

you have to enable receiving on Splunk Enterprise,

then you have to check the route from the Universal Forwarder on port 9997 to the Spunk Enterprise (using telnet),

then you have to configure your outputs.con (as described in the above link) in the Universal Forwarder.

Ciao.

Giuseppe

[Cyner__]

Ok. Now i run the telnet <my-forwarders-ip> 9997 command from my windows pc

The result is "could not open connection to the host. port 9997 .connect failed" 

i run for both private ip and public ip.

My windows firewall is disabled and my forwarders server doesn't even have firewall installed

[Cyner__]

ah also when i clicked "data summary" button from splunk enterprise web,  i only see "waiting for results" 

[Cyner__]

Also i can't find anything in the Splunk Enterprise. Nothing in forwarder management section and no data whatsoever