Splunk Not Receiving Logs From Splunk Forwarder or Syslog-ng what could be the issue with splunk enterprise?

[tks_tman]

I have Splunk setup and it establishes connection with syslog and splunk universal forwarder from a remote server:

I have syslog-ng setup as follows: 

You can see the connections established :

This is the inputs.conf for the splunk universal forwarder: 

But still no data is being received by splunk: 

I was able to use some powershell script to verify that the logs were being sent and delivered to the server with splunk. The issue is with splunk itself.

Am I missing something? And how would I go about troubleshooting the issue and fixing it?

[PickleRick]

Apart from all the questions @gcusello asked, remember that if you simply set your syslog server to forward the events to splunk server's 9997 port, it won't work. Splunk expects s2s communication on 9997, not plain syslog.

Question is whether you're getting anything received by your syslog-ng daemon at all. Does anything get written into the files in /var/log/remote?

Do you in fact get anything in on the 514 port?

Did you verify it in any way?

[tks_tman]

Yes. I am certain that the local logs are generated. What do you mean by "splunk expects s2s communication on 9997"? Does it require some conversion? How would I go about doing that?

[PickleRick]

OK. For the rest of debugging @gcusello already pointed you in the right way. I'll just drop in a few words about this 9997 port.

Port 9997/TCP is used by S2S (splunk to splunk) communication. That is a protocol which is used to forward events from a source splunk machine (typically a forwarder) to a receiving splunk machine (might be an indexer but might be an intermediate forwarder). It is a proprietary protocol and is used only for connectivity between splunk components. So you can't just point your syslog server to send events to splunk server on 9997 and expect it to receive it properly.

As a side note - even though you can set up an input of tcp:// or udp:// type on your splunk forwarder to listen for raw syslog data sent from your sources, you typically don't want to do that. You'd rather use an intermediate syslog server (like you're doing here with syslog-ng writing to files which are then picked up by the UF).

[gcusello · Accepted solution]

Hi @tks_tman,

let me understand: do you want to receive logs from a linux machine where the universal forwarder is installed or do you want to receive logs using syslog?

You spoke of port 9997 that's used  to receive data from a Universal Forwarder installed on another machine and not to receive syslogs.

In this case you don't need syslogs and inputs.conf that you displayed must be located on the Universal Forwarder not in the Splunk server.

If instead you need to receive syslogs, you don't need the inputs.conf you displayed and the 9997 port enabling, but you have to enable a network input using the protocol (UDP/TCP) you prefer.

You don't need also syslog-ng server.

If you want to use syslog-ng server to receive syslogs, you have to enable it ro receive remote syslogs and wring data on file system; then you need an inputs.conf (different from the one you displayed) to read the text files created by syslog-ng.

So wjhat's your requiremen??

Ciao.

Giuseppe

[gcusello]

Hi @tks_tman.,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[tks_tman]

The logs are being sent from a remote device to-> a linux machine (that contains splunk universal forwarder and syslog-ng) ( and stores logs locally) both of these are to send the logs to -> splunk.

Splunk seems to not be accepting the logs from either syslog-ng or the splunk universal forwarder even though the tcp connections are established between both syslog-ng and splunk and splunk universal forwarder and splunk.

The requirement is splunk isn't accepting the logs even though the connections are established.

I also get the following message with the list forward-server command:

[gcusello]

Hi @tks_tman,

debug the problem one by one:

are you receiving internal Splunk logs from the forwarder? you can check this with a simple search

index=_internal host=<your_host>

If yes the problem in in inputs.conf.

In this case in the inputs.conf stanza you have to put the path of the files (written by the syslog-ng server) logs to read and not the "/var/log" path:

[monitor:///<your_data_path>/<your file_name>]

If not the problem cound be:

• in outputs.conf on the Forwarder,
• in an intermediate Firewall,
• in the local Firewall.

What's your outputs.conf?

for more infos see at https://docs.splunk.com/Documentation/Forwarder/9.0.3/Forwarder/Configureforwardingwithoutputs.conf 

it should be something like this:

[tcpout]
defaultGroup=my_indexers

[tcpout:my_indexers]
server=mysplunk_indexer1:9997, mysplunk_indexer2:9996

[tcpout-server://mysplunk_indexer1:9997]
[tcpout-server://mysplunk_indexer2:9997]

To troubleshooting Firewalls, use telnet, from the Forwarder:

telnet <ip_splunk_server> 9997

Ciao.

Giuseppe