Unable to ingest the syslog-data into splunk

[Hemnaath]

Hi All, I am trying to ingest the syslog data into splunk for test POC. In-order to ingests the syslog data, I had followed the below steps

1) created ubuntu 18.04 in Google cloud and opened the firewall port 514.

2) Install syslog-ng on Ubuntu.

$ sudo apt-get install syslog-ng -y

3) Verify installed version of syslog-ng

root@syslog-ng:~# syslog-ng --version

syslog-ng 3 (3.13.2)

Config version: 3.13

Installer-Version: 3.13.2

4) Create a Network input (UDP) in the Splunk console.

Splunk  Settings  Data inputs  UDP  Add New

5) Below details are configured in syslog-ng.conf.

source auth_log { file ("/var/log/secure"); };
destination splunk-demo { network ("35.247.51.122" transport("udp")); }; --> Splunk-demo IP address
log { source(auth_log); destination(s)

6) Restart syslog-ng service to effect the changes

root@syslog-ng:/etc/syslog-ng# systemctl restart syslog-ng

7) Unable to see the syslog-ng data getting ingested into splunk-demo instance which is the indexer server
index = syslog-data

Details:
Splunk Enterprise 8.0.2 Trial Version
Syslog version --> syslog-ng 3 (3.13.2)
Splunk indexer IP address -- > 35.247.51.122
Syslog server IP address --> 35.225.250.86
Firewall Port --> 514 is opened for communication

Kindly guide me how to troubleshoot this issue and where I can see the error details on why the data is not getting ingested.

Thanks in advance.

[PavelP]

Hello @Hemnaath ,

1. double check the communication is allowed. On the syslog server:

   nc -vzu 35.247.51.122 514

2. double check the syslog can read the secure log and send it. On the syslog server start tcpdump as follow and perform a login via ssh (on the same machine) to trigger a log event creation, you should see some packets sent towards the splunk server:

   tcpdump -pnns0 -i any port 514 and host 35.247.51.122

3. do the same on the splunk side:

   tcpdump -pnns0 -i any port 514 and host 35.225.250.86

4. double check that splunk can open this privileged port 514:

   netstat -anup | grep :514

5. if the data is send but you cannot see it, check that the data stored in the index that you can read

Common pitfals: syslog cannot read the log, a communication isn't possible because of the firewall (host firewall, AWS secure groups) , splunk cannot open port, wrong index or non-default index, ....

Let me know if it worked for you.

P.S. I hope it is a test setup and not a productive environment, because this setup doesn't follow some known best practices.

[Hemnaath]

Hi PaveIP, thanks for throwing some lights on this issue, yes I am just trying to do for test POC in google cloud not a production.

On the syslog server:
root@syslog-ng:/etc/syslog-ng# nc -vzu 35.247.51.122 514
Connection to 35.247.51.122 514 port [udp/syslog] succeeded!

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
root@syslog-ng:/etc/syslog-ng#

root@syslog-ng:/etc/syslog-ng# netstat -anup | grep :514
udp        0      0 10.128.15.216:50614     35.247.51.122:514       ESTABLISHED 1002/syslog-ng      
root@syslog-ng:/etc/syslog-ng# 

On splunk indexer instance: 
root@splunk-demo:~# tcpdump -pnns0 -i any port 514 and host 35.239.119.31
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
root@splunk-demo:~# 

root@splunk-demo:~# netstat -anup | grep :514
udp        0      0 0.0.0.0:514             0.0.0.0:*                           1889/splunkd 

Based on above mentioned point, I could get this outputs. I had made sure that firewall port 514 is opened and source IP range is set in such away it can be accessed directly, but still i could see that in Splunk indexer & syslog-ng instances, tcpdump are not getting any data. I am not sure whether the syntax is correct or not, could you please guide me on the same.

thanks in advance.

[PavelP]

if tcpdump isn't showing any data when you creating an entry in /var/log/secure then syslog-ng isn't sending any too. No need to look at the splunk yet. The problem is on the syslog server side. Look in this direction first. Do you see a new entry in /var/log/secure if you login via SSH? What is the output of:

  syslog-ng --syntax-only

[Hemnaath]

Hi PaveIP, I hope there was an issue in logging splunk.com due to which I was unable to comment, by the way today I tried to check whether the telnet connection is happening between the splunk instance and syslog-ng , but it was not happening.

root@syslog-ng:~# sudo ufw disable
Firewall stopped and disabled on system startup
root@syslog-ng:~# telnet 35.247.51.122 514
Trying 35.247.51.122...
^C
root@syslog-ng:~# 

root@splunk-demo:~# sudo ufw disable
Firewall stopped and disabled on system startup
root@splunk-demo:~# telnet 34.71.15.71 514
Trying 34.71.15.71...
^C
root@splunk-demo:~# 

syslog-ng --syntax-only -- Did not fetch any output.

Kindly guide me how to fix this connection issue in google cloud instances, so it can communicate.
thanks in advance.