How to resolve error with VMware ESX/ESXi log files?

[TomP]

Hi,

   I have about 500 hosts to configure syslog.global.loghost on multiple Vcenters.  We are forwarding the logs to a Splunk Universal forwarder. Some ESX host servers keep getting this error. The host "splunklog-domainname:1514" has become unreachable. Remote logging to this host has stopped.

This ends up filling up the Vcenter logs and Vcenter stops responding. Has anyone seen this issue?

Thanks...Tom

[TomP]

Thank  you Shiv.

[shivanshu1593]

You're welcome. If you need more assistance in this, please feel free to reach out or put a new question in the community.

Happy Splunking and best wishes.

[TomP]

If  i create a linux vm with syslog NG point all the ESX hosts to the syslog NG server.  Won't the syslog-ng server get overwhelmed if all  ESX hosts are forwarding there logs to port 1514?

[TomP]

Hi Thanks for the reply    

We are using 2 universal forwarders pointing to one Virtual IP address. I was asked by security to add the forwarder link to all hosts. I've had to revert several times.  When i run the following commands i always get a successful reply.  I'm not familiar with syslog-NG would that be installed on the UF or on each host? Can it be the syslog service on the UF is stuck..hung or not responding?

nc -zv splunklog.domainname 1514

nc -zv pap-splunklogs01.domainname 1514

nc -zv pap-splunklogs02.domainname 1514

[shivanshu1593]

Well, let's gather a bit more information for an effective solution. How are you sending logs from the VM to Splunk UF? Is Splunk UF installed on a linux or windows server? A little more detail will help to identify the set up and then get a good solution.

Regarding syslog-ng or rsyslogd, these are the services which come by default in Linux and can be installed in Windows. They work separately like any service and their job is to listen on ports that you specify in their configuration for incoming logs and then write them to the path in the server which you also specify in their configuration files. 

For your current issue, multiple servers are trying to connect to Splunk on a single port. Splunk is attempting to listen to them but it can only do one at a time, which makes the others wait for their turn and they get the error "Requested server/endpoint cannot be reached". You need to redistribute this work load either by giving each server a different port to connect to Splunk UFs (Ex: 1514 for server A, 1515 for server B and so on) or use a lot of other servers with Splunk on them, neither which is an appropriate solution. So, how do we address this?

Effective solution:  Get a server (Preferably linux), configure either syslog-ng or rsyslogd to listen to the incoming logs from your VMs and have them write down on the disk. Then make Splunk UF monitor those directories using [monitor:///] stanza in the inputs.conf.

Or install Splunk UF on each VM and point them directly to Splunk indexers.

[shivanshu1593]

Requires more context but by the looks of it your UF is getting overwhelmed with the amount of data it is receiving. It is listening to a server sending logs on the port 1514, while other servers are trying to communicate on the same port as well, leading for them to wait for the port to get free and logs to queue up, just like you said. You can either add more servers to outputs.conf and then send the logs there for further ingestion or try sending logs to different ports in Splunk for different servers by configuring inputs.conf accordingly, which IMO is not an effective architecture.

Quick question: Why don't you use a service like Syslog-NG or rsyslogd to listen to the incoming logs via Syslog and write them on the disk of the server and then have the Splunk UF on it monitor those directories and ingest the logs to Splunk. You can configure logrotate to clear out the accumulating logs older than a day or even zip them for higher retention. Easy to maintain and less chances of data getting queued up.

Also, are you using one UF or multiple?

[TomP]

If  i create a linux vm with syslog NG point all the ESX hosts to the syslog NG server.  Won't the syslog-ng server get overwhelmed if all  ESX hosts are forwarding there logs to port 1514?

[shivanshu1593 · Accepted solution]

Depends upon your architecture. You can have both the servers with Splunk UFs acting as syslog servers using syslog-ng. It is very efficient in handling the influx of data coming in. However, if you have a whole boat of servers, then you can look into adding more servers to the list.

As a general practice, using Splunk-tcp and splunk-udp (Making Splunk listen to ports for logs via syslog) is not a good idea outside of the lab environment. Setting up syslog-ng/rsyslogd is a better solution as it is expandable due to its ability to handle a boatload of data coming towards it using multi-threading.

Hope this helps,