Why did syslog stopped sending logs to indexer?

[Lwoods]

Hello,

I have a syslog server that collects logs from various hosts, (esxi).  The syslog is currently receiving the logs each day from the hosts and puts them the  "data/ES/" directory.  I have splunkforwarder installed the syslog and inside the splunkforwarder, I have the esxi add-on app.

Inside the esxi add-on app 

I have created an input stanza that monitors the data and sent to the indexer 

[monitor:///data/ES/]
disabled = false
index = vmware-esxilog
sourcetype = vmw-syslog

The logs stopped sending to the indexer several days ago.  However, my firewall logs are still sending to the indexer.  The firewall logs are sent the same directory "/data/fire/" and then sent to index.  What am I missing?  

Thanks

[gcusello]

Hi @Lwoods,

obvious question: there was change in your firewall routes or configurations in the last days?

In general I always put a file indication in the stanza header, e.g.

[monitor:///data/ES/*]

 Are there logs after the 1st of June or logs stopped to arrive with the end of May?

Ciao.

Giuseppe

[Lwoods]

Hello,  

Firewall logs are still sending logs to syslog, and syslog is forwarding them up to the indexer.   Esxi and other devices have stopped reporting 12 days ago.  8 June.   

What could be wrong?

[gcusello · Accepted solution]

Hi @Lwoods,

if the Forwarder is sending other logs and your configuration worked since few days ago, the easiest solution is that something changed in the intermediate channel: esxi syslog configuration or firewall routes.

I suppose that you already checked them, is it correct?

if you're using tcp as protocol check using telnet the connection between esxi and HF.

then check the traffic through the intermediate firewall and see, using tcpdump, if your HF is receiving from your esxi on your protocol and your port.

Ciao.

Giuseppe

[Lwoods]

Hello,  

Thanks for the response.   The esxi logs add-on installed on the deployment app, didn't match what was on the syslog.  All the deployment apps are pushed down to the syslog.  When configuring inputs.conf (monitor stanza) I didn't mirror those settings in the deployment server.  Once I fixed it, it worked.  

Thanks for all you help and expertise..

Happy Splunking

Lisa

[Lwoods]

This also applies to my rsa logs, which stopped sending logs 7 days ago.

[Lwoods]

The logs stopped sending yesterday.  Firewall logs are still sending

Do you put a wildcard inside the monitor stanza  like this: 

[monitor:///data/ES/*]