topic Re: Syslog stopped sending logs to Indexer in Getting Data In

Why did syslog stopped sending logs to indexer?

Lwoods — Thu, 22 Jun 2023 16:31:57 GMT

Hello,

I have a syslog server that collects logs from various hosts, (esxi). The syslog is currently receiving the logs each day from the hosts and puts them the "data/ES/" directory. I have splunkforwarder installed the syslog and inside the splunkforwarder, I have the esxi add-on app.

Inside the esxi add-on app

I have created an input stanza that monitors the data and sent to the indexer

[monitor:///data/ES/]
disabled = false
index = vmware-esxilog
sourcetype = vmw-syslog

The logs stopped sending to the indexer several days ago. However, my firewall logs are still sending to the indexer. The firewall logs are sent the same directory "/data/fire/" and then sent to index. What am I missing?

Thanks

Re: Syslog stopped sending logs to Indexer

gcusello — Fri, 16 Jun 2023 15:37:57 GMT

Hi @Lwoods,

obvious question: there was change in your firewall routes or configurations in the last days?

In general I always put a file indication in the stanza header, e.g.

[monitor:///data/ES/*]

Are there logs after the 1st of June or logs stopped to arrive with the end of May?

Ciao.

Giuseppe

Re: Syslog stopped sending logs to Indexer

Lwoods — Fri, 16 Jun 2023 15:41:07 GMT

The logs stopped sending yesterday. Firewall logs are still sending

Do you put a wildcard inside the monitor stanza like this:

[monitor:///data/ES/*]

Re: Syslog stopped sending logs to Indexer

Lwoods — Fri, 16 Jun 2023 15:48:24 GMT

This also applies to my rsa logs, which stopped sending logs 7 days ago.

Re: Syslog stopped sending logs to Indexer

Lwoods — Tue, 20 Jun 2023 13:45:55 GMT

Hello,

Firewall logs are still sending logs to syslog, and syslog is forwarding them up to the indexer. Esxi and other devices have stopped reporting 12 days ago. 8 June.

What could be wrong?

Re: Syslog stopped sending logs to Indexer

gcusello — Thu, 22 Jun 2023 06:13:34 GMT

Hi @Lwoods,

if the Forwarder is sending other logs and your configuration worked since few days ago, the easiest solution is that something changed in the intermediate channel: esxi syslog configuration or firewall routes.

I suppose that you already checked them, is it correct?

if you're using tcp as protocol check using telnet the connection between esxi and HF.

then check the traffic through the intermediate firewall and see, using tcpdump, if your HF is receiving from your esxi on your protocol and your port.

Ciao.

Giuseppe

Re: Syslog stopped sending logs to Indexer

Lwoods — Thu, 22 Jun 2023 11:23:20 GMT

Hello,

Thanks for the response. The esxi logs add-on installed on the deployment app, didn't match what was on the syslog. All the deployment apps are pushed down to the syslog. When configuring inputs.conf (monitor stanza) I didn't mirror those settings in the deployment server. Once I fixed it, it worked.

Thanks for all you help and expertise..

Happy Splunking

Lisa