Getting Data In

Heavy Forwarders stopped receiving some logs

vnguyen46
Contributor

Hi,

I have a new HF once accepted logs for about a week, then stopped receiving on almost all logs at a same time.
I compared this HF with the old working one and I don't see rotated logs created on the new HF.

For instance, in log1 directory, I see log1.log and several other copies like log1.log-date1.gz and log1.log-date2.gz and so on, but on the new HF I only see log1.log.

I think not creating rotated logs on the HF could be the issue, but not sure and how to have these rotated logs created.
Anyone can help, I appreciate it.

Thanks,

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Have you verified the new HF is running (splunk status)?

---
If this reply helps you, Karma would be appreciated.
0 Karma

vnguyen46
Contributor

Hi - yes, it's running. I don't see any .gz files in any directories.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Heavy Forwarders typically don't use a directory called "log1" so I wonder if you're looking at a syslog directory. If so, make sure the syslog process is running and data sources are still sending to it (no new firewall rule is blocking them, for instance).

---
If this reply helps you, Karma would be appreciated.
0 Karma

vnguyen46
Contributor

Hi richgalloway - on HF, log stored at: /opt/splunklogs/hostname/hostname.log
I also see some files like hostname.log-timestamp.gz. Are these .gz files created by Splunk and supposed to be there?

Thank you,

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Usually those are created e.g. some syllogism variant not Splunk. You should figure out which tool is used on your environment to deliver / received those logs. Many times it is syslog, syslog-ng or rsyslog. And on network topology there could be a load balancer before those HF hosts to distribute events to all of those hosts.

And probably there is also some log rotation tools to rotate and zip those logs?

R. Ismo

0 Karma
Get Updates on the Splunk Community!

Security Highlights: September 2022 Newsletter

 September 2022 The Splunk App for Fraud Analytics (SFA) is now Splunk SupportedUse your existing Splunk ...

Platform Highlights | September 2022 Newsletter

 September 2022 What’s New in 9.0 and How to UpgradeGet a walk through of what is new Splunk Enterprise 9.0 ...

Observability Highlights | September 2022 Newsletter

 September 2022 Splunk Observability SuiteAccess to "Classic" SignalFx Interface Will be Removed on Sept 30, ...