topic Re: Two syslog servers giving duplicate logs in Deployment Architecture

Two syslog servers giving duplicate logs

splunklearner — Mon, 17 Feb 2025 07:58:53 GMT

We have two syslog standalone servers (both are active) (one named as primary and the other is contingency) with UF installed in it forwards data to Splunk. We have different indexes configured for these two servers.

Now the issue is same log is getting indexed into both servers which resulting in duplication of logs in Splunk.

Syslog 1 --- index = sony_a == Same log

Syslog 2 --- index = sony_b == Same log

When we search with index=sony* it is giving same logs for two indexes which is duplication.

how to avoid two syslog servers from getting indexed same log twice?

Re: Two syslog servers giving duplicate logs

livehybrid — Mon, 17 Feb 2025 08:04:41 GMT

Hi @splunklearner

It sounds like your duplication is coming before it hits Splunk - Its not easy to deduplicate this on the way through, instead you might want to look at how the data is sent to syslog.

Is the data being sent from the origin to both syslog servers at the same time? Is it possible to control this behaviour so it sends only to the primary, or to the standby if it fails?

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will

Re: Two syslog servers giving duplicate logs

splunklearner — Mon, 17 Feb 2025 08:17:09 GMT

Is the data being sent from the origin to both syslog servers at the same time? -- Yes, both syslog servers picking same log and ingesting at the same time.

Is it possible to control this behaviour so it sends only to the primary, or to the standby if it fails? --- How to achieve this?

Re: Two syslog servers giving duplicate logs

livehybrid — Mon, 17 Feb 2025 08:23:28 GMT

Hi, I think ultimately this might depend on the source of the data, what are you sending to the syslog server?

Re: Two syslog servers giving duplicate logs

gcusello — Mon, 17 Feb 2025 08:27:46 GMT

Hi @livehybrid ,

using only Splunk the only way is indexing all the logs and use dedup in your searches, but in this way you pay twice the license, because it isn't possible in Splunk to create a filter to avoid duplicates before indexing.

The only solution is to take logs using an rsyslog and writing logs in files, then preparse the logs using a script, but it's very heavy for the system.

Ciao.

Giuseppe

Re: Two syslog servers giving duplicate logs

splunklearner — Mon, 17 Feb 2025 08:38:13 GMT

F5 WAF logs

Re: Two syslog servers giving duplicate logs

splunklearner — Mon, 17 Feb 2025 12:38:32 GMT

@gcusello can deploying load balancer between syslog servers help us to get rid of same log ingesting in 2 syslog servers?

Re: Two syslog servers giving duplicate logs

gcusello — Mon, 17 Feb 2025 12:59:21 GMT

Hi @splunklearner ,

no, the Load Balancer gives you the condition that you don't lose any logs even if one receiver is down, it's the first condition for HA, but it doesn't give any feature aboud duplicatibg logs.

The only solution is the one I described.

Ciao.

Giuseppe

Re: Two syslog servers giving duplicate logs

splunklearner — Tue, 18 Feb 2025 07:19:13 GMT

The only solution is to take logs using an rsyslog and writing logs in files, then preparse the logs using a script, but it's very heavy for the system. --> Can you please describe more about this and the script I need to use ?