Deployment Architecture

How to prevent duplicate events from occurring when connecting a non-clustered search head with an indexer cluster?

splunkreal
Motivator

Hello guys,

I've setup a separated search head using distributed search with clustered indexers (in order to have a different portal with different authentication)

However if I select more than one indexer then I get duplicate events.

Any idea to search all indexers without duplicate events?

Thanks.

* If this helps, please upvote or accept solution 🙂 *
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi realsplunk,
are you sure that you are sending events to your two indexers in auto load balancing and not in duplication?
verify it in outputs.conf file of your Universal Forwarder:
you are using auto load balancing if you have:

[tcpout:stanza1]
server = xxx.xxx.xxx.xxx:9997, yyy.yyy.yyy.yyy:9997

you aren't using auto load balancing if you have:

[tcpout:stanza1]
server = xxx.xxx.xxx.xxx:9997
[tcpout:stanza2]
server =  yyy.yyy.yyy.yyy:9997

Bye.
Giuseppe

View solution in original post

harsmarvania57
SplunkTrust
SplunkTrust

Hi @realsplunk,

If you want to use standalone searchhead to search data from all clustered indexers without duplicate data then point your standalone search head to cluster master of Indexer cluster, do not add any indexer in Distributed search -> Search Peer.

When you will point standalone search head to cluster master of all indexer, Distibuted search -> Search Peer will be automatically updated with all indexers. Now when you search from this stand alone search head it will not display duplicate events from clustered indexers.

Thanks,
Harshil

splunkreal
Motivator

Thanks Harshil, do you mean I can add that cluster master (it's also our deployment server and is not indexing data) in distributed search or I must add the new search head in the cluster (which I don't want as i want separate authentication, customization...) ?

* If this helps, please upvote or accept solution 🙂 *
0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Not Cluster master in Distributed Search but follow this URL http://docs.splunk.com/Documentation/Splunk/6.5.1/Indexer/Enablethesearchhead to point non-clustred search head to Cluster Master of Indexer cluster.

I am also running Cluster Search Head & Non clustered search head to same indexer cluster for different requirement.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi realsplunk,
are you sure that you are sending events to your two indexers in auto load balancing and not in duplication?
verify it in outputs.conf file of your Universal Forwarder:
you are using auto load balancing if you have:

[tcpout:stanza1]
server = xxx.xxx.xxx.xxx:9997, yyy.yyy.yyy.yyy:9997

you aren't using auto load balancing if you have:

[tcpout:stanza1]
server = xxx.xxx.xxx.xxx:9997
[tcpout:stanza2]
server =  yyy.yyy.yyy.yyy:9997

Bye.
Giuseppe

splunkreal
Motivator

Hi cusello, auto load balancing is OK, there isn't duplicate events if I search on one of the clustered search heads, only on that new separated search head with distributed search.

[tcpout]
defaultGroup = ssl_9997
useACK = true
maxQueueSize = 100MB
[tcpout:ssl_9997]
server = xxx01:9997, yyy01:9997
[tcpout-server://xxx01:9997]
sslCertPath=$SPLUNK_HOME/etc/auth/server.pem
sslPassword=password
sslRootCAPath=$SPLUNK_HOME/etc/apps/APP_OUTPUTS/local/cacert.pem
sslVerifyServerCert=false
[tcpout-server://yyy01:9997]
sslCertPath=$SPLUNK_HOME/etc/auth/server.pem
sslPassword=password
sslRootCAPath=$SPLUNK_HOME/etc/apps/APP_OUTPUTS/local/cacert.pem
sslVerifyServerCert=false

Thanks for your help.

* If this helps, please upvote or accept solution 🙂 *
0 Karma

gcusello
SplunkTrust
SplunkTrust

splunkreal
Motivator

It works, thanks a lot!

I kept 'standalone' mode on that search head and only search head role.

* If this helps, please upvote or accept solution 🙂 *
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...