Splunk Search

Why is clustered indexers duplicating non-load balanced data?

scottrunyon
Contributor

When I run a simple query "index=syslog update sourcetype=fgt_event devname=xxxxx", it returns duplicate (2) events with the only difference being the splunk_server field. The device is sending syslog data to only one of the indexers. I am using the standard UDP:514 Data Input to receive this data.

Splunk setup
2 server indexing cluster
2 non-clustered search heads.

Question 1 - Is this affecting my licence quota? Syslog data my largest source.
Question 2 - How do I clean this up as it is affecting reporting?

Thank you in advance for any help provided.

0 Karma
1 Solution

jdonn_splunk
Splunk Employee
Splunk Employee

Can you double check the values for source and sourcetype? If they are truly duplicated events, I would think that your search head is not configured correctly.

http://docs.splunk.com/Documentation/Splunk/6.4.2/Indexer/Aboutclusters

More specifically - http://docs.splunk.com/Documentation/Splunk/6.4.2/Indexer/Configurethesearchhead

View solution in original post

0 Karma

jdonn_splunk
Splunk Employee
Splunk Employee

Can you double check the values for source and sourcetype? If they are truly duplicated events, I would think that your search head is not configured correctly.

http://docs.splunk.com/Documentation/Splunk/6.4.2/Indexer/Aboutclusters

More specifically - http://docs.splunk.com/Documentation/Splunk/6.4.2/Indexer/Configurethesearchhead

0 Karma

scottrunyon
Contributor

It turns out the the firewall that should be sending data to only one indexer was actually configured to send data to both indexers in the cluster. Splunk was performing as expected.

0 Karma

lycollicott
Motivator

Can you post your 2 events??

0 Karma

scottrunyon
Contributor

It turns out the the firewall that should be sending data to only one indexer was actually configured to send data to both indexers in the cluster. Splunk was performing as expected.

0 Karma

scottrunyon
Contributor

As there are multiple events returned with each search, what are you wanting me to post?

When the search completes and you look at the events tab, all fields are the same except the splunk_server field which has a name of one of the two indexers.

I tried to verify that the data is being indexed only once by using the following search -

sourcetype=fgt_event | eval dupfield=_raw | transaction dupfield maxspan=1s keepevicted=true | where mvcount(sourcetype) > 1

There were no duplicated raw values.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...