I have a single site cluster that contains 5 indexers, 4 search heads, a master node, and a deployer. There are also some universal forwarders with load balancing.
All events in the indexer cluster are from Universal forwarders. The data flow direction is as follows.(The most common cluster architecture)
Server/Host (UF installed here)—————TCP—————>indexer cluster
Server/Host(syslog)—————Universal Forwarder—————TCP—————indexer cluster
Server/Host(UF monitors a file)——————TCP————>Indexer cluster
So the question is coming
Why does it return duplicate events when I search? Is it because I'm using TCP? https://answers.splunk.com/answers/537368/why-is-there-event-duplication-via-tcp-port.html?
I disabled the use_ACK function in the outputs.conf on the UF
What are the common causes of repeated events? Please tell me, I can exclude it one by one. Thank you
Forgive me for my English
Hi xsstest,
Here are some steps to debug,
index=<your index> sourcetype=<sourcetype> source="<source>" host="<host>" | eval bucket=_bkt | eval indextime=_indextime |table _time, indextime, bucket splunk_server _raw | convert ctime(indextime) | stats count list(*) as * by _raw | where count>1 | fields * _raw
posting more information will helpful to assist further.
Hi,
i have a similar problem... at first my config to forward internal splunk indexes...
[tcpout]
forwardedindex.0.whitelist = _.*
forwardedindex.filter.disable = false
defaultGroup = TEST_IDX-CLUSTER
[tcpout:TEST_IDX-CLUSTER]
forceTimebasedAutoLB = true
autoLBFrequency = 30
server = ID01.SPLUNK-TEST.local:9997,ID02.SPLUNK-TEST.local:9997,ID03.SPLUNK-TEST.local:9997
Then i debug with this search...
index="_internal"
| eval bucket=_bkt
| eval indextime=_indextime
| table _time, indextime, bucket splunk_server _raw
| convert ctime(indextime)
| stats count list(*) as * by _raw
| where count>1
| fields * _raw
| sort - indextime
Output:
bucket = every bucket is another in one event
count = 2 or sometimes 3
indextime = every entry is equal
splunk_server = 01,02,03 or 01,01 or 02,03 or 03,03 (many different combinations)
Anyone an idea?
Regards - Markus
What are the common causes of repetitive events? How to carry out one by one investigation?
Hi xsstest,
Here are some steps to debug,
index=<your index> sourcetype=<sourcetype> source="<source>" host="<host>" | eval bucket=_bkt | eval indextime=_indextime |table _time, indextime, bucket splunk_server _raw | convert ctime(indextime) | stats count list(*) as * by _raw | where count>1 | fields * _raw
posting more information will helpful to assist further.
count : 416
count : 500
etc .. Returned a total of 3518 results
Some events are repeated twice, and some events are repeated hundreds of times
Can I use the dedup_time _raw
to exclude duplicate events when I search?
@vasanthmss
The reason was found: because the rsyslog configuration error, the same InputFileFacility
was used
yes, you can filter the duplicate using dedup during the search time. Have you check the source file have duplicate or not? If yes then you can use the dedup other wise try to figure out why the events were duplicated. Any luck with splunk_server and bucket analysis?