I have noticed an issue in my Splunk environment:
Data is getting duplicated twice in indexers. If i do a search in search head, the same events are coming in twice. this issue started 2 days ago, earlier there is no issue with the data.
1)checked the application logs wether same log is existing twice? Answer: No
2)Checked whether this issue is happening to one sourcetype OR only for one index OR one forwarder? Answer: No it is affecting all forwarders and indexers data.
Thanks in advance.
Something changed in your configuration. Did someone change outputs.conf. on the forwarders?
If no one changed the source data files, then someone must have changed a Splunk setting in some .conf file
are the duplicate events coming from the same bucket or different buckets? you can isolate one of the duplicate events, and then check with bucket+splunk_server the event and its duplicates are being returned from
"somedupevent | eval bkt=bkt | fields + bkt,splunkserver"
Seems like an issue from 6.3.x upgrade to newer version after 6.4.x would fix the issue.