Some time we are getting same event more then 5 time, However that log is only available once in log file

[csharm21]

Some time we are getting same event more then 5 time, However that log is only available once in log file.

[harsmarvania57]

If I am understanding your question correctly, you have logfile which contain unique event but that event is available in Splunk 5 times.

As you didn't provided more information how you are ingesting data into Splunk so if I assume that you are ingesting data from Universal Forwarder to Indexer and if you have enabled useACK on Universal Forwarder in that case when Indexer not able to send ACK back to UF it will send same data again. See this documentation for more information http://docs.splunk.com/Documentation/Forwarder/7.2.1/Forwarder/Protectagainstthelossofin-flightdata#...

It will be good if you provide as much information as you can.

[csharm21]

Hi Harsmarvania57,

Yes you understand my issue correctly, but we are using forwarder to send the data from openshift POD,s

OSE containers uses indexer discovery using cluster master connection

[indexer_discovery:prod_cm]
pass4SymmKey = YEhYu124eAfdfdf
master_uri = https://chandra-splunk-cm.test.com:8089

[tcpout:Forward_To_chandra_Splunk_Indexers]
disabled = 0
indexerDiscovery = prod_cm
useClientSSLCompression = true
sslVersions=*,-ssl2
clientCert = $SPLUNK_HOME/etc/apps/chandra_outputs/certs/forwarderCertificate.pem
sslPassword = splunk_forwarder
sslVerifyServerCert = true
sslAltNameToCheck = chandra-splunk-idx.test.com

[tcpout]
defaultGroup = Forward_To_chandra_Splunk_Indexers

[harsmarvania57]

Configuration looks ok, any ERROR or WARNING message on UF or IDX in splunkd.log ? Also can you please check _time and _indextime for those duplicate events ?

Additionally as you are running Indexer Clustering just to double check have you added all Indexers as search peer in Search Head? If yes then based on RF and SF you will see duplicate data (But in this case all data will be displayed SF times however you mentioned that you are seeing duplicate data sometimes), in this case you need to point Search Head to Cluster Master instead of adding search peer in Search Head.

[csharm21]

Hi @Harsmarvania57

event time and index time is looks different. and even no error on logs.

12/13/18
10:05:45.000 PM
2018-12-14T04:05:45+00:00 at=info correlates="faffb766d5337ef24ec8e7eae95f6753" session="0aa63c66-ee21-4f16-a412-dae26f475854" method=GET path="/unavailable" host=host.com port=443 took=30111ms status=500 bytes=75405 uuid=ilike2skip0w
host = pr1-app-89-cta3k source = /log/pr1-app-89-cta3k.log sourcetype = ose:engageui:prd status = 500
12/13/18
10:05:45.000 PM
2018-12-14T04:05:45+00:00 at=info correlates="faffb766d5337ef24ec8e7eae95f6753" session="0aa63c66-ee21-4f16-a412-dae26f475854" method=GET path="/unavailable" host=host.com port=443 took=30111ms status=500 bytes=75405 uuid=ilike2skip0w
host = pr1-app-89-cta3k source = /log/pr1-app-89-cta3k.log sourcetype = ose:engageui:prd status = 500

10:05:45.000 PM
2018-12-14T04:05:45+00:00 at=info correlates="faffb766d5337ef24ec8e7eae95f6753" session="0aa63c66-ee21-4f16-a412-dae26f475854" method=GET path="/unavailable" host=host.com port=443 took=30111ms status=500 bytes=75405 uuid=ilike2skip0w

[hijacob]

Please specify your question. Can you show us a screenshot?