Data getting indexed twice in every field of splunk cluster environment

[architkhanna]

Hi All,

I have a splunk cluster environment where, while pulling data from a source, itgets indexed twice, not as a separate event, but within same event. So all fields have same value coming twice , making it a multivalue field.

Same source code works fine on a standalone splunk server but fails on a cluster. 
I have tried to have props.conf present only in data app of indexer , however, with with that field extraction does not happen. If I keeps props.conf in both HF and data app, field extraction happens but with above issue.

Would appreciate if anyone has any lead on this.

TIA.

[kiran_panchavat]

@architkhanna Can you confirm how your inputs.conf and outputs.conf is configured?

[architkhanna]

@kiran_panchavat 
These are present on server level on Indexers.

[inputs.conf]
[default]
host = 10.100.5.5
[splunktcp://9997]
disabled = 0

[output.conf]

[tcpout]
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup|_configtracker)
forwardedindex.filter.disable = false
indexAndForward = false
blockOnCloning = true
compressed = false
disabled = false
dropClonedEventsOnQueueFull = 5
dropEventsOnQueueFull = -1
heartbeatFrequency = 30
maxFailuresPerInterval = 2
secsInFailureInterval = 1
maxConnectionsPerIndexer = 2
forceTimebasedAutoLB = false
sendCookedData = true
connectionTimeout = 20
readTimeout = 300
writeTimeout = 300
tcpSendBufSz = 0
ackTimeoutOnShutdown = 30
useACK = false
blockWarnThreshold = 100
sslQuietShutdown = false
useClientSSLCompression = true
autoLBVolume = 0
maxQueueSize = auto
connectionTTL = 0
autoLBFrequency = 30
sslVersions = tls1.2
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256
ecdhCurves = prime256v1, secp384r1, secp521r1

[syslog]
type = udp
priority = <13>
maxEventSize = 1024

[rfs]
batchTimeout = 30
batchSizeThresholdKB = 2048
dropEventsOnUploadError = false
compression = zstd
compressionLevel = 3

[kiran_panchavat]

@architkhanna 

If it is possible, provide inputs.conf and outputs.conf from the source side(UF).

Maybe your log files are rotating and splunk is detecting the copy as a new log file to index.

please check if :

you are using the crcSalt option. If you are using "crcSalt=<SOURCE>" with rotated logs, this could also cause duplicates. This happens because the rotated file may stay in the same directory with a different name.

check the rotation of your files, if no first lines are modified during the process.

symlinks, verify that the multiple symlinks are not pointing to the same file/folder

[kiran_panchavat]

@architkhanna Hello, can you please go through this link Solved: Why are there many duplicate events in the indexer... - Splunk Community

[architkhanna]

@kiran_panchavat  This explains and confirms the issue that we do have multiple events in index but does not explain the steps to fix this. Let me know if I'm missing something.