Duplicate logs in Splunk for multiple sourcetypes?

[sekhar463]

Hi All,

Good day, we are getting Duplicate logs in Splunk for multiple sources with same event example below

how to avoid duplicate logs 

index=ivz_unix_linux_events _raw="[2023-02-14 02:22:01.363] [TRACE] shiny-server - Uploading metrics data..."

2/14/23
1:52:01.363 PM
[2023-02-14 02:22:01.363] [TRACE] shiny-server - Uploading metrics data...
host = usapprstdld101source = /var/log/shiny-server.logsourcetype = shiny-server
2/14/23
1:52:01.363 PM
[2023-02-14 02:22:01.363] [TRACE] shiny-server - Uploading metrics data...
host = usapprstdld101source = /var/log/shiny-server.logsourcetype = shiny-server

[sekhar463]

can you tell how can i check this 

[gcusello]

Hi @sekhar463,

at first identify the host that generates duplicated logs,

Are your logs from syslog or from a Forwarder?

if from syslog, probably the issue is that you configured the appliance to send to two Splunk servers and you have to disable one of these sendings or (better) use a load balancer.

if you're receiving logs from two hosts, you have a cluster issue, so you have to choose one source to enable, disabling the other.

If instead you have only one host, see inputs.conf using btool (https://docs.splunk.com/Documentation/Splunk/9.0.3/Troubleshooting/Usebtooltotroubleshootconfigurati...) to understand which configurations generates the duplicated logs.

Ciao.

giuseppe

[sekhar463]

i can see only one input for this logs source 

/opt/splunk/etc/apps/Splunk_TA_nix/local/inputs.conf index = ivz_unix_linux_events
/opt/splunk/etc/apps/Splunk_TA_nix/local/inputs.conf [monitor:///var/log]
/opt/splunk/etc/apps/Splunk_TA_nix/local/inputs.conf disabled = false

[gcusello]

Hi @sekhar463,

let me summarize:

duplicated logs are from a linux server and there isn't crcSalt in your inputs.conf.

So see in the source of your duplicated logs if they come from the same log file.

if from the same log file, open it and see if the log is generated twice from Linux, in this case you have to intervene in Linux.

if from different log files, identify them and see if you have to blacklist one of them.

Check also if one of the duplicated source files is the other in a zipped file.

Ciao.

Giuselle

[sekhar463]

nothing as such as mentioned points 

one more ex events  .see same log 

2/15/23
1:13:13.000 PM
"#includedir",
host = usoraosfclt100source = /etc/insights-client/.cache.jsonsourcetype = unknown-3
2/15/23
1:13:13.000 PM
"#includedir",
host = usoraosfclt100source = /etc/insights-client/.cache.jsonsourcetype = unknown-3
2/15/23
1:09:21.000 PM
"#includedir",
host = usoraosfclt100source = /etc/insights-client/.cache.jsonsourcetype = unknown-3
2/15/23
1:09:21.000 PM
"#includedir",
host = usoraosfclt100source = /etc/insights-client/.cache.jsonsourcetype = unknown-3

[gcusello]

Hi @sekhar463,

"sourcetype = unknown-3" means that in your input sourcetype isn't defines and leaved to Splunk identification.

What's the inputs.conf to take these logs?

Ciao.

Giuseppe

[sekhar463]

can crcsalr resolve this 

if yes what is the syntax to add where should i add this 

[gcusello]

Hi @sekhar463,

crcSalt is useful to reindex already indexed data, because Splunk doesn't index a log twice.

In your case, you have to understand, why you have duplicated logs.

As I said, maybe there's an input with crcSalt so logs are read two times, but follow the debugging steps I hinted.

Ciao.

Giuseppe

[gcusello]

Hi @sekhar463 ,

you should analyze the input that generates this log.

Chjeck if there's an input that uses "crcSalt = <SOUCE>" because, without this option, Splunk doesn't index twice a log.

the, are you ingesting logs from a cluster?

Check also if the log is twicy generated by the log source.

ciao.

Giuseppe