Getting Data In

Duplicate logs in Splunk for multiple sourcetypes?

sekhar463
Path Finder

Hi All,

Good day, we are getting Duplicate logs in Splunk for multiple sources with same event example below

how to avoid duplicate logs 


index=ivz_unix_linux_events _raw="[2023-02-14 02:22:01.363] [TRACE] shiny-server - Uploading metrics data..."

 

2/14/23
1:52:01.363 PM
[2023-02-14 02:22:01.363] [TRACE] shiny-server - Uploading metrics data...
host = usapprstdld101source = /var/log/shiny-server.logsourcetype = shiny-server
2/14/23
1:52:01.363 PM
[2023-02-14 02:22:01.363] [TRACE] shiny-server - Uploading metrics data...
host = usapprstdld101source = /var/log/shiny-server.logsourcetype = shiny-server

Labels (2)
0 Karma

sekhar463
Path Finder

can you tell how can i check this 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @sekhar463,

at first identify the host that generates duplicated logs,

Are your logs from syslog or from a Forwarder?

if from syslog, probably the issue is that you configured the appliance to send to two Splunk servers and you have to disable one of these sendings or (better) use a load balancer.

if you're receiving logs from two hosts, you have a cluster issue, so you have to choose one source to enable, disabling the other.

If instead you have only one host, see inputs.conf using btool (https://docs.splunk.com/Documentation/Splunk/9.0.3/Troubleshooting/Usebtooltotroubleshootconfigurati...) to understand which configurations generates the duplicated logs.

Ciao.

giuseppe

0 Karma

sekhar463
Path Finder

i can see only one input for this logs source 

 

/opt/splunk/etc/apps/Splunk_TA_nix/local/inputs.conf index = ivz_unix_linux_events
/opt/splunk/etc/apps/Splunk_TA_nix/local/inputs.conf [monitor:///var/log]
/opt/splunk/etc/apps/Splunk_TA_nix/local/inputs.conf disabled = false

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @sekhar463,

let me summarize:

duplicated logs are from a linux server and there isn't crcSalt in your inputs.conf.

So see in the source of your duplicated logs if they come from the same log file.

if from the same log file, open it and see if the log is generated twice from Linux, in this case you have to intervene in Linux.

if from different log files, identify them and see if you have to blacklist one of them.

Check also if one of the duplicated source files is the other in a zipped file.

Ciao.

Giuselle

0 Karma

sekhar463
Path Finder

nothing as such as mentioned points 

one more ex events  .see same log 

 

2/15/23
1:13:13.000 PM
"#includedir",
host = usoraosfclt100source = /etc/insights-client/.cache.jsonsourcetype = unknown-3
2/15/23
1:13:13.000 PM
"#includedir",
host = usoraosfclt100source = /etc/insights-client/.cache.jsonsourcetype = unknown-3
2/15/23
1:09:21.000 PM
"#includedir",
host = usoraosfclt100source = /etc/insights-client/.cache.jsonsourcetype = unknown-3
2/15/23
1:09:21.000 PM
"#includedir",
host = usoraosfclt100source = /etc/insights-client/.cache.jsonsourcetype = unknown-3

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @sekhar463,

"sourcetype = unknown-3" means that in your input sourcetype isn't defines and leaved to Splunk identification.

What's the inputs.conf to take these logs?

Ciao.

Giuseppe

0 Karma

sekhar463
Path Finder

can crcsalr resolve this 

if yes what is the syntax to add where should i add this 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @sekhar463,

crcSalt is useful to reindex already indexed data, because Splunk doesn't index a log twice.

In your case, you have to understand, why you have duplicated logs.

As I said, maybe there's an input with crcSalt so logs are read two times, but follow the debugging steps I hinted.

Ciao.

Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @sekhar463 ,

you should analyze the input that generates this log.

Chjeck if there's an input that uses "crcSalt = <SOUCE>" because, without this option, Splunk doesn't index twice a log.

the, are you ingesting logs from a cluster?

Check also if the log is twicy generated by the log source.

ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...