topic Re: Duplicate logs in Splunk for multiple sourcetypes in Getting Data In

Duplicate logs in Splunk for multiple sourcetypes?

sekhar463 — Wed, 15 Feb 2023 15:33:26 GMT

Hi All,

Good day, we are getting Duplicate logs in Splunk for multiple sources with same event example below

how to avoid duplicate logs

index=ivz_unix_linux_events _raw="[34m[2023-02-14 02:22:01.363] [TRACE] shiny-server - [39mUploading metrics data..."

2/14/23
1:52:01.363 PM
[34m[2023-02-14 02:22:01.363] [TRACE] shiny-server - [39mUploading metrics data...
host = usapprstdld101source = /var/log/shiny-server.logsourcetype = shiny-server
2/14/23
1:52:01.363 PM
[34m[2023-02-14 02:22:01.363] [TRACE] shiny-server - [39mUploading metrics data...
host = usapprstdld101source = /var/log/shiny-server.logsourcetype = shiny-server

Re: Duplicate logs in Splunk for multiple sourcetypes

gcusello — Tue, 14 Feb 2023 08:47:32 GMT

Hi @sekhar463 ,

you should analyze the input that generates this log.

Chjeck if there's an input that uses "crcSalt = <SOUCE>" because, without this option, Splunk doesn't index twice a log.

the, are you ingesting logs from a cluster?

Check also if the log is twicy generated by the log source.

ciao.

Giuseppe

Re: Duplicate logs in Splunk for multiple sourcetypes

sekhar463 — Tue, 14 Feb 2023 09:09:07 GMT

can you tell how can i check this

Re: Duplicate logs in Splunk for multiple sourcetypes

gcusello — Tue, 14 Feb 2023 09:22:00 GMT

Hi @sekhar463,

at first identify the host that generates duplicated logs,

Are your logs from syslog or from a Forwarder?

if from syslog, probably the issue is that you configured the appliance to send to two Splunk servers and you have to disable one of these sendings or (better) use a load balancer.

if you're receiving logs from two hosts, you have a cluster issue, so you have to choose one source to enable, disabling the other.

If instead you have only one host, see inputs.conf using btool (https://docs.splunk.com/Documentation/Splunk/9.0.3/Troubleshooting/Usebtooltotroubleshootconfigurations9) to understand which configurations generates the duplicated logs.

Ciao.

giuseppe

Re: Duplicate logs in Splunk for multiple sourcetypes

sekhar463 — Tue, 14 Feb 2023 10:22:06 GMT

i can see only one input for this logs source

/opt/splunk/etc/apps/Splunk_TA_nix/local/inputs.conf index = ivz_unix_linux_events
/opt/splunk/etc/apps/Splunk_TA_nix/local/inputs.conf [monitor:///var/log]
/opt/splunk/etc/apps/Splunk_TA_nix/local/inputs.conf disabled = false

Re: Duplicate logs in Splunk for multiple sourcetypes

gcusello — Tue, 14 Feb 2023 10:29:23 GMT

Hi @sekhar463,

let me summarize:

duplicated logs are from a linux server and there isn't crcSalt in your inputs.conf.

So see in the source of your duplicated logs if they come from the same log file.

if from the same log file, open it and see if the log is generated twice from Linux, in this case you have to intervene in Linux.

if from different log files, identify them and see if you have to blacklist one of them.

Check also if one of the duplicated source files is the other in a zipped file.

Ciao.

Giuselle

Re: Duplicate logs in Splunk for multiple sourcetypes

sekhar463 — Tue, 14 Feb 2023 11:16:24 GMT

can crcsalr resolve this

if yes what is the syntax to add where should i add this

Re: Duplicate logs in Splunk for multiple sourcetypes

gcusello — Tue, 14 Feb 2023 11:21:06 GMT

Hi @sekhar463,

crcSalt is useful to reindex already indexed data, because Splunk doesn't index a log twice.

In your case, you have to understand, why you have duplicated logs.

As I said, maybe there's an input with crcSalt so logs are read two times, but follow the debugging steps I hinted.

Ciao.

Giuseppe

Re: Duplicate logs in Splunk for multiple sourcetypes

sekhar463 — Wed, 15 Feb 2023 08:05:33 GMT

nothing as such as mentioned points

one more ex events .see same log

2/15/23
1:13:13.000 PM
"#includedir",
host = usoraosfclt100source = /etc/insights-client/.cache.jsonsourcetype = unknown-3
2/15/23
1:13:13.000 PM
"#includedir",
host = usoraosfclt100source = /etc/insights-client/.cache.jsonsourcetype = unknown-3
2/15/23
1:09:21.000 PM
"#includedir",
host = usoraosfclt100source = /etc/insights-client/.cache.jsonsourcetype = unknown-3
2/15/23
1:09:21.000 PM
"#includedir",
host = usoraosfclt100source = /etc/insights-client/.cache.jsonsourcetype = unknown-3

Re: Duplicate logs in Splunk for multiple sourcetypes

gcusello — Wed, 15 Feb 2023 08:09:33 GMT

Hi @sekhar463,

"sourcetype = unknown-3" means that in your input sourcetype isn't defines and leaved to Splunk identification.

What's the inputs.conf to take these logs?

Ciao.

Giuseppe