Splunk Search

Double counts in search app

rgeddes
Engager

basic set up:
- splunk 4.2 on ubuntu 10.04
- rsyslog collects logs from other machines, and splunk reads and tabulates event data from /var/log/*

I noticed that a cron.hourly process which normally generates 2 events/hour in the search app, jumped up to 4 events/hour(which were duplicates), for a 24 hour period, and then returned to 2 events/hour.

grepping /var/log/syslog for that particular event does not show duplicates, and when I ask the search app to show the source, the source does not show duplicates either.

Has anyone experienced double counting before? If yes, how did you resolve?

Thanks
Richard

Tags (2)
0 Karma

rgeddes
Engager

I just realized that the duplicate entries are tagged with different sources.

One came from /var/log/syslog, and the other came from /var/log/syslog.2.gz

I grepped though /var/log/syslogs*, and can only find one pair of events with a particular process id... which splunk search app shows as happening twice with the exact same timestamp. The duplication lasts for a 24 hour period.

I was thinking that maybe log rotation created a condition to allow this, but the duplication continues for 24 hours.... far longer than it takes to rotate the logs.

Anyway, still not resolved, but extra information.

R

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...