Getting Data In

avoid duplicate indexing in splunk

c_sahil
New Member

I have a scheduler which logs the data to my log file every hour, the log I use in splunk. Now the problem is every time scheduler runs it appends some row but in the splunk when I query I get (double the no. of rows + added rows as result). how can avoid this duplicate indexing. below ex. will clearly explain my problem.

before scheduler run log have 3 rows. after scheduler run it add 1 more row to log and total no. of rows in log is 4 but in splunk when I query it gives me (3*2+1) 7 rows how to avoid this. please help

Tags (1)
0 Karma
1 Solution

Matthias_BY
Communicator

Hello,

how are you loading in the data? Via Splunk File Monitoring inputs.conf?

Splunk Forwarders + Indexer in File Monitoring are remembering in files when did it stop, up to which line it was captured already etc. this is called "Fish Bucket".

You can read more here:
http://blogs.splunk.com/2008/08/14/what-is-this-fishbucket-thing/

Br
Matthias

View solution in original post

0 Karma

Matthias_BY
Communicator

Hello,

how are you loading in the data? Via Splunk File Monitoring inputs.conf?

Splunk Forwarders + Indexer in File Monitoring are remembering in files when did it stop, up to which line it was captured already etc. this is called "Fish Bucket".

You can read more here:
http://blogs.splunk.com/2008/08/14/what-is-this-fishbucket-thing/

Br
Matthias

0 Karma

Matthias_BY
Communicator

Hi,

great. so just to make clear. your application which did write the log Splunk monitors did write new lines at the bottom instead appending it, right?

0 Karma

c_sahil
New Member

Thanks Matthias...
Fixed the issue after reading the blog, I was adding the data in the log at start instead of adding it at the bottom.
Thank a lot.

0 Karma
Get Updates on the Splunk Community!

Pro Tips for First-Time .conf Attendees: Advice from SplunkTrust

Heading to your first .Conf? You’re in for an unforgettable ride — learning, networking, swag collecting, ...

Raise Your Skills at the .conf25 Builder Bar: Your Splunk Developer Destination

Calling all Splunk developers, custom SPL builders, dashboarders, and Splunkbase app creators – the Builder ...

Hunt Smarter, Not Harder: Discover New SPL “Recipes” in Our Threat Hunting Webinar

Are you ready to take your threat hunting skills to the next level? As Splunk community members, you know the ...