Getting Data In

Monitored txt file keeps re-indexing events

ALXWBR
Path Finder

PLEASE HELP!

This has been driving me mad for days! Every time an event is added, its re-reading the text file from the start and re-indexing events. I am getting hundreds of duplicate events and have tried a variety of combos in the inputs.conf, but still cant solve it!

I am monitoring a series of text files. Each day a new .txt file is created and events are written into this text continuously throughout the day, until the beginning of the next, where again a new file is created. the files are named as follows.

Statistics_20211104_034330_840.txt

The contents of the file is as follows

QPS statistics: SW-Version:3.64 [UTC+00:00]
time,id,valid,invalid,mode,......[ETC ETC ETC]
2021-11-04T03:43:19+00:00,248559,1,0,A,....[ETC ETC ETC]
2021-11-04T03:43:19+00:00,248560,1,0,A,....[ETC ETC ETC]

This is what I currently have in the inputs.conf

[monitor://\\Lgwnasapp002\bsr$\]
disabled = false
index = idx_security_scanner
sourcetype = QPSdata
whitelist = .+Statistics_\d{8}_\d{6}_\d{1,5}\.txt
crcSalt = <SOURCE>

Any ideas?

Labels (3)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Have you tried adding crcSalt = <SOURCE> to the monitor stanza?

---
If this reply helps you, Karma would be appreciated.
0 Karma

ALXWBR
Path Finder

Yup, no joy.

I've just been monitoring it more carefully. It actually looks like its indexing each event ~100 times in the first place. I ran a real time search to see each event come in live and this one for example indexed 99 times.

 

Capture.PNG

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...