Monitored txt file keeps re-indexing events

[ALXWBR]

PLEASE HELP!

This has been driving me mad for days! Every time an event is added, its re-reading the text file from the start and re-indexing events. I am getting hundreds of duplicate events and have tried a variety of combos in the inputs.conf, but still cant solve it!

I am monitoring a series of text files. Each day a new .txt file is created and events are written into this text continuously throughout the day, until the beginning of the next, where again a new file is created. the files are named as follows.

Statistics_20211104_034330_840.txt

The contents of the file is as follows

QPS statistics: SW-Version:3.64 [UTC+00:00]
time,id,valid,invalid,mode,......[ETC ETC ETC]
2021-11-04T03:43:19+00:00,248559,1,0,A,....[ETC ETC ETC]
2021-11-04T03:43:19+00:00,248560,1,0,A,....[ETC ETC ETC]

This is what I currently have in the inputs.conf

[monitor://\\Lgwnasapp002\bsr$\]
disabled = false
index = idx_security_scanner
sourcetype = QPSdata
whitelist = .+Statistics_\d{8}_\d{6}_\d{1,5}\.txt
crcSalt = <SOURCE>

Any ideas?

[richgalloway]

Have you tried adding crcSalt = <SOURCE> to the monitor stanza?

[ALXWBR]

Yup, no joy.

I've just been monitoring it more carefully. It actually looks like its indexing each event ~100 times in the first place. I ran a real time search to see each event come in live and this one for example indexed 99 times.