Problem to index the entire CSV file

[harry521]

we use csv to track app's performance. I added the csv to forwarder and keep monitoring it. The problem is that while app is running it keeps writing to the CSV, however, only the few minutes at the beginning will be indexed but not the rest. If I move the whole file, while no more update, to another location, I can get everything indexed. Any suggestions?

inputs.conf
[monitor:../logs/performance.csv]
blacklist = -\d
index = test
sourcetype = csv
crcSalt = &ltSOURCE&gt

[somesoni2]

You want to index the entire file or just index what's new?? Does your application re-writes the entire file frequently (daily/hourly etc).

[gcusello]

Hi,
Try to Delete the crcSalt row and restart Splunk.
Remember that to correctly Index csv files, you have to deploy on forwarder also props.conf.
Bye.
Giuseppe

[harry521]

Thanks Cusello. I should have noticed the special char being removed when I posted it. It should be "crcSalt = &ltSOURCE&gt". I did try with/without that line. Doesn't help.

I looked into that little bit more, and found out that the data seemed to have been indexed, but, only the first few minutes are correct, and at certain point, the time stamp all messed up which being set months back.

I then created a custom csv sourcetype with specifying the timestamp format and field. That seems to have addressed my problem. I was not thinking of doing that because the timestamp being logged doesn't have millisecond, nor a date.