topic Re: Problem to index the entire CSV file in Getting Data In

Problem to index the entire CSV file

harry521 — Mon, 02 Oct 2017 14:47:05 GMT

we use csv to track app's performance. I added the csv to forwarder and keep monitoring it. The problem is that while app is running it keeps writing to the CSV, however, only the few minutes at the beginning will be indexed but not the rest. If I move the whole file, while no more update, to another location, I can get everything indexed. Any suggestions?

inputs.conf
[monitor:../logs/performance.csv]
blacklist = -\d
index = test
sourcetype = csv
crcSalt = &ltSOURCE&gt

Re: Problem to index the entire CSV file

gcusello — Mon, 02 Oct 2017 16:15:41 GMT

Hi,
Try to Delete the crcSalt row and restart Splunk.
Remember that to correctly Index csv files, you have to deploy on forwarder also props.conf.
Bye.
Giuseppe

Re: Problem to index the entire CSV file

somesoni2 — Mon, 02 Oct 2017 16:24:29 GMT

You want to index the entire file or just index what's new?? Does your application re-writes the entire file frequently (daily/hourly etc).

Re: Problem to index the entire CSV file

harry521 — Mon, 02 Oct 2017 16:41:02 GMT

Thanks Cusello. I should have noticed the special char being removed when I posted it. It should be "crcSalt = &ltSOURCE&gt". I did try with/without that line. Doesn't help.

I looked into that little bit more, and found out that the data seemed to have been indexed, but, only the first few minutes are correct, and at certain point, the time stamp all messed up which being set months back.

I then created a custom csv sourcetype with specifying the timestamp format and field. That seems to have addressed my problem. I was not thinking of doing that because the timestamp being logged doesn't have millisecond, nor a date.