Logs sent over FTP are indexed with nulls \x00

[Dan]

My Bluecoat logs are sent over ftp every 5 minutes to a ftp server (linux), and my Splunk indexer (linux) is nfs mounting and monitoring the drop location. The issue is when a new file appears, it is padded with 0 bytes, and Splunk indexes this as \x00*. After a second, real data is written over the bytes, and Splunk indexes those events correctly (for the most part, although sometimes the first few linebreaks are garbled).

Has anyone experienced the same issue with ftp? Could this be a configuration issue on the Bluecoats? i.e. ftp transfer is in binary mode, not ascii?

Thanks!

[ahall_splunk]

This isn't actually an issue with the Blue Coat ProxySG. It's an issue with the FTP Server. Following the RFC, the Blue Coat device will connect via FTP and do any authentication, then send the command "ALLO size" where size is the size of the log file. This allocates space for the file. This allows the Blue Coat device to be reasonably sure that when it actually sends the FTP STOR command (to actually send the file), the file will be received in its entirety.

To fix this on Linux, you can use ProFTPD (instead of the more normal vsftpd) and use the HiddenStores directive in /etc/proftpd.conf.