For some reason Splunk is not consistently breaking on date. I have tried using a REGEX instead of breaking on date. I have also re-arranged so that the line "Unix_process_pid" is the start of the event. The same inconsistent behaviour remains. Around 90% are correctly broken into events and the remainder are truncated. Examples:
On further inspection the script which outputs the data to the file prints each line individually. So although it takes milliseconds it looks like Splunk's tail on the file is consuming the data before the entire event has been printed to the file.
I've just tried this out by manually adding data to the file and it is breaking at random points.