We are currently using HUNK and MapR to read in some dummy data which is basically a collection of repeating events. We are trying to test the data ingestion of MapR.
In my script, at first, I had the event divided by one newline. Then this error happened. So I added a new newline thinking that maybe a few events were being concatenated. But the error still came back.
The question is.. how do we avoid seeing this error? It says we can set mapreduce.input.linrecordreader.line.maxlength to a lower value but as far as I can see the only place that is located is in a java file. Is there an xml properties files this is located in I can change?
Anyone else had an error like this?
IOException - Out of memory error while reading a very large single line input record. To skip this record set mapreduce.input.linerecordreader.line.maxlength to a lower value. Current value: 2147483647, jvm heap size: 508035072, potential value: 31752192
