We are currently using HUNK and MapR to read in some dummy data which is basically a collection of repeating events. We are trying to test the data ingestion of MapR.
In my script, at first, I had the event divided by one newline. Then this error happened. So I added a new newline thinking that maybe a few events were being concatenated. But the error still came back.
The question is.. how do we avoid seeing this error? It says we can set mapreduce.input.linrecordreader.line.maxlength to a lower value but as far as I can see the only place that is located is in a java file. Is there an xml properties files this is located in I can change?
Anyone else had an error like this?
IOException - Out of memory error while reading a very large single line input record. To skip this record set mapreduce.input.linerecordreader.line.maxlength to a lower value. Current value: 2147483647, jvm heap size: 508035072, potential value: 31752192
This was discovered to be produced from an artifact in our rotate logs script. We recreated the perl script in Python and it doesnt happen anymore.
This was discovered to be produced from an artifact in our rotate logs script. We recreated the perl script in Python and it doesnt happen anymore.