Remove Trailing Data After 65K bytes

[ShaneNewman]

I am using Splunk DB Connect to push data into a Teradata database, the limitation of the ODBC driver is that is can only handle 64K bytes of data in a single field... which on this data set (HL7) is fairly common. Is there a command I can use in Splunk to strip out the remaining data so this will not fail? I was thinking SED but I do not know how to set that up for bytes...

[jcoates_splunk · Accepted solution]

Hi Shane,

Are you trying to emit raw records or tabular data? If a table, you can just eval down to a smaller data set.

[ShaneNewman]

For lab and rad results the "column" containing the HL7 message can exceed 500K characters. For our purposes, we are thinking that most of the data we need will be in the first 64K characters. I will give this a shot tomorrow.

[araitz]

Yeah, something like:

... | eval field = substr(field, 1, 64000)

I didn't test, so you might need to do an if statement to only truncate if there are more than 64k characters. As for doing this for every field in arbitrary results, this would be quite hard.