topic Re: Remove Trailing Data After 65K bytes in Knowledge Management

Remove Trailing Data After 65K bytes

ShaneNewman — Wed, 26 Mar 2014 21:31:40 GMT

I am using Splunk DB Connect to push data into a Teradata database, the limitation of the ODBC driver is that is can only handle 64K bytes of data in a single field... which on this data set (HL7) is fairly common. Is there a command I can use in Splunk to strip out the remaining data so this will not fail? I was thinking SED but I do not know how to set that up for bytes...

Re: Remove Trailing Data After 65K bytes

jcoates_splunk — Wed, 26 Mar 2014 22:41:05 GMT

Hi Shane,

Are you trying to emit raw records or tabular data? If a table, you can just eval down to a smaller data set.

Re: Remove Trailing Data After 65K bytes

araitz — Thu, 27 Mar 2014 03:21:54 GMT

Yeah, something like:

... | eval field = substr(field, 1, 64000)

I didn't test, so you might need to do an if statement to only truncate if there are more than 64k characters. As for doing this for every field in arbitrary results, this would be quite hard.

Re: Remove Trailing Data After 65K bytes

ShaneNewman — Fri, 28 Mar 2014 01:08:33 GMT

For lab and rad results the "column" containing the HL7 message can exceed 500K characters. For our purposes, we are thinking that most of the data we need will be in the first 64K characters. I will give this a shot tomorrow.