Knowledge Management

Remove Trailing Data After 65K bytes

ShaneNewman
Motivator

I am using Splunk DB Connect to push data into a Teradata database, the limitation of the ODBC driver is that is can only handle 64K bytes of data in a single field... which on this data set (HL7) is fairly common. Is there a command I can use in Splunk to strip out the remaining data so this will not fail? I was thinking SED but I do not know how to set that up for bytes...

0 Karma
1 Solution

jcoates_splunk
Splunk Employee
Splunk Employee

Hi Shane,

Are you trying to emit raw records or tabular data? If a table, you can just eval down to a smaller data set.

View solution in original post

jcoates_splunk
Splunk Employee
Splunk Employee

Hi Shane,

Are you trying to emit raw records or tabular data? If a table, you can just eval down to a smaller data set.

ShaneNewman
Motivator

For lab and rad results the "column" containing the HL7 message can exceed 500K characters. For our purposes, we are thinking that most of the data we need will be in the first 64K characters. I will give this a shot tomorrow.

0 Karma

araitz
Splunk Employee
Splunk Employee

Yeah, something like:

... | eval field = substr(field, 1, 64000)

I didn't test, so you might need to do an if statement to only truncate if there are more than 64k characters. As for doing this for every field in arbitrary results, this would be quite hard.

Get Updates on the Splunk Community!

.conf25 Community Recap

Hello Splunkers, And just like that, .conf25 is in the books! What an incredible few days — full of learning, ...

Splunk App Developers | .conf25 Recap & What’s Next

If you stopped by the Builder Bar at .conf25 this year, thank you! The retro tech beer garden vibes were ...

Congratulations to the 2025-2026 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...