Knowledge Management

Remove Trailing Data After 65K bytes

ShaneNewman
Motivator

I am using Splunk DB Connect to push data into a Teradata database, the limitation of the ODBC driver is that is can only handle 64K bytes of data in a single field... which on this data set (HL7) is fairly common. Is there a command I can use in Splunk to strip out the remaining data so this will not fail? I was thinking SED but I do not know how to set that up for bytes...

0 Karma
1 Solution

jcoates_splunk
Splunk Employee
Splunk Employee

Hi Shane,

Are you trying to emit raw records or tabular data? If a table, you can just eval down to a smaller data set.

View solution in original post

jcoates_splunk
Splunk Employee
Splunk Employee

Hi Shane,

Are you trying to emit raw records or tabular data? If a table, you can just eval down to a smaller data set.

ShaneNewman
Motivator

For lab and rad results the "column" containing the HL7 message can exceed 500K characters. For our purposes, we are thinking that most of the data we need will be in the first 64K characters. I will give this a shot tomorrow.

0 Karma

araitz
Splunk Employee
Splunk Employee

Yeah, something like:

... | eval field = substr(field, 1, 64000)

I didn't test, so you might need to do an if statement to only truncate if there are more than 64k characters. As for doing this for every field in arbitrary results, this would be quite hard.

Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In January, the Splunk Threat Research Team had one release of new security content via the Splunk ES Content ...

Expert Tips from Splunk Professional Services, Ensuring Compliance, and More New ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Observability Release Update: AI Assistant, AppD + Observability Cloud Integrations & ...

This month’s releases across the Splunk Observability portfolio deliver earlier detection and faster ...