Knowledge Management

Remove Trailing Data After 65K bytes

ShaneNewman
Motivator

I am using Splunk DB Connect to push data into a Teradata database, the limitation of the ODBC driver is that is can only handle 64K bytes of data in a single field... which on this data set (HL7) is fairly common. Is there a command I can use in Splunk to strip out the remaining data so this will not fail? I was thinking SED but I do not know how to set that up for bytes...

0 Karma
1 Solution

jcoates_splunk
Splunk Employee
Splunk Employee

Hi Shane,

Are you trying to emit raw records or tabular data? If a table, you can just eval down to a smaller data set.

View solution in original post

jcoates_splunk
Splunk Employee
Splunk Employee

Hi Shane,

Are you trying to emit raw records or tabular data? If a table, you can just eval down to a smaller data set.

ShaneNewman
Motivator

For lab and rad results the "column" containing the HL7 message can exceed 500K characters. For our purposes, we are thinking that most of the data we need will be in the first 64K characters. I will give this a shot tomorrow.

0 Karma

araitz
Splunk Employee
Splunk Employee

Yeah, something like:

... | eval field = substr(field, 1, 64000)

I didn't test, so you might need to do an if statement to only truncate if there are more than 64k characters. As for doing this for every field in arbitrary results, this would be quite hard.

Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...