Getting Data In

How to truncate the entire event to a max number of characters?

skirven
Communicator

Hi folks,
  I'm trying to see how I can truncate the entire event to a max number of characters. So basically, if this is my test event (including new lines), and I wanted to capture say the first 10 characters ("Mary had a"), i can't seem to do it.

 

Mary had a little lamb,
little lamb, little lamb.
Mary had a little lamb, its fleece was white as snow.
And everywhere that Mary went,
Mary went, Mary went,
and everywhere that Mary went, the lamb was sure to go.

 

 

 

I don't seem to be able to use TRUNCATE, because it seems to evaluate *each line* versus the event as a whole. And MAX_EVENTS would not work either, because it would roll to the next event. (I would be OK with MAX_EVENTS if the behavior was to discard the extra.

I have tried this transform, and it seems to want to match each line, and even breaks the events into single line events, as I can't seem to pattern match the newline character.

 

 

[truncate_raw_10]
SOURCE_KEY = _raw
REGEX = ^(.{0,10})
DEST_KEY = _raw
FORMAT = $1

 

 

Does anyone have any insight?
Thanks!

Labels (3)
0 Karma
1 Solution

skirven
Communicator

After the vacation, and getting back to this, I found that my valid use case was truncating at 4096 characters. This lead me down to see why, and I found the LOOKAHEAD parameter in transforms.conf. So I made it this, and it works!

[truncate_raw_20000]
LOOKAHEAD = 20000
SOURCE_KEY = _raw
#REGEX = (^[\s\S])
REGEX = (^[\s\S]{0,20000})
DEST_KEY = _raw
FORMAT = $1

View solution in original post

0 Karma

skirven
Communicator

After the vacation, and getting back to this, I found that my valid use case was truncating at 4096 characters. This lead me down to see why, and I found the LOOKAHEAD parameter in transforms.conf. So I made it this, and it works!

[truncate_raw_20000]
LOOKAHEAD = 20000
SOURCE_KEY = _raw
#REGEX = (^[\s\S])
REGEX = (^[\s\S]{0,20000})
DEST_KEY = _raw
FORMAT = $1
0 Karma

skirven
Communicator

Hi,

  Thanks you for your response. I did actually partially get this working in my development environment. I need to set up some time to validate next week in Production again.

Here's what I came up with:

https://regex101.com/r/pGxZWU/1

 

[truncate_raw_500]
SOURCE_KEY = _raw
REGEX = (^[\s\S]{0,500})
DEST_KEY = _raw
FORMAT = $1

Thank you,
Stephen

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

Have you try to add regex modifiers on your REGEX in props.conf (?mgx) ?

https://regex101.com/r/asBFDf/1

r. Ismo

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...