How to truncate the entire event to a max number of characters?

[skirven]

Hi folks,
  I'm trying to see how I can truncate the entire event to a max number of characters. So basically, if this is my test event (including new lines), and I wanted to capture say the first 10 characters ("Mary had a"), i can't seem to do it.

Mary had a little lamb,
little lamb, little lamb.
Mary had a little lamb, its fleece was white as snow.
And everywhere that Mary went,
Mary went, Mary went,
and everywhere that Mary went, the lamb was sure to go.

I don't seem to be able to use TRUNCATE, because it seems to evaluate *each line* versus the event as a whole. And MAX_EVENTS would not work either, because it would roll to the next event. (I would be OK with MAX_EVENTS if the behavior was to discard the extra.

I have tried this transform, and it seems to want to match each line, and even breaks the events into single line events, as I can't seem to pattern match the newline character.

[truncate_raw_10]
SOURCE_KEY = _raw
REGEX = ^(.{0,10})
DEST_KEY = _raw
FORMAT = $1

Does anyone have any insight?
Thanks!

[skirven · Accepted solution]

After the vacation, and getting back to this, I found that my valid use case was truncating at 4096 characters. This lead me down to see why, and I found the LOOKAHEAD parameter in transforms.conf. So I made it this, and it works!

[truncate_raw_20000]
LOOKAHEAD = 20000
SOURCE_KEY = _raw
#REGEX = (^[\s\S])
REGEX = (^[\s\S]{0,20000})
DEST_KEY = _raw
FORMAT = $1

[skirven]

Hi,

  Thanks you for your response. I did actually partially get this working in my development environment. I need to set up some time to validate next week in Production again.

Here's what I came up with:

https://regex101.com/r/pGxZWU/1

[truncate_raw_500]
SOURCE_KEY = _raw
REGEX = (^[\s\S]{0,500})
DEST_KEY = _raw
FORMAT = $1

Thank you,
Stephen

[isoutamo]

Hi

Have you try to add regex modifiers on your REGEX in props.conf (?mgx) ?

https://regex101.com/r/asBFDf/1

r. Ismo