Hi folks,
I'm trying to see how I can truncate the entire event to a max number of characters. So basically, if this is my test event (including new lines), and I wanted to capture say the first 10 characters ("Mary had a"), i can't seem to do it.
Mary had a little lamb,
little lamb, little lamb.
Mary had a little lamb, its fleece was white as snow.
And everywhere that Mary went,
Mary went, Mary went,
and everywhere that Mary went, the lamb was sure to go.
I don't seem to be able to use TRUNCATE, because it seems to evaluate *each line* versus the event as a whole. And MAX_EVENTS would not work either, because it would roll to the next event. (I would be OK with MAX_EVENTS if the behavior was to discard the extra.
I have tried this transform, and it seems to want to match each line, and even breaks the events into single line events, as I can't seem to pattern match the newline character.
[truncate_raw_10]
SOURCE_KEY = _raw
REGEX = ^(.{0,10})
DEST_KEY = _raw
FORMAT = $1
Does anyone have any insight?
Thanks!
After the vacation, and getting back to this, I found that my valid use case was truncating at 4096 characters. This lead me down to see why, and I found the LOOKAHEAD parameter in transforms.conf. So I made it this, and it works!
[truncate_raw_20000]
LOOKAHEAD = 20000
SOURCE_KEY = _raw
#REGEX = (^[\s\S])
REGEX = (^[\s\S]{0,20000})
DEST_KEY = _raw
FORMAT = $1
After the vacation, and getting back to this, I found that my valid use case was truncating at 4096 characters. This lead me down to see why, and I found the LOOKAHEAD parameter in transforms.conf. So I made it this, and it works!
[truncate_raw_20000]
LOOKAHEAD = 20000
SOURCE_KEY = _raw
#REGEX = (^[\s\S])
REGEX = (^[\s\S]{0,20000})
DEST_KEY = _raw
FORMAT = $1
Hi,
Thanks you for your response. I did actually partially get this working in my development environment. I need to set up some time to validate next week in Production again.
Here's what I came up with:
https://regex101.com/r/pGxZWU/1
[truncate_raw_500]
SOURCE_KEY = _raw
REGEX = (^[\s\S]{0,500})
DEST_KEY = _raw
FORMAT = $1
Thank you,
Stephen
Hi
Have you try to add regex modifiers on your REGEX in props.conf (?mgx) ?
https://regex101.com/r/asBFDf/1
r. Ismo