topic Re: filter data that are more than hundred K bytes in Getting Data In

filter data that are more than hundred K bytes

SplunkCSIT — Mon, 28 Sep 2020 16:24:57 GMT

1212341234gfgdfggsgsdf-sgsdgsdgreg-3333

props.conf

[test]
BREAK_ONLY_BEFORE =<xml>
KV_MODE = xml
TRANSFORMS-test1 = content
transforms.conf

[content]
SOURCE_KEY=_raw
REGEX=(.?)<body>.?</body>(.*)
DEST_KEY=_raw
FORMAT=$1$2

it works for above xml but if the content in the are more than few hundred K bytes, it will not work. Any suggestion to resolve this? thks

Re: filter data that are more than hundred K bytes

lguinn2 — Thu, 17 Apr 2014 17:53:21 GMT

I believe that your problem is not that the .conf files are wrong, but that your events are larger than the Splunk defaults.
You can fix this by adding the following lines to your stanza in props.conf

TRUNCATE = 0
MAX_EVENTS = 10000

TRUNCATE is the maximum number of bytes in an event. Setting it to 0 means "no limit." You might want to set it to an actual maximum size.

MAX_EVENTS is the maximum number of lines in an event. The default is 256. I arbitrarily set it to 10,000 in the example.

Also, see the answer to your original question:

http://answers.splunk.com/answers/132023/propsconf-and-transformsconf-does-not-work

Re: filter data that are more than hundred K bytes

Lowell — Thu, 17 Apr 2014 18:10:23 GMT

Something else to keep in mind is the LOOKAHEAD value in transforms.conf. If you expect your regex to go more than 4K into the event, then the value will need to be increased. (LOOKAHEAD defaults to 4096 bytes.) Also be careful with the performance implications of changing LOOKAHEAD or the TRUCATE/MAX_EVENTS settings. It's fine to test with arbitrarily high numbers, but sometimes "tests" leak into production when they can kill performance. 😉

Re: filter data that are more than hundred K bytes

SplunkCSIT — Fri, 18 Apr 2014 02:09:27 GMT

it does not work, any other alternative? thks