Re: filter data that are more than hundred K bytes

SplunkCSIT · ‎04-17-2014

1212341234gfgdfggsgsdf-sgsdgsdgreg-3333

props.conf

[test]
BREAK_ONLY_BEFORE =<xml>
KV_MODE = xml
TRANSFORMS-test1 = content
transforms.conf

[content]
SOURCE_KEY=_raw
REGEX=(.?)<body>.?</body>(.*)
DEST_KEY=_raw
FORMAT=$1$2

it works for above xml but if the content in the are more than few hundred K bytes, it will not work. Any suggestion to resolve this? thks

lguinn2 · ‎04-17-2014

I believe that your problem is not that the .conf files are wrong, but that your events are larger than the Splunk defaults.
You can fix this by adding the following lines to your stanza in props.conf

TRUNCATE = 0
MAX_EVENTS = 10000

TRUNCATE is the maximum number of bytes in an event. Setting it to 0 means "no limit." You might want to set it to an actual maximum size.

MAX_EVENTS is the maximum number of lines in an event. The default is 256. I arbitrarily set it to 10,000 in the example.

Also, see the answer to your original question:

http://answers.splunk.com/answers/132023/propsconf-and-transformsconf-does-not-work

SplunkCSIT · ‎04-17-2014

it does not work, any other alternative? thks

Lowell · ‎04-17-2014

Something else to keep in mind is the LOOKAHEAD value in transforms.conf. If you expect your regex to go more than 4K into the event, then the value will need to be increased. (LOOKAHEAD defaults to 4096 bytes.) Also be careful with the performance implications of changing LOOKAHEAD or the TRUCATE/MAX_EVENTS settings. It's fine to test with arbitrarily high numbers, but sometimes "tests" leak into production when they can kill performance. 😉

filter data that are more than hundred K bytes

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Free Professional Services for .conf26 Attendees

Defend at Machine Speed: Your Guide to Security Sessions at .conf26

Where Innovation Takes Flight: The Splunk4Aviation Flight Sim Lands at .conf26

Join the Conversation