Regex in transforms only matches parts of the data.

[royhvaara]

in inputs.conf:

[tcp://:9995]
connection_host = dns
sourcetype = tcp:9995
source = tcp:9995

in props.conf:

[source::tcp:9995]
TRANSFORMS = streamsourcetype, streamsource, streamrawextract

in transforms.conf:

[streamsource]
REGEX = ^source=(\S+)
DEST_KEY = MetaData:Source
FORMAT = source::$1

[streamsourcetype]
REGEX = ^source=\S+ sourcetype=(\S+)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::$1

[streamrawextract]
REGEX = (?s)^[^\n]+\n(.*)
DEST_KEY = _raw
FORMAT = $1

Expected input is xml with the first line being in a special format:

source=<source> sourcetype=<sourcetype>\n
<input><entry host="example.com">1234</entry><entry host="static.example.com">95959</entry></input>

What's above is just an example. There is a lot of xmldata being fed to splunk. This is truncated at about 4030 chars when fed through streamrawextract, however it is not truncated when streamrawextract is not applied. I obviously don't want it to be truncated in the middle of the data. Right now the streamrawextract is invalidating my xml... I'm sure I'm missing some configuration setting somewhere, but I just can't seem to find out which one.

Also: it's insanely frustrating to have to restart splunk every time i make a change to props.conf or transforms.conf. Is there any way to have splunk reload the configuration without doing a restart (like a reload on most other services)?

[royhvaara · Accepted solution]

I think I should ask questions here more often. Minutes after i asked the question I found the answer to my own question. In transforms.conf i added LOOKAHEAD = 10000 under [streamrawextract]. Now it matches all of it. I decided to answer my own question and not delete it as I haven't found a lot of info about this on the interwebs. Maybe it's just obvious to other people...

If anyone knows how to fix the last part (reloading the configs without restarting splunk) feel free to shout out! 🙂