conditional whitespace in transform

[wowbaggerHU]

Hello everyone!

I am experimenting with the SC4S transforms that are posted here:
https://splunk.github.io/splunk-connect-for-syslog/main/sources/vendor/Splunk/heavyforwarder/

My goal is to send the logs to a syslog-ng instance running with a custom config.
My current problem is that the SC4S config contains a part where it checks for subseconds, and appends the value to the timestamp, if found.

[metadata_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::(.*)$
FORMAT = _s=$1 $0
DEST_KEY = _raw

[metadata_sourcetype]
SOURCE_KEY = MetaData:Sourcetype
REGEX = ^sourcetype::(.*)$
FORMAT = _st=$1 $0
DEST_KEY = _raw

[metadata_index]
SOURCE_KEY = _MetaData:Index
REGEX = (.*)
FORMAT = _idx=$1 $0
DEST_KEY = _raw

[metadata_host]
SOURCE_KEY = MetaData:Host
REGEX = ^host::(.*)$
FORMAT = _h=$1 $0
DEST_KEY = _raw

[metadata_time]
SOURCE_KEY = _time
REGEX = (.*)
FORMAT = _ts=$1$0
DEST_KEY = _raw

[metadata_subsecond]
SOURCE_KEY = _meta
REGEX = \_subsecond\:\:(\.\d+)
FORMAT = $1 $0
DEST_KEY = _raw

In my case however, when it's not found, the timestamp field will not get a whitespace appended, and thus it will be practically concatenated with the following field, which is not what I want.
How could set up the config so that there will always be a whitespace before the next field (the host/_h field)?
I tried adding an extra whitespace in front of the _h in the FORMAT part of the metadata_host stanza, but that seems to be ignored.

This is what I see:

05:58:07.270973 lo    In  ifindex 1 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 16712: (tos 0x0, ttl 64, id 49071, offset 0, flags [DF], proto TCP (6), length 16692)
    127.0.0.1.49916 > 127.0.0.1.cslistener: Flags [.], cksum 0x3f29 (incorrect -> 0x5743), seq 1:16641, ack 1, win 260, options [nop,nop,TS val 804630966 ecr 804630966], length 16640
        0x0000:  0800 0000 0000 0001 0304 0006 0000 0000  ................
        0x0010:  0000 0000 4500 4134 bfaf 4000 4006 3c12  ....E.A4..@.@.<.
        0x0020:  7f00 0001 7f00 0001 c2fc 2328 021a 7392  ..........#(..s.
        0x0030:  486d 209f 8010 0104 3f29 0000 0101 080a  Hm......?)......
        0x0040:  2ff5 b1b6 2ff5 b1b6 5f74 733d 3137 3336  /.../..._ts=1736
        0x0050:  3931 3730 3739 5f68 3d73 706c 756e 6b2d  917079_h=splunk-
        0x0060:  6866 205f 6964 783d 5f6d 6574 7269 6373  hf._idx=_metrics
        0x0070:  205f 7374 3d73 706c 756e 6b5f 696e 7472  ._st=splunk_intr

 This is the interesting part:

        0x0040:  2ff5 b1b6 2ff5 b1b6 5f74 733d 3137 3336  /.../..._ts=1736
        0x0050:  3931 3730 3739 5f68 3d73 706c 756e 6b2d  917079_h=splunk-
        0x0060:  6866 205f 6964 783d 5f6d 6574 7269 6373  hf._idx=_metrics

The _h will come right after the end of the _ts field, without any clear separation.

[wowbaggerHU · Accepted solution]

In the end, after struggling with this for several days, I thought to myself, this is leading me nowhere.
I wanted to have the fields to follow each other like this:

(TIME)(SUBSECOND) (HOST)

I had the idea to concentrate on adding the whitespace before HOST, and not after TIME or SUBSECOND.
This approach had also its problems because spaces at the start of the FORMAT string seem to be ignored, but here I managed to get around that:
https://community.splunk.com/t5/Getting-Data-In/Force-inclusion-of-space-character-as-a-first-charac...

This way I could let go the question of addomg whitespaces conditionally. Though I could not solve this very problem, my overall problam is now solved.

[PickleRick]

This "glues" the extracted timestamp with the remainder of the event. Add a space between $1 and $0

FORMAT = _ts=$1$0

[wowbaggerHU]

Yes, it is in deed. I thought of that, but I assume that the creators at SC4S probably wanted the timestamp to have the fraction seconds added to it, if there is a metadata variable holding that. In that case, the decimal point and the fraction seconds need to follow the timestamp without any whitespace. That is the reason the whitespace is missing where you pointed it out.

If there isn't a variable holding the fraction seconds however, like in my case, there will be no trailing space added to the timestamp, and the host key-value pair will follow it right without a whitespace.

Any idea how I could add a whitespace conditionally?

[PickleRick]

You could try something like this

[metadata_subsecond]
SOURCE_KEY = _meta
REGEX = \_subsecond\:\:(\.\d+)
FORMAT = $1 $0
DEST_KEY = subsecond_temp

[metadata_fix_subsecond]
INGEST_EVAL = _raw=if(isnull(subsecond_temp),_raw,subsecond_temp." "._raw)

 Of course you need to add the metadata_fix_subsecond transform into TRANSFORMS-zza-syslog before metadata_subsecond

The number of backslashes in that subsecond regex is surprisingly high. Those characters shouldn't normally need escaping.

[wowbaggerHU]

Yes, I noticed the excess escaping too, but if it's incorrect then the SC4S suggested config is wrong too.
Either way, I will try it both ways, and get back to you. Thank you so much for your help!

[PickleRick]

It shouldn't hurt. If you escape something that doesn't need escaping, nothing bad should happen.

It's just ugly. 😉

[wowbaggerHU]

I have been experimenting further, and found the following...
This is my latest test config:

[md_time]
SOURCE_KEY = _time
REGEX = (.*)
FORMAT = _ts=$1
DEST_KEY = time_temp

[md_subsecond]
SOURCE_KEY = _meta
REGEX = _subsecond=(\.\d+)
FORMAT = $1
DEST_KEY = subsecond_temp

[md_fix_subsecond]
INGEST_EVAL = _raw=if(isnull(time_temp), "aaa" . _raw, "bbb" . _raw)
#INGEST_EVAL = _raw=if(isnull(subsecond_temp), time_temp . " " . _raw, time_temp . subsecond_temp . " " . _raw)

Both md_time and md_subsecond are in the list, and before md_fix_subsecond.
If in md_fix_subsecond I check for the null-ness of either time_temp or subsecond_temp, then they are both reported as null. So for some reason they are not available in the EVAL_RAW for some reason.
And as they are both null, referencing them resulted in an error, and so no log was output.

How could we resolve this?

[wowbaggerHU]

I even set up the Windows box to emit metadata that matches the regex in the transform, because none of my logs seemed to have that subsecond data.
Now my Windows test machine sends among its metadata the string

time_subsecond=.123456

And interestingly enough the subsecond transform doesn't get triggered. My latest version is:

[metadata_subsecond]
SOURCE_KEY = _meta
REGEX = _subsecond::(\.[0-9]+)
FORMAT = $1$0
DEST_KEY = _raw

But nothing seems to happen.

[wowbaggerHU]

I tried what you suggested, but did not seem to help. It seemed as if the fix_subsecond stanza wouldn't be executed at all. The _h KV pair followed _ts's value without a whitespace.

After experimenting a bit more, I now have this, but this doesn't work either:

[md_time]
SOURCE_KEY = _time
REGEX = (.*)
FORMAT = _ts=$1 $0
DEST_KEY = time_temp

[md_subsecond]
SOURCE_KEY = _meta
REGEX = _subsecond::(\.\d+)
FORMAT = $1
DEST_KEY = subsecond_temp

[md_fix_subsecond]
INGEST_EVAL = _raw=if(isnull(subsecond_temp),time_temp+" "+_raw,time_temp+subsecond_temp+" "+_raw)

Plus props.conf:

[default]
ADD_EXTRA_TIME_FIELDS = none
ANNOTATE_PUNCT = false
SHOULD_LINEMERGE = false
TRANSFORMS-zza-syslog = syslog_canforward, reformat_metadata, md_add_separator, md_source, md_sourcetype, md_index, md_host, md_subsecond, md_time, md_fix_subsecond, discard_empty_msg
# The following applies for TCP destinations where the IETF frame is required
TRANSFORMS-zzz-syslog = syslog_octet_count, octet_count_prepend
# Comment out the above and uncomment the following for udp
#TRANSFORMS-zzz-syslog-udp = syslog_octet_count, octet_count_prepend, discard_empty_msg

[audittrail]
# We can't transform this source type its protected
TRANSFORMS-zza-syslog =
TRANSFORMS-zzz-syslog =

However this now breaks logging and I'm getting no logs forwarded to sylsog-ng. The connection is up, but no meaningful data arrives, just "empty" packages.

What may be the problem? Did I break the sequence of the stanzas? (I did not seem to understand it in the first place, as they seem to be in backward order compared to how the KV pairs follow each other in the actual log message.)

[wowbaggerHU]

[md_time]
SOURCE_KEY = _time
REGEX = (.*)
FORMAT = _ts=$1
DEST_KEY = time_temp

[md_subsecond]
SOURCE_KEY = _meta
REGEX = _subsecond::(\.\d+)
FORMAT = $1
DEST_KEY = subsecond_temp

[md_fix_subsecond]
INGEST_EVAL = _raw=if(isnull(subsecond_temp), time_temp + " " + _raw, time_temp + subsecond_temp + " " + _raw)

[md_time_default]
SOURCE_KEY = _time
REGEX = (.*)
FORMAT = _ts=$1 $0
DEST_KEY = _raw

The problem seems to be somewhere in md_time, md_subsecond or md_fix_subsecond, because if I use md_time_default, it works (though without subseconds), and if I enable these three instead of md_time_default, then I get no output: the packets emitted by Splunk seem to be empty: without a payload.