topic Re: Force inclusion of space character as a first character in FORMAT string in transforms.conf in Getting Data In

Force inclusion of space character as a first character in FORMAT string in transforms.conf

wowbaggerHU — Sat, 18 Jan 2025 16:31:21 GMT

Hello everyone!

I am experimenting with the SC4S transforms that are posted here:
https://splunk.github.io/splunk-connect-for-syslog/main/sources/vendor/Splunk/heavyforwarder/

My problem is that I am trying to reformat fields, and in one particular place I would need to ensure that a space preceeds the _h= part in the transform stanza below.

[md_host] SOURCE_KEY = MetaData:Host REGEX = ^host::(.*)$ FORMAT = _h=$1 $0 DEST_KEY = _raw

However if I add multiple whitespaces in the FORMAT string, right after the equals sign in the above example, they will be ignored.
Should put the whole thing betweem quotes? Wouldn't the quotes be included in the _raw string?

What would be the right solution for this?

Re: Force inclusion of space character as a first character in FORMAT string in transforms.conf

tscroggins — Sat, 18 Jan 2025 23:34:38 GMT

Hi @wowbaggerHU,

You can use INGEST_EVAL as a workaround:

# transforms.conf [md_host] INGEST_EVAL = _raw:=" h=\"".host."\" "._raw

Re: Force inclusion of space character as a first character in FORMAT string in transforms.conf

PickleRick — Sun, 19 Jan 2025 08:36:20 GMT

Apart from @tscroggins 's solution you could try escaping your initial space. It should show the config parser that there is a non-space character so your key-value pair in config is split properly but since the space doesn't normally need escaping it shouldn't hurt.

Re: Force inclusion of space character as a first character in FORMAT string in transforms.conf

wowbaggerHU — Sun, 19 Jan 2025 09:37:38 GMT

Good idea, I tried it, but unfortunately it doesn't seem to work. I have this configured:

[md_host] SOURCE_KEY = MetaData:Host REGEX = ^host::(.*)$ FORMAT = \ _h=$1 $0 DEST_KEY = _raw [md_subsecond_default] SOURCE_KEY = _meta REGEX = _subsecond=(\.\d+) FORMAT = $1$0 DEST_KEY = _raw [md_time_default] SOURCE_KEY = _time REGEX = (.*) FORMAT = _ts=$1$0 DEST_KEY = _raw

And I get this:

0x0040: e073 339e e073 339e 3232 3738 205f 7473 .s3..s3.2278._ts 0x0050: 3d31 3733 3732 3739 3038 315c 205f 683d =1737279081\._h= 0x0060: 7370 6c75 6e6b 2d68 6620 5f69 6478 3d5f splunk-hf._idx=_

But I agree, this would have been the most elegant solution.

Re: Force inclusion of space character as a first character in FORMAT string in transforms.conf

wowbaggerHU — Mon, 20 Jan 2025 13:22:59 GMT

@tscrogginsThanks for the idea. It worked, though I added my own set of modifications to it.

As a final touch I would like to put the relevant part of my config here, so as to contribute it back to the community:

[md_host] INGEST_EVAL = _raw=" _h=".host." "._raw [md_subsecond] SOURCE_KEY = _meta REGEX = _subsecond=(\.\d+) FORMAT = $1$0 DEST_KEY = _raw [md_time] SOURCE_KEY = _time REGEX = (.*) FORMAT = _ts=$1$0 DEST_KEY = _raw

Re: Force inclusion of space character as a first character in FORMAT string in transforms.conf

tscroggins — Sun, 19 Jan 2025 21:02:52 GMT

Looks good! Just be mindful of the difference between the plus (+) operator and the dot (.) operator. Plus concatenates strings and adds numbers, but dot concatenates both strings and numbers as strings. If you're unsure of the order of operations or the variable value in other contexts, you can wrap the variables in the tostring() function.

It's interesting that Splunk also assumed a leading space would work in the SC4S transforms:

[metadata_time] SOURCE_KEY = _time REGEX = (.*) FORMAT = t="$1$0 DEST_KEY = _raw

@PickleRick's suggestion to escape the space with i.e. a blackslash just evaluates to a literal backslash followed by a space.

Re: Force inclusion of space character as a first character in FORMAT string in transforms.conf

wowbaggerHU — Mon, 20 Jan 2025 08:08:15 GMT

Thanks for the explanation, it was educational. I have corrected the plus operators where appropriate.
On the other hand, I have now triple-checked, and in deed, multiple leading whitespaces are ignored in the FORMAT string.

But yes, it would seem that Splunk or whoever wrote the SC4S config assumed that they would be honored.