About wowbaggerHU

wowbaggerHU

Well, we work with our own syslog-ng based solution on the receiving side, and we have the know-how to make use of all of its features. The thing is, we don't want to venture into the legally gray area surrounding the reverse engineering of S2S, and SC4S already has some pre-made open source configs in place to achieve the goal of sending logs (with specially crafted headers and framing) from a Splunk HWF to syslog-ng. We are just taking it a few steps further. We just noticed this problem, and aside ours, their approach has this flaw too, so I notified them of this problem.

wowbaggerHU

We are doing some magic in transforms.cond, and wan to send out log data with a forged syslog header and framing. For the framing we need the byte count and and not the character count. Your example illustrates the problem very well. I guess most of the characters were 2 byte ones, perhaps the spaces were single byte, so that's the reason why it's not the exact duplicate of the length. However one needs to handle 3 and 4 byte UTF-8 characters too.

wowbaggerHU

Thanks for the suggestion, we are looking at this possibility. Unfortunately there are 4, 3 and 2 byte long UTF-8 characters, so we will likely have to do separate passes for all of those. I hope there'll be a better approach because this likely has a less-than-ideal performance impact. But apart from that this vay may prove to be viable.

wowbaggerHU

Dear Members, We are developing a Splunk add-on, and we need to find the octet (byte) count of a data field in an INGEST_EVAL statement in tranforms.conf. Currently we are using the length(_raw) statement, but it is problematic because if the data to be transferred contains UTF-8 characters then the number of characters returned by length() will not match the actual number of bytes used to store the message. Is there a function that returns the byte-count in a single step, or do we have to employ some kind of magic to get this information? Thanks in advance!

wowbaggerHU · ‎10-28-2025

@livehybrid Thanks for the suggestion and help! I opted for the hard way: I have updated the necessary spec files, and the Dockerfile that I have in place for building slim. It works now as expected!

wowbaggerHU · ‎10-27-2025

Hello Everyone, I am at the stage of considering to submit a Splunk app to SplunkBase, and before submitting our code, I am running validation steps via `slim validate`: docker run --volume /home/USER/Development/WORK/SPLUNK-ingest/splunk-app:/apps slim-pip:latest slim package /apps/APP_NAME slim package: Packaging app at "/apps/APP_NAME" slim package: [WARNING] /apps/APP_NAME/default/app.conf, line 12: Undefined setting in app.conf, stanza [ui]: supported_themes slim package: [WARNING] /apps/APP_NAME/default/transforms.conf, line 30: Undefined setting in transforms.conf, stanza [all_in_one]: INGEST_EVAL slim package: [WARNING] /apps/APP_NAME/default/transforms.conf, line 33: Undefined setting in transforms.conf, stanza [syslog_octet_count]: INGEST_EVAL slim package: [WARNING] /apps/APP_NAME/default/transforms.conf, line 36: Undefined setting in transforms.conf, stanza [octet_count_prepend]: INGEST_EVAL slim package: [WARNING] /apps/APP_NAME/default/transforms.conf, line 39: Undefined setting in transforms.conf, stanza [discard_empty_msg]: INGEST_EVAL slim package: [WARNING] /apps/APP_NAME/default/transforms.conf, line 42: Undefined setting in transforms.conf, stanza [discard_internals]: INGEST_EVAL slim package: [INFO] Source package exported to "/apps/APP_NAME-0.7.1.tar.gz" docker run --volume /home/USER/Development/WORK/SPLUNK-ingest/splunk-app:/apps slim-pip:latest slim validate /apps/APP_NAME-0.7.1.tar.gz slim validate: Validating app at "/apps/APP_NAME-0.7.1.tar.gz"... slim validate: [WARNING] /root/.config/slim/repository/slim.cache_xndmoh9d/APP_NAME-0.7.1.source/APP_NAME/default/app.conf, line 12: Undefined setting in app.conf, stanza [ui]: supported_themes slim validate: [WARNING] /root/.config/slim/repository/slim.cache_xndmoh9d/APP_NAME-0.7.1.source/APP_NAME/default/transforms.conf, line 30: Undefined setting in transforms.conf, stanza [all_in_one]: INGEST_EVAL slim validate: [WARNING] /root/.config/slim/repository/slim.cache_xndmoh9d/APP_NAME-0.7.1.source/APP_NAME/default/transforms.conf, line 33: Undefined setting in transforms.conf, stanza [syslog_octet_count]: INGEST_EVAL slim validate: [WARNING] /root/.config/slim/repository/slim.cache_xndmoh9d/APP_NAME-0.7.1.source/APP_NAME/default/transforms.conf, line 36: Undefined setting in transforms.conf, stanza [octet_count_prepend]: INGEST_EVAL slim validate: [WARNING] /root/.config/slim/repository/slim.cache_xndmoh9d/APP_NAME-0.7.1.source/APP_NAME/default/transforms.conf, line 39: Undefined setting in transforms.conf, stanza [discard_empty_msg]: INGEST_EVAL slim validate: [WARNING] /root/.config/slim/repository/slim.cache_xndmoh9d/APP_NAME-0.7.1.source/APP_NAME/default/transforms.conf, line 42: Undefined setting in transforms.conf, stanza [discard_internals]: INGEST_EVAL slim validate: [INFO] App validation complete What may be the reason of `slim validate` complaining about `INGEST_EVAL`? On my 9.2.1+ Splunk instance that I tested the app with, seem to work fine as is. Any ideas?

wowbaggerHU · ‎03-11-2025

We took a few steps and looked at how the config files worked. It seemed as if the content of the different config types were virtually merged (each type of config with its own kind). Therefore we reasoned that we could use the settings we were setting up via the GUI for the forwarder, and use our outputs.conf from the app, to add/override the settings we needed, and it turned out that this approach works! So now we have the possibility to set up the forwarding via the Web UI, and also have those settings augmented with our own extra settings. This seems to solve our initial problem.

wowbaggerHU · ‎03-10-2025

I tried putting my props.conf and transforms.conf to $SPLUNK_HOME/etc/apps/yourAppName/local/ but the settings don't seem to take effect for some reason. I created a tcpout destination from the web UI, but it nevertheless tries to send stuff over S2S, disregarding the things I've set in transforms.conf. Though I have to admit, I need to have something like this in the outputs.conf: #Because audit trail is protected and we can't transform it we can not use default we must use tcp_routing [tcpout] defaultGroup = NoForwarding [tcpout:nexthop] server = localhost:9000 sendCookedData = false But if I set up the destination from the Forwarding and receiving page, then I get something like this: [tcpout] defaultGroup = default-autolb-group [tcpout:default-autolb-group] server = localhost:9000 [tcpout-server://localhost:9000]

wowbaggerHU · ‎03-07-2025

I remember being able to install apps from a zip file from the web GUI on physical Splunk installations. On the other hand, I got an idea. It may be a stupid question , but is it possible to configure a tcpout output on the Splunk web UI? If yes, then there is no need for a separate second app. Then I would only need to add the transforms and props configs, and have the users configure the tcpout on their own, and that's it. Just to give you an idea, I want to package something similar to the SC4S heavy forwarder configs described here: https://splunk.github.io/splunk-connect-for-syslog/main/sources/vendor/Splunk/heavyforwarder/

wowbaggerHU · ‎03-07-2025

@livehybridand @isoutamo Thanks for both of your answers! I more or less know what I need to put in those files, so have that part figured out already. Yes, as far as my understanding goes, the app is supposed to go on a heavy forwarder node. We have no plans in place for using a deployment server. For the initial POC phase, I believe that adding the app az a simple zip file would suffice. As for outputs.conf, is it possible to somehow dinamically generate its content? I mean, to ask the user for a hostname or IP address, and then use that value for the server value. I will try adding my configs to the app, and will report back in a few days. Now looking at the page https://dev.splunk.com/enterprise/docs/developapps/extensionpoints It does mention props.conf and transforms.conf, but there seems to be no mention of outputs.conf. Is it possible to have an outputs.conf in the app, and force Splunk to somehow use it regardless of not being present in the list above?

wowbaggerHU · ‎03-07-2025

Dear Members, I have a use case where I would need to update or insert configuration to transforms.conf, props.conf and outputs.conf. I was told that it is possible to do this via a creating an app. That would make it easier for users to make the necessary changes, instead of doing it via the error-prone manual procedure. Nevertheless, I haven't come across any documentation that would illustrate and explain how to do it. Does someone have any experience with that? Or perhaps can someone point me to the relevant documentation? Thanks in advance!

wowbaggerHU · ‎01-20-2025

Thanks for the explanation, it was educational. I have corrected the plus operators where appropriate. On the other hand, I have now triple-checked, and in deed, multiple leading whitespaces are ignored in the FORMAT string. But yes, it would seem that Splunk or whoever wrote the SC4S config assumed that they would be honored.

wowbaggerHU · ‎01-19-2025

In the end, after struggling with this for several days, I thought to myself, this is leading me nowhere. I wanted to have the fields to follow each other like this: (TIME)(SUBSECOND) (HOST) I had the idea to concentrate on adding the whitespace before HOST, and not after TIME or SUBSECOND. This approach had also its problems because spaces at the start of the FORMAT string seem to be ignored, but here I managed to get around that: https://community.splunk.com/t5/Getting-Data-In/Force-inclusion-of-space-character-as-a-first-character-in/m-p/709157 This way I could let go the question of addomg whitespaces conditionally. Though I could not solve this very problem, my overall problam is now solved.

wowbaggerHU · ‎01-19-2025

@tscrogginsThanks for the idea. It worked, though I added my own set of modifications to it. As a final touch I would like to put the relevant part of my config here, so as to contribute it back to the community: [md_host] INGEST_EVAL = _raw=" _h=".host." "._raw [md_subsecond] SOURCE_KEY = _meta REGEX = _subsecond=(\.\d+) FORMAT = $1$0 DEST_KEY = _raw [md_time] SOURCE_KEY = _time REGEX = (.*) FORMAT = _ts=$1$0 DEST_KEY = _raw

wowbaggerHU · ‎01-19-2025

Good idea, I tried it, but unfortunately it doesn't seem to work. I have this configured: [md_host] SOURCE_KEY = MetaData:Host REGEX = ^host::(.*)$ FORMAT = \ _h=$1 $0 DEST_KEY = _raw [md_subsecond_default] SOURCE_KEY = _meta REGEX = _subsecond=(\.\d+) FORMAT = $1$0 DEST_KEY = _raw [md_time_default] SOURCE_KEY = _time REGEX = (.*) FORMAT = _ts=$1$0 DEST_KEY = _raw And I get this: 0x0040: e073 339e e073 339e 3232 3738 205f 7473 .s3..s3.2278._ts 0x0050: 3d31 3733 3732 3739 3038 315c 205f 683d =1737279081\._h= 0x0060: 7370 6c75 6e6b 2d68 6620 5f69 6478 3d5f splunk-hf._idx=_ But I agree, this would have been the most elegant solution.

wowbaggerHU · ‎01-18-2025

Hello everyone! I am experimenting with the SC4S transforms that are posted here: https://splunk.github.io/splunk-connect-for-syslog/main/sources/vendor/Splunk/heavyforwarder/ My problem is that I am trying to reformat fields, and in one particular place I would need to ensure that a space preceeds the _h= part in the transform stanza below. [md_host] SOURCE_KEY = MetaData:Host REGEX = ^host::(.*)$ FORMAT = _h=$1 $0 DEST_KEY = _raw However if I add multiple whitespaces in the FORMAT string, right after the equals sign in the above example, they will be ignored. Should put the whole thing betweem quotes? Wouldn't the quotes be included in the _raw string? What would be the right solution for this?

wowbaggerHU · ‎01-17-2025

I have been experimenting further, and found the following... This is my latest test config: [md_time] SOURCE_KEY = _time REGEX = (.*) FORMAT = _ts=$1 DEST_KEY = time_temp [md_subsecond] SOURCE_KEY = _meta REGEX = _subsecond=(\.\d+) FORMAT = $1 DEST_KEY = subsecond_temp [md_fix_subsecond] INGEST_EVAL = _raw=if(isnull(time_temp), "aaa" . _raw, "bbb" . _raw) #INGEST_EVAL = _raw=if(isnull(subsecond_temp), time_temp . " " . _raw, time_temp . subsecond_temp . " " . _raw) Both md_time and md_subsecond are in the list, and before md_fix_subsecond. If in md_fix_subsecond I check for the null-ness of either time_temp or subsecond_temp, then they are both reported as null. So for some reason they are not available in the EVAL_RAW for some reason. And as they are both null, referencing them resulted in an error, and so no log was output. How could we resolve this?

wowbaggerHU · ‎01-17-2025

I even set up the Windows box to emit metadata that matches the regex in the transform, because none of my logs seemed to have that subsecond data. Now my Windows test machine sends among its metadata the string time_subsecond=.123456 And interestingly enough the subsecond transform doesn't get triggered. My latest version is: [metadata_subsecond] SOURCE_KEY = _meta REGEX = _subsecond::(\.[0-9]+) FORMAT = $1$0 DEST_KEY = _raw But nothing seems to happen.

wowbaggerHU · ‎01-16-2025

[md_time] SOURCE_KEY = _time REGEX = (.*) FORMAT = _ts=$1 DEST_KEY = time_temp [md_subsecond] SOURCE_KEY = _meta REGEX = _subsecond::(\.\d+) FORMAT = $1 DEST_KEY = subsecond_temp [md_fix_subsecond] INGEST_EVAL = _raw=if(isnull(subsecond_temp), time_temp + " " + _raw, time_temp + subsecond_temp + " " + _raw) [md_time_default] SOURCE_KEY = _time REGEX = (.*) FORMAT = _ts=$1 $0 DEST_KEY = _raw The problem seems to be somewhere in md_time, md_subsecond or md_fix_subsecond, because if I use md_time_default, it works (though without subseconds), and if I enable these three instead of md_time_default, then I get no output: the packets emitted by Splunk seem to be empty: without a payload.

wowbaggerHU · ‎01-15-2025

I tried what you suggested, but did not seem to help. It seemed as if the fix_subsecond stanza wouldn't be executed at all. The _h KV pair followed _ts's value without a whitespace. After experimenting a bit more, I now have this, but this doesn't work either: [md_time] SOURCE_KEY = _time REGEX = (.*) FORMAT = _ts=$1 $0 DEST_KEY = time_temp [md_subsecond] SOURCE_KEY = _meta REGEX = _subsecond::(\.\d+) FORMAT = $1 DEST_KEY = subsecond_temp [md_fix_subsecond] INGEST_EVAL = _raw=if(isnull(subsecond_temp),time_temp+" "+_raw,time_temp+subsecond_temp+" "+_raw) Plus props.conf: [default] ADD_EXTRA_TIME_FIELDS = none ANNOTATE_PUNCT = false SHOULD_LINEMERGE = false TRANSFORMS-zza-syslog = syslog_canforward, reformat_metadata, md_add_separator, md_source, md_sourcetype, md_index, md_host, md_subsecond, md_time, md_fix_subsecond, discard_empty_msg # The following applies for TCP destinations where the IETF frame is required TRANSFORMS-zzz-syslog = syslog_octet_count, octet_count_prepend # Comment out the above and uncomment the following for udp #TRANSFORMS-zzz-syslog-udp = syslog_octet_count, octet_count_prepend, discard_empty_msg [audittrail] # We can't transform this source type its protected TRANSFORMS-zza-syslog = TRANSFORMS-zzz-syslog = However this now breaks logging and I'm getting no logs forwarded to sylsog-ng. The connection is up, but no meaningful data arrives, just "empty" packages. What may be the problem? Did I break the sequence of the stanzas? (I did not seem to understand it in the first place, as they seem to be in backward order compared to how the KV pairs follow each other in the actual log message.)

wowbaggerHU · ‎01-15-2025

Yes, I noticed the excess escaping too, but if it's incorrect then the SC4S suggested config is wrong too. Either way, I will try it both ways, and get back to you. Thank you so much for your help!

wowbaggerHU · ‎01-15-2025

Yes, it is in deed. I thought of that, but I assume that the creators at SC4S probably wanted the timestamp to have the fraction seconds added to it, if there is a metadata variable holding that. In that case, the decimal point and the fraction seconds need to follow the timestamp without any whitespace. That is the reason the whitespace is missing where you pointed it out. If there isn't a variable holding the fraction seconds however, like in my case, there will be no trailing space added to the timestamp, and the host key-value pair will follow it right without a whitespace. Any idea how I could add a whitespace conditionally?

wowbaggerHU · ‎01-14-2025

Thanks for your help with this. In the meantime I've run into another problem. Could you please help me? This is the topic: https://community.splunk.com/t5/Getting-Data-In/conditional-whitespace-in-transform/m-p/708831

wowbaggerHU · ‎01-14-2025

Hello everyone! I am experimenting with the SC4S transforms that are posted here: https://splunk.github.io/splunk-connect-for-syslog/main/sources/vendor/Splunk/heavyforwarder/ My goal is to send the logs to a syslog-ng instance running with a custom config. My current problem is that the SC4S config contains a part where it checks for subseconds, and appends the value to the timestamp, if found. [metadata_source] SOURCE_KEY = MetaData:Source REGEX = ^source::(.*)$ FORMAT = _s=$1 $0 DEST_KEY = _raw [metadata_sourcetype] SOURCE_KEY = MetaData:Sourcetype REGEX = ^sourcetype::(.*)$ FORMAT = _st=$1 $0 DEST_KEY = _raw [metadata_index] SOURCE_KEY = _MetaData:Index REGEX = (.*) FORMAT = _idx=$1 $0 DEST_KEY = _raw [metadata_host] SOURCE_KEY = MetaData:Host REGEX = ^host::(.*)$ FORMAT = _h=$1 $0 DEST_KEY = _raw [metadata_time] SOURCE_KEY = _time REGEX = (.*) FORMAT = _ts=$1$0 DEST_KEY = _raw [metadata_subsecond] SOURCE_KEY = _meta REGEX = \_subsecond\:\:(\.\d+) FORMAT = $1 $0 DEST_KEY = _raw In my case however, when it's not found, the timestamp field will not get a whitespace appended, and thus it will be practically concatenated with the following field, which is not what I want. How could set up the config so that there will always be a whitespace before the next field (the host/_h field)? I tried adding an extra whitespace in front of the _h in the FORMAT part of the metadata_host stanza, but that seems to be ignored. This is what I see: 05:58:07.270973 lo In ifindex 1 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 16712: (tos 0x0, ttl 64, id 49071, offset 0, flags [DF], proto TCP (6), length 16692) 127.0.0.1.49916 > 127.0.0.1.cslistener: Flags [.], cksum 0x3f29 (incorrect -> 0x5743), seq 1:16641, ack 1, win 260, options [nop,nop,TS val 804630966 ecr 804630966], length 16640 0x0000: 0800 0000 0000 0001 0304 0006 0000 0000 ................ 0x0010: 0000 0000 4500 4134 bfaf 4000 4006 3c12 ....E.A4..@.@.<. 0x0020: 7f00 0001 7f00 0001 c2fc 2328 021a 7392 ..........#(..s. 0x0030: 486d 209f 8010 0104 3f29 0000 0101 080a Hm......?)...... 0x0040: 2ff5 b1b6 2ff5 b1b6 5f74 733d 3137 3336 /.../..._ts=1736 0x0050: 3931 3730 3739 5f68 3d73 706c 756e 6b2d 917079_h=splunk- 0x0060: 6866 205f 6964 783d 5f6d 6574 7269 6373 hf._idx=_metrics 0x0070: 205f 7374 3d73 706c 756e 6b5f 696e 7472 ._st=splunk_intr This is the interesting part: 0x0040: 2ff5 b1b6 2ff5 b1b6 5f74 733d 3137 3336 /.../..._ts=1736 0x0050: 3931 3730 3739 5f68 3d73 706c 756e 6b2d 917079_h=splunk- 0x0060: 6866 205f 6964 783d 5f6d 6574 7269 6373 hf._idx=_metrics The _h will come right after the end of the _ts field, without any clear separation.

wowbaggerHU · ‎01-14-2025

Thanks! That is understandable. Based on your answers so far, I will think through what would work best, and will get back to you. But either way, I think I got all the answers I needed.

Posts	36
Solutions	2
Karma Given	15
Karma Received	1
Member Since	‎01-12-2025

Online Status	Offline
Date Last Visited	7 hours ago

byte count of a text field

Validating a Splunk App: Undefined setting in tran...

update transforms.conf and props.conf from an app

Force inclusion of space character as a first char...

conditional whitespace in transform

Using transforms.conf to change metadata format fr...

Re: byte count of a text field

Re: byte count of a text field

Re: byte count of a text field

byte count of a text field

Re: Validating a Splunk App: Undefined setting in ...

Validating a Splunk App: Undefined setting in tran...

Re: update transforms.conf and props.conf from an ...

Re: update transforms.conf and props.conf from an ...

Re: update transforms.conf and props.conf from an ...

Re: update transforms.conf and props.conf from an ...

update transforms.conf and props.conf from an app

Re: Force inclusion of space character as a first ...

Re: conditional whitespace in transform

Re: Force inclusion of space character as a first ...

Re: Force inclusion of space character as a first ...

Force inclusion of space character as a first char...

Re: conditional whitespace in transform

Re: conditional whitespace in transform

Re: conditional whitespace in transform

Re: conditional whitespace in transform

Re: conditional whitespace in transform

Re: conditional whitespace in transform

Re: Using transforms.conf to change metadata forma...

conditional whitespace in transform

Re: Using transforms.conf to change metadata forma...

Are you a member of the Splunk Community?