I even set up the Windows box to emit metadata that matches the regex in the transform, because none of my logs seemed to have that subsecond data. Now my Windows test machine sends among its metadata the string time_subsecond=.123456 And interestingly enough the subsecond transform doesn't get triggered. My latest version is: [metadata_subsecond] SOURCE_KEY = _meta REGEX = _subsecond::(\.[0-9]+) FORMAT = $1$0 DEST_KEY = _raw But nothing seems to happen. ... View more

[md_time] SOURCE_KEY = _time REGEX = (.*) FORMAT = _ts=$1 DEST_KEY = time_temp [md_subsecond] SOURCE_KEY = _meta REGEX = _subsecond::(\.\d+) FORMAT = $1 DEST_KEY = subsecond_temp [md_fix_subsecond] INGEST_EVAL = _raw=if(isnull(subsecond_temp), time_temp + " " + _raw, time_temp + subsecond_temp + " " + _raw) [md_time_default] SOURCE_KEY = _time REGEX = (.*) FORMAT = _ts=$1 $0 DEST_KEY = _raw   The problem seems to be somewhere in md_time, md_subsecond or md_fix_subsecond, because if I use md_time_default, it works (though without subseconds), and if I enable these three instead of md_time_default, then I get no output: the packets emitted by Splunk seem to be empty: without a payload. ... View more

I tried what you suggested, but did not seem to help. It seemed as if the fix_subsecond stanza wouldn't be executed at all. The _h KV pair followed _ts's value without a whitespace. After experimenting a bit more, I now have this, but this doesn't work either: [md_time] SOURCE_KEY = _time REGEX = (.*) FORMAT = _ts=$1 $0 DEST_KEY = time_temp [md_subsecond] SOURCE_KEY = _meta REGEX = _subsecond::(\.\d+) FORMAT = $1 DEST_KEY = subsecond_temp [md_fix_subsecond] INGEST_EVAL = _raw=if(isnull(subsecond_temp),time_temp+" "+_raw,time_temp+subsecond_temp+" "+_raw) Plus props.conf: [default] ADD_EXTRA_TIME_FIELDS = none ANNOTATE_PUNCT = false SHOULD_LINEMERGE = false TRANSFORMS-zza-syslog = syslog_canforward, reformat_metadata, md_add_separator, md_source, md_sourcetype, md_index, md_host, md_subsecond, md_time, md_fix_subsecond, discard_empty_msg # The following applies for TCP destinations where the IETF frame is required TRANSFORMS-zzz-syslog = syslog_octet_count, octet_count_prepend # Comment out the above and uncomment the following for udp #TRANSFORMS-zzz-syslog-udp = syslog_octet_count, octet_count_prepend, discard_empty_msg [audittrail] # We can't transform this source type its protected TRANSFORMS-zza-syslog = TRANSFORMS-zzz-syslog = However this now breaks logging and I'm getting no logs forwarded to sylsog-ng. The connection is up, but no meaningful data arrives, just "empty" packages. What may be the problem? Did I break the sequence of the stanzas? (I did not seem to understand it in the first place, as they seem to be in backward order compared to how the KV pairs follow each other in the actual log message.) ... View more

Yes, I noticed the excess escaping too, but if it's incorrect then the SC4S suggested config is wrong too. Either way, I will try it both ways, and get back to you. Thank you so much for your help! ... View more

Yes, it is in deed. I thought of that, but I assume that the creators at SC4S probably wanted the timestamp to have the fraction seconds added to it, if there is a metadata variable holding that. In that case, the decimal point and the fraction seconds need to follow the timestamp without any whitespace. That is the reason the whitespace is missing where you pointed it out. If there isn't a variable holding the fraction seconds however, like in my case, there will be no trailing space added to the timestamp, and the host key-value pair will follow it right without a whitespace. Any idea how I could add a whitespace conditionally? ... View more

Thanks for your help with this. In the meantime I've run into another problem. Could you please help me? This is the topic: https://community.splunk.com/t5/Getting-Data-In/conditional-whitespace-in-transform/m-p/708831 ... View more

Thanks! That is understandable. Based on your answers so far, I will think through what would work best, and will get back to you. But either way, I think I got all the answers I needed. ... View more

Thanks for clarifying. You helped a lot! That means there are two options for me: do this conversion on the syslog-ng side and that won't hurt the splunk side of things forward the logs to yet another splunk instance that will only do this conversion, thereby isolating the "production" Splunk instance from these transforms ... View more

Okay, I reverted to using INGEST_EVAL, that works as well. On the other hand, I have an additional question: If a given Splunk node is already forwarding logs to another node over S2S or S2S over HEC, and I want to add this configuration to send the logs to yet another destination (a node running syslog-ng), then will this configuration break the other pre-existing destinations' log format? Or is it safe to use from this perspective? ... View more

I can confirm, this type of setup does not work for the Windows logs:   [sanitize_metadata] EVAL-EEEE =replace(_meta,"::","=") [metadata_meta] SOURCE_KEY = EEEE REGEX = (?ims)(.*) FORMAT = $1__-__$0 DEST_KEY = _raw The problem is that with this the Windows logs only contain the eventlog message part, as if they did not have any metadata attached.   ... View more

I have played around a bit more... This is what seems to be working for me: [sanitize_metadata] EVAL-_meta=replace(_meta,"::","=") [metadata_meta] SOURCE_KEY = _meta REGEX = (?ims)(.*) FORMAT = $1__-__$0 DEST_KEY = _raw Note: __-__ is just a placeholder for a separator. I found an article that is aiming at a marginally similar thing as I do: https://zchandikaz.medium.com/alter-splunk-data-at-indexing-time-a10c09713f51 There, the individual uses EVAL instead of INGEST_EVAL. Is there any significant difference? Also, I changed your example because it worked differently if I did not use _meta as a target variable in the INGEST_EVAL. I noticed that with your version, the logs that originated from the Windows machine with the UF on it, were missing the metadata assigned there. When I use my version, all the metadata set on the UF (static key-value pairs) is there in the log. Any idea why that might be? Either way, thanks so much for your effort to help me! I really appreciate it! ... View more

No, it's a custom configured syslog-ng instance. that I set up. After looking at the logs arriving, I saw that the logs that previously had the metadata part included, now have nothing instead and the separators (~~~EM~~~ and ~~~SM~~~) are missing too. ... View more

I am forwarding the logs from the Splunk HF to a syslog-ng instance, that I configured myself so it doesn't matter here. ... View more

I checked it, but unfortunately it does not seem to work. Now I can't seem to find logs that contain any metadata, so I assume they are being dropped due to some problem. Where should I look for clues? ... View more

Hello, I have a Windows machine with UF that sends its logs to a HF, which has the SC4S derived config loaded (see the opening entry's link). That allows to reformat the logs that passed through the HF to IETF 5424 syslog (with framing enabled) and forward them to a syslog instance. That reformatting pretty much alters most parts of the original message. In the output you will generally see the first half of the message (not counting the SDATA part) will contain the metadata fields in the key::value format. I would like to change that in the syslog output generated by the config on the HF node. ... View more

What you are referring to is the syslog serialized data or SDATA (see RFC 5424) portion of the message. That consists of only 5 values (same as the Splunk JSON envelope's 5 top-level fields). And yes, those use the equals sign as a separator. On the other hand the main part of the message will look like this: ~~~SM~~~env::env01~~~EM~~~11/29/2024 02:01:55 PM\nLogName=Security\nEventCode=4624\nEventType=0\nComputerName=DESKTOP-OOU0O6E\nSourceName=Microsoft Windows security auditing.\nType=Information\nRecordNumber=49513\nKeywords=Audit Success\nTaskCategory=Logon\nOpCode=Info\nMessage=An account was successfully logged on.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tNT AUTHORITY\\SYSTEM\r\n\tAccount Name:\t\tDESKTOP-OOU0O6E$\r\n\tAccount Domain:\t\tWORKGROUP\r\n\tLogon ID:\t\t0x3E7\r\n\r\nLogon Information:\r\n\tLogon Type:\t\t5\r\n\tRestricted Admin Mode:\t-\r\n\tVirtual Account:\t\tNo\r\n\tElevated Token:\t\tYes\r\n\r\nImpersonation Level:\t\tImpersonation\r\n\r\nNew Logon:\r\n\tSecurity ID:\t\tNT AUTHORITY\\SYSTEM\r\n\tAccount Name:\t\tSYSTEM\r\n\tAccount Domain:\t\tNT AUTHORITY\r\n\tLogon ID:\t\t0x3E7\r\n\tLinked Logon ID:\t\t0x0\r\n\tNetwork Account Name:\t-\r\n\tNetwork Account Domain:\t-\r\n\tLogon GUID:\t\t{00000000-0000-0000-0000-000000000000}\r\n\r\nProcess Information:\r\n\tProcess ID:\t\t0x2d4\r\n\tProcess Name:\t\tC:\\Windows\\System32\\services.exe\r\n\r\nNetwork Information:\r\n\tWorkstation Name:\t-\r\n\tSource Network Address:\t-\r\n\tSource Port:\t\t-\r\n\r\nDetailed Authentication Information:\r\n\tLogon Process:\t\tAdvapi \r\n\tAuthentication Package:\tNegotiate\r\n\tTransited Services:\t-\r\n\tPackage Name (NTLM only):\t-\r\n\tKey Length:\t\t0\r\n\r\nThis event is generated when a logon session is created. It is generated on the computer that was accessed.\r\n\r\nThe subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.\r\n\r\nThe logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).\r\n\r\nThe New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.\r\n\r\nThe network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.\r\n\r\nThe impersonation level field indicates the extent to which a process in the logon session can impersonate.\r\n\r\nThe authentication information fields provide detailed information about this specific logon request.\r\n\t- Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.\r\n\t- Transited services indicate which intermediate services have participated in this logon request.\r\n\t- Package name indicates which sub-protocol was used among the NTLM protocols.\r\n\t- Key length indicates the length of the generated session key. This will be 0 if no session key was requested. I would like to have the first part of the syslog message to have the metadata as env=env01 or env:env01. As I understand the SC4S derived config allows you to modify most parts of the message. But is it possible for the metadata part too? If yes, how do I match to the metadata key-value pairs? ... View more

I don't want to change how fields are indexed. I just want to reformat the metadata (to use different key-value separators) via the transforms.conf prior to being forwarded to syslog-ng. ... View more