topic Re: Is it possible to change the delimiter on the UF side during transfer? (UF側で、転送時に区切り文字を変更できますか?) in Splunk Enterprise

Is it possible to change the delimiter on the UF side during transfer? (UF側で、転送時に区切り文字を変更できますか?)

tsudatyou — Thu, 06 Oct 2022 19:29:59 GMT

Hi
(お世話になっております)

An application logs to "/var/log/messages".
(ある既製のアプリケーションから、/var/log/messages にログが出力されています。)

However, unfortunately, the delimiter is \x09.
(但し、区切り文字が、\x09 となっています。)

Is it possible to replace the delimiter with a space or comma on the "suplunk Universal forwarder" side and forward it?
("suplunk universal fowarder" 側で、区切り文字をスペースやカンマに置き換えてから転送することは可能でしょうか？)

The version of 'splunk' is unknown.
("splunk"のバージョンは不明です。)

The version of "suplunk Universal forwarder" is "9.0.1".
("suplunk universal fowarder"のバージョンは、"9.0.1"です。)

"suplunk Universal forwarder" is installed in RHEL8.5.
("suplunk universal fowarder"は、RHEL8.5にインストールしています。)

Thanks!
(よろしくお願いいたします)

Re: Is it possible to change the delimiter on the UF side during transfer? (UF側で、転送時に区切り文字を変更できますか?)

richgalloway — Sun, 09 Oct 2022 14:29:31 GMT

Try these untested settings in your props.conf file on the UF.

[mysourcetype] force_local_processing = true SEDCMD-notab = s/\x09/ /g

Re: Is it possible to change the delimiter on the UF side during transfer? (UF側で、転送時に区切り文字を変更できますか?)

PickleRick — Sun, 09 Oct 2022 14:54:45 GMT

Never thought SEDCMD would work on UF. Have you ever tried SEDCMD on UF? I was always under the impression that there is no event modification on the UF (apart from indexed extractions).

Re: Is it possible to change the delimiter on the UF side during transfer? (UF側で、転送時に区切り文字を変更できますか?)

richgalloway — Mon, 10 Oct 2022 12:42:30 GMT

I haven't tried this myself, which is why I labeled the answer as "untested". The force_local_processing setting is supposed to allow the UF to perform regex replacements, so it might work.

Re: Is it possible to change the delimiter on the UF side during transfer? (UF側で、転送時に区切り文字を変更できますか?)

PickleRick — Mon, 10 Oct 2022 11:26:18 GMT

Yeah. Just tested it and it seems to be working. Good to know!

It's still quite limited from what I see (you can't seem to be able to do the more complicated stuff like index-time evals) but for this case it seems to be the thing.

Re: Is it possible to change the delimiter on the UF side during transfer? (UF側で、転送時に区切り文字を変更できますか?)

tsudatyou — Tue, 11 Oct 2022 08:15:15 GMT

Thank you for your answer.

Since I am a beginner, it will take some time to report the results.
I'll test it.

Re: Is it possible to change the delimiter on the UF side during transfer? (UF側で、転送時に区切り文字を変更できますか?)

tsudatyou — Tue, 11 Oct 2022 08:17:35 GMT

Thank you for your answer.

I tried it.

But , nothing changed on the Splunk Server side.

Oct 11 16:47:17 SPLUNK-UF OneApp[2417]: [notice] 2022/10/11 16:47:17\x091665474437.6657\x09uid=user999\x09domain=local\x09level=notice\x09code=54201\x09message=OneApp Authentication succeeded.\x09host_ip=192.168.0.52\x09client_ip=192.168.0.5\x09client_ua=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36\x09client_cookie=\x09admin_id=

I installed UF as follows:

# export SPLUNK_HOME="/opt/splunkforwarder"
# vi .bash_profile
# mkdir $SPLUNK_HOME

# tar xvzf splunkforwarder-9.0.1-82c987350fde-Linux-x86_64.tgz -C /opt
# useradd -m splunk
# passwd splunk
# chown -R splunk:splunk $SPLUNK_HOME

# $SPLUNK_HOME/bin/splunk start --accept-license
# $SPLUNK_HOME/bin/splunk stop
# $SPLUNK_HOME/bin/splunk disable boot-start
# $SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 1
# $SPLUNK_HOME/bin/splunk start

# $SPLUNK_HOME/bin/splunk add forward-server 192.168.0.51:9997

# $SPLUNK_HOME/bin/splunk add monitor /var/log/messages
# $SPLUNK_HOME/bin/splunk restart

Currently there are only three files in the "$SPLUNK_HOME/etc/system/local" folder:

I don't know what to do. What additional settings are required?

# ls $SPLUNK_HOME/etc/system/local
README outputs.conf props.conf server.conf
#

# cat $SPLUNK_HOME/etc/system/local/outputs.conf
[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 192.168.0.51:9997

[tcpout-server://192.168.0.51:9997]
#

# cat $SPLUNK_HOME/etc/system/local/server.conf
[general]
serverName = SPLUNK-UF
pass4SymmKey = ***

[sslConfig]
sslPassword = ***

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
peers = *
quota = MAX
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
peers = *
quota = MAX
stack_id = free
#

# cat $SPLUNK_HOME/etc/system/local/props.conf
[mysourcetype]
force_local_processing = true
SEDCMD-notab = s/\x09/ /g

Re: Is it possible to change the delimiter on the UF side during transfer? (UF側で、転送時に区切り文字を変更できますか?)

PickleRick — Tue, 11 Oct 2022 10:21:41 GMT

You have to change the

[mysourcetype]

stanza to the sourcetype that corresponds to the sourcetype you're ingesting your files with.

If I remember correctly, if you add monitor without explicitly stated sourcetype you're telling Splunk to guess the sourcetype which... isn't the best idea. You should add monitor input with a specific sourcetype (and destination index) and name your props.conf stanza with the same sourcetype name.

For example:

$SPLUNK_HOME/bin/splunk add monitor /var/log/messages -sourcetype unix_messages -index my_unix_inde

Of course you'd need to have an index my_unix_index created for this to work properly.

And then you'd need to add [unix_messages] stanza to your props.conf instead of [mysourcetype].

Re: Is it possible to change the delimiter on the UF side during transfer? (UF側で、転送時に区切り文字を変更できますか?)

tsudatyou — Tue, 11 Oct 2022 13:50:32 GMT

Thanks for following me.

I tried:
<<< server side >>>
# $SPLUNK_HOME/bin/splunk add index my_unix_index

<<< UF side >>>

# $SPLUNK_HOME/bin/splunk remove monitor /var/log/messages
# $SPLUNK_HOME/bin/splunk add monitor /var/log/messages -sourcetype unix_messages -index my_unix_index

# cat $SPLUNK_HOME/etc/system/local/props.conf
[unix_messages]
force_local_processing = true
SEDCMD-notab = s/\x09/ /g
#
# $SPLUNK_HOME/bin/splunk restart

then Result is:
Oct 11 22:37:32 SPLUNK-UF OneApp[3007]: [notice] 2022/10/11 22:37:32\x091665495452.2401\x09uid=tx_pt999\x09domain=local\x09level=notice\x09code=54201\x09message=OneApp Authentication succeeded.\x09host_ip=192.168.0.52\x09client_ip=192.168.0.5\x09client_ua=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36\x09client_cookie=\x09admin_id=

　host = SPLUNK-UF source = /var/log/messages sourcetype = unix_messages

What settings am I missing?

Re: Is it possible to change the delimiter on the UF side during transfer? (UF側で、転送時に区切り文字を変更できますか?)

PickleRick — Tue, 11 Oct 2022 14:39:12 GMT

I think there's some misunderstanding here.

If your delimiter is literarily a string saying "\x09" you need to do double backslash in your SEDCMD

Like:

SEDCMD-notab = s/\\x09/ /g

I think both I and @richgalloway assumed that you were talking about a single character with a code of 0x09 (a tab character if I remember correctly).

Re: Is it possible to change the delimiter on the UF side during transfer? (UF側で、転送時に区切り文字を変更できますか?)

tsudatyou — Wed, 12 Oct 2022 01:34:33 GMT

I really appreciate it！

finally
I achieved my goal.

index="my_unix_index"

Oct 12 09:37:27 SPLUNK-UF OneApp[1579]: [notice] 2022/10/12 09:37:27 1665535047.3298 uid=tx_pt999 domain=local level=notice code=54201 message=OneApp Authentication succeeded. host_ip=192.168.0.52 client_ip=192.168.0.5 client_ua=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36 client_cookie= admin_id=
host = SPLUNK-UF | source = /var/log/messages | sourcetype = unix_messages