topic Re: Timestamp and masking syslog through universal forwarder in Splunk Data Stream Processor

How to get timestamp and mask syslog through universal forwarder?

marellasunil_3 — Wed, 23 Mar 2022 22:14:37 GMT

I have a systemout.log file and I am indexing using pretrained sourcetype websphere_trlog_sysout. Currently there is an issue with masking

I have created props.conf in the deployment app and deployed to universal forwarder as below. Created transforms.conf to mask the data.

Seems the issue is still same.

[websphere_trlog_sysout] TIME_FORMAT = %d-%m-%y %H:%M:%S:%3Q %Z TIME_PREFIX = ^\[ FORMAT = $1-$2-$3 $4:$5:$6:$7 $8 TRANSFORMS-anonymize = session-anonymizer [session-anonymizer] REGEX = XXXXX FORMAT = $1XXXX$2 DEST_KEY = _raw

Re: Timestamp and masking syslog through universal forwarder

venky1544 — Wed, 23 Mar 2022 11:04:06 GMT

Hi @marellasunil_3

you can try the sedcmd in props.conf and dont require in transform but this should go to indexer
SED script works at index time and executed on _raw field. so should be in indexer

first you can test the sedcmd in a rex in a search: to check if masking is working

|makeresults |eval date="2022-03-23 10:24:19:695+0000"|rex mode=sed field=date max_match=0 "s/(([\d+\-]+) ([\d+:]+))/XXXXXX/"

Just write this is in props.conf . you do not need to write transforms.conf.

[websphere_trlog_sysout]

SEDCMD-replace=/(([\d+\-]+) ([\d+:]+))/XXXXXX/

and then restart the server.

Hope this helps

Re: Timestamp and masking syslog through universal forwarder

marellasunil_3 — Wed, 23 Mar 2022 12:47:12 GMT

Hi Venky
Thanks for the reply I am trying to mask card details, that does not have field

Ex: [23/03/2020 13:45:20:123 IST Systemout ..... Card 1234-5678-9012-3456]

I would like to mask card details.

Re: Timestamp and masking syslog through universal forwarder

venky1544 — Wed, 23 Mar 2022 13:03:41 GMT

Hi @marellasunil_3

try the below search

|makeresults |eval data="23/03/2020 13:45:20:123 IST Systemout ..... Card 1234-5678-9012-3456"|rex mode=sed field=data "s/Card ((\d+\-)+)/Card XXXX-XXXX-XXXX/"

and try it in props.conf

[websphere_trlog_sysout]

SEDCMD-replace=s/Card ((\d+\-)+)/Card XXXX-XXXX-XXXX/

---
If this reply helps you, an upvote/karma would be appreciated.

Re: Timestamp and masking syslog through universal forwarder

marellasunil_3 — Wed, 23 Mar 2022 14:49:03 GMT

Thanks Venky.

It is resolved the issue partially. I found Card details in other location as well

As sed command have limitation to use only once, I would like to use regex.

I tried regex command as stated above in the custom app deployed in forwarder. But this is not working at all.

where do you think I did the mistake?

Props.comf

[websphere_trlog_sysout]
TRANSFORMS-anonymize = session-anonymizer

Transforms.conf

[session-anonymizer]
REGEX = XXXXX
FORMAT = $1XXXX$2
DEST_KEY = _raw

Re: Timestamp and masking syslog through universal forwarder

venky1544 — Wed, 23 Mar 2022 15:24:27 GMT

Hi @marellasunil_3

Hey not sure how you came to conclusion that As "sed command have limitation to use only once" there is no such limitation it would replace multiple occurences of the matched regex if the pattern is same you just have to add global attribute "g" at the end

|rex mode=sed field=data "s/Card ((\d+\-)+)/Card XXXX-XXXX-XXXX/g"

example

|makeresults |eval data="23/03/2020 13:45:20:123 IST Systemout ..... Card 1234-5678-9012-3456 Card 1234-5678-9012-3456" |rex mode=sed field=data "s/Card ((\d+\-)+)/Card XXXX-XXXX-XXXX/g"

Note this would only not work in case the pattern is not matching for the second occcurence

And about your Transforms.conf

[session-anonymizer]
REGEX = XXXX ( you have to define the regex which would be the identifier )
FORMAT = $1XXXX$2 ($1 and $2) are the captured named group
DEST_KEY = _raw

[session-anonymizer]
REGEX = (Card) ((\d+\-)+) ( this is the regex)
FORMAT = $1XXXX (here there wont be $2 because there is only one group here and your are masking 2nd group so dont include $2
DEST_KEY = _raw

if you still have trouble i would share more details with screenshot

--------

If you find the answer was helpful, an upvote/karma is appreciated and please accept as solution it would help others as well