Why won't my regex mask my data?

[wsveum]

Hi,
i have a challenge masking out password data from the ps-source/sourcetype events at indexing time. 

We have made an application with a props.conf file and a transforms.conf file. This application is distributed to all indexers, and when we use btool to list which settings are in use, it all seems ok. The indexers has also been restarted after pushing the bundle to them, although a restart was not necessary according to the validate  cluster-bundle command.

My regex works fine in regex101, but nevertheless the passwords still remains unmasked after trying to activate it.

From props.conf:
# Remove password from source:ps for wlp-servers
[ps]
TRANSFORMS-anonymize = ps_password-anonymizer

From transforms.conf:
[ps_password-anonymizer]
REGEX = (?m)^(.*?password=|.*?PASSWORD=).*?_(-.*)$
FORMAT = $1XXXX_$2
DEST_KEY = _raw

From btool:
/opt/splunk/etc/peer-apps/nt_anonymizer/default/transforms.conf [ps_password-anonymizer]
/opt/splunk/etc/system/default/transforms.conf CAN_OPTIMIZE = True
/opt/splunk/etc/system/default/transforms.conf CLEAN_KEYS = True
/opt/splunk/etc/system/default/transforms.conf DEFAULT_VALUE =
/opt/splunk/etc/system/default/transforms.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/peer-apps/nt_anonymizer/default/transforms.conf DEST_KEY = _raw
/opt/splunk/etc/peer-apps/nt_anonymizer/default/transforms.conf FORMAT = $1XXXX_$2
/opt/splunk/etc/system/default/transforms.conf KEEP_EMPTY_VALS = False
/opt/splunk/etc/system/default/transforms.conf LOOKAHEAD = 4096
/opt/splunk/etc/system/default/transforms.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/transforms.conf MV_ADD = False
/opt/splunk/etc/peer-apps/nt_anonymizer/default/transforms.conf REGEX = (?m)^(.*?password=|.*?PASSWORD=).*?_(-.*)$
/opt/splunk/etc/system/default/transforms.conf SOURCE_KEY = _raw

Any ideas why this wont work as expected? Isn't it possible to do this on the indexers? Does it have to be done on a HF?

[richgalloway]

If the data passes through an HF then the transforms MUST be done on the HF.

When testing expressions on regex101.com, be sure to select the PCRE regex engine.  Also, Splunk's regex processor is not the same as regex101's so there will be things you can do on one and not the other.

Try simplifying the regex.  There's no need for the multiline flag.  Using ^.* is meaningless as is .*$ because they're implied in almost every regex.

REGEX = (password=|PASSWORD=).*?_(-.*)

[wsveum]

Thanks for your answer, @richgalloway 

The data does'nt pass through any HF, it comes directly from UFs.

I've tried your suggestion, but unfortunately without any difference. 
It seems like the REGEX totally is overlooked, and no transformation is done.

Any other suggestions?

[richgalloway]

It would seem the regex is not matching the data so the transform is not applied.  To resolve that, we need to see some sample (sanitized) data.

[wsveum]

This is a short extraction form the source, with one of the events (the 2. one starting with "wasadmin") containing 2 occurences of passwords that needs to be masked. In this expample I've already changed the original password to something else 😉

root 70273 1 0.0 00:00:00 0.0 0 0 ? S 02:22 [kworker/1:1] <noArgs>
splunk 76840 1 0.0 00:00:00 0.0 1708 118784 ? S 00:00 sh /opt/splunkforwarder/etc/apps/Splunk_TA_nix_700/bin/ps.sh
splunk 76862 0 0.0 00:00:00 0.0 1844 155452 ? R 00:00 ps -wweo_uname:32,pid,psr,pcpu,cputime,pmem,rsz,vsz,tty,s,etime,args
splunk 76863 1 0.0 00:00:00 0.0 676 108056 ? S 00:00 tee /dev/null
splunk 76864 1 0.0 00:00:00 0.0 1240 113648 ? S 00:00 awk {NR_==_1_&&_$0_=_header}_{sub("^_",_"",_$1);_if_(NF>12)_{args=$13;_for_(j=14;_j<=NF;_j++)_args_=_args_"_"_$j}_else_args="<noArgs>";_sub("^[^\134[:_-]*/",_"",_$12)}_(NR>1)_{if_($4<0_||_$4>100)_$4=0;_if_($6<0_||_$6>100)_$6=0}_{if_(NR_==_1)_{print_$0}_else_{printf_"%32.32s_%6s_%4s_%6s_%12s_%6s_%8s_%8s_%-7.7s_%1.1s_%12s_%-100.100s_%s\n",_$1,_$2,_$3,_$4,_$5,_$6,_$7,_$8,_$9,_$10,_$11,_$12,_args}}_header=USER_PID_PSR_pctCPU_CPUTIME_pctMEM_RSZ_KB_VSZ_KB_TTY_S_ELAPSED_COMMAND_ARGS
root 100889 0 0.0 00:00:00 0.0 0 0 ? S 38:18 [kworker/u256:1] <noArgs>
root 115532 1 0.0 00:00:04 0.0 0 0 ? S 1-06:12:01 [kworker/u256:0] <noArgs>
root 118622 1 0.0 00:00:00 0.0 676 115812 ? S 92-18:08:00 rhsmcertd <noArgs>
wasadmin 119789 0 0.2 04:21:11 3.9 477620 3004144 ? S 79-18:30:08 machine-agent/jre/bin/java -Xmx256m_-Dlog4j.configuration=file:/opt/appdynamics/machine-agent/conf/logging/log4j.xml_-jar_/opt/appdynamics/machine-agent/machineagent.jar
wasadmin 126422 0 1.2 04:54:16 4.3 525304 2199416 ? S 15-21:58:41 java -javaagent:/opt/IBM/Paymentv4-20354/wlp/bin/tools/ws-javaagent.jar_-Djava.awt.headless=true_-Djdk.attach.allowAttachSelf=true_-verbose:gc_-Xverbosegclog:/var/log/websphere/Paymentv4-20354/verbosegc-Paymentv4-20354.log,10,10000_-Xms128m_-Xmx256m_-DEnvironmentName=PROD01_-DGetEnvironmentPropertiesURL=http://vip-esb2.prod01.norsk-tipping.no/GetEnvironmentProperties_-Dappdynamics.agent.applicationName=Payment_-Dappdynamics.agent.logs.dir=/var/log/websphere/Paymentv4-appdynamics_-Dappdynamics.agent.tierName=Paymentv4_-Dbuypass.clientId=100018_-Dbuypass.keystore.password=dfhrKWdw38674s%w_-Dbuypass.keystore.path=certs/Buypass-ID-100018.p12_-Dbuypass.scopes=nt-reconciliation-api_-Dcustomerbalance.redis.database=2_-Dcustomerbalance.redis.master=master01_-Dcustomerbalance.redis.maxidlepoolclients=100_-Dcustomerbalance.redis.maxtotalpoolclients=200_-Dcustomerbalance.redis.password=hkdERDTG5467&ll_-Dcustomerbalance.redis.sentinels=p1reds500.prod01.norsk-tipping.no:26379,p1reds501.prod01.norsk-tipping.no:26379,p2reds500.prod01.norsk-tipping.no:26379_-Dcustomerbalance.redis.timeout=2000_-Dcustomerbalance.ttl_sec=300_-Dfeature.toggles.reconciliation=true_-Dfeign.timeout.connection.BuypassClient=10000_-Dfeign.timeout.connection.DoPaymentClient=10000_-Dfeign.timeout.read.BuypassClient=10000_-Dfeign.timeout.read.DoPaymentClient=10000_-Dnt.envprop.override.Authenticationv1URL=http://authenticationv1-prod01.apps.ocpprod03.norsk-tipping.no_-Dnt.envprop.override.ReconciliationURL=https://api.nt.vpn.buypass.no/nt-reconciliation-api_-Dorg.springframework.boot.logging.LoggingSystem=none_-javaagent:/opt/appdynamics/AppServerAgent-1.8-22.12.0.34603/javaagent.jar_--add-exports_java.base/sun.security.action=ALL-UNNAMED_--add-exports_java.naming/com.sun.jndi.ldap=ALL-UNNAMED_--add-exports_java.naming/com.sun.jndi.url.ldap=ALL-UNNAMED_--add-exports_jdk.naming.dns/com.sun.jndi.dns=ALL-UNNAMED_--add-exports_java.security.jgss/sun.security.krb5.internal=ALL-UNNAMED_--add-exports_jdk.attach/sun.tools.attach=ALL-UNNAMED_--add-opens_java.base/java.util=ALL-UNNAMED_--add-opens_java.base/java.lang=ALL-UNNAMED_--add-opens_java.base/java.util.concurrent=ALL-UNNAMED_--add-opens_java.base/java.io=ALL-UNNAMED_--add-opens_java.naming/javax.naming.spi=ALL-UNNAMED_--add-opens_java.naming/com.sun.naming.internal=ALL-UNNAMED_--add-opens_jdk.naming.rmi/com.sun.jndi.url.rmi=ALL-UNNAMED_--add-opens_java.naming/javax.naming=ALL-UNNAMED_--add-opens_java.rmi/java.rmi=ALL-UNNAMED_--add-opens_java.sql/java.sql=ALL-UNNAMED_--add-opens_java.management/javax.management=ALL-UNNAMED_--add-opens_java.base/java.lang.reflect=ALL-UNNAMED_--add-opens_java.desktop/java.awt.image=ALL-UNNAMED_--add-opens_java.base/java.security=ALL-UNNAMED_--add-opens_java.base/java.net=ALL-UNNAMED_--add-opens_java.base/java.text=ALL-UNNAMED_--add-opens_java.base/sun.net.www.protocol.https=ALL-UNNAMED_--add-exports_jdk.management.agent/jdk.internal.agent=ALL-UNNAMED_--add-exports_java.base/jdk.internal.vm=ALL-UNNAMED_-jar_/opt/IBM/Paymentv4-20354/wlp/bin/tools/ws-server.jar_Paymentv4 
root 1 1 0.0 01:05:20 0.0 4512 199604 ? S 98-23:15:58 systemd --switched-root_--system_--deserialize_22

[wsveum · Accepted solution]

Fixed this issue with below files, after a little help from Splunk support!

props.conf

[ps]
TRANSFORMS-anonymize = ps_password-anonymizer
SEDCMD-mask = s/(password=|PASSWORD=)(.*?_)/\1xxxx_/g

transforms.conf

[ps_password-anonymizer]
REPEAT_MATCH = true