topic Re: How to remove everything BEFORE the second comma AND before indexing? in Splunk Enterprise

How to remove everything BEFORE the second comma AND before indexing?

newsplunker1 — Fri, 05 May 2023 16:08:10 GMT

I have the following events

<190>May 4 20:20:36 data.test.com 1,2023/05/04 20:20:35,013001101002958,test,end,2305,2023/05/04

I want to remove everything before the second comma (including the comma)

Since i dont want it to be indexed , im using the props and transforms on my HF to do that . My regex seems to work but when i try to implement it ,it does not filter anything

props.conf

[source::/var/log/splunk/IP/syslog.log]
TRANSFORMS-null = remove_before_comma

transforms.conf

[remove_before_comma]
REGEX = ^([^,]*,[^,]*),
DEST_KEY = queue
FORMAT = nullQueue

Here is the regex

https://regex101.com/r/Lxqgue/1

Any idea why this is not working properly

Thanks

Re: How to remove everything BEFORE the second comma AND before indexing?

richgalloway — Fri, 05 May 2023 16:42:13 GMT

The existing transform moves any event that matches the regex to nullQueue. Try these settings to re-write the event.

[remove_before_comma] REGEX = ^([^,]*,[^,]*),(.*) DEST_KEY = _raw FORMAT = $2

Re: How to remove everything BEFORE the second comma AND before indexing?

newsplunker1 — Fri, 05 May 2023 17:39:00 GMT

@richgalloway That seems to do the trick ( I ll mark yours as the asnwer ) - Do you have any idea on how to use SED to replace the strings before the comma with a number lets say 0 for example

this is the original event

<190>May 4 20:20:36 data.test.com 1,2023/05/04 20:20:35,013001101002958

becomes like

0,0,013001101002958

Re: How to remove everything BEFORE the second comma AND before indexing?

richgalloway — Fri, 05 May 2023 17:39:18 GMT

SED is search-time rather than index-time, but you could do it with this props:

SEDCMD-replaceUpToComma = s/^([^,]*,[^,]*),/0/