Splunk Search

Transform to remove the first column of a CSV if it matches a set of criteria?

dennywebb
Path Finder

i have logs coming in as CSV files, but sometimes junk data is truncated on the front by the system generating them, sometimes not. they are otherwise identical. i have no control on this system.

the upside is the junk is always of a certain pattern:
blah:blah,gooddata,gooddata,gooddata,gooddata,gooddata
versus the clean ones which are just:
gooddata,gooddata,gooddata,gooddata,gooddata

how do i get a transform to drop that first column before indexing ONLY IF it has X:X as a value?

been beating my head against this for 2 days... regex is not my strong point.

Tags (3)
1 Solution

Ayn
Legend

Well a regex that will match anything at the beginning of the line until the first comma, with a : character in it would look like this:

^[^,]+:[^,]+,

View solution in original post

Ayn
Legend

Well a regex that will match anything at the beginning of the line until the first comma, with a : character in it would look like this:

^[^,]+:[^,]+,

dennywebb
Path Finder

an update, installed and working perfectly.

0 Karma

dennywebb
Path Finder

you sir/mam, are the man/woman. the end result:
^([^,]+:[^,]+,)?(.*)
works great. i should be able to just feed $2 back into the raw for either type and always have the same result.

0 Karma

Ayn
Legend

Sorry, forgot two + signs in my regex. Editing my answer with a correct regex.

0 Karma

dennywebb
Path Finder

though when i put what you provided into http://regexlib.com/RETester.aspx
as: ^[^,]:[^,],(.*)
with data as: something:anything,stuff1,stuff2,stuff3:stuff4,stuff5

nothing comes back.

0 Karma

dennywebb
Path Finder

and then i would use the Dest_Key=_Raw in the transform stanza to replace the raw log with my newly cleaned one i presume?

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...