Splunk Search

Transform to remove the first column of a CSV if it matches a set of criteria?

dennywebb
Path Finder

i have logs coming in as CSV files, but sometimes junk data is truncated on the front by the system generating them, sometimes not. they are otherwise identical. i have no control on this system.

the upside is the junk is always of a certain pattern:
blah:blah,gooddata,gooddata,gooddata,gooddata,gooddata
versus the clean ones which are just:
gooddata,gooddata,gooddata,gooddata,gooddata

how do i get a transform to drop that first column before indexing ONLY IF it has X:X as a value?

been beating my head against this for 2 days... regex is not my strong point.

Tags (3)
1 Solution

Ayn
Legend

Well a regex that will match anything at the beginning of the line until the first comma, with a : character in it would look like this:

^[^,]+:[^,]+,

View solution in original post

Ayn
Legend

Well a regex that will match anything at the beginning of the line until the first comma, with a : character in it would look like this:

^[^,]+:[^,]+,

dennywebb
Path Finder

an update, installed and working perfectly.

0 Karma

dennywebb
Path Finder

you sir/mam, are the man/woman. the end result:
^([^,]+:[^,]+,)?(.*)
works great. i should be able to just feed $2 back into the raw for either type and always have the same result.

0 Karma

Ayn
Legend

Sorry, forgot two + signs in my regex. Editing my answer with a correct regex.

0 Karma

dennywebb
Path Finder

though when i put what you provided into http://regexlib.com/RETester.aspx
as: ^[^,]:[^,],(.*)
with data as: something:anything,stuff1,stuff2,stuff3:stuff4,stuff5

nothing comes back.

0 Karma

dennywebb
Path Finder

and then i would use the Dest_Key=_Raw in the transform stanza to replace the raw log with my newly cleaned one i presume?

0 Karma
Get Updates on the Splunk Community!

Expert Tips from Splunk Professional Services, Ensuring Compliance, and More New ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Observability Release Update: AI Assistant, AppD + Observability Cloud Integrations & ...

This month’s releases across the Splunk Observability portfolio deliver earlier detection and faster ...

Stay Connected: Your Guide to February Tech Talks, Office Hours, and Webinars!

💌Keep the new year’s momentum going with our February lineup of Community Office Hours, Tech Talks, ...