topic Re: CSV Field Extraction in Dashboards & Visualizations

CSV Field Extraction

teco_akelly — Thu, 11 Feb 2021 21:07:52 GMT

I've got a non-standard CSV log file that has no headers. Depending on the first field of each line determines what number and order of the fields are behind it. How can I have splunk assign field names to the data depending on the first field in a given row?

An example would be:

Operation Record,Timestamp,Tagname,Decription,Station

Alarm Confirm,Timestamp,Tagname,Station,Severity,

Process Alarm,Timestamp,Tagname,Description,Value

Re: CSV Field Extraction

richgalloway — Fri, 12 Feb 2021 14:44:16 GMT

That's not a true CSV file, even though the values are separated by commas.

I would try using transforms with REGEX to parse those rows.

Re: CSV Field Extraction

teco_akelly — Thu, 18 Feb 2021 15:33:24 GMT

Do you know if there are any tools out there or someone I can get to assist in writing this if I provide the csv? I am completely unfamiliar with transforms and REGEX.

Re: CSV Field Extraction

richgalloway — Thu, 18 Feb 2021 19:31:03 GMT

Many of us like regex101.com for help writing and testing regular expressions.

I think you'll need a transform for each type of records in the CSV file. The transforms would look something like this.

[parseOperation] REGEX = Operation Record,([^,]+),([^,]+),([^,]+),([^,]+) FORMAT = Timestamp::$1 Tagname::$2 Description::$3 Station::$4 [parseAlarm] REGEX = Alarm Confirm,([^,]+),([^,]+),([^,]+),([^,]+) FORMAT = Timestamp::$1 Tagname::$2 Station::$3 Severity::$4 [parseProcess] REGEX = Process Alarm,([^,]+),([^,]+),([^,]+),([^,]+) FORMAT = Timestamp::$1 Tagname::$2 Description::$3 Value::$4

Then a props.conf attribute would reference the transforms.

[mysourcetype] TRANSFORMS-parsers = parseOperation, parseAlarm, parseProcess

See the Admin Manual for more information about transforms.