How do I configure my sourcetype to deal with a log that creates events with fixed field lengths?

[andrewtrobec]

Hello all,

I have a structured log that doesn't contain a headers but contains fields with a fixed lengths. Here is a simplified example that considers 4 fields with names exit_code, id, description, and timestamp:

3 40023452 This is the description that gets trunca 2019-31-10 11:01:00
3 40016468 Shorter description no truncation        2019-31-10 11:02:00
3 40045418                                          2019-31-10 11:03:00

The samples above shows that all fields are fixed length even when they don't have values:

• exit_code is always 1 character from columns 1 to 2
• id is always 8 characters from columns 3 to 11
• description is always 40 characters from columns 12 to 52
• timestamp is always 19 characters from columns 53 to 72

With this in mind, is there any way in props.conf to specify the column ranges that define the fields? If not, is there a combination of props.conf and transforms.conf that will allow me to do this, maybe using regular expressions?

Any suggestions would be greatly appreciated!

Thank you and best regards,

Andrew

[richgalloway]

This regex works with your sample data. Test it in search to verify it works.

(?<exit_code>\d+)\s+(?<id>\d+)\s(?<description>.{40})\s(?<timestamp>.*)

Put it into your transforms.conf like this:

[foo]
REGEX = (\d+)\s+(\d+)\s(.{40})\s(*)
FORMAT = exit_code::$1 id::$2 description::$3 timestamp::$4

[andrewtrobec]

@richgalloway Thanks Rich! Follow-up questions: must the stanza in trasforms.conf have the same name as the stanza in props.conf? Also. must I update props.conf with some sort of parameter so that it refers to transforms.conf?

[richgalloway]

The props and transforms stanza names do not have to match and usually don't.
Reference the transform from props like this:

[mysourcetype]
TRANSFORMS-sometext = foo