How to apply multiple search time patterns to a single sourcetype where the sourcetype has slight variations in the logging pattern?

[markaperdue]

Hi,

I have a sourcetype I am trying to apply some search-time extractions to. The log statements often contain additional fields that I would like to extract and I unfortunately cannot modify the existing pattern.

Some example log lines that show up in this single .log file would look like:

2017-05-24 15:09:11,374 INFO (10.10.173.210:8080-5) (123456789) My log message here
2017-05-24 15:09:12,374 DEBUG (10.10.173.210:8080-4) (987654321) (uuid: abc-123) My log message here
2017-05-24 15:09:13,374 DEBUG (10.10.173.210:8080-4) (485746289) (uuid: foo-123) (gid: foo-bar) My log message here

My props.conf file contains a list of the extract patterns with the most strict pattern at the top. The patterns include some lookaheads to try to cut down on parsing errors when the uuid or gids include parentheses

[jboss_source]
category = Application
pulldown_type = 1
BREAK_ONLY_BEFORE = \d+\-\d+\-\d+\s+\d+:\d+:\d+,\d+
EXTRACT-patt1 = (?P<timestamp>\d+\-\d+\-\d+\s+\d+:\d+:\d+,\d+)\s(?P<log_level>\w+)\s\((?P<thread>.*(?=\)\s\())\)\s\((?P<log_ms>\d+)\)\s\(uuid\:\s(?P<uuid>.*(?=\)\s+\())\)\s+\(gid\:\s(?P<gid>.*(?=\)\s+))\)\s(?P<message>.+)
EXTRACT-patt2 = (?P<timestamp>\d+\-\d+\-\d+\s+\d+:\d+:\d+,\d+)\s(?P<log_level>\w+)\s\((?P<thread>.*(?=\)\s\())\)\s\((?P<log_ms>\d+)\)\s\(uuid\:\s(?P<uuid>.*(?=\)\s))\)\s(?P<message>.+)
EXTRACT-patt3 = (?P<timestamp>\d+\-\d+\-\d+\s+\d+:\d+:\d+,\d+)\s(?P<log_level>\w+)\s\((?P<thread>.*(?=\)\s\())\)\s\((?P<log_ms>\d+)\)\s(?P<message>.+)

I cannot find the documentation where I recall reading it but I seem to recall that for search time extractions the first matching pattern would be applied so I had been using that to have the most exact pattern get applied during searches. Is there a better way to do this since the approach above has been unreliable?

Thanks

[micahkemp]

You could accomplish the key/value extractions with a REPORT- line in props.conf and matching stanza in transforms.conf:

props.conf:

[<sourcetype_name>]
REPORT-keyvalues = keyvalues

transforms.conf:

[keyvalues]
REGEX = \((?<_KEY_1>[^:]+): (?<_VAL_1>[^)]+)\)
MV_ADD = true

The regexes above haven't actually been tested, but should at least show the concepts. Note the _KEY_1 and _VAL_1 capture group names. These are special cases detailed in the transforms.conf documentation.