Can you sanity check my transforms.conf config?

[daniel333]

All,

Can I have a quick sanity check on this transforms.conf? Basically I want to keep any log which has fatal, critical and error and drop the rest. THEN turn around and drop some of those logs which are "known".

# transforms.conf

[XYZ_keep]
  REGEX=(?!error)|(?!fatal)|(?!crit)|(?!splunk)
  DEST_KEY=queue
  FORMAT=nullQueue

[XYZ_nullqueue]
  REGEX=priority=INFO app_name=SolrCloud|location=MissingListingAttributeList|JmxHealthCheck
  DEST_KEY=queue
  FORMAT=nullQueue

[woodcock]

This will NOT work, it will send everything to nullQueue.

[woodcock · Accepted solution]

You need something like this in props.conf:

TRANSFORMS-this_string_does_not_matter = XYZ_null_all, XYZ_keep, XYZ_null_some

Then in transforms.conf:

[XYZ_null_all]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[XYZ_keep]
REGEX=(?!error)|(?!fatal)|(?!crit)|(?!splunk)
DEST_KEY=queue
FORMAT=indexQueue

[XYZ_null_some]
REGEX=priority=INFO app_name=SolrCloud|location=MissingListingAttributeList|JmxHealthCheck
DEST_KEY=queue
FORMAT=nullQueue

[richgalloway]

I recommend avoiding the double-negative logic in XYZ_keep to prevent mis-understandings by future maintainers. I added the case-insensitive flag to allow for "Error", "ERROR", and "error".

[XYZ_keep]
   REGEX=(?i)(error)|(fatal)|(crit)|(splunk)
   DEST_KEY=queue
   FORMAT=indexQueue