Regex to match two fields in transforms.conf

[danbrook]

I'm looking to match against two fields in transforms.conf. I would like to match against a customer _meta field and the source field then route to a specific index based on that. There is a good reason for me not matching on inputs.conf that I won't go into here.

I would like to match

regex=MyCustomField::somestring AND regex=source::syslog

I'd also like to be able to test this in search before committing it to transforms.conf

[DalJeanis · Accepted solution]

Okay, you have to make sure that your extraction transforms are happening in the right order. This can't really be tested in "search", so I recommend you set yourself up a sandbox instance.

There are lots of good answers on overriding metadata and sending to a new index. But a workable final architecture is going to depend on knowing your data and your existing transforms.

1) Is MyCustomField::somestring going to occur in any source other than syslog?
2) When does MyCustomField get extracted?

Your new transform will need to be called and evaluated AFTER MyCustomField exists, but only for events in source=syslog.

If you can make that happen, then the stanza would look something like this...

  [index_reset_for_MyCustomField]
  SOURCE_KEY = MyCustomField
  DEST_KEY =  _MetaData:index
  REGEX = somestring
  FORMAT = mynewindexname

If you CANNOT determine an order where that would happen, then you might have to do place a ricochet shot. You can't really PROGRAM in a stanza, or concatenate two fields.

[danbrook]

Ah ha! Yes I add the MyCustomField on the input from a Universal Forwarder. I want to set the index name based on the value in MyCustomField so maybe I can match in props.conf for the source and then pull out the value for MyCustomField to build the Index name.

props.conf

[source::syslog]
TRANSFORMS-Index-Syslog = Set-Index-Syslog

transforms.conf

[Set-Index-Syslog]    
SOURCE_KEY = MetaData:MyCustomField    
REGEX = (.*)
DEST_KEY = _MetaData:Index
FORMAT = index-$1-Syslog

[DalJeanis]

The two requirements for that to work are (1) MyCustomField must be extracted before the [source::syslog] stanza is reached, and (2) the value of MyCustomField must have been set to whatever you want sandwiched into the index name.

I would probably use

 REGEX = (.+)

since the other would match an empty (but not a null) field.

[DalJeanis]

I assume "now matching" was supposed to be "not matching"?

[danbrook]

Yes, edited.