Splunk Search

Regex to match two fields in transforms.conf

danbrook
Explorer

I'm looking to match against two fields in transforms.conf. I would like to match against a customer _meta field and the source field then route to a specific index based on that. There is a good reason for me not matching on inputs.conf that I won't go into here.

I would like to match

regex=MyCustomField::somestring AND regex=source::syslog

I'd also like to be able to test this in search before committing it to transforms.conf

0 Karma
1 Solution

DalJeanis
SplunkTrust
SplunkTrust

Okay, you have to make sure that your extraction transforms are happening in the right order. This can't really be tested in "search", so I recommend you set yourself up a sandbox instance.

There are lots of good answers on overriding metadata and sending to a new index. But a workable final architecture is going to depend on knowing your data and your existing transforms.

1) Is MyCustomField::somestring going to occur in any source other than syslog?
2) When does MyCustomField get extracted?

Your new transform will need to be called and evaluated AFTER MyCustomField exists, but only for events in source=syslog.

If you can make that happen, then the stanza would look something like this...

  [index_reset_for_MyCustomField]
  SOURCE_KEY = MyCustomField
  DEST_KEY =  _MetaData:index
  REGEX = somestring
  FORMAT = mynewindexname

If you CANNOT determine an order where that would happen, then you might have to do place a ricochet shot. You can't really PROGRAM in a stanza, or concatenate two fields.

View solution in original post

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Okay, you have to make sure that your extraction transforms are happening in the right order. This can't really be tested in "search", so I recommend you set yourself up a sandbox instance.

There are lots of good answers on overriding metadata and sending to a new index. But a workable final architecture is going to depend on knowing your data and your existing transforms.

1) Is MyCustomField::somestring going to occur in any source other than syslog?
2) When does MyCustomField get extracted?

Your new transform will need to be called and evaluated AFTER MyCustomField exists, but only for events in source=syslog.

If you can make that happen, then the stanza would look something like this...

  [index_reset_for_MyCustomField]
  SOURCE_KEY = MyCustomField
  DEST_KEY =  _MetaData:index
  REGEX = somestring
  FORMAT = mynewindexname

If you CANNOT determine an order where that would happen, then you might have to do place a ricochet shot. You can't really PROGRAM in a stanza, or concatenate two fields.

0 Karma

danbrook
Explorer

Ah ha! Yes I add the MyCustomField on the input from a Universal Forwarder. I want to set the index name based on the value in MyCustomField so maybe I can match in props.conf for the source and then pull out the value for MyCustomField to build the Index name.

props.conf

[source::syslog]
TRANSFORMS-Index-Syslog = Set-Index-Syslog

transforms.conf

[Set-Index-Syslog]    
SOURCE_KEY = MetaData:MyCustomField    
REGEX = (.*)
DEST_KEY = _MetaData:Index
FORMAT = index-$1-Syslog
0 Karma

DalJeanis
SplunkTrust
SplunkTrust

The two requirements for that to work are (1) MyCustomField must be extracted before the [source::syslog] stanza is reached, and (2) the value of MyCustomField must have been set to whatever you want sandwiched into the index name.

I would probably use

 REGEX = (.+)

since the other would match an empty (but not a null) field.

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

I assume "now matching" was supposed to be "not matching"?

0 Karma

danbrook
Explorer

Yes, edited.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...