topic Re: Transforming events twice in Getting Data In

Transforming events twice

OL — Mon, 28 Sep 2020 14:34:47 GMT

Hello all,

Would anyone know if there is a way to apply a transform twice on two different sourcetype. Explanation: events are received with sourcetype syslog. I use a "transforms" on sourcetype syslog to change the sourcetype of some events to firewall_syslog. I then would like to change the index of the events with sourcetype firewall_syslog to index=firewall. The first transform is being applied, but not the second one. Here is how I do it:

props.conf

[syslog]
TRANSFORMS-01 = sourcetype2fw

[firewall_syslog]
TRANSFORMS-02 = index2fw

transforms.conf

[sourcetype2fw]
REGEX = 10.12.14.*
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::firewall_syslog

[index2fw]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = index::firewall

I know that I could use the second transform in the syslog sourcetype and this is working, but this would imply that we need to run the same REGEX twice (once for the change of sourcetype and once for the change of index). The way I thought and which doesn't seem to work would run the REGEX only once (on ethe sourcetype) when the second REGEX would accept everything.

Any idea?

Regards,
Olivier

Re: Transforming events twice

starcher — Thu, 15 Aug 2013 20:25:50 GMT

Why would you do it this way when you can just assign the correct sourcetype and index to the input source?

Re: Transforming events twice

bmacias84 — Thu, 15 Aug 2013 21:02:59 GMT

Question are you picking your syslog data from a flat file or UDP stream? Is every syslog entry being sent to one syslog?

Re: Transforming events twice

OL — Thu, 15 Aug 2013 21:18:09 GMT

The reason for doing it is that syslog data are not only one type: firewall, routers, linux servers and so on. So, in this particular case, you cannot set the sourcetype and index at the input.

Re: Transforming events twice

starcher — Fri, 16 Aug 2013 10:55:37 GMT

You must be pulling all syslog into a network input on your indexer. Best practice is to bring syslog to a receiver like a syslog-ng box. Use the universal forwarder to pick up. If you have the logs write to a folder by host you can assign pickup by each folder specifying index and sourcetype. This also means when you add indexers later you can leverage autoLB (load balance) on the forwarder.