Getting Data In

Transforming events twice

OL
Communicator

Hello all,

Would anyone know if there is a way to apply a transform twice on two different sourcetype. Explanation: events are received with sourcetype syslog. I use a "transforms" on sourcetype syslog to change the sourcetype of some events to firewall_syslog. I then would like to change the index of the events with sourcetype firewall_syslog to index=firewall. The first transform is being applied, but not the second one. Here is how I do it:

props.conf

[syslog]
TRANSFORMS-01 = sourcetype2fw

[firewall_syslog]
TRANSFORMS-02 = index2fw

transforms.conf

[sourcetype2fw]
REGEX = 10.12.14.*
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::firewall_syslog

[index2fw]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = index::firewall

I know that I could use the second transform in the syslog sourcetype and this is working, but this would imply that we need to run the same REGEX twice (once for the change of sourcetype and once for the change of index). The way I thought and which doesn't seem to work would run the REGEX only once (on ethe sourcetype) when the second REGEX would accept everything.

Any idea?

Regards,
Olivier

Tags (1)
0 Karma

bmacias84
Champion

Question are you picking your syslog data from a flat file or UDP stream? Is every syslog entry being sent to one syslog?

0 Karma

starcher
Influencer

Why would you do it this way when you can just assign the correct sourcetype and index to the input source?

0 Karma

starcher
Influencer

You must be pulling all syslog into a network input on your indexer. Best practice is to bring syslog to a receiver like a syslog-ng box. Use the universal forwarder to pick up. If you have the logs write to a folder by host you can assign pickup by each folder specifying index and sourcetype. This also means when you add indexers later you can leverage autoLB (load balance) on the forwarder.

0 Karma

OL
Communicator

The reason for doing it is that syslog data are not only one type: firewall, routers, linux servers and so on. So, in this particular case, you cannot set the sourcetype and index at the input.

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...

Beyond Detection: How Splunk and Cisco Integrated Security Platforms Transform ...

Financial services organizations face an impossible equation: maintain 99.9% uptime for mission-critical ...

Customer success is front and center at .conf25

Hi Splunkers, If you are not able to be at .conf25 in person, you can still learn about all the latest news ...