If I change an event's sourcetype, can it then be processed as that sourcetype? Also, can an indexer transform forwarded events?

[tomburnell]

It seems that the transformation layer only processes an event once. If the factors that influence which props.conf stanza are applied, this does not cause the event to continue to be processed. Is there any mechanism to make this happen?

The usecase is for a file that has multiple event types inside. I first determine the sourcetype as a series of transforms and then would like to set the properties of that sourcetype in props.conf.

I have tried using a heavy forwarder in front to do the initial sourcetype decision making but the indexer seems not to apply any transformations to forwarded events. Is there a way to make this happen?

Thanks, Tom

[woodcock]

It depends on how/where you change it and what you settings you are trying to invoke (which parser handles them). If you "change" it with rename, which is a search-time operation, then definitely not do anything at all. The Splunk_TA_paloalto TA from SplunkBase take stuff that comes in with sourcetype=pan:logs and breaks it out (successfully) like this:

From props.conf:

[pan:log]
TRANSFORMS-sourcetype = pan_threat, pan_traffic, pan_system, pan_config, pan_hipmatch, pan_endpoint

From transforms.conf:

[pan_threat]
DEST_KEY = MetaData:Sourcetype
REGEX = ^[^,]+,[^,]+,[^,]+,THREAT,
FORMAT = sourcetype::pan:threat

And then later in props.conf for some stuff:

[pan:threat]
TIME_PREFIX = ...
REPORT-foo...
FIELDALIAS-foo ...
EVAL-foo ...
LOOKUP-foo ...

The bottom line, though, is that you are never going to get a second change to go through the TRANSFORMS- parser unless you cook it twice (Index it twice). Once it has been cooked, the only decision to be made is where to write it to disk. Can you transform it with syslog-ng before sending it to Splunk (this is a very common way to handle this kind of thing)? Another possibility might be manipulation with HTTP Event Collector; as I recall, it is very special in how it cooks the data.