Can I forward all events and selectively index some events?

[Dan]

We are having an issue where we would like to route all events from a specific source to a third-party (ArcSight) but then index only some of these events.

We are doing fine routing to ArcSight, but when we add in "null_alert" below to our props.conf TRANSFORMS-routing line, those events are neither forwarded or indexed.

Here are the configs:

# props.conf
[source::udp:514]
TRANSFORMS-routing=routeArcSight,null_alert

# transforms.conf
[routeArcSight]
REGEX=(.)
DEST_KEY=_SYSLOG_ROUTING
FORMAT=ArcSightGroup

[null_alert]
REGEX = \w+\s+\d+\s+\d+\:\d+\:\d+\s+([^,]+\,){3}THREAT,url,.*,alert,
DEST_KEY = queue
FORMAT = nullQueue

# outputs.conf
[syslog]
indexAndForward=true
defaultGroup=ArcSightGroup

[syslog:ArcSightGroup]
type=UDP
disabled=false
server=10.21.47.123:514

[Dan]

I don't think thats going to work.

If the syslog processor is configured with indexAndForward=true, all events sent to the processor (i.e _SYSLOG_ROUTING=routeArcSight) will necessarily be indexed. I'm not sure how you would conditionally disable that. Setting queue=nullQueue would just prevent the event from being sent to the syslog processor in the first place.

You might try to configure the syslog processor with indexAndForward=false and then for those events that you want indexed also add queue=indexQueue. But I think you would bump up against the same issue - if you use transforms to send an event to an output processor, you can't also send it to another queue.