Index Time Sourcetyping

[tprzelom]

props.conf:
[pan_event]
TRANSFORMS-traffic = traffic_source

transforms.conf:
[traffic_source]
REGEX = (,TRAFFIC,)
FORMAT = sourcetype::pan_traffic
DEST_KEY = MetaData:Sourcetype

I've got data being indexed as sourcetype=pan_event. I have distributed the above props and transforms to the indexers and restarted them, but I am unable to create a pan_traffic sourcetype. Where am I going wrong?

[tprzelom · Accepted solution]

The issue was the light forwarder was being managed by cfengine3 and was transitioned to chef and something was lost in the transition which resulted in partial functionality.

My solution was rip all that out and use a universal forwarder that has a deploymentclient.conf sent to it by chef. Then use the deployment server from there.

[_d_]

You don't need parens, nor do you need to escape the commas: REGEX= TRAFFIC would be fine.

Is the data coming to the indexers with sourcetype=pan_event or is it transformed to be become that? If the latter, you need to scope the transform on source instead. Is it perhaps coming from a heavy forwarder?

[ShaneNewman]

REGEX= \,TRAFFIC\,

[tprzelom]

I'm using
REGEX = ,TRAFFIC,
with no success

There is no space after the comma. This feels like something more sinister that just a regex. I had a much more complex system in place that all of a sudden stopped working, so I dialed it back to basics to try and uncover the problem. These logs are coming through an rsyslog tier, could there be some kind of metadata affecting the logs?

[sdaniels]

Yep, forget those and it seems like you might have a space with the comma before traffic? Not sure though. You could remove those or go with REGEX= ,\sTRAFFIC,

[kristian_kolb]

I think you can skip the parenthesis in the REGEX.

/k

[tprzelom]

Nov 5 15:32:38 hostname.net 1,2013/11/05 15:32:38,0002C100698,TRAFFIC,end,1,2013/11/05 15:32:37,IP_ADDR,IP_ADDR,IP_ADDR,IP_ADDR,firewall_rule,,,incomplete,vsys1,trust,untrust,ethernet1/2,ethernet1/1,anycast_vip_IP_ADDR,2013/11/05 15:32:37,159998,1,47031,3978,47031,3978,0x400000,tcp,allow,74,74,0,1,2013/11/05 15:32:33,0,any,0,12676624,0x0,IP_RANGE,IP_RANGE,0,1,0

hot off the press

[sdaniels]

Can you post a couple of your events. Typically this is regex related because the rest of what you have looks good.