topic Re: Index Time Sourcetyping in Getting Data In

Index Time Sourcetyping

tprzelom — Mon, 28 Sep 2020 15:12:53 GMT

props.conf:
[pan_event]
TRANSFORMS-traffic = traffic_source

transforms.conf:
[traffic_source]
REGEX = (,TRAFFIC,)
FORMAT = sourcetype::pan_traffic
DEST_KEY = MetaData:Sourcetype

I've got data being indexed as sourcetype=pan_event. I have distributed the above props and transforms to the indexers and restarted them, but I am unable to create a pan_traffic sourcetype. Where am I going wrong?

Re: Index Time Sourcetyping

sdaniels — Tue, 05 Nov 2013 23:31:05 GMT

Can you post a couple of your events. Typically this is regex related because the rest of what you have looks good.

Re: Index Time Sourcetyping

tprzelom — Mon, 28 Sep 2020 15:12:56 GMT

Nov 5 15:32:38 hostname.net 1,2013/11/05 15:32:38,0002C100698,TRAFFIC,end,1,2013/11/05 15:32:37,IP_ADDR,IP_ADDR,IP_ADDR,IP_ADDR,firewall_rule,,,incomplete,vsys1,trust,untrust,ethernet1/2,ethernet1/1,anycast_vip_IP_ADDR,2013/11/05 15:32:37,159998,1,47031,3978,47031,3978,0x400000,tcp,allow,74,74,0,1,2013/11/05 15:32:33,0,any,0,12676624,0x0,IP_RANGE,IP_RANGE,0,1,0

hot off the press

Re: Index Time Sourcetyping

kristian_kolb — Wed, 06 Nov 2013 00:58:09 GMT

I think you can skip the parenthesis in the REGEX.

Re: Index Time Sourcetyping

sdaniels — Wed, 06 Nov 2013 14:08:53 GMT

Yep, forget those and it seems like you might have a space with the comma before traffic? Not sure though. You could remove those or go with REGEX= ,\sTRAFFIC,

Re: Index Time Sourcetyping

tprzelom — Wed, 06 Nov 2013 16:09:51 GMT

I'm using
REGEX = ,TRAFFIC,
with no success

There is no space after the comma. This feels like something more sinister that just a regex. I had a much more complex system in place that all of a sudden stopped working, so I dialed it back to basics to try and uncover the problem. These logs are coming through an rsyslog tier, could there be some kind of metadata affecting the logs?

Re: Index Time Sourcetyping

ShaneNewman — Fri, 08 Nov 2013 03:19:52 GMT

REGEX= \,TRAFFIC\,

Re: Index Time Sourcetyping

_d_ — Fri, 08 Nov 2013 03:58:01 GMT

You don't need parens, nor do you need to escape the commas: REGEX= TRAFFIC would be fine.

Is the data coming to the indexers with sourcetype=pan_event or is it transformed to be become that? If the latter, you need to scope the transform on source instead. Is it perhaps coming from a heavy forwarder?

Re: Index Time Sourcetyping

tprzelom — Fri, 15 Nov 2013 17:02:09 GMT

The issue was the light forwarder was being managed by cfengine3 and was transitioned to chef and something was lost in the transition which resulted in partial functionality.

My solution was rip all that out and use a universal forwarder that has a deploymentclient.conf sent to it by chef. Then use the deployment server from there.