topic Rule based source typing in Getting Data In

Rule based source typing

FaceF18 — Mon, 28 Sep 2020 15:43:16 GMT

I'm trying to set the sourcetype on some events I get based on their contents, and then I want to send each of those differentiated sourcetypes to their own indexes. I've tried a bunch of different ways, and none of my approaches seem to work quite like the docs say they should.

So, for starters, source typeing. I feel like what I'm trying to do is simple. If the string FlightEvent occurs anywhere in the event, it should be a FlightEvent. Flight and Event are actually separate xml opening tags, but I can't seem to get less-than and greater-than symbols to display in markdown. I don't know if that has any impact in props or transforms.conf.

In props.conf

[FlightEvent]
TRANSFORMS-flighteventtrans = flighteventformat

In transforms.conf

[flighteventformat]

REGEX = FlightEvent

LOOKAHEAD = 16

DEST_KEY = MetaData:Sourcetype

FORMAT = sourcetype:FlightEvent

No good.

I tried setting up rule based source typing.

In props.conf

[rule::flighteventrule]

sourcetype=FlightEvent

MORE_THAN_1 = FlightEvent

No good. I also can't get sourcetypes to go to the correct indexes, or actually any index other than main, but I guess I'll try to deal with that when I get source typing figured out.

Re: Rule based source typing

lguinn2 — Sat, 25 Jan 2014 07:00:55 GMT

The symbols < and > are special characters in regular expressions. Also, your props.conf has a problem - you are asking it to process events of sourcetype FlightEvent - before you have assigned the sourcetype of FlightEvent!!

I suggest this in your props.conf

[source::yourinputsourcefilehere]
TRANSFORMS-flighteventtrans = flighteventformat

In transforms.conf

[flighteventformat]
REGEX = \<Flight\>\<Event\>
LOOKAHEAD = 16
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype:FlightEvent

Skip the rule-based sourcetyping, you don't need it. Also the REGEX above may be wrong, because I don't entirely understand the format of your events. If you could post a sample of the raw data (anonymized of course), that would be really helpful. If you paste it into the question, you can highlight it and then click on the 101 icon to keep the markdown from messing with it.

Re: Rule based source typing

FaceF18 — Mon, 27 Jan 2014 18:30:59 GMT

I get flight events from a number of different sources and hosts, so I'm not sure how to write the first statement so that it applies in this case. I tried both source and host * and .* with the hope that this would apply that transform to every event, and then I edited the regex to escape any special characters, and I still didn't have any luck.

Also, I can't edit my post right now because every captcha is coming back invalid, but basically I want any event that has this substring "<Flight><Event>" in it anywhere to be set as a FlightEvent. Thanks for the help.