I would like filter certain known data events into three different indexes (possibly more in the future).
Events have an embedded field called "AppName". The possible values happen to be the names of my indexes.
Is this the right approach? Does it look correct?
in transforms.conf
[RedirectToAppIndex]
REGEX = .*AppName="(?i)(app1|app2|app3)"
DEST_KEY = _MetaData:Index
FORMAT = $1
This looks ok.
make sure that the indexes app1 app2 and app3 exists with the proper case.
This looks ok.
make sure that the indexes app1 app2 and app3 exists with the proper case.
yes, each event has to be compared to the regex and routed, it adds load to the indexing.
By that you just mean...index time transforms take time and resources...add complexity, etc. Right?
But the best solution is still to specify the index at the input level.
If you have distinct inputs per sources of course.
Thanks for calling out that they need to exist. This transform is actually part of my strategy for dealing with indexes that logically should exist but haven't yet been created for whatever reason.
All my events from a group of machines have an initial default index defined in the index meta data field (think catchall). The log events can include a preference for destination index by specifying AppName, but if it doesn't exist the log events fall back on the default index.
My goal on the matching side was to have the "(?i)" make the regex case-insensitive.
On the index name side, my experience has suggested that the ingestion does not care about case (all my indexes show up in the manager as lower case and events coming in as App1 or aPP1 all make it into app1).
Does that not line up with your experience/understanding?