I have created a few very straight-forward eventtype (ET) definitions.
I noticed that when I use a join that combines the events from both eventtypes, that it marks the resulting event with the eventtype of the subsearch.
index=myindex sourcetype=myst1 <search terms> | join c_id [search index=myindex sourcetype=myst2 <search terms>]
All events get marked as eventtype=ET2
I believe this is because the sourcetype that gets attached to the resulting events is that of the subsearch. Based on this behavior I created a workflow that uses my ET2.
When I installed the *NIX app on my searchhead, a few of the global eventtypes started getting applied to my results. This also caused the original behavior, where eventtype was set to ET2, to change. Now I have the eventtype=ET1, nix-all-logs and nix_errors.
This caused my workflow to break, since the eventtype was now ET1 rather than ET2, and the workflow was defined for ET2. On close inspection the events were still being marked as having sourcetype=myst2, which would imply that the eventtype should have been ET2 as they were before the *NIX install.
To solve this I simply disabled the two *nix global eventtypes (nix-all-logs and nix_errors), and now the eventtype identification works the same as before, and my workflow works correctly.
This inconsistent behavior seems to indicate a bug. Has anyone been able to figure out a pattern to the behavior?
Not sure why precedence would have an effect on the evaluated eventtype for a result.
If the eventtype comes out as ET2 for my result, but then I add two global eventtypes on a different app, the exact same result should still have ET2 + the two new global eventtypes. But what I'm seeing is that the behavior changes to now show eventtype ET1.
Also the name of my app starts with a capital D, which would imply that it would take precedence over the unix app. So I don't think it's precedence causing the problem.
It is probably not the eventtype, but the app that contains the eventtypes.conf.
Splunk uses the ASCII sort order for evaluating things of the same context, and in this case I believe it is the app.
This sort order goes like this: A>Z>a>z, the A is used before the Z. The unix app starts with "u", which means it will take precedence if your eventtype is in an app that starts with v-z.
Also, search time is a user context, so the apps directory will take precedence over the system directory. And a local app folder will take precedence over a default folder.