Tag definition not immediately applied

[FRoth]

I noticed that a tag definition doesn't get immediately applied but takes up 1 min to become active.

E.g. I define a tag false_positive on the search head (we have a single indexer):
MD5=4bc41dc57d4ababc2810b9905b91ac2f

Then I run a search and don't see the messages tagged false_positive. I run another search - still no tagging.
After a minute or so I run the same search and see the events tagged.

Why is that?

The same strange behaviour applies to cases in which I delete a tag definition. The search still returns the tagged log lines and it takes around a minute until the tag disappears.

[rafaelsalazar]

I've noticed pretty much the same behavior but my deployment is a production deployment with clusters and 5+ indexers and massive amount of data. 1 minute for you is sometimes 20 minutes for me.

I don't know the technical specification for this, but when Splunk says "Eventtypes and tags run at search time" it refers that when you run your search request it will look for the rules that apply for your particular search and then perform them. My primary suspect is that splunk uses a more static than dynamic way to store this rules so that they are available as soon as anyone needs them, and the time it takes to update them based on changes to the splunk UI are related to the availability of both the cached set of rules to apply to searches and the memory/cpu resources in the deployment.

So let me explain why I think this, because if the job manager is running constantly and overloading the machines and using constantly the rules, it would be hard to splunk to say "okay, now is the right time to alter the rules without impacting other Jobs.

I recall a time when I updated a lookup by removing the old one and uploading the new one.. and the users reported 15~ minutes later that it wasn't finding the lookup, but it was there, and permissions were correctly assigned, just the system didn't updated itself with the new lookup reference during that time.

So that's my educated opinion on this, maybe if I get to ask an splunk technician from Splunk I would definitely ask this kind of questions on how they manage internally the availability of the search time rules.