Knowledge Management

Tag definition not immediately applied

FRoth
Contributor

I noticed that a tag definition doesn't get immediately applied but takes up 1 min to become active.

E.g. I define a tag false_positive on the search head (we have a single indexer):
MD5=4bc41dc57d4ababc2810b9905b91ac2f

Then I run a search and don't see the messages tagged false_positive. I run another search - still no tagging.
After a minute or so I run the same search and see the events tagged.

Why is that?

The same strange behaviour applies to cases in which I delete a tag definition. The search still returns the tagged log lines and it takes around a minute until the tag disappears.

rafaelsalazar
Path Finder

I've noticed pretty much the same behavior but my deployment is a production deployment with clusters and 5+ indexers and massive amount of data. 1 minute for you is sometimes 20 minutes for me.

I don't know the technical specification for this, but when Splunk says "Eventtypes and tags run at search time" it refers that when you run your search request it will look for the rules that apply for your particular search and then perform them. My primary suspect is that splunk uses a more static than dynamic way to store this rules so that they are available as soon as anyone needs them, and the time it takes to update them based on changes to the splunk UI are related to the availability of both the cached set of rules to apply to searches and the memory/cpu resources in the deployment.

So let me explain why I think this, because if the job manager is running constantly and overloading the machines and using constantly the rules, it would be hard to splunk to say "okay, now is the right time to alter the rules without impacting other Jobs.

I recall a time when I updated a lookup by removing the old one and uploading the new one.. and the users reported 15~ minutes later that it wasn't finding the lookup, but it was there, and permissions were correctly assigned, just the system didn't updated itself with the new lookup reference during that time.

So that's my educated opinion on this, maybe if I get to ask an splunk technician from Splunk I would definitely ask this kind of questions on how they manage internally the availability of the search time rules.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Kick the Tires Before You Commit: A Hands-On Tour of the Splunk Observability Cloud ...

Evaluating an enterprise observability platform usually goes like this: fill out a form, get a free trial with ...

Deep insights, no barriers: Splunk Observability Cloud Free Edition

As software delivery cycles continue to accelerate, observability shouldn’t be a luxury — it should be a ...

Monitoring AI Agents with Splunk Observability Cloud

Let’s say I’m running a travel planning AI app in production. A user asks for three concise hotel options in ...