So the reason for the observed error, only reported by the heavy forwarders, seems clear. The TA is not installed on the heavy forwarders. I might be missing some dependency but I fail to see where this date would fit? The HF complains about the incorrectly genereated dates as being "out of bounds", but the output from the script does not contain "year" or a reasonable "timestamp" (as the year is missing). PRINTF='{printf "%-30s %-30.30s %-s\n", username, from, latest}' if [ "$KERNEL" = "Linux" ] ; then CMD='last -iw' # shellcheck disable=SC2016 FILTER='{if ($0 == "") exit; if ($1 ~ /reboot|shutdown/ || $1 in users) next; users[$1]=1}' # shellcheck disable=SC2016 FORMAT='{username = $1; from = (NF==10) ? $3 : "<console>"; latest = $(NF-6) " " $(NF-5) " " $(NF-4) " " $(NF-3)}' The incorrect years are not included in the indexed events, so it seems that there is little to gain by trying to "fix" this on the HF layer? The set-up is Universal Forwarder (TA nix) --> Heavy Forwarder () --> Indexer () --> Search Head (TA nix). The documentations seems to indicate installing the plugin on HF as "optional" (Install the Splunk Add-on for Unix and Linux - Splunk Documentation)? The intention would not be "data collection" from the HF server and I don't see that this would necessarily fix the issue? If these events from lastlog.sh should not contain years and do not contain a readable timestamp, then should this error not be automatically supressed by the HF? ... View more

Checking the "host" details these complaints are all coming from the heavy forwarders, so I assumed this is where I should be checking. Running the command on a heavy forwarder produced the following output: /opt/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml  While on any search head or indexer the output is as you suggested: /opt/splunk/etc/apps/Splunk_TA_nix/default/props.conf DATETIME_CONFIG = CURRENT There is no '/etc/datetime.xml', however there is an '/opt/splunk/etc/datetime.xml' file. I have no idea of the configuration is reffering to a non-existing file or the built in Splunk one. I don't know if or why anyone would have modified this setting, or the consequences of doing such. I'll do some reasearch on my own but any feedback and/or suggestions are more than welcome ... View more

You know, sometimes you know something but until you really test all options you're just not sure. WIth a "count", like | tstats count WHERE index="<index>" earliest="-5min" latest=now() | `<mail_macro>` | rename count as "Events" There will always be at least one result "0" (zero). It also does not matter if the count is 0 or 99999999, there is exactly 1 result. So the email macro does work, the condition "Number of results = 0" just fails, and it will fail producing false positives with ">=1" as well. I forgot about "custom trigger conditions" though, which is likely the best solution for the intended usecase. | tstats count WHERE index="<index>" earliest="-5min" latest=now() | eval Information = if(count="0", "Currently f-d","Working") | `<mail_macro>` | rename count as "Events" Then using a "custom trigger" like 'search Events = "Currently f-d"' works just as well as the solution outputing only results where there were events last x minutes but are zero events current x minutes. Probably more effective as well. Thank you both for your help, the community here is fantastic ... View more

Sweet I re-wrote the "if 0 events" to "if there were events but now there are none" This way I suppose the alert should be set to if there are "1 events"? If the initial search returns 0 events then there will be 1 generated event with the field "description". Thanx   ... View more

Good to hear your back up and running! It sure does feel like a breaking change/incompatibility between alert manager and Splunk 9.3. Maybe we'll hold off on updating until 9.3.1 🙂 All the best ... View more

So in theory doable but practically ridiculous, gotcha 🙂 ... View more

I just have to know, did a rollback solve the issue or does it persist?  ... View more

Sorry I could not be of more assistance. If file permissions are not the culprit I still think there might be some issue with the python version handling. Just cant figure ut what that would be though, sorry. ... View more

Sorru, I jumped the gun a bit there... This seems to be a problem with the manual execution of house_keeping.py, there is a shebang there which should use the correct version #!/usr/bin/env python3.7 So that is most likely just a problem with the manual execution of the script. Then we're back to checking the "permissions to execute scripts". That may still be the issue 🙂 ... View more

So I think I figured it out. /opt/splunk/bin/splunk cmd python3 house_keeping.py --scheme Traceback (most recent call last): File "/opt/splunk/etc/apps/alert_manager_enterprise/bin/house_keeping.py", line 21, in <module> from datapunctum.factory_logger import Logger File "/opt/splunk/etc/apps/alert_manager_enterprise/bin/../lib/datapunctum/factory_logger.py", line 10, in <module> from pydantic import BaseModel File "/opt/splunk/etc/apps/alert_manager_enterprise/bin/../lib/pydantic/__init__.py", line 372, in __getattr__ module = import_module(module_name, package=package) File "/opt/splunk/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/opt/splunk/etc/apps/alert_manager_enterprise/bin/../lib/pydantic/main.py", line 11, in <module> import pydantic_core File "/opt/splunk/etc/apps/alert_manager_enterprise/bin/../lib/pydantic_core/__init__.py", line 6, in <module> from ._pydantic_core import ( ModuleNotFoundError: No module named 'pydantic_core._pydantic_core' However /opt/splunk/bin/splunk cmd python3.7 house_keeping.py --scheme <scheme><title>AME house keeping</title><description>This task does everything, that has to be done in the background, like mapping users and checking the events time to live.</description><use_external_validation>true</use_external_validation><use_single_instance>false</use_single_instance><streaming_mode>xml</streaming_mode><endpoint><args><arg name="default"><description>Unused Default Argument</description><data_type>string</data_type><required_on_edit>false</required_on_edit><required_on_create>true</required_on_create></arg></args></endpoint></scheme>splunk@gdovm:~/etc/apps/alert_manager_enterprise/bin$ And then there is this: https://docs.splunk.com/Documentation/Splunk/9.3.0/ReleaseNotes/MeetSplunk n this release, the default Python interpreter is set to Python version 3.9. The Python.Version settings has been updated so that the parameter is set to value of force_python3, this forces all Python extension points to use Python 3.9 including overriding any application specified settings. "This is designed to be secure-by-default for new customers. If the value is set to python3.9, the default interpreter is set to Python 3.9 but applications can choose to use a different value. Python 3.7 continues to be available in the build for customers' private apps." So the problem is a change in the Python version handling when upgrading to Splunk 9.3.0 Either you can try to figure out how to force the app to revert to python 3.7 or you can get a hold of the developers and inform them the app may not be working with 9.3.0 anymore. Sorry, not fun to have to be the bearer of bad news ... View more

Hold on a second, there is no permission to execute any of the python scripts, right? Might be something I'm missing but I suspect that might be a problem at least: -rwxrw-rw- 1 splunk splunk 5,7K jun 28 11:32 alertqueue_consumer.py -rwxrw-rw- 1 splunk splunk 11K jun 28 11:32 command_ameenrich.py -rwxrw-rw- 1 splunk splunk 4,6K jun 28 11:32 command_ameevents.py -rwxrw-rw- 1 splunk splunk 12K jun 28 11:32 create_alert.py -rwxrw-rw- 1 splunk splunk 1,2K jun 28 11:32 _env.py -rwxrw-rw- 1 splunk splunk 5,9K jun 28 11:32 handler_license.py -rwxrw-rw- 1 splunk splunk 4,4K jun 28 11:32 handler_logging.py -rwxrw-rw- 1 splunk splunk 3,2K jun 28 11:32 handler_minit.py -rwxrw-rw- 1 splunk splunk 3,4K jun 28 11:32 handler_proxy.py -rwxrw-rw- 1 splunk splunk 441 jun 28 11:32 handler.py -rwxrw-rw- 1 splunk splunk 3,5K jun 28 11:32 handler_role_utils.py -rwxrw-rw- 1 splunk splunk 5,4K jun 28 11:32 house_keeping.py -rwxrw-rw- 1 splunk splunk 273 jun 28 11:32 __init__.py -rwxrw-rw- 1 splunk splunk 3,8K jun 28 11:32 notificationqueue_consumer.py drwxrwxrwx 2 splunk splunk 4,0K jun 28 11:32 persistconn -rwxrw-rw- 1 splunk splunk 3,6K jun 28 11:32 tag_keeping.py  So maybe a chmod u+x may solve your problems? ... View more

OK, assuming you are running Splunk as root that seems to check out. Splunk python cannot itself import pydantic, though it seems to be bundled with the app so maybe that is just how the command is executed (which is pretty bad, but still). I see a bunch of "from pathlib import Path" so I'm assuming this is to import from localy bundled versions. Not sure what the issue is then, I'll see if I can recreate the error on a local box   ... View more

Hm, so it seems that pydantic (or at least pydantic_core) is not available for import. Not sure that house_keeping.py suffers the same problems as tag_keeping.py, but it is a problem for sure. Did the file permissions look OK? ... View more

This solution still works for most cases, however if you need an alert where number of events is 0 then this solution will not work, not "as is" at least. As a search for, lets say a problem with logshipping, should alert on 0 returned events. Then there is no way to hitch on a hidden field to anything as there are no results. So to fill "all my needs" here, I would have to come up with something completely different, it would need to become a feature in Splunk or I have to sort out an manage a number of group recipients in exchange. ... View more

That is a really great explanation, thank you! In other words, there would be little to no gain by using my suggested base search as it it would retain a lot of excess data from entire events. What I could, in theory, do would be to run a basesearch keeping only the 3-4 fields all subsequent panels would use. This would however put a strain on the SH cluster. So for any real gain here, I would need to rewrite all panels that could use an effective base search to work with something like calculated daily averages and process these for each panel. However, and I'm sorry for being a stickler, this does not really answer the question regarding using a base search with subserches. I can run the base search and have a panel use that base with a query. But can you reference a base search withing the query using the base search? The example below is pretty crappy but hopefully a bit clearer then in my initial post? <search id="base_search"> <query> index="_internal" | stats count by <something> </query> </search> ... ... <search base="base_search"> <query> search <field>=<value> | join type=outer _time [ <search referencing the same base_search> | stats count something ] </query> </search> In a search which uses a base search (an effective one 😉 ) can I reference the same (or another ) base search inside a "subsearch"/"nested search"? ... View more

The seems to be a python script "alert_manager_enterprise/bin/tag_keeping.py" which fails to run/execute causing the error. The script offers a way to test output through: /opt/splunk/bin/splunk cmd python3 house_keeping.py --scheme I don't use this app hence this is more "in general" and "best guess", initially I would check that the script has the correct permissions set to be run. Then (assuming it is safe to run the "test command" above) see if you can manually execute the command. It's possible that the update caused some problem with permissions for script execution, or (have not checked) there was an update to the python version which now is incompatible with the script bundled with the alert manager app (probably less likely though).   ... View more

Just wanted to add the "final touch" which made the solution work as intended: Solved: Re: Defining a global token for alert recipients - Splunk Community ... View more

Sweet relief after so much trial and error, I could kiss you! Yes, this solution finally works! savedsearches.conf <basesearch> | table <something> | `macro`  macro.conf [macro] definition = eval _recipients="email1@email.com, email2@email.com" and finally in the savedsearches.conf (or To: field in the UI) action.email.to = $result._recipients$ And it finnaly works as intended!!! 👏 Whish I could reward 100 karma for this 😄 Still think that this should be a "built in" thing available both in the GUI and config files, "email groups", but I'm to happy to care right now ... View more

👍Which also explains why the "z" version was not available for download 🙂 Well, then all is well that ends well I suppose! Thank you for both help and clarification, much appreciated! ... View more

Yes and no. I guess my question could have been a bit more concrete and clear. It does list the details regarding premium services ([ Contracts ] - /support/contracts.html (openssl.org)) inkluding LTS for 1.0.2. But it does not list any premium customers. I'm struggling to validate that the LTS versions is what is shipped with Splunk. I have not found  information documenting this as a fact by Splunk representatives yet. If you can point me to the documentation regaring Splunk being a premium customer I'd appreciate it very much. Then I have something to lean on while ignoring the alerts 🙂 ... View more