"There's an app for that" 😄 Thank you! ... View more

I feel content accepting that bundle checksums are by design present to visualise the different states and not to detect modifications between validation and application. Thank you for the link though, I will definetly have a look at the presentation. With a little luck it might even be available online somewhere. All the best ... View more

While I still find this "counter intuitive" speaking of checksums, I understand that this is by design. The validated bundle is modified upon application and hence the checksum changes. This is not to prevent pushing bundles that have been modified after validation but rather to keep track of states of changes/bundles. Thank you for the explanation ... View more

Thank you @sainag_splunk  I do understand the "how" so to speak, but I do not understand the "why". If I validate a bundle and then push that bundle I would assume that the checksum for the validated bundle after a push should be the same as the new and current one. Otherwise I cannot validate that the last validated bundle is exactly what has been pushed and applied. But maybe I'm missing the point? If the checksums are not there to validate changes but rather just to track the presence of unapplied changes, then its fine. I suspect that at this point the logical follow-up question is, how are bundles modified after validation but before being applied? Or are the checksums only for the changed files/settings and the final cheksum is for the entire bundle? All the best   ... View more

So it is now flagged to include "z" versions of OpenSSL, meaning that all prior and current versions were and are indeed affected? Could you provide a link to the supporting information? ... View more

Sorry for the delay Exporting the scan results did provide additional information, as with most other apps the problem is with "backups" of older versions of the app ".../default.old.20240828…i/views/attribution.xml" So URA is triggering on "old" folders which are no longer active. The remaining questions hence is "to delete or not to delete"? I know I've participated in these discussions before. For "private" apps I could normally just ignore a specific search path for an app, this is not possible for the "splunk base app". So either I have to ingore the "failing" (false positives) apps completely, or manually delete "old" folders. What is the "best praxis" here? ... View more

So the reason for the observed error, only reported by the heavy forwarders, seems clear. The TA is not installed on the heavy forwarders. I might be missing some dependency but I fail to see where this date would fit? The HF complains about the incorrectly genereated dates as being "out of bounds", but the output from the script does not contain "year" or a reasonable "timestamp" (as the year is missing). PRINTF='{printf "%-30s %-30.30s %-s\n", username, from, latest}' if [ "$KERNEL" = "Linux" ] ; then CMD='last -iw' # shellcheck disable=SC2016 FILTER='{if ($0 == "") exit; if ($1 ~ /reboot|shutdown/ || $1 in users) next; users[$1]=1}' # shellcheck disable=SC2016 FORMAT='{username = $1; from = (NF==10) ? $3 : "<console>"; latest = $(NF-6) " " $(NF-5) " " $(NF-4) " " $(NF-3)}' The incorrect years are not included in the indexed events, so it seems that there is little to gain by trying to "fix" this on the HF layer? The set-up is Universal Forwarder (TA nix) --> Heavy Forwarder () --> Indexer () --> Search Head (TA nix). The documentations seems to indicate installing the plugin on HF as "optional" (Install the Splunk Add-on for Unix and Linux - Splunk Documentation)? The intention would not be "data collection" from the HF server and I don't see that this would necessarily fix the issue? If these events from lastlog.sh should not contain years and do not contain a readable timestamp, then should this error not be automatically supressed by the HF? ... View more

Checking the "host" details these complaints are all coming from the heavy forwarders, so I assumed this is where I should be checking. Running the command on a heavy forwarder produced the following output: /opt/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml  While on any search head or indexer the output is as you suggested: /opt/splunk/etc/apps/Splunk_TA_nix/default/props.conf DATETIME_CONFIG = CURRENT There is no '/etc/datetime.xml', however there is an '/opt/splunk/etc/datetime.xml' file. I have no idea of the configuration is reffering to a non-existing file or the built in Splunk one. I don't know if or why anyone would have modified this setting, or the consequences of doing such. I'll do some reasearch on my own but any feedback and/or suggestions are more than welcome ... View more

You know, sometimes you know something but until you really test all options you're just not sure. WIth a "count", like | tstats count WHERE index="<index>" earliest="-5min" latest=now() | `<mail_macro>` | rename count as "Events" There will always be at least one result "0" (zero). It also does not matter if the count is 0 or 99999999, there is exactly 1 result. So the email macro does work, the condition "Number of results = 0" just fails, and it will fail producing false positives with ">=1" as well. I forgot about "custom trigger conditions" though, which is likely the best solution for the intended usecase. | tstats count WHERE index="<index>" earliest="-5min" latest=now() | eval Information = if(count="0", "Currently f-d","Working") | `<mail_macro>` | rename count as "Events" Then using a "custom trigger" like 'search Events = "Currently f-d"' works just as well as the solution outputing only results where there were events last x minutes but are zero events current x minutes. Probably more effective as well. Thank you both for your help, the community here is fantastic ... View more

Sweet I re-wrote the "if 0 events" to "if there were events but now there are none" This way I suppose the alert should be set to if there are "1 events"? If the initial search returns 0 events then there will be 1 generated event with the field "description". Thanx   ... View more

Good to hear your back up and running! It sure does feel like a breaking change/incompatibility between alert manager and Splunk 9.3. Maybe we'll hold off on updating until 9.3.1 🙂 All the best ... View more

So in theory doable but practically ridiculous, gotcha 🙂 ... View more

I just have to know, did a rollback solve the issue or does it persist?  ... View more

Sorry I could not be of more assistance. If file permissions are not the culprit I still think there might be some issue with the python version handling. Just cant figure ut what that would be though, sorry. ... View more

Sorru, I jumped the gun a bit there... This seems to be a problem with the manual execution of house_keeping.py, there is a shebang there which should use the correct version #!/usr/bin/env python3.7 So that is most likely just a problem with the manual execution of the script. Then we're back to checking the "permissions to execute scripts". That may still be the issue 🙂 ... View more

So I think I figured it out. /opt/splunk/bin/splunk cmd python3 house_keeping.py --scheme Traceback (most recent call last): File "/opt/splunk/etc/apps/alert_manager_enterprise/bin/house_keeping.py", line 21, in <module> from datapunctum.factory_logger import Logger File "/opt/splunk/etc/apps/alert_manager_enterprise/bin/../lib/datapunctum/factory_logger.py", line 10, in <module> from pydantic import BaseModel File "/opt/splunk/etc/apps/alert_manager_enterprise/bin/../lib/pydantic/__init__.py", line 372, in __getattr__ module = import_module(module_name, package=package) File "/opt/splunk/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/opt/splunk/etc/apps/alert_manager_enterprise/bin/../lib/pydantic/main.py", line 11, in <module> import pydantic_core File "/opt/splunk/etc/apps/alert_manager_enterprise/bin/../lib/pydantic_core/__init__.py", line 6, in <module> from ._pydantic_core import ( ModuleNotFoundError: No module named 'pydantic_core._pydantic_core' However /opt/splunk/bin/splunk cmd python3.7 house_keeping.py --scheme <scheme><title>AME house keeping</title><description>This task does everything, that has to be done in the background, like mapping users and checking the events time to live.</description><use_external_validation>true</use_external_validation><use_single_instance>false</use_single_instance><streaming_mode>xml</streaming_mode><endpoint><args><arg name="default"><description>Unused Default Argument</description><data_type>string</data_type><required_on_edit>false</required_on_edit><required_on_create>true</required_on_create></arg></args></endpoint></scheme>splunk@gdovm:~/etc/apps/alert_manager_enterprise/bin$ And then there is this: https://docs.splunk.com/Documentation/Splunk/9.3.0/ReleaseNotes/MeetSplunk n this release, the default Python interpreter is set to Python version 3.9. The Python.Version settings has been updated so that the parameter is set to value of force_python3, this forces all Python extension points to use Python 3.9 including overriding any application specified settings. "This is designed to be secure-by-default for new customers. If the value is set to python3.9, the default interpreter is set to Python 3.9 but applications can choose to use a different value. Python 3.7 continues to be available in the build for customers' private apps." So the problem is a change in the Python version handling when upgrading to Splunk 9.3.0 Either you can try to figure out how to force the app to revert to python 3.7 or you can get a hold of the developers and inform them the app may not be working with 9.3.0 anymore. Sorry, not fun to have to be the bearer of bad news ... View more

Hold on a second, there is no permission to execute any of the python scripts, right? Might be something I'm missing but I suspect that might be a problem at least: -rwxrw-rw- 1 splunk splunk 5,7K jun 28 11:32 alertqueue_consumer.py -rwxrw-rw- 1 splunk splunk 11K jun 28 11:32 command_ameenrich.py -rwxrw-rw- 1 splunk splunk 4,6K jun 28 11:32 command_ameevents.py -rwxrw-rw- 1 splunk splunk 12K jun 28 11:32 create_alert.py -rwxrw-rw- 1 splunk splunk 1,2K jun 28 11:32 _env.py -rwxrw-rw- 1 splunk splunk 5,9K jun 28 11:32 handler_license.py -rwxrw-rw- 1 splunk splunk 4,4K jun 28 11:32 handler_logging.py -rwxrw-rw- 1 splunk splunk 3,2K jun 28 11:32 handler_minit.py -rwxrw-rw- 1 splunk splunk 3,4K jun 28 11:32 handler_proxy.py -rwxrw-rw- 1 splunk splunk 441 jun 28 11:32 handler.py -rwxrw-rw- 1 splunk splunk 3,5K jun 28 11:32 handler_role_utils.py -rwxrw-rw- 1 splunk splunk 5,4K jun 28 11:32 house_keeping.py -rwxrw-rw- 1 splunk splunk 273 jun 28 11:32 __init__.py -rwxrw-rw- 1 splunk splunk 3,8K jun 28 11:32 notificationqueue_consumer.py drwxrwxrwx 2 splunk splunk 4,0K jun 28 11:32 persistconn -rwxrw-rw- 1 splunk splunk 3,6K jun 28 11:32 tag_keeping.py  So maybe a chmod u+x may solve your problems? ... View more

OK, assuming you are running Splunk as root that seems to check out. Splunk python cannot itself import pydantic, though it seems to be bundled with the app so maybe that is just how the command is executed (which is pretty bad, but still). I see a bunch of "from pathlib import Path" so I'm assuming this is to import from localy bundled versions. Not sure what the issue is then, I'll see if I can recreate the error on a local box   ... View more

Hm, so it seems that pydantic (or at least pydantic_core) is not available for import. Not sure that house_keeping.py suffers the same problems as tag_keeping.py, but it is a problem for sure. Did the file permissions look OK? ... View more

This solution still works for most cases, however if you need an alert where number of events is 0 then this solution will not work, not "as is" at least. As a search for, lets say a problem with logshipping, should alert on 0 returned events. Then there is no way to hitch on a hidden field to anything as there are no results. So to fill "all my needs" here, I would have to come up with something completely different, it would need to become a feature in Splunk or I have to sort out an manage a number of group recipients in exchange. ... View more