About schose

schose · Upgrades to 9.4.15, 10.0.10, 10.2.7 and 10.4.3 fai...

Hi all, when upgrading to Splunk Enterprise 9.4.15, 10.0.10, 10.2.7 or 10.4.3 with the .rpm or .deb package, the upgrade may fail on hardened systems with the following error: runuser: failed to execute /var/tmp/splunk-preinstall.EDCpCC/temp_splunk-preinstall: Permission denied Root cause The preinstall script in these packages was changed to address an internal Splunk security issue tracked as VULN-81607. Before the change, the script extracted the embedded splunk-preinstall binary into $SPLUNK_HOME/bin and ran it as root. Now it extracts the binary into a temporary directory under /var/tmp (falling back to /tmp) and runs it either as root or as the owner of the Splunk installation, depending on the ownership of the install tree. On CIS-hardened systems, /var/tmp and /tmp are mounted with the noexec option. The kernel refuses to execute any binary from these file systems, so the preinstall check fails and the package manager aborts the whole upgrade. The script only handles the /tmp case in its comment and does not consider that /var/tmp is commonly hardened the same way. The relevant CIS recommendations are, for example in the CIS Red Hat Enterprise Linux 8 Benchmark v2.0: - Linux 1.1.4.1: Ensure that a separate partition exists for /var/tmp - Linux 1.1.4.2: Ensure that the noexec option is set on the /var/tmp partition Reference: https://www.ibm.com/docs/en/powersc-standard/2.2.0?topic=scac-cis-red-hat-enterprise-linux-8-benchmark-v20 Here is the relevant snippet from the preinstall script: # VULN-81607: Root is trusted, but a malicious service account may control existing install-tree paths. # Only run existing install content as root when the full launcher ancestry is root-owned and not group/world-writable. # splunk-preinstall is package content, but it calls the existing launcher during KVStore checks. if [ -x "$SPLUNK_HOME/bin/splunk" ] ; then echo "This looks like an upgrade of an existing Splunk Server. Checking to see what component we are installing" if [ "$COMPONENT" = "splunk" ]; then echo "extracting splunk_preinstall_base64 into a temporary directory" # Prefer /var/tmp because some hardened hosts mount /tmp noexec. splunk_execution_owner="$(resolve_splunk_execution_owner || true)" preinstall_tmpdir="" for preinstall_tmpbase in /var/tmp /tmp; do preinstall_tmpdir="$(mktemp -d "$preinstall_tmpbase/splunk-preinstall.XXXXXX" 2>/dev/null)" && break done Who is affected Only upgrades of an existing installation are affected, because the check runs only when $SPLUNK_HOME/bin/splunk already exists. Fresh installations are not affected. The paths are hard-coded in the script, so setting TMPDIR does not help. Workaround Temporarily remount /var/tmp with exec for the duration of the upgrade, then restore the hardened mount options: mount -o remount,exec /var/tmp rpm -Uvh splunk-10.4.3-4174a2deda5d.x86_64.rpm # or dpkg -i for .deb mount -o remount,noexec /var/tmp

schose · ‎03-30-2026

Hi @livehybrid , right, i just want to share information, as in containerized environments you might run the official image with supported OS (default: rhel8.10) but do not have influence on running kernel. furthermore i guess it's just a matter of time, that the issue have to be addressed. Ubuntu 26.04 lts will be released soon and folks will ask for support. regards, Andreas

schose · ‎03-30-2026

Hi all, I just want to share some information: there seem to be an issue with mongod v8.0 with recent linux kernels. https://www.mongodb.com/docs/manual/release-notes/8.0/ "MongoDB 8.0 Incompatible with Kernel 6.19 Due to an incompatibility between a new kernel release and the currently vendored version of TCMalloc, running MongoDB 8.0 or newer with Linux kernel version 6.19 can cause MongoDB to crash on startup. This applies to all MongoDB packages, including those obtained from the MongoDB website, or obtained from package managers or Docker." In my containerized environment the crash happens after ~30seconds. I captured it here: https://asciinema.org/a/7BqfstSfHykgqirG . I guess this have to be addressed soon and might be worth to be mentioned at "known issues" for v10.2 best regards, Andreas

schose · ‎07-31-2025

Hi @livehybrid, Well, strange thing is that mongo7 is working for splunk v10 when installing vanilla. I would strongly guess that the update even does not work on intel macs also, because the mongo binaries needed for migration are not the in macos intel tarball. Cheers, Andreas

schose · ‎07-30-2025

Hi all, When upgrading from v9.4.1 to a newer version (including 10) on MacOS (arm) i receive the error message: -> Currently configured KVSTore database path="/Users/andreas/splunk/var/lib/splunk/kvstore" -> Currently used KVSTore version=4.2.22. Expected version=4.2 or version=7.0 CPU Vendor: GenuineIntel CPU Family: 6 CPU Model: 44 CPU Brand: \x AVX Support: No SSE4.2 Support: Yes AES-NI Support: Yes There seems to be an issue with determine AVX correctly thru rosetta?! - Anyway, i tried to upgrade on v9.4.1using ~/splunk/bin/splunk start-standalone-upgrade kvstore -version 7.0 -dryRun true and receive the error In handler 'kvstoreupgrade': Missing Mongod Binaries :: /Users/andreas/splunk/bin/mongod-7.0; /Users/andreas/splunk/bin/mongod-6.0; /Users/andreas/splunk/bin/mongod-5.0; /Users/andreas/splunk/bin/mongod-4.4; Please make sure they are present under :: /Users/andreas/splunk/bin before proceeding with upgrade. Upgrade Path = /Users/andreas/splunk/bin/mongod_upgrade not found Please make sure upgrade tool binary exists under /Users/andreas/splunk/bin The error that mongod-4.4, mongod-5.0, mongod-6.0 and mongod-7.0 are missing is correct, the files are not there. There are not in delivered splunk .tgz for macos. The linux tarball includes them.. any hints? best regards, Andreas

schose · ‎07-15-2025

Hi all, When creating a systemd unit file for and old UF (<9.1) using "splunk enable boot-start -systemd-managed 1 -user .. " a systemd file is created with content: [Service] ExecStartPost=/bin/bash -c "chown -R splunkfwd:splunkfwd /sys/fs/cgroup/cpu/system.slice/%n" ExecStartPost=/bin/bash -c "chown -R splunkfwd:splunkfwd /sys/fs/cgroup/memory/system.slice/%n" This is also documented in here: https://help.splunk.com/en/splunk-enterprise/forward-and-process-data/universal-forwarder-manual/9.4/working-with-the-universal-forwarder/manage-a-linux-least-privileged-user In "Reference unit file template". Does anyone have an idea why this is done? The paths are using cgroupv1 which only exists on old linux systems, on up-to-date systems this chown fails, but service starts anyway. When creating a systemd config with recent UFs these ExecStartPost Parameters are not set anymore. BUT when installing Splunk Enterprise this line is set in systemd unit ExecStartPost=-/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/system.slice/%n" AFAIK splunk core uses cgroups for Workspace Management, but not on UF. Is the reference unit file template for UF just old&false and the settings never had a sense or is there any good reason? thanks for your help and best regards, Andreas

schose · ‎04-22-2025

Hi all, We want to test if a cluster bundle on cluster manager needs to restart the cluster peers using the REST API. In the first step we run a POST against: https://CLM:8089/services/cluster/manager/control/default/validate_bundle?output_mode=json check-restart=true in body and check for json.entry[0].content.checksum to get the checksum of the new bundle. If there is no checksum, there is no new bundle. Second we check the checksum against GET: https://CLM:8089/services/cluster/manager/info?output_mode=json json.entry[0].content.last_validated_bundle.checksum json.entry[0].content.last_dry_run_bundle.checksum to verify if the bundle check and test of the restart is completed and consider json.entry[0].content.last_check_restart_bundle_result if the restart is nessary or not. Unfurtunatly we see that the value of json.entry[0].content.last_check_restart_bundle_result changes, even if last_dry_run_bundle.checksum and last_dry_run_bundle.checksum are set to the correct values. to make a long story short we see that the red value is changing, while green is not changing. which is unexprected for us. tested against v9.2.5 and v9.4.1. At the moment is looks like a timing issue for me and i want to avoid sleep() code. Is there a more solid way to check if restart is necessary or not? best regards, Andreas

schose · ‎09-30-2024

Hi @marnall , I had set web.conf to [settings] cacheEntriesLimit = 0 cacheBytesLimit = 0 js_no_cache = 1 .. no difference.

schose · ‎09-30-2024

Hi all, Since v9.3 there seem to be a different method for displaying nav menus. When you update the tag <label> tag of a view from external editor, those changes are not updated in navigation until a local storage object is deleted. /debug/refresh or restart splunk doesn't refresh the navigation. I was able to to update navigation when deleting the following object -> chrome -> developer tools -> applications -> "local storage" -> splunk-appnav:MYAPP:admin:en-GB:UUID containing the following datastructure.. { "nav": [ { "label": "Search", "uri": "/en-GB/app/testapp/search", "viewName": "search", "isDefault": true }, { "label": "testview", "uri": "/en-GB/app/testapp/testview", "viewName": "testview" } ], "color": null, "searchView": "search", "lastModified": 1727698963355 } I'm wondering why content of the nav is now saved on client side. This is a different behaviour than on v9.1 and v9.2. If i need to guess, they tried to improve response time of the webui. But how do i ensure that every user is receiving the latest version of navigation menu in an app? Best regards, Andreas

schose · ‎08-12-2024

Hi Joe, yes, you can download the app, patch it and upload it as a private app. Cheers, Andreas

schose · ‎08-10-2024

Hi Joe, there is a command documentation in default/searchbnf.conf [curl-command] syntax = CURL [choice:URI=<uri> OR URIFIELD=<urifield>] [optional: METHOD=<GET|PATCH|POST|PUT|DELETE> VERIFYSSL=<TRUE|FALSE> DATAFIELD=<field_name> DATA=<data> HEADERFIELD=<json_header_field_name> HEADERS=<json_header> USER=<user> PASS=<password> DEBUG=<true|false> SPLUNKAUTH=<true|false> SPLUNKPASSWDNAME=<username_in_passwordsconf> SPLUNKPASSWDCONTEXT=<appcontext (optional)> TIMEOUT=<float>] -k = "VERIFYSSL=FALSE" headers="{\"content-type\":\"application/json\"}" best regards, Andreas

schose · ‎07-27-2023

Hi all, To ensure that CAP_DAC_READ_SEARCH is not set, systemd overwrite mechanism could be used. create /etc/systemd/system/SplunkForwarder.service.d/override.conf with content: [Service] NoNewPrivileges=yes AmbientCapabilities= this ensures, that even if /etc/systemd/system/SplunkForwarder.service is rewritten - which still looks like an issue to me - the AmbientCapabilities are still empty. Best regards, Andreas

schose · ‎07-03-2023

Hi all, Splunk UF since 9.x is setting [Service] NoNewPrivileges=yes AmbientCapabilities=CAP_DAC_READ_SEARCH in systemd unit file (/etc/systemd/system/SplunkForwarder.service). This enables splunkforwarder to bypass Filesystems permissions and acls and read every file on harddisk - yes, every file: every ssh key, every private key, confidential data.. the opposite of the "least-to-know" principle. As we have correct filesystem permissions in place we decided to remove those settings from systemd unit file. When we now run e.g.: "/opt/splunkforwarder/bin/splunk stop" command the systemd file is rewritten by the splunk command. This will start splunkforwarder with enabled CAP_DAC_READ_SEARCH capability. To make is more visual we uploaded a video to https://asciinema.org/a/FAYFPJYrKaizfL3alzvm3uNGF . Are you able to reproduce the issue? What do you think? For us this looks like a secuity issue, as we would never expect a command like "splunk stop" manipulate systemd files. I'm also not aware which other command might rewrite the systemd unit. I also do not seed any usecase for this. steps to reproduce: install-splunkuf.sh #!/bin/bash # break if errors set -e # add system user sudo groupadd splunk sudo useradd splunk --system --home-dir /opt/splunk --create-home -g splunk wget -O /tmp/splunkuf.tgz https://download.splunk.com/products/universalforwarder/releases/9.1.0/linux/splunkforwarder-9.1.0-1c86ca0bacc3-Linux-x86_64.tgz #wget -O /tmp/splunkuf.tgz https://download.splunk.com/products/universalforwarder/releases/9.0.5/linux/splunkforwarder-9.0.5-e9494146ae5c-Linux-armv8.tgz tar zxfv /tmp/splunkuf.tgz -C /opt echo -e "[user_info]\nUSERNAME=admin\nPASSWORD=Password01" > /opt/splunkforwarder/etc/system/local/user-seed.conf /opt/splunkforwarder/bin/splunk start --accept-license && /opt/splunkforwarder/bin/splunk stop -f /opt/splunkforwarder/bin/splunk enable boot-start -user splunk -group splunk -systemd-managed 1 # remove capabilities from systemd service sed -i '/^NoNewPrivileges\|^AmbientCapabilities/s/^/#/' /etc/systemd/system/SplunkForwarder.service systemctl daemon-reload systemctl start SplunkForwarder systemctl status SplunkForwarder # systemd file is still fine echo -n "systemd unit file after starting splunk" cat /etc/systemd/system/SplunkForwarder.service pid=$(systemctl show -p MainPID --value SplunkForwarder.service) && getpcaps $pid when you now run /opt/splunkforwarder/bin/splunk stop cat /etc/systemd/system/SplunkForwarder.service you see that lines NoNewPrivileges=yes AmbientCapabilities=CAP_DAC_READ_SEARCH are re-added to /etc/systemd/system/SplunkForwarder.service and next time the service is started caps are set. A backup file is also placed /etc/systemd/system/SplunkForwarder.service_TIMESTAMP. when running a strace strace -s 0 -o /tmp/910stop.strace -f /opt/splunkforwarder/bin/splunk stop we clearly see the splunk process manipulating the systemd file. 2120 rename("/etc/systemd/system/SplunkForwarder.service", "/etc/systemd/system/SplunkForwarder.service_2023_07_03_21_47_00") = 0 2120 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7feb05354f10) = 2122 2120 wait4(2122, 2122 set_robust_list(0x7feb05354f20, 24) = 0 This happens on all 9.x versions of UF. best regards, Andreas

schose · ‎06-16-2023

Hi Forum, We have an issue with UF 9.0.5. When starting or stopping the filesytem group permissions are changed to the primary group of the technical user running splunk. when splunk is started we always see the message: Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk_tech_user /opt/splunkforwarder" This does not chown the user but also the group to the primary group of the user. Any chance to skip this? version 8.* does not show this issue. best regards, Andreas

schose · ‎05-26-2023

Hi @rkantamaneni , Well, i can rerun the tests with a 9.x version, but wouldn't expect different results with the same level setting. I would be also interested in behaviour for different sourcetypes, as we see huge differences there. Do you have an idea for good high cardinality logfiles? Best regards, Andreas

schose · ‎05-08-2023

Hi @AdrianMa , hope you mean splunk enterprise x86_64 Linux? Here is a way: docker pull splunk/splunk:7.3.6 docker run -ti --entrypoint /bin/bash splunk/splunk:7.3. tar zcfv /tmp/splunk-7.3.6.tar.gz splunk docker ps -> get container id docker cp CONTAINERID:/tmp/splunk-7.3.6.tar.gz . regards, Andreas

schose · ‎04-24-2023

Hi @VatsalJagani , updating the app using REST API or GUI doesn't make a difference. We will open a call for this behaviour. as a workaround it's possible to "version" your savedsearches -> putting 0001,2,3 at the end. Not very user friendly, but working. best regards, Andreas

schose · ‎04-19-2023

Hi all, I have an testapp with a savesearch containing: [testsearch] alert.suppress = 0 alert.track = 0 counttype = number of events cron_schedule = 1 * * * 1 dispatch.earliest_time = -7d@d dispatch.latest_time = -0d@d display.general.type = statistics display.page.search.mode = verbose display.page.search.tab = statistics enableSched = 1 quantity = 0 relation = greater than search = | makeresults when i install the app using gui/api or commandline i see the schedule for next monday 1am. When i now change the schedule and settings to: [testsearch] alert.suppress = 0 alert.track = 0 counttype = number of events cron_schedule = 5 * * * * dispatch.earliest_time = -16m@m dispatch.latest_time = -1m@m display.general.type = statistics display.page.search.mode = verbose display.page.search.tab = statistics enableSched = 1 quantity = 0 relation = greater than search = | makeresults and checking by | rest /servicesNS/-/testsearch/saved/searches | table title cron_schedule eai:acl.app cron_schedule next_scheduled_time | rename eai:acl.app as app | search title=*testsearch i see that next_scheduled_time is still next monday. running a /debug/refresh or curl -k -u admin:XXX https://localhost:8089/servicesNS/nobody/testsearch/admin/localapps/_reload does not fix the issue. It seems only restarting the searchhead, wait until next schedule (multiple day in this case) or change using the gui fixes the issue. In our case non of the above is possible, as we are deploying our apps using a cicd pipeline. Any hint or workaround? Best regards, Andreas

schose · ‎03-28-2023

Hi @danielbb , It's EventCode, not EventID. 😉 | tstats count where index=wineventlog* TERM(EventCode=4688) by _time span=1m best regards, Andreas

schose · ‎12-16-2022

Hi NIranjan, can you send an ansible-playbook debug while the task is hanging? I'm using .tgz files as, .rpm always seems a blackbox for me - i cannot see from outside what post scripts are running in. there. best regards, Andreas

schose · ‎12-15-2022

Hi Niranjan, yes, i simply run - name: accept license tags: - install shell: /opt/splunkforwarder/bin/splunk status --accept-license --answer-yes become: yes when: - splunk_path.stat.exists == false or current_version.rc != 0 ignore_errors: true before.. cheers, Andreas

schose · ‎12-14-2022

Hi Niranjan, Mongo is no component started by UFW but only by Splunk Enterprise.. So this is surely a bug.. it also see to make a difference if you use .tgz or .deb. I had the issue using .tgz. My workaround is this task: - name: ensure kvstore is disabled on UFW ini_file: dest: /opt/splunkforwarder/etc/system/local/server.conf section: kvstore option: disabled value: true regards, Andreas

schose · ‎12-14-2022

Hi, Same here.. for some reason the update script tries to upgrade mongo-db on UFW and get stuck. seems like a bug. disable kvstore in server.conf works for me. https://community.splunk.com/t5/Splunk-Enterprise/Splunk-9-Universal-Forwarder-getting-quot-App-Key-Value-Store/m-p/613838#M13969 regards, Andreas

schose · ‎12-08-2022

Hi @ojay , there doesn't seem to be a ready-to-go iLO TA. We ingest data by sending iLO data to syslog and using UFW on syslog to ingest to splunk. best regards, Andreas

schose · ‎12-08-2022

Hi @jcourses , right, the iowait check in distributed health reporter is quite aggressive. You can raise the limits by configure health.conf We changed iowait to: [feature:iowait] display_name = IOWait indicator:single_cpu__max_perc_last_3m:description = This indicator tracks the IOWait percentage for the single most bottle-necked CPU on the machine running the Splunk Enterprise instance, over the last 3 minute window. By default, this indicator will turn Yellow if the percentage exceeds 5% and Red if it exceeds 10% during this window. indicator:single_cpu__max_perc_last_3m:red = 20 indicator:single_cpu__max_perc_last_3m:yellow = 10 meaning iowait is yellow for 10% io wait, red for 20% io wait in a 3 min period. best regards, Andreas

Posts	198
Solutions	24
Karma Given	20
Karma Received	93
Member Since	‎06-28-2012

Online Status	Offline
Date Last Visited	Thursday

Upgrades to 9.4.15, 10.0.10, 10.2.7 and 10.4.3 fai...

mongodb/kvstore v8 on splunk v10.2 - segfault on 6...

upgrade to kvstore v7 impossible on MacOS?!

what is ExecStartPost in systemd unit itended for?

test of rolling-restart using rest api

Splunk v9.3 not refreshing nav correctly

Security issue?! Splunk UF v9.x is re-adding reada...

Universal Forwarder 9.0.* changing filesystem grou...

How to refresh cron schedule of a savesearch from ...

Why does merge-buckets only merge up to 300 bucket...

Upgrades to 9.4.15, 10.0.10, 10.2.7 and 10.4.3 fai...

Re: mongodb/kvstore v8 on splunk v10.2 - segfault ...

mongodb/kvstore v8 on splunk v10.2 - segfault on 6...

Re: upgrade to kvstore v7 impossible on MacOS?!

upgrade to kvstore v7 impossible on MacOS?!

what is ExecStartPost in systemd unit itended for?

test of rolling-restart using rest api

Re: Splunk v9.3 not refreshing nav correctly

Splunk v9.3 not refreshing nav correctly

Re: TA-WebTools - Syntax question

Re: TA-WebTools - Syntax question

Re: Security issue?! Splunk UF v9.x is re-adding r...

Security issue?! Splunk UF v9.x is re-adding reada...

Universal Forwarder 9.0.* changing filesystem grou...

Re: index.conf- Can anyone explain me tsidxWriting...

Re: Does anyone have a Splunk 7.3.6 download?

Re: refresh cron schedule of a savesearch from cli...

How to refresh cron schedule of a savesearch from ...

Re: How can we use tstats with TERM and PREFIX?

Re: Why is splunk UF version 9.0.1 stuck while acc...

Re: Why is splunk UF version 9.0.1 stuck while acc...

Re: Why is splunk UF version 9.0.1 stuck while acc...

Re: Why is splunk UF version 9.0.1 stuck while acc...

Re: What add-on can I use for integration of HPE i...

Re: How to disable IOWAIT?

Join the Conversation