Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
×
Join the Conversation
Without signing in, you're just watching from the sidelines. Sign in or Register to connect, share, and be part of the Splunk Community.
- Find Answers
- :
- About schose
schose
Builder
Member since:
06-28-2012
2 weeks ago
Community Statistics
| Posts | 199 |
| Solutions | 24 |
| Karma Given | 21 |
| Karma Received | 97 |
| Member Since | 06-28-2012 |
Activity Feed
- Got Karma for Re: Upgrades to 9.4.15, 10.0.10, 10.2.7 and 10.4.3 fail on CIS-hardened systems (noexec on /var/tmp). Sunday
- Got Karma for Upgrades to 9.4.15, 10.0.10, 10.2.7 and 10.4.3 fail on CIS-hardened systems (noexec on /var/tmp). Sunday
- Got Karma for Upgrades to 9.4.15, 10.0.10, 10.2.7 and 10.4.3 fail on CIS-hardened systems (noexec on /var/tmp). 3 weeks ago
- Karma Re: Upgrades to 9.4.15, 10.0.10, 10.2.7 and 10.4.3 fail on CIS-hardened systems (noexec on /var/tmp) for kodddk. 3 weeks ago
- Posted Re: Upgrades to 9.4.15, 10.0.10, 10.2.7 and 10.4.3 fail on CIS-hardened systems (noexec on /var/tmp) on Splunk Enterprise. 3 weeks ago
- Got Karma for Upgrades to 9.4.15, 10.0.10, 10.2.7 and 10.4.3 fail on CIS-hardened systems (noexec on /var/tmp). a month ago
- Posted Upgrades to 9.4.15, 10.0.10, 10.2.7 and 10.4.3 fail on CIS-hardened systems (noexec on /var/tmp) on Splunk Enterprise. a month ago
- Got Karma for Re: mongodb/kvstore v8 on splunk v10.2 - segfault on 6.19.x kernels. 07-29-2026 07:56 AM
- Got Karma for mongodb/kvstore v8 on splunk v10.2 - segfault on 6.19.x kernels. 07-29-2026 07:56 AM
- Got Karma for mongodb/kvstore v8 on splunk v10.2 - segfault on 6.19.x kernels. 05-15-2026 03:32 PM
- Got Karma for Re: mongodb/kvstore v8 on splunk v10.2 - segfault on 6.19.x kernels. 04-01-2026 03:37 AM
- Posted Re: mongodb/kvstore v8 on splunk v10.2 - segfault on 6.19.x kernels on Splunk Enterprise. 03-30-2026 03:04 PM
- Posted mongodb/kvstore v8 on splunk v10.2 - segfault on 6.19.x kernels on Splunk Enterprise. 03-30-2026 01:07 PM
- Got Karma for upgrade to kvstore v7 impossible on MacOS?!. 08-15-2025 05:30 AM
- Got Karma for Re: upgrade to kvstore v7 impossible on MacOS?!. 08-15-2025 03:17 AM
- Posted Re: upgrade to kvstore v7 impossible on MacOS?! on Splunk Enterprise. 07-31-2025 01:22 AM
- Posted upgrade to kvstore v7 impossible on MacOS?! on Splunk Enterprise. 07-30-2025 02:52 PM
- Posted what is ExecStartPost in systemd unit itended for? on Deployment Architecture. 07-15-2025 02:58 PM
- Got Karma for Re: Splunk v9.3 not refreshing nav correctly. 06-18-2025 10:34 AM
- Got Karma for Splunk v9.3 not refreshing nav correctly. 06-18-2025 05:47 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 3 | |||
| 2 | |||
| 1 | |||
| 0 | |||
| 0 | |||
| 1 | |||
| 1 | |||
| 0 | |||
| 0 | |||
| 0 |
3 weeks ago
1 Karma
hey kodddk, Can you please provide your ticket nr.? You can send it in PM if you want to. We will also open one and would have a good reference for it as this blocks our default upgrade procedures. Best regards, Andreas
... View more
a month ago
3 Karma
Hi all, when upgrading to Splunk Enterprise 9.4.15, 10.0.10, 10.2.7 or 10.4.3 with the .rpm or .deb package, the upgrade may fail on hardened systems with the following error: runuser: failed to execute /var/tmp/splunk-preinstall.EDCpCC/temp_splunk-preinstall: Permission denied Root cause The preinstall script in these packages was changed to address an internal Splunk security issue tracked as VULN-81607. Before the change, the script extracted the embedded splunk-preinstall binary into $SPLUNK_HOME/bin and ran it as root. Now it extracts the binary into a temporary directory under /var/tmp (falling back to /tmp) and runs it either as root or as the owner of the Splunk installation, depending on the ownership of the install tree. On CIS-hardened systems, /var/tmp and /tmp are mounted with the noexec option. The kernel refuses to execute any binary from these file systems, so the preinstall check fails and the package manager aborts the whole upgrade. The script only handles the /tmp case in its comment and does not consider that /var/tmp is commonly hardened the same way. The relevant CIS recommendations are, for example in the CIS Red Hat Enterprise Linux 8 Benchmark v2.0: - Linux 1.1.4.1: Ensure that a separate partition exists for /var/tmp - Linux 1.1.4.2: Ensure that the noexec option is set on the /var/tmp partition Reference: https://www.ibm.com/docs/en/powersc-standard/2.2.0?topic=scac-cis-red-hat-enterprise-linux-8-benchmark-v20 Here is the relevant snippet from the preinstall script: # VULN-81607: Root is trusted, but a malicious service account may control existing install-tree paths.
# Only run existing install content as root when the full launcher ancestry is root-owned and not group/world-writable.
# splunk-preinstall is package content, but it calls the existing launcher during KVStore checks.
if [ -x "$SPLUNK_HOME/bin/splunk" ] ; then
echo "This looks like an upgrade of an existing Splunk Server. Checking to see what component we are installing"
if [ "$COMPONENT" = "splunk" ]; then
echo "extracting splunk_preinstall_base64 into a temporary directory"
# Prefer /var/tmp because some hardened hosts mount /tmp noexec.
splunk_execution_owner="$(resolve_splunk_execution_owner || true)"
preinstall_tmpdir=""
for preinstall_tmpbase in /var/tmp /tmp; do
preinstall_tmpdir="$(mktemp -d "$preinstall_tmpbase/splunk-preinstall.XXXXXX" 2>/dev/null)" && break
done Who is affected Only upgrades of an existing installation are affected, because the check runs only when $SPLUNK_HOME/bin/splunk already exists. Fresh installations are not affected. The paths are hard-coded in the script, so setting TMPDIR does not help. Workaround Temporarily remount /var/tmp with exec for the duration of the upgrade, then restore the hardened mount options: mount -o remount,exec /var/tmp
rpm -Uvh splunk-10.4.3-4174a2deda5d.x86_64.rpm # or dpkg -i for .deb
mount -o remount,noexec /var/tmp
... View more
- Tags:
- installation
- upgrade
Labels
- Labels:
-
installation
-
upgrade
03-30-2026
03:04 PM
2 Karma
Hi @livehybrid , right, i just want to share information, as in containerized environments you might run the official image with supported OS (default: rhel8.10) but do not have influence on running kernel. furthermore i guess it's just a matter of time, that the issue have to be addressed. Ubuntu 26.04 lts will be released soon and folks will ask for support. regards, Andreas
... View more
03-30-2026
01:07 PM
2 Karma
Hi all, I just want to share some information: there seem to be an issue with mongod v8.0 with recent linux kernels. https://www.mongodb.com/docs/manual/release-notes/8.0/ "MongoDB 8.0 Incompatible with Kernel 6.19 Due to an incompatibility between a new kernel release and the currently vendored version of TCMalloc, running MongoDB 8.0 or newer with Linux kernel version 6.19 can cause MongoDB to crash on startup. This applies to all MongoDB packages, including those obtained from the MongoDB website, or obtained from package managers or Docker." In my containerized environment the crash happens after ~30seconds. I captured it here: https://asciinema.org/a/7BqfstSfHykgqirG . I guess this have to be addressed soon and might be worth to be mentioned at "known issues" for v10.2 best regards, Andreas
... View more
Labels
- Labels:
-
installation
-
troubleshooting
07-31-2025
01:22 AM
1 Karma
Hi @livehybrid, Well, strange thing is that mongo7 is working for splunk v10 when installing vanilla. I would strongly guess that the update even does not work on intel macs also, because the mongo binaries needed for migration are not the in macos intel tarball. Cheers, Andreas
... View more
07-30-2025
02:52 PM
1 Karma
Hi all, When upgrading from v9.4.1 to a newer version (including 10) on MacOS (arm) i receive the error message: -> Currently configured KVSTore database path="/Users/andreas/splunk/var/lib/splunk/kvstore"
-> Currently used KVSTore version=4.2.22. Expected version=4.2 or version=7.0
CPU Vendor: GenuineIntel
CPU Family: 6
CPU Model: 44
CPU Brand: \x
AVX Support: No
SSE4.2 Support: Yes
AES-NI Support: Yes There seems to be an issue with determine AVX correctly thru rosetta?! - Anyway, i tried to upgrade on v9.4.1using ~/splunk/bin/splunk start-standalone-upgrade kvstore -version 7.0 -dryRun true and receive the error In handler 'kvstoreupgrade': Missing Mongod Binaries :: /Users/andreas/splunk/bin/mongod-7.0; /Users/andreas/splunk/bin/mongod-6.0; /Users/andreas/splunk/bin/mongod-5.0; /Users/andreas/splunk/bin/mongod-4.4; Please make sure they are present under :: /Users/andreas/splunk/bin before proceeding with upgrade.
Upgrade Path = /Users/andreas/splunk/bin/mongod_upgrade not found Please make sure upgrade tool binary exists under /Users/andreas/splunk/bin The error that mongod-4.4, mongod-5.0, mongod-6.0 and mongod-7.0 are missing is correct, the files are not there. There are not in delivered splunk .tgz for macos. The linux tarball includes them.. any hints? best regards, Andreas
... View more
Labels
- Labels:
-
upgrade
07-15-2025
02:58 PM
Hi all, When creating a systemd unit file for and old UF (<9.1) using "splunk enable boot-start -systemd-managed 1 -user .. " a systemd file is created with content: [Service]
ExecStartPost=/bin/bash -c "chown -R splunkfwd:splunkfwd /sys/fs/cgroup/cpu/system.slice/%n"
ExecStartPost=/bin/bash -c "chown -R splunkfwd:splunkfwd /sys/fs/cgroup/memory/system.slice/%n" This is also documented in here: https://help.splunk.com/en/splunk-enterprise/forward-and-process-data/universal-forwarder-manual/9.4/working-with-the-universal-forwarder/manage-a-linux-least-privileged-user In "Reference unit file template". Does anyone have an idea why this is done? The paths are using cgroupv1 which only exists on old linux systems, on up-to-date systems this chown fails, but service starts anyway. When creating a systemd config with recent UFs these ExecStartPost Parameters are not set anymore. BUT when installing Splunk Enterprise this line is set in systemd unit ExecStartPost=-/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/system.slice/%n" AFAIK splunk core uses cgroups for Workspace Management, but not on UF. Is the reference unit file template for UF just old&false and the settings never had a sense or is there any good reason? thanks for your help and best regards, Andreas
... View more
Labels
- Labels:
-
universal forwarder
04-22-2025
02:29 PM
Hi all, We want to test if a cluster bundle on cluster manager needs to restart the cluster peers using the REST API. In the first step we run a POST against: https://CLM:8089/services/cluster/manager/control/default/validate_bundle?output_mode=json check-restart=true in body and check for json.entry[0].content.checksum to get the checksum of the new bundle. If there is no checksum, there is no new bundle. Second we check the checksum against GET: https://CLM:8089/services/cluster/manager/info?output_mode=json json.entry[0].content.last_validated_bundle.checksum json.entry[0].content.last_dry_run_bundle.checksum to verify if the bundle check and test of the restart is completed and consider json.entry[0].content.last_check_restart_bundle_result if the restart is nessary or not. Unfurtunatly we see that the value of json.entry[0].content.last_check_restart_bundle_result changes, even if last_dry_run_bundle.checksum and last_dry_run_bundle.checksum are set to the correct values. to make a long story short we see that the red value is changing, while green is not changing. which is unexprected for us. tested against v9.2.5 and v9.4.1. At the moment is looks like a timing issue for me and i want to avoid sleep() code. Is there a more solid way to check if restart is necessary or not? best regards, Andreas
... View more
Labels
- Labels:
-
configuration
09-30-2024
01:25 PM
1 Karma
Hi @marnall , I had set web.conf to [settings]
cacheEntriesLimit = 0
cacheBytesLimit = 0
js_no_cache = 1 .. no difference.
... View more
09-30-2024
08:59 AM
1 Karma
Hi all, Since v9.3 there seem to be a different method for displaying nav menus. When you update the tag <label> tag of a view from external editor, those changes are not updated in navigation until a local storage object is deleted. /debug/refresh or restart splunk doesn't refresh the navigation. I was able to to update navigation when deleting the following object -> chrome -> developer tools -> applications -> "local storage" -> splunk-appnav:MYAPP:admin:en-GB:UUID containing the following datastructure.. {
"nav": [
{
"label": "Search",
"uri": "/en-GB/app/testapp/search",
"viewName": "search",
"isDefault": true
},
{
"label": "testview",
"uri": "/en-GB/app/testapp/testview",
"viewName": "testview"
}
],
"color": null,
"searchView": "search",
"lastModified": 1727698963355
} I'm wondering why content of the nav is now saved on client side. This is a different behaviour than on v9.1 and v9.2. If i need to guess, they tried to improve response time of the webui. But how do i ensure that every user is receiving the latest version of navigation menu in an app? Best regards, Andreas
... View more
Labels
- Labels:
-
Classic dashboard
08-12-2024
03:30 PM
1 Karma
Hi Joe, yes, you can download the app, patch it and upload it as a private app. Cheers, Andreas
... View more
08-10-2024
06:29 AM
Hi Joe,
there is a command documentation in default/searchbnf.conf
[curl-command]
syntax = CURL [choice:URI=<uri> OR URIFIELD=<urifield>] [optional: METHOD=<GET|PATCH|POST|PUT|DELETE> VERIFYSSL=<TRUE|FALSE> DATAFIELD=<field_name> DATA=<data> HEADERFIELD=<json_header_field_name> HEADERS=<json_header> USER=<user> PASS=<password> DEBUG=<true|false> SPLUNKAUTH=<true|false> SPLUNKPASSWDNAME=<username_in_passwordsconf> SPLUNKPASSWDCONTEXT=<appcontext (optional)> TIMEOUT=<float>]
-k = "VERIFYSSL=FALSE"
headers="{\"content-type\":\"application/json\"}"
best regards,
Andreas
... View more
07-27-2023
08:22 AM
1 Karma
Hi all, To ensure that CAP_DAC_READ_SEARCH is not set, systemd overwrite mechanism could be used. create /etc/systemd/system/SplunkForwarder.service.d/override.conf with content: [Service]
NoNewPrivileges=yes
AmbientCapabilities= this ensures, that even if /etc/systemd/system/SplunkForwarder.service is rewritten - which still looks like an issue to me - the AmbientCapabilities are still empty. Best regards, Andreas
... View more
07-03-2023
02:51 PM
1 Karma
Hi all, Splunk UF since 9.x is setting [Service]
NoNewPrivileges=yes
AmbientCapabilities=CAP_DAC_READ_SEARCH in systemd unit file (/etc/systemd/system/SplunkForwarder.service). This enables splunkforwarder to bypass Filesystems permissions and acls and read every file on harddisk - yes, every file: every ssh key, every private key, confidential data.. the opposite of the "least-to-know" principle. As we have correct filesystem permissions in place we decided to remove those settings from systemd unit file. When we now run e.g.: "/opt/splunkforwarder/bin/splunk stop" command the systemd file is rewritten by the splunk command. This will start splunkforwarder with enabled CAP_DAC_READ_SEARCH capability. To make is more visual we uploaded a video to https://asciinema.org/a/FAYFPJYrKaizfL3alzvm3uNGF . Are you able to reproduce the issue? What do you think? For us this looks like a secuity issue, as we would never expect a command like "splunk stop" manipulate systemd files. I'm also not aware which other command might rewrite the systemd unit. I also do not seed any usecase for this. steps to reproduce: install-splunkuf.sh #!/bin/bash
# break if errors
set -e
# add system user
sudo groupadd splunk
sudo useradd splunk --system --home-dir /opt/splunk --create-home -g splunk
wget -O /tmp/splunkuf.tgz https://download.splunk.com/products/universalforwarder/releases/9.1.0/linux/splunkforwarder-9.1.0-1c86ca0bacc3-Linux-x86_64.tgz
#wget -O /tmp/splunkuf.tgz https://download.splunk.com/products/universalforwarder/releases/9.0.5/linux/splunkforwarder-9.0.5-e9494146ae5c-Linux-armv8.tgz
tar zxfv /tmp/splunkuf.tgz -C /opt
echo -e "[user_info]\nUSERNAME=admin\nPASSWORD=Password01" > /opt/splunkforwarder/etc/system/local/user-seed.conf
/opt/splunkforwarder/bin/splunk start --accept-license && /opt/splunkforwarder/bin/splunk stop -f
/opt/splunkforwarder/bin/splunk enable boot-start -user splunk -group splunk -systemd-managed 1
# remove capabilities from systemd service
sed -i '/^NoNewPrivileges\|^AmbientCapabilities/s/^/#/' /etc/systemd/system/SplunkForwarder.service
systemctl daemon-reload
systemctl start SplunkForwarder
systemctl status SplunkForwarder
# systemd file is still fine
echo -n "systemd unit file after starting splunk"
cat /etc/systemd/system/SplunkForwarder.service
pid=$(systemctl show -p MainPID --value SplunkForwarder.service) && getpcaps $pid when you now run /opt/splunkforwarder/bin/splunk stop
cat /etc/systemd/system/SplunkForwarder.service you see that lines NoNewPrivileges=yes
AmbientCapabilities=CAP_DAC_READ_SEARCH are re-added to /etc/systemd/system/SplunkForwarder.service and next time the service is started caps are set. A backup file is also placed /etc/systemd/system/SplunkForwarder.service_TIMESTAMP. when running a strace strace -s 0 -o /tmp/910stop.strace -f /opt/splunkforwarder/bin/splunk stop we clearly see the splunk process manipulating the systemd file. 2120 rename("/etc/systemd/system/SplunkForwarder.service", "/etc/systemd/system/SplunkForwarder.service_2023_07_03_21_47_00") = 0
2120 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7feb05354f10) = 2122
2120 wait4(2122,
2122 set_robust_list(0x7feb05354f20, 24) = 0 This happens on all 9.x versions of UF. best regards, Andreas
... View more
06-16-2023
07:42 AM
Hi Forum, We have an issue with UF 9.0.5. When starting or stopping the filesytem group permissions are changed to the primary group of the technical user running splunk. when splunk is started we always see the message: Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk_tech_user /opt/splunkforwarder" This does not chown the user but also the group to the primary group of the user. Any chance to skip this? version 8.* does not show this issue. best regards, Andreas
... View more
Labels
- Labels:
-
universal forwarder
-
upgrade
05-26-2023
06:24 AM
Hi @rkantamaneni , Well, i can rerun the tests with a 9.x version, but wouldn't expect different results with the same level setting. I would be also interested in behaviour for different sourcetypes, as we see huge differences there. Do you have an idea for good high cardinality logfiles? Best regards, Andreas
... View more
05-08-2023
04:54 AM
Hi @AdrianMa , hope you mean splunk enterprise x86_64 Linux? Here is a way: docker pull splunk/splunk:7.3.6 docker run -ti --entrypoint /bin/bash splunk/splunk:7.3. tar zcfv /tmp/splunk-7.3.6.tar.gz splunk docker ps -> get container id docker cp CONTAINERID:/tmp/splunk-7.3.6.tar.gz . regards, Andreas
... View more
04-24-2023
04:12 AM
Hi @VatsalJagani , updating the app using REST API or GUI doesn't make a difference. We will open a call for this behaviour. as a workaround it's possible to "version" your savedsearches -> putting 0001,2,3 at the end. Not very user friendly, but working. best regards, Andreas
... View more
04-19-2023
12:06 PM
Hi all,
I have an testapp with a savesearch containing:
[testsearch]
alert.suppress = 0
alert.track = 0
counttype = number of events
cron_schedule = 1 * * * 1
dispatch.earliest_time = -7d@d
dispatch.latest_time = -0d@d
display.general.type = statistics
display.page.search.mode = verbose
display.page.search.tab = statistics
enableSched = 1
quantity = 0
relation = greater than
search = | makeresults
when i install the app using gui/api or commandline i see the schedule for next monday 1am. When i now change the schedule and settings to:
[testsearch]
alert.suppress = 0
alert.track = 0
counttype = number of events
cron_schedule = 5 * * * *
dispatch.earliest_time = -16m@m
dispatch.latest_time = -1m@m
display.general.type = statistics
display.page.search.mode = verbose
display.page.search.tab = statistics
enableSched = 1
quantity = 0
relation = greater than
search = | makeresults
and checking by
| rest /servicesNS/-/testsearch/saved/searches | table title cron_schedule eai:acl.app cron_schedule next_scheduled_time | rename eai:acl.app as app | search title=*testsearch
i see that next_scheduled_time is still next monday. running a /debug/refresh or curl -k -u admin:XXX https://localhost:8089/servicesNS/nobody/testsearch/admin/localapps/_reload does not fix the issue. It seems only restarting the searchhead, wait until next schedule (multiple day in this case) or change using the gui fixes the issue.
In our case non of the above is possible, as we are deploying our apps using a cicd pipeline.
Any hint or workaround?
Best regards,
Andreas
... View more
Labels
- Labels:
-
configuration
-
development
-
upgrade
03-28-2023
10:05 AM
Hi @danielbb , It's EventCode, not EventID. 😉 | tstats count where index=wineventlog* TERM(EventCode=4688) by _time span=1m best regards, Andreas
... View more
12-16-2022
12:39 AM
Hi NIranjan, can you send an ansible-playbook debug while the task is hanging? I'm using .tgz files as, .rpm always seems a blackbox for me - i cannot see from outside what post scripts are running in. there. best regards, Andreas
... View more
12-15-2022
06:31 AM
Hi Niranjan, yes, i simply run - name: accept license
tags:
- install
shell: /opt/splunkforwarder/bin/splunk status --accept-license --answer-yes
become: yes
when:
- splunk_path.stat.exists == false or current_version.rc != 0
ignore_errors: true before.. cheers, Andreas
... View more
12-14-2022
08:16 AM
Hi Niranjan, Mongo is no component started by UFW but only by Splunk Enterprise.. So this is surely a bug.. it also see to make a difference if you use .tgz or .deb. I had the issue using .tgz. My workaround is this task: - name: ensure kvstore is disabled on UFW
ini_file:
dest: /opt/splunkforwarder/etc/system/local/server.conf
section: kvstore
option: disabled
value: true regards, Andreas
... View more
12-14-2022
04:11 AM
Hi, Same here.. for some reason the update script tries to upgrade mongo-db on UFW and get stuck. seems like a bug. disable kvstore in server.conf works for me. https://community.splunk.com/t5/Splunk-Enterprise/Splunk-9-Universal-Forwarder-getting-quot-App-Key-Value-Store/m-p/613838#M13969 regards, Andreas
... View more
12-08-2022
07:20 AM
Hi @ojay , there doesn't seem to be a ready-to-go iLO TA. We ingest data by sending iLO data to syslog and using UFW on syslog to ingest to splunk. best regards, Andreas
... View more
