Hello, we tried to enable TLS validation with Splunk 9.0.2 as described in the Splunk documentation. Unfortunately, this caused our distributed Splunk system consisting of index cluster and searchhead cluster to stop working. Specifically, searches could no longer be performed. We discovered the following messages in the search head log: 11-06-2022 23:59:59.839 +0100 ERROR DistributedPeerManagerHeartbeat [95344 DistributedPeerMonitorThread] - Send failure while pushing public key to search peer = https://10.10.10. 10:8089 , error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed - please check the output of the `openssl verify` command for the certificates involved; note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name. This makes us wonder, because actually the connection for the TLS connection should be established over fqdn and not over an ip address. When establishing a connection over the IP, it is also logical that the TLS name check against the host name fails, since only the FQDN and no IP address is stored in the TLS certificate. We therefore wondered why the search head addresses our index peer 10.10.10.10 via the IP and not the fqdn. The Splunk documentation states that the search head gets the list of index peers from the cluster manager. However, in the cluster manager, our index peers also report by name, at least that is what the output of the cluster master suggests: >> splunk/bin/splunk show cluster-status WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details. Replication factor met Search factor met All data is searchable Indexing Ready YES HA Mode: Disabled indexpeer01.bla.fasel 132BC25B-7774-40D9-AAED-22F9795C8E3F site2 Searchable YES Status Up Bucket Count=2412 indexpeer02.bla.fasel 9B5E23F6-3D53-4AAF-805F-DEDF3ACF9D87 site2 Searchable YES Status Up Bucket Count=2467 indexpeer03.bla.fasel CCBC4D24-025E-45FB-A68D-9C1A14219D3F site1 Searchable YES Status Up Bucket Count=2384 indexpeer04.bla.fasel D649A1C7-9E86-4E48-9B47-144C70029C15 site1 Searchable YES Status Up Bucket Count=2475 On our search heads, our search peers are listed by IP-Adress in the attribute PeerURI instead of the fqdn. How can we change it to the fqdn? Do other users have the problem as well? Or does TLS validation work even though IP addresses are listed there? Thanks! ... View more

Hi,   our splunk  universal forwarder is runnning under a non-root service account, which is defined in our central ldap.   We upgraded our universal forwarder from 7.3.7 to 8.0.5 and now our forwarder cannot monitor our file anymore.   The file to be monitored is set as following: #ls -al /tmp/ldap-group-test -rw-r-----. 1 root ldapgroup 100 Sep 1 09:27 /tmp/ldap-group-test   Our splunk service account user is a member of the group ldapgroup: #id ldapsplunk uid=100007(ldapsplunk) gid=100008(ldapsplunkgroup) groups=100008(ldapsplunkgroup),100009(ldapgroup)   In Splunk with a universal forwarder in version 8.0.5 we get a permission denied. If we use a scripted input (inputs.conf) to display the user and group context of the currently running splunk forwarder session: bin/display_groups.sh #/bin/bash hostname=$(hostname) time=$(date +%s) id=$(id) rc=$? echo "${time} - ${hostname} - ${id} - ${rc}" we get as output: 1598967093 - splunkhost.bla.fasel.de - uid=100007(ldapsplunk) gid=100008(ldapsplunkgroup) groups=100008(ldapsplunkgroup) context=system_u:system_r:unconfined_service_t:s0 - 0   As you can see, the membership of ldapgroup is missing here. It seems that during the start process of the universal forwarder, the permissions for ldap groups aren't passed correctly.   Has anyone already noticed this? Are there any workarounds other than downgrading to version 7.3? Thanks - Lorenz   ... View more

Hi Splunk community, we have an Heavy Forwarder which mostly ingests syslog data via tcp. The volume of ingested data is rather constant. In order to check whether we could get could more performance we disabled useACK in the outputs.conf of the HF. We were quite surprised that instead of increasing throughput by saving overhead, we observed a sharp drop in ingestion, including events from the internal logs of the Heavy Forwarder . We went back enabling acknowledgement after an hour. Can you give us some advice, that could explain this behvaiour? Could this be a hint, that there's a deeper problem in out Splunk infrastructure that is smoothed out by the acknowledgement feature? Thanks in advance! Kindly regards, Jens Wunder ... View more

Hello Splunk Community, in order to honour privacy policies we need to limit the searches of most users/roles of an index to events younger than seven days. In cases of an emergency we want to empower an user/role to search without restriction for the full retention period of the index. Is there any way to implement this without indexing the data twice in seperate indexes? Thanks in advance for your help! Regards, Jens Wunder ... View more

Hi, today i upgraded MLTK to version 3.4 and PSC to version 1.3 on our splunk search head running 7.0.5. During a restart of the splunk process i get the following error: $SPLUNK_HOME/bin/splunk restart [...] Checking conf files for problems... Invalid key in stanza [default] in $SPLUNK_HOME/etc/apps/Splunk_ML_Toolkit/default/scorings.conf, line 2: package (value: scorings). Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug' Done Can i ignore this error? What should i do? Thanks! ... View more