topic Re: Why is splunk UF version 9.0.1 stuck while accepting license? in Splunk Enterprise

Why is Splunk UF version 9.0.1 stuck while accepting license?

nbommu — Thu, 15 Dec 2022 14:43:07 GMT

We have existing UF 8.2.2 in all instance and managed by ansible, when we are trying upgrade to 9.0.1 ansible stuck which accepting license, if I run that command manually on the target host, it is working fine, I think ansible is waiting for response after ran the command on the target hots which has changed with version 9.0.1.

task to accept license:

- name: Accept Splunk license and set up init script command: cmd: /opt/splunkforwarder/bin/splunk status --accept-license --answer-yes --no-prompt

Re: Why is splunk UF version 9.0.1 stuck while accepting license?

schose — Wed, 14 Dec 2022 12:11:41 GMT

Hi,

Same here.. for some reason the update script tries to upgrade mongo-db on UFW and get stuck. seems like a bug.

disable kvstore in server.conf works for me.

https://community.splunk.com/t5/Splunk-Enterprise/Splunk-9-Universal-Forwarder-getting-quot-App-Key-Value-Store/m-p/613838#M13969

regards,

Andreas

Re: Why is splunk UF version 9.0.1 stuck while accepting license?

nbommu — Wed, 14 Dec 2022 14:30:36 GMT

thanks for checking Andreas, looks like I'm not alone 🙂

I opened kvstore bug as well as ansible, let see if we get quick workaround on them.

Thanks

Niranjan

Re: Why is splunk UF version 9.0.1 stuck while accepting license?

schose — Wed, 14 Dec 2022 16:16:19 GMT

Hi Niranjan,

Mongo is no component started by UFW but only by Splunk Enterprise.. So this is surely a bug.. it also see to make a difference if you use .tgz or .deb.

I had the issue using .tgz. My workaround is this task:

- name: ensure kvstore is disabled on UFW ini_file: dest: /opt/splunkforwarder/etc/system/local/server.conf section: kvstore option: disabled value: true

regards,

Andreas

Re: Why is splunk UF version 9.0.1 stuck while accepting license?

nbommu — Wed, 14 Dec 2022 16:30:40 GMT

thanks Andreas.

I was able to start Splunk UF ver 9.0.1 after disabling kvstore and accepting license manually, but stuck while accepting license via anisble, will post if I have more info from splunk support on this.

thanks

Niranjan

Re: Why is splunk UF version 9.0.1 stuck while accepting license?

nbommu — Wed, 14 Dec 2022 21:51:08 GMT

Hi Andreas,

how you overcome that issue with accepting license via ansible task or do you have a workaround?

thanks

Niranjan

Re: Why is splunk UF version 9.0.1 stuck while accepting license?

schose — Thu, 15 Dec 2022 14:31:04 GMT

Hi Niranjan,

yes, i simply run

- name: accept license tags: - install shell: /opt/splunkforwarder/bin/splunk status --accept-license --answer-yes become: yes when: - splunk_path.stat.exists == false or current_version.rc != 0 ignore_errors: true

before..

cheers,

Andreas

Re: Why is splunk UF version 9.0.1 stuck while accepting license?

nbommu — Thu, 15 Dec 2022 19:39:39 GMT

I have been trying with version 9.0.0, 9.0.1 and 9.0.2, all are stuck while accepting license via ansible task but when I run that command manually on that target host it is working fine, I think with version 9 there are some changes in the splunk command causing this.

rpm I have been using:

splunkforwarder-9.0.2-17e00c557dc1-linux-2.6-x86_64.rpm

thanks

NIranjan

Re: Why is splunk UF version 9.0.1 stuck while accepting license?

schose — Fri, 16 Dec 2022 08:39:15 GMT

Hi NIranjan,

can you send an ansible-playbook debug while the task is hanging? I'm using .tgz files as, .rpm always seems a blackbox for me - i cannot see from outside what post scripts are running in. there.

best regards,

Andreas

Re: Why is Splunk UF version 9.0.1 stuck while accepting license?

VatsalJagani — Fri, 16 Dec 2022 15:28:28 GMT

@nbommu , @schose - Note that there was a issue like that with Splunk 9.0.0 and 9.0.1 which was fixed in 9.0.2 or 9.0.3.

Release notes - https://docs.splunk.com/Documentation/Splunk/9.0.1/ReleaseNotes/KnownIssues

Please consider upvoting/accepting the answer if this helps!!!

Re: Why is Splunk UF version 9.0.1 stuck while accepting license?

likedasplunk — Tue, 03 Jan 2023 18:03:28 GMT

Yeah,

9.0.3 still has this exact same issue. This issue is definitely NOT fixed.

Re: Why is splunk UF version 9.0.1 stuck while accepting license?

nbommu — Tue, 03 Jan 2023 23:16:54 GMT

definitely something has changed with version 9 splunk binary, when we run license task from ansible its hung forever and I don't see an obvious stdout to check what is going on there, interestingly it works fine with version 8+

Re: Why is Splunk UF version 9.0.1 stuck while accepting license?

VatsalJagani — Wed, 04 Jan 2023 09:52:52 GMT

@likedasplunk - Have you tried disabling the kvstore from the server.conf on the UF as suggested by @schose ?

Re: Why is Splunk UF version 9.0.1 stuck while accepting license?

likedasplunk — Wed, 04 Jan 2023 13:37:23 GMT

@VatsalJagani

I did try disabling kvstore in server.conf and it did not fix the issue.

Re: Why is splunk UF version 9.0.1 stuck while accepting license?

likedasplunk — Wed, 04 Jan 2023 21:47:27 GMT

I'm also having the same issue with 9.0.3. I also replied to this thread:

https://community.splunk.com/t5/Splunk-Enterprise/Splunk-9-Universal-Forwarder-getting-quot-App-Key-Value-Store/m-p/613899/highlight/false#M13975

Re: Why is Splunk UF version 9.0.1 stuck while accepting license?

nbommu — Sat, 07 Jan 2023 13:46:46 GMT

I opened a support ticket with all the details, they are redirecting to ansible since it was stuck there, even I clearly mentioned that it is not ansible issue and there is something to do with version 9, support is eager to close the ticket instead of understanding the issue and debug 🙂

Re: Why is Splunk UF version 9.0.1 stuck while accepting license?

likedasplunk — Mon, 09 Jan 2023 13:43:50 GMT

@nbommu This isn't an issue with Ansible either. I use the installer in a bash script and it does the same thing. I try to just install and use the switches to accept the license manually and it does the same thing.

It's a bug in the nix UF installer.

Re: Why is Splunk UF version 9.0.1 stuck while accepting license?

Jordan1 — Wed, 11 Jan 2023 14:45:44 GMT

Did you find a solution to this?

As I'm experiencing the same issue

Re: Why is Splunk UF version 9.0.1 stuck while accepting license?

Jordan1 — Wed, 11 Jan 2023 14:48:12 GMT

Hi all,

Has anyone found a solution to this issue as I've had a similar experience. I'm using Tanium to upgrade Splunk UF's and getting this same issue of the license not getting accepted on Linux.

I also opened a Splunk support case which got closed due to Splunk saying the issue was with Tanium.

I have tried adding the KVStore stanza to the server.conf and disabling = true but this hasn't made any difference.

Re: Why is Splunk UF version 9.0.1 stuck while accepting license?

likedasplunk — Wed, 11 Jan 2023 14:50:24 GMT

@Jordan1

I did not.

Splunk support is actually telling me that they don't have any issues in their "lab". I've sent over all the documentation and screenshots that I can take. I have a meeting with them some time tomorrow, but there's not doubt that this is still an issue.

The suggested work arounds here do not work. I'm almost certain that this is a bug in the UF installer.