Splunk AI Toolkit / Splunk_SA_Scientific_Python - challenging upgrade

[NullZero]

IHAC with a C13 SVA instance, I recently upgraded them from Splunk MLTK 5.x to Splunk AI Toolkit 6.0.2 to modernise and neutralise the recently published CVE exposure. Splunk Enterprise is on version 10.2.6 on RHEL 8 so all recent and properly up-to-date. You also have a dependency of the 'Splunk_SA_Scientific_Python_linux_x86_64'  with this new version.

At first there was a problem pushing the large bundle for the SA from the SH Deployer to the SHC, and I received the following error which is easily solvable in the same way that a large ES image install is fixed:

Error while deploying apps to target=https://<server>:8089 with members=3: Error while updating app=Splunk_SA_Scientific_Python_linux_x86_64 on target=https://<server>:8089: Non-200/201 status_code=413; {\"messages\":[{\"type\":\"ERROR\",\"text\":\"Content-Length of 2147496687 too large (maximum is 2147483648)\"}]}, Error while updating app=Splunk_SA_Scientific_Python_linux_x86_64 on target=https://<server>:8089: Non-200/201 status_code=413; {\"messages\":[{\"type\":\"ERROR\",\"text\":\"Content-Length of 2147496687 too large (maximum is 2147483648)

Fix 1: Updated the Search Head TA and pushed this from the deployer after temporarily removing the SA from the deployment folder. I ended up making the changes on the deployer and the SH as it didn't work the first time and nothing lost

web.conf
[settings]
max_upload_size = 4096

 
This change got me further forward but it would still not allow the large push and I saw a new error. I also noted the the size on disk on the Deployer was approx. 3.6GB and on the SHC 1.5GB.

Error while deploying apps to target=https://<server>:8089 with members=3: Error while updating app=Splunk_SA_Scientific_Python_linux_x86_64 on target=https://<server>:8089: Error in JSON response: Unexpected EOF, Error while updating app=Splunk_SA_Scientific_Python_linux_x86_64 on target=https://<server>:8089: Error in JSON response: Unexpected EOF

Fix 2: update server.conf and http stanza:

server.conf
[httpServer]
max_content_length = 5000000000

 I did make a silly mistake here and had a zero missing making it 0.5GB upload as opposed to 5GB. After that the push worked and the error message on the dependency in the UI went away (the message related to the requirement for the Splunk_SA_Scientific_Python).

Final problem:
I am unable to see any models in the UI, my understanding is that the models are just the .csv files in the lookups folder in the AI toolkit app, but I have not personally been the end user.

- I updated app.conf on the deployer for the AI toolkit and added:
deployer_lookups_push_mode =always_overwrite
- This did push the updated csv's and I can see all the dates match.
- I have checked the docs and I can only see remediation action required for windows
- It could conceivably be related to RBAC, but I've change one item and I'm not seeing a difference as an admin.
- I have also just noticed that there is a new version as of 25 August, I will upgrade to that to eliminate possible problems: Splunk AI Toolkit | Splunkbase

[Coach_CND · Accepted solution]

Looking at that error I would recommend reviewing if the roles that need the ability to list machine learning models have the list_models capability.

Solution: You must have thelist_models capability turned on to view available machine learning models when you upgrade to AITK version 6.0.2. The Audit - MLTK models saved search is configured to run under the admin namespace, so you must turn on the list_models capability for the admin user.

Troubleshoot upgrading AI Toolkit in Splunk Enterprise Security | Platform (last updated 2026-09-01T...

[NullZero]

Thanks @Coach_CND this capability "list_models" was indeed the fix. 

[NullZero]

For the benefit of the community I found this further data:

Troubleshoot upgrading AI Toolkit in Splunk Enterprise Security

Machine learning models not displayed upon upgrade

Issue: Splunk Enterprise Security version 8.7 is compatible with Splunk AI Toolkit (AITK) version 6.0.2 and Python Scientific Computing (PSC) version 3.3.3. However, after upgrading to AITK 6.0.2, you can't view machine learning models by default. The Machine learning models table on the Machine learning audit dashboard is empty and custom searches that list machine learning models stop working.

Cause: The list_models capability must be turned on for roles that need the ability to list machine learning models.

Solution: You must have thelist_models capability turned on to view available machine learning models when you upgrade to AITK version 6.0.2. The Audit - MLTK models saved search is configured to run under the admin namespace, so you must turn on the list_models capability for the admin user.

I would note that the capability is not listed generally in Splunk by default:

Define roles on the Splunk platform with capabilities - Splunk 10.4

[NullZero]

Still no improvement with the updated version from Splunkbase