I opened a support ticket, and they discovered that my Okta/SAML/SSO had a "role" (for my SH (search head)) that was not present on my DS or HF (it wasn't necessary).  However, my account was inheriting the "admin" role so IMO it shouldn't have mattered.  I added the "role" to my HF and DS and was able to proceed.   But then...it happened again in another part of the UI.  I opened another support ticket and the advice was not to use Add-On Builder in a distributed environment...which didn't make sense, what would having a distributed environment have to do with it?   What Add-On Builder does not like, I suspect, is basically any instance of Splunk with "roles" associated to accounts that are not the local/default admin and ONLY the local/default admin. I deployed a stand-alone, local, Enterprise instance on my laptop NOT integrated with Okta/SAML/SSO and everything works fine...so that is the solution 😋 ... View more

Here are the basic steps for setting up a new deployment.  Build your Deployment Server first through step 8.  You can use the WebUI on the Deployment server to upload your license file.  Build your indexers, heavy forwarders, & search head with the same steps adding items 9-13.  Be sure you also configure your Deployment server with your indexes, SSL, and forward the internal logs to your indexers.  These steps were done on Ubuntu so commands may vary slightly on other *nix flavors.  You may have other ways (systemd) to do some of the *nix admin such as the limits and THP.   login via cli & elevate to root increase system limits   vi /etc/security/limits.conf * hard nofile 64000 * hard nproc 16000 * hard fsize -1   disable THP https://www.mongodb.com/docs/manual/tutorial/transparent-huge-pages/ System V Init (service)   vi /etc/init.d/disable-transparent-hugepages #!/bin/bash ### BEGIN INIT INFO # Provides: disable-transparent-hugepages # Required-Start: $local_fs # Required-Stop: # X-Start-Before: splunk # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Disable Linux transparent huge pages ### END INIT INFO echo 'never' | tee /sys/kernel/mm/transparent_hugepage/enabled > /dev/null echo 'never' | tee /sys/kernel/mm/transparent_hugepage/defrag > /dev/null chmod 755 /etc/init.d/disable-transparent-hugepages /etc/init.d/disable-transparent-hugepages start update-rc.d disable-transparent-hugepages defaults   reboot, login, & elevate to root create the splunk user   useradd -m splunk   install splunk & configure to run as non-root "splunk" user at system boot   cd /opt wget -O splunk-9.0.4.1-419ad9369127-Linux-x86_64.tgz "https://download.splunk.com/products/splunk/releases/9.0.4.1/linux/splunk-9.0.4.1-419ad9369127-Linux-x86_64.tgz" tar zxfv splunk-9.0.4.1-419ad9369127-Linux-x86_64.tgz chown -R splunk:splunk /opt/splunk /opt/splunk/bin/splunk enable boot-start -user splunk   reboot, login, & elevate t root verify configs https://docs.splunk.com/Documentation/Splunk/9.0.4/RESTREF/RESTintrospect   ulimit -a cat /sys/kernel/mm/transparent_hugepage/enabled cat /sys/kernel/mm/transparent_hugepage/defrag ps -ef | grep splunk   switch to splunk user   su - splunk   add deploymentclient.conf   vi /opt/splunk/etc/system/local/deploymentclient.conf [deployment-client] [target-broker:deploymentServer] targetUri = https://deploymentserver.yourdomain.com:8089   add the Splunk license, restart Splunk, & check licensing   /opt/splunk/bin/splunk edit licenser-localpeer -manager_uri 'https://deploymentserver.yourdomain.com:8089' /opt/splunk/bin/splunk restart /opt/splunk/bin/splunk list licenser-localpeer   add to appropriate Deployment Server Class(es) & deploy appropriate apps enable (SH, HF, DS) or disable (indexers) the WebUI enable cooked Splunk port 9997 inputs on the indexers forward _* internal logs to the indexers define indexes enable SSL WebUI, 8089 mgmt, 9997 cooked input enable email (SMTP) for SearchHead only via the SH WebUI  ... View more

Here are some CIM mappings for the KB and the Host Detections.   [qualys:knowledgebase] FIELDALIAS-qualys_alias_1 = QID AS qid TITLE AS signature SEVERITY AS severity_id CATEGORY AS category SOLUTION AS solution CONSEQUENCE AS impact DIAGNOSIS AS description EVAL-published = substr('PUBLISHED_DATETIME',1,10) EVAL-modified = substr('LAST_SERVICE_MODIFICATION_DATETIME',1,10) EVAL-cvss = max('CVSS_BASE','CVSS_TEMPORAL','CVSS_V3_BASE','CVSS_V3_TEMPORAL') EVAL-cve = mvsort(trim(split('CVE',","))) EVAL-threat = mvsort(trim(split('THREAT_INTEL_VALUES',","))) EVAL-remote = lower('DISCOVERY_REMOTE') EVAL-patchable = lower('PATCHABLE') EVAL-pci = lower('PCI_FLAG') EVAL-bugtraq = mvsort(trim(split('BUGTRAQ_IDS',","))) EVAL-authentication = mvsort(trim(split('AUTHENTICATION',","))) EVAL-xref = mvsort(trim(split('VENDOR_REFERENCE',","))) LOOKUP-qualys_severity_lookup = qualys_severity_lookup severity_id AS SEVERITY OUTPUT severity [qualys:hostDetection] FIELDALIAS-qualys_alias_2 = QID AS qid FIRST_FOUND_DATETIME AS first_found LAST_FOUND_DATETIME AS last_found LAST_FIXED_DATETIME AS last_fixed HOST_ID AS dest_host_id IP AS dest_ip OS AS os EVAL-status = lower('STATUS') EVAL-type = lower('TYPE') EVAL-first_found_date = substr('FIRST_FOUND_DATETIME',1,10) EVAL-first_found_epoch = substr(strptime('FIRST_FOUND_DATETIME',"%F"),1,10) EVAL-last_found_date = substr('LAST_FOUND_DATETIME',1,10) EVAL-last_found_epoch = substr(strptime('LAST_FOUND_DATETIME',"%F"),1,10) EVAL-last_fixed_date = substr('LAST_FIXED_DATETIME',1,10) EVAL-last_fixed_epoch = substr(strptime('LAST_FIXED_DATETIME',"%F"),1,10) EVAL-dest_host = upper(coalesce('HOSTNAME','NETBIOS')) EVAL-dest = upper(coalesce('HOSTNAME','NETBIOS','IP')) EVAL-tags = mvsort(trim(split('TAGS',","))) LOOKUP-qualys_kb_lookup = qualys_kb_lookup qid AS QID OUTPUT published modified signature cvss cve threat patchable remote category pci bugtraq xref authentication solution impact description ... View more