We upgraded to 8.1.2 and want to use workload manager, workload manager requires systemd.  With 8.1.x you can allow the splunk user to stop/start the systemd splunk service, which works fine however it seems to be to broad of a configuration and also allows for stopping/starting other systemd services as well.  Is there a way to lock down the polkit rule where it doesn't grant beyond the splunk service?  I'll do more research on polkit to see if I can find a way but wondering if others have done this.   sh-4.2$ sudo /apps/splunk/bin/splunk enable boot-start -systemd-managed 1 -create-polkit-rules 1 -user splunk CAUTION: The system has systemd version < 237 and polkit version > 105. With this combination, polkit rule created for this user will enable this user to manage all systemd services.Are you sure you want to continue [y/n]? y Systemd unit file installed at /etc/systemd/system/Splunkd.service. Polkit rules file installed at /etc/polkit-1/rules.d/10-Splunkd.rules. Configured as systemd managed service. sh-4.2$ sudo su - splunk splunk@qasshd$ systemctl stop amazon-ssm-agent.service splunk@qasshd$ systemctl status amazon-ssm-agent.service ● amazon-ssm-agent.service - amazon-ssm-agent Loaded: loaded (/etc/systemd/system/amazon-ssm-agent.service; enabled; vendor preset: disabled) Active: inactive (dead) since Wed 2021-02-10 22:19:39 UTC; 7s ago Process: 1130 ExecStart=/usr/bin/amazon-ssm-agent (code=exited, status=0/SUCCESS) Main PID: 1130 (code=exited, status=0/SUCCESS) splunk@qasshd$ systemctl start amazon-ssm-agent.service splunk@qasshd$ systemctl status amazon-ssm-agent.service ● amazon-ssm-agent.service - amazon-ssm-agent Loaded: loaded (/etc/systemd/system/amazon-ssm-agent.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2021-02-10 22:19:55 UTC; 3s ago Main PID: 5087 (amazon-ssm-agen) Memory: 30.6M CGroup: /system.slice/amazon-ssm-agent.service ├─5087 /usr/bin/amazon-ssm-agent └─5101 /usr/bin/ssm-agent-worker splunk@qasshd$ This is our rules file: /etc/polkit-1/rules.d/10-Splunkd.rules polkit.addRule(function(action, subject) { if (action.id == "org.freedesktop.systemd1.manage-units" && subject.user == "splunk") { return polkit.Result.YES; } }); ... View more

We recently installed the Splunk app and add-on for Infrastructure and have found that the app is very similar to Analytics Workspace, will one eventually replace the other?  Analytics Workspace comes with Splunk.  What are the differences between the two?  Outside of the entities and grouping entities I don't see many differences.   Thanks...Alisa ... View more

Using Splunk Enterprise 7.3.0 and ITSI 4.3.0 Goal is to configure adaptive thresholding for some metric based KPIs Using collectd to collect 17 CPU, memory, disk, etc metrics on Unix servers into index=unixperf_metrics We are unable to see data in the Thresholding section of the KPIs When choosing a search type of "metric" and choose from the drop downs the index and metric I can see results in the "Thresholding" section and can see overall results in the service For better search efficiency we created base searches with something like this for the CPU metrics that lists all of the names per each metric type: | mstats avg(cpu.percent.idle.value) as cpu_idle avg(cpu.percent.interrupt.value) as cpu_interrupt avg(cpu.percent.nice.value) as cpu_nice avg(cpu.percent.softirq.value) as cpu_soft avg(cpu.percent.steal.value) as cpu_steal avg(cpu.percent.system.value) as cpu_system avg(cpu.percent.user.value) as cpu_user WHERE index=unixperf_metrics by host After creating these base searches and KPI's we receive results in the service but none in the Thresholding section In opting to remove base search from the equation we selected a search type of base search, same behavior, no results under the Thresholding section however results show in the service, search is: | mstats avg(cpu.percent.system.value) as cpu_percent WHERE index=unixperf_metrics by host I think I need to address this to get adaptive thresholding to work, any ideas as to what I'm missing? ... View more

Bulk entity import worked but trying to understand why recurring ITSI imports aren't occurring: https://docs.splunk.com/Documentation/ITSI/4.3.0/Configure/Recurringimport Working on single instance Under step 5 there is a note: The recurring import search executes as splunk-system-user, which returns entities from datasets that exist in indexes that the user creating the import might not have access to. No errors when looking in _audit index in regards to splunk-system-user, how can I determine if this is the issue since I can't assign the role capabilities? Using REPLACE as the update type and log level set to DEBUG and see it run but no errors, my test is deleting an alias in one of the entities to see if it will get added. When running the saved search either scheduled or by itself it returns data as needed, it seems like the replace option isn't working. I don't have another example to compare to so I'm not sure what the logs should look like for a successful import, below is the log file. 2019-08-01 14:00:00,550 DEBUG [itsi.csv_import] [itoa_storage] [is_available] [104610] Querying if KV store is available: True 2019-08-01 14:00:00,550 INFO [itsi.csv_import] [itoa_storage] [wait_for_storage_init] [104610] KV store has been initialized. 2019-08-01 14:00:00,550 INFO [itsi.csv_import] [itsi_csv_import] [do_run] [104610] import_from_search: True 2019-08-01 14:00:00,550 INFO [itsi.csv_import] [itsi_csv_import] [do_run] [104610] import_info: {"import_from_search": "1", "service": {"titleField": "Service Title", "serviceEnabled": "1", "criticality": "", "descriptionColumns": ["Service Description"], "serviceSecurityGroup": "default_itsi_security_group", "backfillEnabled":"0"}, "search_string": "| savedsearch \"SPLUNK:firewall_entities_search\"", "service_dependents": [], "selected_services": null, "interval": "0 */2 * * *","index_latest": "now", "updateType": "replace", "index_earliest": "-60m", "log_level": "DEBUG", "entity": {"identifyingFields": ["host"], "informationalFields": ["bunit", "owner", "os", "category"], "titleField": "device_hostname", "mergeField": "undefined", "service_column": [], "fieldMapping": {}, "descriptionColumns": ["description"]}, "selectedServices": [], "service_rel": [], "template": {}} 2019-08-01 14:00:17,272 DEBUG [itsi.csv_import] [itsi_csv_import] [import_via_search] [104610] Done running search. Modular input will now try to import your entities/services. 2019-08-01 14:00:17,294 INFO [itsi.csv_import] [itoa_bulk_import_specification] [_get_fields_to_import] [104610] Fields to Import: SpecFields(entity_fields=['os', 'device_hostname', 'description', 'category', 'bunit', 'owner', 'host'], service_fields=['', 'Service Title', 'Service Description'], entity_relationship_fields=[]) 2019-08-01 14:00:17,300 INFO [itsi.csv_import] [itoa_bulk_import] [_bulk_import] [104610] CSV data load initializing mark start=1564668017.3 ... View more