Since I realized it existed, I've setup my environment to source the $SPLUNK_HOME/share/splunk/cli-command-completion.sh script to allow tab completion of Splunk commands. Recently, we upgraded to 8.2.2 after previously being on 8.0.3. After the upgrade, the sourcing of the file no longer works, giving the following 2 stderr messages: cli-command-completion.sh: line 83: verb_to_objects: bad array subscript cli-command-completion.sh: line 85: verb_to_objects[$verb]: bad array subscript It looks like the script was originally a Splunk Answers post by a Splunk Dev that was later included in the Splunk distribution, and has not changed since then: https://community.splunk.com/t5/Deployment-Architecture/CLI-command-completion-Yes-and-here-s-how-For-bash-4-0-and/m-p/82552 However it looks like @V_at_Splunk is no longer active on the community and likely no longer at Splunk, their last post being in 2014. Is anyone still using this script? Has anyone run into these issues and determined their cause? I suspect something the script references changed, but I'm unsure what was changed. This was such a nice QoL thing to have it'd be a shame if I had to let it die. ... View more

I have a multi-site cluster, and am planning on decommissioning one to transform it into a single-site cluster. Looking over these two guides: https://docs.splunk.com/Documentation/Splunk/8.0.2/Indexer/Decommissionasite https://docs.splunk.com/Documentation/Splunk/8.1.2/Indexer/Converttosinglesite And trying to see how to do both, preferably at the same time. When converting to a single-site, it states to stop the entire cluster, update the configurations, then start the cluster back up. Is there any issue with doing the configurations changes necessary for decommissioning the old site while everything is offline, and only bringing up the remaining site? Basically, current plan is: Stop all nodes Update the Manager Configs Set multi-site to false Set single site search/rep factors Remove site attribute Remove available_sites attribute/site mappings Update Search Head Configs Set multi-site to false Remove site attribute Start nodes that are remaining from new site Would this work, or would it cause conflicts in replication somehow? Do I need to use Splunk commands on the cluster manager to remove the old indexers? ... View more

Question -  If I wanted to prevent SAML/SSO configurations from replicating to other SHs in a cluster, could I use the 'conf_replication_blacklist.<name>' or something similar to exclude the authentication.conf? Or would that cause more issues outside of just preventing SAML/SSO configs to be unsyncd? Context -  We are migrating from onprem servers to AWS servers. The current configurations for SSO/SAML only work for the onprem servers, and we will need new configs for the AWS servers. The configs are in the etc/system/local/authentication.conf, so already at highest precendence. However, while working on those configurations we don't want to break the working SSO for onprem. We don't want to make it a separate cluster, cause then we'd have to get all the searches/lookups replicated across some other way. I came across the 'conf_replication_summary.blacklist' and 'conf_replication_include.<conf_file_name> = <boolean>' in the server.conf spec, and was wondering if anyone has any experience using these for authentication.conf and if there are complications I should be aware of? Cause if we could use these to temporarily pause the replication with no real ill effects, that'd be great. ... View more

I've added the Splunk TA for Unix/Linux to my indexers and have been trying to get iostat data feeding in from the indexers. The script run, but only the headers are present Sample _raw: Device rReq_PS wReq_PS rKB_PS wKB_PS avgWaitMillis avgSvcMillis bandwUtilPct Running 'iostat -xk 1 2' as the script is setup to do does return results. Running the script normally/debug only shows the headers output to screen, but the debug file show it pulling recording the statistics: debug--iostat.sh file Output:   Linux 4.18.0-147.8.1.el8_1.x86_64 (hostname) 06/07/2021 _x86_64_ (64 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 16.16 0.00 3.60 1.60 0.00 78.63 Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sda 52.92 212.80 1675.92 26670.15 7.49 152.92 12.40 41.81 2.40 0.46 0.12 31.67 125.33 0.35 9.19 sdc 138.11 22.04 13308.18 5149.92 0.09 0.26 0.07 1.15 0.33 2.62 0.03 96.36 233.70 0.57 9.19 sdb 548.22 401.94 56049.28 19294.08 0.87 665.61 0.16 62.35 0.10 0.12 0.11 102.24 48.00 0.08 7.70 dm-0 42.67 324.87 1604.43 26506.50 0.00 0.00 0.00 0.00 2.36 0.27 0.06 37.60 81.59 0.24 8.96 dm-1 17.87 40.86 71.48 163.44 0.00 0.00 0.00 0.00 3.01 1.05 0.10 4.00 4.00 0.07 0.39 dm-2 0.00 0.00 0.00 0.21 0.00 0.00 0.00 0.00 40.68 4.93 0.00 81.97 239.35 0.26 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 19.52 0.00 3.07 1.08 0.00 76.33 Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sda 0.00 11.00 0.00 150.00 0.00 0.00 0.00 0.00 0.00 0.09 0.00 0.00 13.64 1.00 1.10 sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 6825.00 50.00 27312.00 232.00 0.00 0.00 0.00 0.00 0.32 0.04 0.83 4.00 4.64 0.15 99.80 dm-0 0.00 11.00 0.00 150.00 0.00 0.00 0.00 0.00 0.00 0.09 0.00 0.00 13.64 1.09 1.20 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Cmd = [iostat -xk 1 2]; | awk 'BEGIN {print "Device rReq_PS wReq_PS rKB_PS wKB_PS avgWaitMillis avgSvcMillis bandwUtilPct"} /^$/ {next} /^Device:/ {for (i = 1; i <= NF; i++) {if ($i == "svctm") { svctm=i; } else if ($i == "%util") {putil=i;} } reportOrd++; next} (reportOrd<2) {next} {device=$1; rReq_PS=$4; wReq_PS=$5; rKB_PS=$6; wKB_PS=$7; avgWaitMillis=$10; avgSvcMillis=$svctm; bandwUtilPct=$putil} {printf "%-10s %11s %11s %12s %12s %13s %13s %13s\n", device, rReq_PS, wReq_PS, rKB_PS, wKB_PS, avgWaitMillis, avgSvcMillis, bandwUtilPct}' header="Device rReq_PS wReq_PS rKB_PS wKB_PS avgWaitMillis avgSvcMillis bandwUtilPct"    The only thing I can think is that there is some issue with the awk statement maybe? So I be manually adjusting the script? ... View more

I'm at my wits end here, everything seems to indicate what I'm doing should work, yet it's not.  I have Azure firewall logs feeding in through a storage account using the Microsoft Cloud Services app. These come in as standard JSON, which is being extracted fine by Splunk. There is a nested field in the JSON, "properties.msg", that has the actual firewall log message including source/destination information, IPs/ports, whether it was allowed/denied, and what firewall rule was referenced.  For reference, this thread discusses a nearly similar case/problem -https://community.splunk.com/t5/Splunk-Search/Azure-Firewall-Log-Field-Extraction-Help/m-p/411148  The added wrinkle I have is that I am trying to get the fields extracted to work with CIM data models, not just get  the extractions as results from a search. This honestly seemed easy enough, but for some reason none of my field extractions are working. Here are some facts/things I have tried This is in Splunk Cloud I created a regex to extract all the fields from the properties.msg to named capture groups The regex shows correct in Regex101 The regex extracts all the fields if used in the 'rex' command in search Using the regex inside the Field Extractor tool and checking with preview function shows the fields extracted I've saved the extraction as being shared Globally, Private, and App only (even tried different apps other than search) I've tried saving as a inline extraction, and as a transform applying to both the _raw and individual properties.msg as SOURCE_KEY I'm not seeing any errors or warnings when trying to do any of these changes that would make me thing something was wrong None of this seems to work, none of the fields are extracted. I tried doing a field alias for 'properties.msg' to 'msg', and that worked so it's not like its  (but didn't help me because I still can't extract the data from within that message.) I honestly don't get how I can see the regex working in the Field Extractor, hit 'Save', see it saved in the configurations, but not extract fields. EDIT: Sample _raw log (more in updated link posted above) { "category": "AzureFirewallApplicationRule", "time": "2021-05-04T15:41:59.8967610Z", "resourceId": "/SUBSCRIPTIONS/REDACTED/RESOURCEGROUPS/REDACTED/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/SOMEFW", "operationName": "AzureFirewallApplicationRuleLog", "properties": {"msg":"HTTPS request from 192.168.0.1:8888 to subdomain.x99.blob.storage.azure.net:443. Action: Allow. Rule Collection: AllowOutbound. Rule: AllowOutbound-AA-AA-A"}} { "category": "AzureFirewallApplicationRule", "time": "2021-05-04T15:41:58.6369780Z", "resourceId": "/SUBSCRIPTIONS/REDACTED/RESOURCEGROUPS/REDACTED/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/SOMEFW", "operationName": "AzureFirewallApplicationRuleLog", "properties": {"msg":"HTTPS request from 192.168.0.1:8888 to subdomain.x99.blob.storage.azure.net:443. Action: Allow. Rule Collection: AllowOutbound. Rule: AllowOutbound-AA-AA-A"}} { "category": "AzureFirewallNetworkRule", "time": "2021-05-07T15:05:59.8277330Z", "resourceId": "/SUBSCRIPTIONS/REDACTED/RESOURCEGROUPS/REDACTED/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/SOMEFW", "operationName": "AzureFirewallNetworkRuleLog", "properties": {"msg":"TCP request from 192.168.0.1:8888 to 8.8.8.8:8888. Action: Deny. "}} Regex \"(?<protocol>\w+)\s[rR]equest\D+(?<src>[^\:]+)\:(?<src_port>\d+) to (?<dest>[^\:]+)\:((?<dest_port>\d+))?\.\sAction\: (?<action>\w+)\.(?: Rule Collection\: (?<cat>\w+)\. Rule\: (?<rule>[^\"]+))? ... View more