I see below log in operation_install showing continuous failure to connect to https://gravity-site.kube-system.svc.cluster.local:3009/healthz. ================ Wed Nov 10 02:40:41 UTC [INFO] [DAPD02] Executing postInstall hook for site:6.1.48. Created Pod "site-app-post-install-125088-zqsmd" in namespace "kube-system". Container "post-install-hook" created, current state is "waiting, reason PodInitializing". Pod "site-app-post-install-125088-zqsmd" in namespace "kube-system", has changed state from "Pending" to "Running". Container "post-install-hook" changed status from "waiting, reason PodInitializing" to "running". ^[[31m[ERROR]: failed connecting to https://gravity-site.kube-system.svc.cluster.local:3009/healthz Get https://gravity-site.kube-system.svc.cluster.local:3009/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) ^[[0mContainer "post-install-hook" changed status from "running" to "terminated, exit code 255". Container "post-install-hook" restarted, current state is "running". ^[[31m[ERROR]: failed connecting to https://gravity-site.kube-system.svc.cluster.local:3009/healthz Get https://gravity-site.kube-system.svc.cluster.local:3009/healthz: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) ^[[0mContainer "post-install-hook" changed status from "running" to "terminated, exit code 255". Container "post-install-hook" changed status from "terminated, exit code 255" to "waiting, reason CrashLoopBackOff". ================ The gravity cluster status after the installation failure: ================ [root@DAPD02 crashreport]# gravity status Cluster name: charmingmeitner2182 Cluster status: degraded (application status check failed) Application: dsp, version 1.2.1 Gravity version: 6.1.48 (client) / 6.1.48 (server) Join token: b9b088ce63c0a703ee740ba5dfb380d Periodic updates: Not Configured Remote support: Not Configured Last completed operation: * 3-node install ID: 46614e3c-fcd1-4974-8cd7-dc404d1880b Started: Wed Nov 10 02:33 UTC (1 hour ago) Completed: Wed Nov 10 02:35 UTC (1 hour ago) Cluster endpoints: * Authentication gateway: - 10.69.80.1:32009 - 10.69.80.2:32009 - 10.69.89.3:32009 * Cluster management URL: - https://10.69.80.1:32009 - https://10.69.80.2:32009 - https://10.69.89.3:32009 Cluster nodes: Masters: * DAPD02 / 10.69.80.1 / master Status: healthy [!] overlay packet loss for node 10.69.89.3 is higher than the allowed threshold of 20% (current packet loss at 100%) [!] overlay packet loss for node 10.69.80.2 is higher than the allowed threshold of 20% (current packet loss at 100%) Remote access: online * DWPD03 / 10.69.80.2 / master Status: healthy [!] overlay packet loss for node 10.69.80.1 is higher than the allowed threshold of 20% (current packet loss at 100%) [!] overlay packet loss for node 10.69.89.3 is higher than the allowed threshold of 20% (current packet loss at 100%) Remote access: online * DDPD04 / 10.69.89.3 / master Status: healthy [!] overlay packet loss for node 10.69.80.2 is higher than the allowed threshold of 20% (current packet loss at 100%) [!] overlay packet loss for node 10.69.80.1 is higher than the allowed threshold of 20% (current packet loss at 100%) Remote access: online ================ ... View more

Functionality worked before the upgrade. After upgrade logon shows "Tab-completion of "splunk <verb> <object> is available". Like, $ source ./setSplunkEnv Tab-completion of "splunk <verb> <object>" is available.   But when utilized it returns "Splunk re-bash: completion: function 'fSplunkComplete' not found. File is in "/opt/splunk/bin/setSplunkEnv" "/opt/splunk/share/splunk/cli-command-completion.sh" location and referenced files are both available. ... View more

pushed apps to the search head cluster to 3 Searchheads 2 of them working good but one searched URL is not accessible and cannot run searches. The searchhead appeared to be in restarting loop, logs as below;   ... View more

I was able to manage/view XML for private user dashboards till last friday. Since today I realised that I am not able to even list those private dashboards under Settings > UI > Views, OR, Settings > All Configs. As platform admins, we promote user made objects to global sharing. I compared with my teammates listed below, we all share the same role (admin) and yet they are able to view those same private dashboards that I cannot view/list. SH Version 8.0.5 in SHC. ... View more

We are testing our upgrade from Splunk 7.3.1 to 8.1.2. Once we upgrade the SH regardless of the indexers' version 7.3.1 or 8.1.2, we are unable to run any searches. Every time we attempt a search, Splunk crashes 100% - crash logs looks like below; - Resource usage is low as in CPU IDLE 99% memory usages is less than 10%.  - systemd and bootstart enabled   [build 545206cc9f70] 2021-02-22 17:44:41 Received fatal signal 6 (Aborted). Cause: Signal sent by PID 15454 running under UID 1863478. Crashing thread: RemoteTimelineReadThread Registers: RIP: [0x00007F8BE1E67F77] gsignal + 55 (libc.so.6 + 0x34F77) RDI: [0x0000000000003C5E] .. Backtrace (PIC build): [0x00007F8BE1E67F77] gsignal + 55 (libc.so.6 + 0x34F77) [0x00007F8BE1E6934A] abort + 314 (libc.so.6 + 0x3634A) [0x000056191C24136E] ? (splunkd + 0x128F36E) [0x000056191E355416] _ZN10__cxxabiv111__terminateEPFvvE + 6 (splunkd + 0x33A3416) [0x000056191E355461] ? (splunkd + 0x33A3461) [0x000056191E355594] ? (splunkd + 0x33A3594) [0x000056191C2632C6] ? (splunkd + 0x12B12C6) [0x000056191C263724] ? (splunkd + 0x12B1724) [0x000056191D28257B] _ZN6ThreadC1EPKcz + 443 (splunkd + 0x22D057B) [0x000056191DB44181] _ZN39EventDownloadInitializeCollectionThreadC1EP20EventDownloadManager + 33 (splunkd + 0x2B92181) [0x000056191DB490A8] _ZN20EventDownloadManagerC2ERK3Str + 600 (splunkd + 0x2B970A8) [0x000056191D7DF4BF] _ZN9TimelinerC2ERK3Str + 319 (splunkd + 0x282D4BF) [0x000056191C523782] _ZN24RemoteTimelineReadThread4mainEv + 114 (splunkd + 0x1571782) [0x000056191D281F87] _ZN6Thread8callMainEPv + 135 (splunkd + 0x22CFF87) [0x00007F8BE21E1724] ? (libpthread.so.0 + 0x8724) [0x00007F8BE1F1FEED] clone + 109 (libc.so.6 + 0xECEED) Linux / ito044165 / 4.4.180-94.130-default / #1 SMP Fri Sep 4 09:11:24 UTC 2020 (e5e5142) / x86_64 C++ exception: exception_addr=0x7f8be158bfc0 typeinfo=0x56191f0bf6e8, name=15ThreadException what(): RemoteTimelineReadThread: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 20 threads active ... View more

Our Splunk  SH cluster scheduler stopping, users complaining that alerts/scheduled reporting not running or processing. We disabled and enabled the scheduler on the captain but that didnt work. We decided to switch captaincy to another and that worked - scheduling/processing resumed. Today we had reoccurrence but on a different Search head cluster - we switched captains and that remediated issue again.  Version 8.0.6 and recent change - cascading bundle replication enabled around a month ago. ... View more

We have data ingesting into Splunk via HEC token, and observed the time parsing of the event is not taking properly. Example - In the event the timestamp looks like 2020-12-01 09:59:18.0674, but in the Splunk it was capturing 12/1/20 9:59:18.000 AM. Here missing the millisecond in the Splunk time but it's not limited to the millisecond.. sometimes the second field are not correct.. We tried applying the time format and time prefix for the source and sourcetype as below, but it is not fixing the issue. TIME_PREFIX = "Date": " TIME_FORMAT = %Y-%m-%d %H:%M:%S.%4N And also tried the props.conf below; [the_sourcetype] AUTO_KV_JSON = false INDEXED_EXTRACTIONS = json TIMESTAMP_FIELDS = Date We use collector/event REST endpoint. Splunk version 7.2.8. ... View more

We are seeing delay in indexing - this started to happen after the AWS TA 5.0.3 upgrade from 4.0.6. In the TA log there are many ERROR messages which appears to be much relevant.   ... View more

Using both 8.0.1 and 8.0.6, I am unable to redeploy apps when attempting to deploy Splunk_ML_Toolkit with Splunk_SA_Scientific_Python_linux_x86_64. The same error occurs in both environments in which I have tried it: Error while deploying apps to first member, aborting apps deployment to all members: Error while updating app=Splunk_SA_Scientific_Python_linux_x86_64 on target=https://10.8.8.13:8089:Network-layer error: Connection reset by peer This occurs almost immediately after trying command below; $ splunk apply shcluster-bundle -target https://10.8.8.13:8089 ... View more