I was able to manage/view XML for private user dashboards till last friday. Since today I realised that I am not able to even list those private dashboards under Settings > UI > Views, OR, Settings > All Configs. As platform admins, we promote user made objects to global sharing. I compared with my teammates listed below, we all share the same role (admin) and yet they are able to view those same private dashboards that I cannot view/list. SH Version 8.0.5 in SHC. ... View more

We are testing our upgrade from Splunk 7.3.1 to 8.1.2. Once we upgrade the SH regardless of the indexers' version 7.3.1 or 8.1.2, we are unable to run any searches. Every time we attempt a search, Splunk crashes 100% - crash logs looks like below; - Resource usage is low as in CPU IDLE 99% memory usages is less than 10%.  - systemd and bootstart enabled   [build 545206cc9f70] 2021-02-22 17:44:41 Received fatal signal 6 (Aborted). Cause: Signal sent by PID 15454 running under UID 1863478. Crashing thread: RemoteTimelineReadThread Registers: RIP: [0x00007F8BE1E67F77] gsignal + 55 (libc.so.6 + 0x34F77) RDI: [0x0000000000003C5E] .. Backtrace (PIC build): [0x00007F8BE1E67F77] gsignal + 55 (libc.so.6 + 0x34F77) [0x00007F8BE1E6934A] abort + 314 (libc.so.6 + 0x3634A) [0x000056191C24136E] ? (splunkd + 0x128F36E) [0x000056191E355416] _ZN10__cxxabiv111__terminateEPFvvE + 6 (splunkd + 0x33A3416) [0x000056191E355461] ? (splunkd + 0x33A3461) [0x000056191E355594] ? (splunkd + 0x33A3594) [0x000056191C2632C6] ? (splunkd + 0x12B12C6) [0x000056191C263724] ? (splunkd + 0x12B1724) [0x000056191D28257B] _ZN6ThreadC1EPKcz + 443 (splunkd + 0x22D057B) [0x000056191DB44181] _ZN39EventDownloadInitializeCollectionThreadC1EP20EventDownloadManager + 33 (splunkd + 0x2B92181) [0x000056191DB490A8] _ZN20EventDownloadManagerC2ERK3Str + 600 (splunkd + 0x2B970A8) [0x000056191D7DF4BF] _ZN9TimelinerC2ERK3Str + 319 (splunkd + 0x282D4BF) [0x000056191C523782] _ZN24RemoteTimelineReadThread4mainEv + 114 (splunkd + 0x1571782) [0x000056191D281F87] _ZN6Thread8callMainEPv + 135 (splunkd + 0x22CFF87) [0x00007F8BE21E1724] ? (libpthread.so.0 + 0x8724) [0x00007F8BE1F1FEED] clone + 109 (libc.so.6 + 0xECEED) Linux / ito044165 / 4.4.180-94.130-default / #1 SMP Fri Sep 4 09:11:24 UTC 2020 (e5e5142) / x86_64 C++ exception: exception_addr=0x7f8be158bfc0 typeinfo=0x56191f0bf6e8, name=15ThreadException what(): RemoteTimelineReadThread: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 20 threads active ... View more

Our Splunk  SH cluster scheduler stopping, users complaining that alerts/scheduled reporting not running or processing. We disabled and enabled the scheduler on the captain but that didnt work. We decided to switch captaincy to another and that worked - scheduling/processing resumed. Today we had reoccurrence but on a different Search head cluster - we switched captains and that remediated issue again.  Version 8.0.6 and recent change - cascading bundle replication enabled around a month ago. ... View more

We have data ingesting into Splunk via HEC token, and observed the time parsing of the event is not taking properly. Example - In the event the timestamp looks like 2020-12-01 09:59:18.0674, but in the Splunk it was capturing 12/1/20 9:59:18.000 AM. Here missing the millisecond in the Splunk time but it's not limited to the millisecond.. sometimes the second field are not correct.. We tried applying the time format and time prefix for the source and sourcetype as below, but it is not fixing the issue. TIME_PREFIX = "Date": " TIME_FORMAT = %Y-%m-%d %H:%M:%S.%4N And also tried the props.conf below; [the_sourcetype] AUTO_KV_JSON = false INDEXED_EXTRACTIONS = json TIMESTAMP_FIELDS = Date We use collector/event REST endpoint. Splunk version 7.2.8. ... View more

We are seeing delay in indexing - this started to happen after the AWS TA 5.0.3 upgrade from 4.0.6. In the TA log there are many ERROR messages which appears to be much relevant.   ... View more

Using both 8.0.1 and 8.0.6, I am unable to redeploy apps when attempting to deploy Splunk_ML_Toolkit with Splunk_SA_Scientific_Python_linux_x86_64. The same error occurs in both environments in which I have tried it: Error while deploying apps to first member, aborting apps deployment to all members: Error while updating app=Splunk_SA_Scientific_Python_linux_x86_64 on target=https://10.8.8.13:8089:Network-layer error: Connection reset by peer This occurs almost immediately after trying command below; $ splunk apply shcluster-bundle -target https://10.8.8.13:8089 ... View more

On all SearchHead cluster members with ver 8.0.2,  every day we are observing that CPU utilization grows. After roughly two days CPU load grapsh looks like "climbing". After our analysis we found that several queries are "zombied" and it looks like Splunk does not control them. These processes runs on Operating System level endlessly like consuming more and more CPU over time. In UI there is message that "Search auto-canceled" Always on the end of search.log for such process we see ; 09-28-2020 14:52:57.907 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=CANCEL 09-28-2020 14:52:57.907 INFO DispatchExecutor - User applied action=CANCEL while status=3 09-28-2020 14:52:58.906 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=CANCEL 09-28-2020 14:52:58.906 INFO DispatchExecutor - User applied action=CANCEL while status=3 09-28-2020 14:52:59.906 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=CANCEL   Please help. ... View more

As described in the subject.  Most of panels in Overview against the older version instances which are yet to be upgraded to 8.0.6 are not populated correctly.  Resource Usage tab also shows many panels and information are broken too. When I tried to run the same broken searches  seprately, it has error messages like, [indexer01] Failed to fetch REST endpoint uri=https://127.0.0.1:8089/services/server/info?count=0&strict=false from server https://127.0.0.1:8089. Check that the URI path provided exists in the REST API     ... View more