there was another script on the HF that updated the kvstore on the shc... so that was the mystery Thanks ... View more

@livehybrid  Thank you for the sanity check, I was not understanding how this would have worked, I was not familiar if that sort of replication was even possible (HF to SHC). Apparently some other mechanism (to transfer the file.csv to the SHC) is missing, broken, or unknown at this time. I will keep you posted Thanks ... View more

@isoutamo  Yes you are correct.  The acceleration detail has an Summary Id , which does correspond to the savedsearch_name  _ACCELERATE_<redacted>_search_nobody_<Summary Id>_ACCELERATE_ This confirms the issue is the License Usage Data Cube  cube report/acceleration. I will need to adjust the search resources to prevent the skipping. Thank you!!! ... View more

Thank you, I am aware of that modal in MC but it gives me the same arcane name for example  >>> _ACCELERATE_111111-22222-333-4444-123456789_search_nobody_123456978_ACCELERATE_" However, the origin host is my dedicated MC splunk server and there is only 1 accelerate report icon listed for >License Usage Data Cube, so I assume that is the culprit.    But why is it skipping?  I clicked the accelerate option, perhaps I need to adjust the max scheduled searches? Yes I found a number of garbage scheduled reports from years ago eating up resources and starving the accelerated report for the License Usage Data Cube.   I incorrectly assumed that report would have priority to resources. Thank you for your help. ... View more

I think all the suggestions that spawned from my original question are worth considering in the context of the individual environments a user might have.  For instance, I was not able to spin up a duplicate SHC or IDXC and thus chose a different option.    However, I suggest that new threads be created for each solution, with a detailed explanation.  This thread is getting a bit long. ... View more

@iankitkumar  Yes I did get it to work.  But depends on how the disks and mount points are setup on the existing host.  If the OS disk and Splunk disk are separate with distinct Volume Groups then a disk swap should work.   For example, on a linux host where sda is the OS disk and sdb is the Splunk disk with mountpoint /opt/splunk.   Here are the steps.   Note: I install with .tgz files not rpm.  Of course test in your dev environment 1st!! 1 install same version of Splunk on the new host with new Linux OS version (I use same username/pwd that I used to install Splunk on the existing host) 2 enable boot-start with systemd managed and verify install runs correctly 3 shut down the old and new hosts 4 swap the disk 5 rename/reIP and open required firewalld ports (if needed for your environment) to verify its working. IF you cannot do a disk swap (because the host is not configured to do so) you can always tar up /opt/splunk and untar over /opt/splunk on the new host (on top of the fresh install of splunk (same version)).  Documentation describes this method >>>  Migrate a Splunk Enterprise instance from one physical machine to another - Splunk Documentation Your Splunk Enterprise installation is on an operating system that either your organization or Splunk no longer supports, and you want to move it to an operating system that does have support. Good Luck!   ... View more

@siemless  This may be best to discuss in the Slack Users but I could not find you so I will respond here. In some cases, I have boxes where the /opt/splunk dir is mounted to a separate drive w/ mount point /opt/splunk.  In that case you can just swap the disk, I learned this method from AWS support.  But that takes preplanning. In some cases, I have boxes that are jacked up, either volume issues across multiple disks or  just not setup to swap disks.  In that case you can use the Splunk docs >>> https://docs.splunk.com/Documentation/Splunk/9.2.1/Installation/MigrateaSplunkinstance I argued with Splunk about the documentation steps but they claim the steps are correct, although I still believe confusing.  FWIW this is what I did... 1 >Create a new host with new OS (in my case I  rename /re-IP to the original afterward). 2 > Install the same version of Splunk on new host (I used a .tar), set systemd, set same admin pwd, then stop Splunkd, maybe test a restart and reboot, to verify. 3 > Stop Splunkd on old host, tar up /opt/splunk, copy over the old.tar to new box, untar over the new install, then start Splunkd. That worked for me, and going fwd all new hosts will be configured for the disk-swappable process. Good luck ... View more

Ok thank you for the clarity... I think someone should revise those steps then, its ambiguous. ... View more

Thank you for the reply. RE: "swap" method, yeah I thought about that and also share your apprehension. RE: " deploy new component", yeah I agree with that method for idxc peers and shc members... But check this out... please LMK what you think Per Splunk docs >>> docs.splunk.com/Documentation/Splunk/9.2.0/Installation/MigrateaSplunkinstance "Migrate a Splunk Enterprise instance from one physical machine to another" "When to migrate" "Your Splunk Enterprise installation is on an operating system that either your organization or Splunk no longer supports, and you want to move it to an operating system that does have support." "How to migrate" The Steps say >>> Stop Splunk Enterprise services on the host from which you want to migrate. Copy the entire contents of the $SPLUNK_HOME directory from the old host to the new host. Copying this directory also copies the mongo subdirectory. Install Splunk Enterprise on the new host. The way I read this is... 1) Stop Splunk on the old box 2) tar up the /opt/splunk on the old box e.g. > tar -cjvf $(hostname)_splunk-og.tar.bz2 --exclude=./var/* --exclude=./$(hostname)*bz2 ./ 3) move and untar the .bz2 file on the new box in /opt  e.g. > tar -xjvf <hostname>_splunk-og.tar.bz2 -C /opt/splunk 4 ) install a clean copy (downloaded from Splunk) of the same version of Splunk on top of the old copies Apparently someone that documented this believes this is the way to go... What do you think? RE: my --exclude=./var/* that is for boxes that don't contain indexed data RE: my --exclude=./$(hostname)*bz2./  this is because I am running the tar from /opt/splunk dir Thank you ... View more

The numbers are not exact, from the DS Forwarder Management > 1275, dc(h) from metrics > 1287, and the total stats count from the final query > 1166  so its not accurate.  I will need to create a lookup of UFs. Thank you for your support. ... View more

      index=_internal source=*license_usage.log earliest=-1d@d latest=now [search index=_internal source=*metrics.log fwdType=uf earliest=-1d@d latest=now | rename hostname as h | fields h] | stats sum(b) as total_usage_bytes by h | eval total_usage_gb = round(total_usage_bytes/1024/1024/1024, 2) | fields - total_usage_bytes | addcoltotals label="Total" labelfield="h" total_usage_gb   I think this is what I wanted, unless someone thinks its inaccurate?  Please advise. TY   ... View more

Thank you for the reply.  I also looked at this log but it requires curating an exact list of the UFs, bc I have some pollution, e.g. h= HFs, SC4S, etc.  The license_usage log may be the best route if I can put together a lookup of just UFs. ... View more

Thank you for your reply, do you have a method of querying to get an answer for my question? I am not finding the key logs containing UF data thruput or ingest information.   ... View more

yes thx I accepted that over a year ago ... View more

Curious, is TZ=GB for UK valid or did I misread something? ... View more