Restart splunkd on CM worked fine for me when "Previously scheduled Remove Excess Buckets is running", been hanging for days. ... View more

A very old post but still relevant if the log timestamp format cannot be change. If exact timestamp is not needed I would set this to current or none for the specific sourcetype in props.conf. It is a very quick fix. DATETIME_CONFIG = [CURRENT | NONE]  Alternative just extract the time and not the date. Works fine as long as the events are indexed the same day as they are written. TIME_FORMAT = %H:%M:%S As last alternative datetime.xml is possible but maybe not so easy. In this case it might be enough to modify the "litmonth" attributes. Just remember to copy the file and rename it to something else and use that modified file for this specific sourcetype only. Modifying datetime.xml would impact all transactions on so do not do that. ... View more

I had this exact issue in one environment. Versioning turned off in AWS S3 and turned on in Splunk. It works perfectly fine until a index bucket needs to freeze. Then Splunk is not able to remove any index bucket related files on S3, and splunkd will log errors and warnings. This event gives a hint of the issue: 03-26-2024 18:53:54.640 +0100 WARN S3Client [118080 FilesystemOpExecutorWorker-0] - Error removing object name=splunk01/index01/db/9c/b2/1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF/receipt.json(0,-1,0,) as versions of the object could not be listed   These are all events related to failing to freeze a index bucket on S3: 03-26-2024 18:53:54.640 +0100 INFO BucketMover [118080 FilesystemOpExecutorWorker-0] - RemoteStorageAsyncFreezer freeze completed succesfully for bid=index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF 03-26-2024 18:53:54.640 +0100 WARN DatabaseDirectoryManager [118080 FilesystemOpExecutorWorker-0] - failed to request CacheManager to remove remote data for bucket, cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|", exception=Error removing bucket with cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|" 03-26-2024 18:53:54.640 +0100 ERROR CacheManager [118080 FilesystemOpExecutorWorker-0] - cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|", issue="Failed to remove receipt remoteId=splunk01/index01_ccd/db/9c/b2/1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF/receipt.json(0,-1,0,)" 03-26-2024 18:53:54.640 +0100 ERROR CacheManager [118080 FilesystemOpExecutorWorker-0] - Remove bucket cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|" with receiptId=splunk01/index01_ccd/db/9c/b2/1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF/receipt.json(0,-1,0,) failed 03-26-2024 18:53:54.640 +0100 WARN S3Client [118080 FilesystemOpExecutorWorker-0] - Error removing object name=splunk01/index01_ccd/db/9c/b2/1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF/receipt.json(0,-1,0,) as versions of the object could not be listed 03-26-2024 18:53:54.558 +0100 INFO CacheManager [118080 FilesystemOpExecutorWorker-0] - will remove cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|" from remote storage 03-26-2024 18:53:54.545 +0100 INFO CacheManager [118080 FilesystemOpExecutorWorker-0] - will remove cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|" removeRemote=1 03-26-2024 18:53:54.379 +0100 INFO BucketMover [118080 FilesystemOpExecutorWorker-0] - RemoteStorageAsyncFreezer trying to freeze bid=index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF, freezeInitiatedByAnotherPeer=false 03-26-2024 18:53:54.379 +0100 INFO DatabaseDirectoryManager [118080 FilesystemOpExecutorWorker-0] - cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|" found to be on remote storage v   ... View more

If syslog is sent with TCP or TLS then this is an known issue in CyberArk. If you are using SC4S (Splunk Connect for Syslog) the solution is to add a line feed https://cyberark-customers.force.com/s/article/00004289 ... View more

Yes it is applicable for standard HF functionality as well. At least according to Docs. Some special cases where the HF is configured differently then just forward events it might be different. https://docs.splunk.com/Documentation/VersionCompatibility/current/Matrix/Compatibilitybetweenforwardersandindexers ... View more

Thanks @gcusello  Is it true, what I am stating in the question, that its not possible to limit max concurrent searches on the search peer? Based on what you're saying I guess we should never hit these limits if we follow the reference hardware sizing. It's just that it is cheaper and quicker to change thing with software and config. This environment works fine under normal conditions but there are exceptions where this type of limit would keep the load on a level so that indexing and replication works well. As a workaround we reduced search load manually during high load situations. ... View more

DS init failed: Deployment Server not available on a dedicated forwarder. This is not a real error on an Universal Forwarder. There is no Deployment Server on an Universal Forwarder. There is just a Deployment Client on the UF.  I think this event is show everytime the UF starts. This is how it looks in version 7.1.2 and 7.36, both Windows and Linux: INFO DS_DC_Common - Deployment Server not available on a dedicated forwarder. ... View more

Did you manage to resolve this issue? You need to change the thresholds, or disable iowait, on each enterprise instance. See Answers https://community.splunk.com/t5/Splunk-Enterprise/Where-do-I-configure-the-health-conf-so-that-I-can... ... View more

You can change the thresholds on each enterprise instance. Most of what is described here is locally configured on each instance. See Answers https://community.splunk.com/t5/Splunk-Enterprise/Where-do-I-configure-the-health-conf-so-that-I-can-disable-the/m-p/596827     ... View more

In addition you can use these searches to benchmark iowait performance over time, so you can set relevant thresholds for your environment. Just replace the hostname: CPU IOwait average index=_internal  source="*/splunk/var/log/splunk/health.log"   feature=IOWait component=PeriodicHealthReporter node_type=indicator indicator=avg_cpu__max_perc_last_3m host=ind0* | timechart span=30s max(measured_value) min(due_to_threshold_value) by host CPU IOwait single CPU index=_internal  source="*/splunk/var/log/splunk/health.log"   feature=IOWait component=PeriodicHealthReporter node_type=indicator indicator=single_cpu__max_perc_last_3m host=ind0* | timechart span=300s max(measured_value) min(due_to_threshold_value) by host CPU IOwait top3 CPU index=_internal  source="*/splunk/var/log/splunk/health.log"   feature=IOWait component=PeriodicHealthReporter node_type=indicator indicator=sum_top3_cpu_percs__max_last_3m host=ind0* | timechart span=30s max(measured_value) min(due_to_threshold_value) by host   ... View more

The Health feature has caused some confusion regards local vs. distributed config. I have investigated this and it is very flexible to configure even though the docs is not so clear about it. So far I have not found any Answers posts that isn't possible to solve using standard config. If you do configuration locally on Monitoring Console (DMC), as you described, that threshold will only be valid for the DMC local host. There is no distributed threashold. You need to configure the threshold on each and every enterprise instance (e.g. your standalone indexers). Either you do config in Splunk Web under Settings menu on each enterprise instance and just click Save. Or toggle Status Disable/Enable. This will take direct effect and does not require restart. If you instead configure health.conf on each instance, example disable iowait, put this in health.conf [feature:iowait] disabled = 1 And then you need to do a reload e.g. http://<your_splunk>:<splunk_port>/debug/refresh   If you have a index cluster, applying a cluster bundle, this will trigger a restart of the peers. Version 8.2.4 Hope this solves your issue. ... View more

Windows 2012 and 2012 R2 is supported again from version 8.2.0, until current release as of today, version 9.0.0. According to both docs and download pages https://docs.splunk.com/Documentation/Splunk/9.0.0/Installation/Systemrequirements ... View more

The release note of ES lists the preferred CIM version. For ES there is no longer any info about supported CIM versions in Splunkbase. https://docs.splunk.com/Documentation/ES/7.0.1/RN/Enhancements#Updated_add-ons ... View more

Same issue in version 8.0.3 and Red Hat Enterprise Linux Server release 7.9 (Maipo). Looks like some customers have this issue but not all. The free/available figures in the REST call are incorrect but the capacity is correct. | rest splunk_server=* /services/server/status/partitions-space   ... View more

Correct, Instance Name is not searchable in Forwarder Management (and cannot be used in serverclass.conf). Host Name in Forwarder Management is the real server name as perceived by Splunk and cannot be configured. Host field in an event is coming from the host stanza in inputs.conf If host in inputs.conf is set to a static value e.g. the server hostname, and the server hostname is changed, then event Host field and Host Name in Forwarder Management will differ. That situation is problematic. ... View more

Only cacert.pem shall be placed in the certificate store. That file contains only the root certificate (public). server.pem contains also the private server key and ca.pem contains also the private root key and that will compromise the security. Not that it really matters in this case since Splunk default root CA is used. In case of using a trusted signed cert, keep this in mind. ... View more

Does your Remedy setup have BMC Remedy ITSM (IT Service Management module) or is it a standalone Remedy? Johan is referring to BMC Remedy ITSM and not Splunk ITSI. ... View more

Not true, according to the outputs.conf manual since version 7 at least. One cloned output group should be enough to keep the event flow running. Whether or not the TcpOutputProcessor should wait until at least one of the cloned output groups receives events before attempting to send more events. * If set to "true", the TcpOutputProcessor blocks until at least one of the cloned groups receives events. The definition of a cloned group is according to the manual, when there are two ore more groups in the defaultGroup attribute. https://docs.splunk.com/Documentation/Forwarder/8.1.3/Forwarder/Configureforwardingwithoutputs.conf This is so strange, since the real behavior is like Rich says. That's my experience as well. https://community.splunk.com/t5/Getting-Data-In/Any-data-forwarding-issue-using-data-cloning-and-different/m-p/496072 If there is one or no groups in defaultGroup you might have some different behavior, since then you must use _TCP_ROUTING instead, and the event metadata is tagged with the route in that case, which is probably not the case if you use two groups in defaultGroup.  Anyone with any practical experience, please share. ... View more

The simple answer is: Host Name in Forwarder Management is the same as you would get using the hostname shell command in both *nix and Windows. The answer marked Solution is not entirely true. If you restart splunk you will also see this hostname in the splunkd.log. I don't think there is and should be any way to override that value. Also it is very useful to see the real hostname. 02-04-2021 22:05:22.836 +0100 INFO ServerConfig - My hostname is "smarttest-old.band.com".   ... View more

Are you changing the hostname field in the Splunk code or is this configurable? I thought, "Host Name" on Forwarder Management page, was hostname shortname set in inputs.conf, even if you configured host=$decideOnStartup and hostnameOption = fullyqualifiedname for Windows. But it is not. The Host Name in Forwarder Management is the same as you would get using the hostname shell command in both *nix and Windows. It is very useful to see the real hostname together with Client Name (which is either GUID by default or coming from deploymentclient.conf) and Instance Name (which is serverName in server.conf). Also, if you edit clients in a Serverclass, you will see something called DNS Name. It is equivalent to DNS response on Deployment Server e.g. using shell command nslookup <hostname>. (The Deployment Server does a reverse lookup using the IP address of the incoming TCP packets. If it fails, DNS Name will have the IP address instead.) I have not seen any proper documentation of all these different names anywhere, so this is an area with lots of misunderstandings. Servers with Forwarders installed on easily gets incorrect host name and serverName once servers are cloned or renamed, which happens all the time. It can easily be a mess. ... View more

So no conclusion on this subject? I have seen this type of mismatch as well. Here is an example:   -bash-4.2$ df /data3 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/data3vg-data3lv 12882810880 11133090120 1749720760 87% /data3     From Monitoring Console, Resource Usage:Machine Mount Point File System Type Disk Usage (GB) /data3 xfs 9746.51 / 12286.01   It is over 1TB difference (around 10%). This filesystem only contains cold indexes. Nothing else.   Monitoring Console page Resource Usage:Machine Disk Usage, uses REST /services/server/status/partitions-space and this sometimes give incorrect numbers. The REST call used in Volume Details /services/data/index-volumes gives however correct figures. ... View more

Edit the JavaScripts are no longer required. If above solution does not solve it, you need to remove user specific config. See solution provided in Why is my ui-prefs.conf change to make the default search time range 15 minutes in all apps not being respected  ... View more