We are planning to upgrade our Splunk hardware. We currently have below(multisite indexer cluster with independant search head clusters) and we are facing problems with low cpu count and high disk latency(we currently have HDDs). We primarily index data through HEC.   Type Site Number of nodes CPU p/v (per node) memory GB (per node) SH cluster 1 4 16/32 128 Indexer cluster 1 11 4/8 64 Indexer manager/License master 1 1 16/32 128 SH cluster 2 4 16/32 128 Indexer cluster 2 11 4/8 64 Indexer manager/License master 2 1 16/32 128   Daily indexing/license usage 400-450GB which may grow further in near future Search concurrency example for one instance from 4 node SH cluster   We are trying to come up with the best hardware configuration that can support such load.   Looking at Splunk recommended settings, we have comeup with below config. Can someone shed more light on if this is an optimal config and also advise on the number of SH machines and indexer machines needed with such new hardware Site1: 3 node SH clusters, 7 node idx cluster Site2:  As we are using site2 for searching and indexing only during unavailability of site1, may be it can be smaller? Role CPU (p/v) Memory Indexer 24/48 64G Non indexer 32/64 64G ... View more

Hi all, Is it possible to get informations on the cluster manager config bundle through rest api? I am specifically looking for active bundle hash/Active Bundle ID. ... View more

Hi all, We have an index say index1 with a log retention of 7 days where we receive logs for different applications. Now we have a requirement to copy all ERROR logs from an application to another index, say index2 continuously.  The intention here is to have index2 with a higher retention so we can have access to the error logs for longer period.   What is the best way to implement such a mechanism. Its okay to run this job every day, or every 6 hour or so. Would be best to retain all fields and field extractions in logs in target index as well.   ... View more

Hi all, We are indexing different topics from our kafka cluster to an index say, index1. But we now have a requirement to retain a subset of those topics for longer period of time. Is there a way to implement this while we still get all the data into the same index? I can think of the subset of topics that I need longer retention being searched, filtered out and collected to a new index. But this involves licensing if we want to retain the source/sourcetype and other fields I believe which is not practical for us. We want to retain the original source/sourcetype etc and have the subset of topics that we need longer retention be copied over to another index, say index2, that has longer retention. We also need the original copy in index1 as we have lot of dependant searches and alerts that use this index to search for the same data.     ... View more

Hi All, I have a report running every 6 hour with below search query. This is fetching hourly availability of haproxy backends based on http response code as shown below. I need to accelerate this report, but I think the bucket section of the search is disqualifying this for report acceleration. Can someone help with modifying this search so that it can be accelerated or are there any other work arounds to do this to get the exact same table as shown?   index=haproxy (backend="backend1" OR backend="backend2") | bucket _time span=1h | eval result=if(status >= 500, "Failure", "Success") | stats count(result) as totalcount, count(eval(result="Success")) as success, count(eval(result="Failure")) as failure by backend, _time | eval availability=tostring(round((success/totalcount)*100,3)) + "%" | fields _time, backend, success, failure, totalcount, availability   _time backend success failure totalcount availability 2024-06-07 04:00 backend1 28666 0 28666 100.000% 2024-06-07 05:00 backend1 28666 0 28666 100.000% 2024-06-07 06:00 backend1 28712 0 28712 100.000% 2024-06-07 07:00 backend1 28697 0 28697 100.000% 2024-06-07 08:00 backend1 28678 0 28678 100.000% 2024-06-07 09:00 backend1 28714 0 28714 100.000% 2024-06-07 04:00 backend2 618 0 618 100.000% 2024-06-07 05:00 backend2 179 0 179 100.000% 2024-06-07 06:00 backend2 555 0 555 100.000% 2024-06-07 07:00 backend2 103 0 103 100.000% 2024-06-07 08:00 backend2 1039 0 1039 100.000% ... View more

We recently upgraded from 9.0.2 to 9.2.1 and started seeing some new errors on all indexer peer nodes as shown below. -------- 05-17-2024 14:35:07.225 +0000 ERROR DispatchCommandProcessor [949840 TcpChannelThread] - Search results may be incomplete, peer <indexer peer ip>'s search ended prematurely. Error = Peer <indexer peer hostname> will not return any results for this search, because the search head is using an outdated generation (search head gen_id=4626; peer gen_id=4969). This can be caused by the peer re-registering and the search head not yet updating to the latest generation. This should resolve itself shortly. -------- The master has logs like below. -------- splunkd.log.1:05-17-2024 12:06:59.491 +0000 WARN CMMaster [950487 CMMasterServiceThread] - got a large jump in gen_id suggestion=4921 current pending=1 reason=event=addPeerParallel Success guid=xxx adding_peers=7 -------- I tried suggestion actions from below discussion but no luck so far and ERROR is continuing for days now. https://community.splunk.com/t5/Splunk-Enterprise/Why-am-I-receiving-this-error-quot-The-search-head-is-using-an/td-p/599044 It looks like the problem is with the primary master as we could see that when switching to the standby master, the error goes away. Can anyone advise on this? What is a generation/gen_id and if there is a way to reset this to fix the issue? ... View more

Hi all,   Im trying to understand how rotation certificates used for SSO works in a search head cluster. We have a searchhead cluster where we have SSO working already. As for initial setup, I understand we can download SPmetadata.xml file from splunk SAML settings page. However, during rotation, how do we create this as we are using a cert thats already existing and we want to rotate the server side certificate? If we just download SPmetadata.xml for creating request for IDP, this will have same cert as we are using. If we rotate the cert first at our side so we can download SPmetadata.xml  to create request for IDP, then this will end up in error as IDP wont detect server side certificate during this, obviously.   ... View more

Hi all,   Im analysing event counts for a specific search criteria and I want to know how the count of values changed over time.  Below search is not good enough to see whats going on as many usernames have huge number of events and some with small numbers are barely noticeable (Im interested in rate of change and not count itself) ``` index=test_index "search string" | timechart span=10m count(field1) by username ``` So I want to see a rate of change of the count rather than simple count, by username field. How can we achieve this? ... View more

Hi all, We are seeing a scenario where there are a lot of unoptimised searches, dashboards etc which when run are exhausting our CPU on indexers. If some users run resource intensive adhoc searches/dashboards etc simultaneously, this is becoming a problem as so many searches running together resulting in 'server busy' error at indexer.   1. Is there any way we can throttle CPU/memory usage per user/role/searches? 2. Are there any documents on optimising searches for better performance and less resource usage? ... View more

Hi all, We have been facing some errors with Splunk indexers, where it says something like below. ``` Failed processing http input, token name=<HECtoken>, channel=n/a, source_IP=, reply=9, events_processed=62, http_input_body_size=47326, parsing_err="Server is busy" ``` And I found in some discussions that increasing queue sizes may help sometimes. We are indexing ~400GB per day and it makes sense to increase the queue sizes as default values might not be good enough in this case. However, the splunk docs doesnt have a detailed explanation of which queues can be set in server.conf and what are the proportions that we need consider. Can someone help with understanding this? ... View more

Hi All, Im looking for a way to share a non expiring search with other users. If we use the ''share job" option or just use the URL from address bar - it will get expired once the job expires. But I want to share a link that will not expire. Of course in such a case the search needs to run again from the users end, but the time stamps and search query are the things I want to share with a link. Is there a way to do this? ... View more