About beneteos

beneteos · ‎06-24-2025

Thanks @livehybrid It's not working for me for the moment, but I will be patient 🙂 (even a simple login try give me a 500)

beneteos · ‎06-24-2025

Hello, I'm trying to load download page and login (or vice versa) Unable to load the page, got a saml redirection and next error 500 (I even had an akamai error). I tried on several browsers, private mode on, and also on another computer. Does anyone have any information?

beneteos · ‎01-09-2024

But step 3 you mentioned is optional, in the sense that it's not required to request statuses for events to be indexed (I can verify my data is present, and events logged). So I didn't expect this behavior. After this max number of pending events reached, channel for the related token go on busy status, and leads to loss of logs until I restart service. I tried to increase max_number_of_acked_requests_pending_query, but it will only allow me to postpone the deadline, and set a huge value could perhaps also have negative impact on servers health. As I cannot control anything on client except channel header and authorization header, and as client doesn't seem do status requests (firewall logs), I will try to update maxIdleTime under 60, as client send data every 60 seconds. Thanks

beneteos · ‎01-09-2024

Hello, Thanks for your answer but I don't have the same understanding of Splunk documentation as you. If you were right, HEC service would be down a few hours after startup, or less. As explained in Splunk documentation (see the graph), HEC responds with an ACK for each event thrown, but you can send a request for a particular event to verify the status : "Each time a client sends a request to the HEC endpoint using a token with indexer acknowledgment enabled (1), HEC returns an acknowledgment identifier to the client (2)." https://docs.splunk.com/Documentation/Splunk/9.1.2/Data/AboutHECIDXAck#Query_for_indexing_status 1. Client send HEC request with event data 2. HEC acks the request once event is indexed HEC clients don't need to ask for status for events to get indexed well (millions each day), but after a while, the indexers become busy due to the maximum number of pending requests. I already increased this value so now I need to understand why this pending queries So my problem is something with pending requests and why they are increasing like that. I don't see any errors with the metrics, but they don't seem to be cumulative (Because Splunk Enterprise deletes status information after clients retrieve it) : I cannot control HEC client behavior beyond basic settings (for information, this is Akamai DataStream).

beneteos · ‎01-08-2024

Hello, We set HEC http input for several flows of data and related tokens, and we added ACK feature to this configuration. (following https://docs.splunk.com/Documentation/Splunk/9.1.2/Data/AboutHECIDXAck) We work with a distributed infra, 1 Search Head, two indexers (no cluster) All was Ok with HEC but after some time we got our first error event : ERROR HttpInputDataHandler [2576842 HttpDedicatedIoThread-0] - Failed processing http input, token name=XXXX [...] reply=9, events_processed=0 INFO HttpInputDataHandler [2576844 HttpDedicatedIoThread-2] - HttpInputAckService not in healthy state. The maximum number of ACKed requests pending query has been reached. Server busy error (reply=9) leads to unavailability of HEC, but only for the token(s) where maximum number of ACKed requests pending query have been reached. Restarting the indexer is enough to get rid of the problem, but after many logs have been lost. We did some search and tried to customize some settings, but we only succeeded in delaying the 'server busy' problem (1 week to 1 month). Has anyone experienced the same problem ? How can we avoid increasing those pending query counter ? Thanks a lot for any help. etc/system/local/limits.conf [http_input] # The max number of ACK channels. max_number_of_ack_channel = 1000000 # The max number of acked requests pending query. max_number_of_acked_requests_pending_query = 10000000 # The max number of acked requests pending query per ACK channel. max_number_of_acked_requests_pending_query_per_ack_channel = 4000000 etc/system/local/server.conf [queue=parsingQueue] maxSize=10MB maxEventSize = 20MB maxIdleTime = 400 channel_cookie = AppGwAffinity (this one because we are using load balancer, so cookie is also set on LB)

beneteos · ‎11-07-2023

Hi @gcusello and thank you for your answer ! We do not use DS for Splunk Forwarders management (we do that with puppet), so we only have 2 clients, our indexers, in order to replicate HEC configs and so get the same tokens on both indexers. Normally we don't need to change these configs except to add new HEC inputs, so really ponctually. As for the Monitoring Console, we are basically afraid of warnings saying that it could lead to incomplete search results. But as I said, we only have two indexers that are already configured as search peers on the SH, so it's exactly the same pool of indexers that we want to integrate in MC distributed mode. In terms of load, our search head is used very sparingly, with only a few searches per day. I think there's a maximum of ten connections/searches per day. So if I understand you correctly, there are no real risks of functionality loss, the question is more about load, right ?

beneteos · ‎11-07-2023

Hello, We have migrated our standalone installation of Splunk Enterprise to a "Small enterprise distributed deployment". This is a really small distributed deployment because the load is essentially on indexing capacity, even though it's less than 100Go daily (our licence allows 80Go) and search load is really low. So we have : - 1 Search Head - 2 indexers (no cluster) The search head also acts as license master and deployment server (just HEC configs and indexes replication to indexers). Now the question is : Is it possible to install Monitoring Console on the Search Head node ? We've well seen the recommandation here, and especially : "When you set up the monitoring console in distributed mode, it creates one search group for each server role, identified cluster, or custom group. Unless you use a "splunk_server_group" or the "splunk_server" option, only search peers that are members of the indexer group are searched by default. Because all searches that run on the monitoring console instance follow this behavior, non-monitoring console searches might have incomplete results." I'm not sure I really understand this, but as we only have 2 indexers and since they are the nodes that we want to put in the indexer group on the MC side, could it really leads to incomplete searchs ? It seems that this is the same advice given on dashboard, via the MC general setup page when trying to activate in distributed mode : "Do not configure the DMC in distributed mode if this is a production search head. Doing so can change the behavior of all searches on this instance. This is dangerous and unsupported." As already said, load consideration is secondary because we do not have a heavy searching activity. Thanks a lot.

Posts	7
Solutions	0
Karma Given	7
Karma Received	0
Member Since	‎09-12-2023

Online Status	Offline
Date Last Visited	‎06-24-2025 07:02 AM

unable to login and download

HEC HttpInputAckService pending queries

Install Monitoring Console on the Search Head node...

Re: unable to login and download

unable to login and download

Re: HEC HttpInputAckService pending queries

Re: HEC HttpInputAckService pending queries

HEC HttpInputAckService pending queries

Re: Install Monitoring Console on the Search Head ...

Install Monitoring Console on the Search Head node...

Are you a member of the Splunk Community?