cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question
beneteos
Explorer
Member since: ‎09-12-2023
‎01-10-2024
Community Statistics
Posts 5
Solutions 0
Karma Given 7
Karma Received 0
Member Since ‎09-12-2023
Activity Feed
View All

Re: HEC HttpInputAckService pending queries

by in Getting Data In
‎01-09-2024 06:53 AM
‎01-09-2024 06:53 AM
But step 3 you mentioned is optional, in the sense that it's not required to request statuses for events to be indexed (I can verify my data is present, and events logged). So I didn't expect this behavior. After this max number of pending events reached, channel for the related token go on busy status, and leads to loss of logs until I restart service. I tried to increase max_number_of_acked_requests_pending_query, but it will only allow me to postpone the deadline, and set a huge value could perhaps also have negative impact on servers health. As I cannot control anything on client except channel header and authorization header, and as client doesn't seem do status requests (firewall logs), I will try to update maxIdleTime under 60, as client send data every 60 seconds. Thanks     ... View more

Re: HEC HttpInputAckService pending queries

by in Getting Data In
‎01-09-2024 04:24 AM
‎01-09-2024 04:24 AM
Hello, Thanks for your answer but I don't have the same understanding of Splunk documentation as you. If you were right, HEC service would be down a few hours after startup, or less. As explained in Splunk documentation (see the graph), HEC responds with an ACK for each event thrown, but you can send a request for a particular event to verify the status : "Each time a client sends a request to the HEC endpoint using a token with indexer acknowledgment enabled (1), HEC returns an acknowledgment identifier to the client (2)." https://docs.splunk.com/Documentation/Splunk/9.1.2/Data/AboutHECIDXAck#Query_for_indexing_status 1. Client send HEC request with event data 2. HEC acks the request once event is indexed HEC clients don't need to ask for status for events to get indexed well (millions each day), but after a while, the indexers become busy due to the maximum number of pending requests. I already increased this value so now I need to understand why this pending queries So my problem is something with pending requests and why they are increasing like that. I don't see any errors with the metrics, but they don't seem to be cumulative (Because Splunk Enterprise deletes status information after clients retrieve it) : I cannot control HEC client behavior beyond basic settings (for information, this is Akamai DataStream).     ... View more

HEC HttpInputAckService pending queries

by in Getting Data In
‎01-08-2024 07:26 AM
‎01-08-2024 07:26 AM
Hello, We set HEC http input for several flows of data and related tokens, and we added ACK feature to this configuration. (following https://docs.splunk.com/Documentation/Splunk/9.1.2/Data/AboutHECIDXAck) We work with a distributed infra, 1 Search Head, two indexers (no cluster) All was Ok with HEC but after some time we got our first error event : ERROR HttpInputDataHandler [2576842 HttpDedicatedIoThread-0] - Failed processing http input, token name=XXXX [...] reply=9, events_processed=0 INFO HttpInputDataHandler [2576844 HttpDedicatedIoThread-2] - HttpInputAckService not in healthy state. The maximum number of ACKed requests pending query has been reached. Server busy error (reply=9) leads to unavailability of HEC, but only for the token(s) where maximum number of ACKed requests pending query have been reached. Restarting the indexer is enough to get rid of the problem, but after many logs have been lost. We did some search and tried to customize some settings, but we only succeeded in delaying the 'server busy' problem (1 week to 1 month). Has anyone experienced the same problem ? How can we avoid increasing those pending query counter ? Thanks a lot for any help. etc/system/local/limits.conf [http_input] # The max number of ACK channels. max_number_of_ack_channel = 1000000 # The max number of acked requests pending query. max_number_of_acked_requests_pending_query = 10000000 # The max number of acked requests pending query per ACK channel. max_number_of_acked_requests_pending_query_per_ack_channel = 4000000 etc/system/local/server.conf [queue=parsingQueue] maxSize=10MB maxEventSize = 20MB maxIdleTime = 400 channel_cookie = AppGwAffinity (this one because we are using load balancer, so cookie is also set on LB) ... View more
Labels

Re: Install Monitoring Console on the Search Head ...

by in Monitoring Splunk
‎11-07-2023 06:18 AM
‎11-07-2023 06:18 AM
Hi @gcusello and thank you for your answer ! We do not use DS for Splunk Forwarders management (we do that with puppet), so we only have 2 clients, our indexers, in order to replicate HEC configs and so get the same tokens on both indexers. Normally we don't need to change these configs except to add new HEC inputs, so really ponctually. As for the Monitoring Console, we are basically afraid of warnings saying that it could lead to incomplete search results. But as I said, we only have two indexers that are already configured as search peers on the SH, so it's exactly the same pool of indexers that we want to integrate in MC distributed mode. In terms of load, our search head is used very sparingly, with only a few searches per day. I think there's a maximum of ten connections/searches per day. So if I understand you correctly, there are no real risks of functionality loss, the question is more about load, right ? ... View more

Install Monitoring Console on the Search Head node...

by in Monitoring Splunk
‎11-07-2023 05:10 AM
‎11-07-2023 05:10 AM
Hello, We have migrated our standalone installation of Splunk Enterprise to a "Small enterprise distributed deployment". This is a really small distributed deployment because the load is essentially on indexing capacity, even though it's less than 100Go daily (our licence allows 80Go) and search load is really low. So we have : - 1 Search Head - 2 indexers (no cluster) The search head also acts as license master and deployment server (just HEC configs and indexes replication to indexers). Now the question is : Is it possible to install Monitoring Console on the Search Head node ? We've well seen the recommandation here, and especially : "When you set up the monitoring console in distributed mode, it creates one search group for each server role, identified cluster, or custom group. Unless you use a "splunk_server_group" or the "splunk_server" option, only search peers that are members of the indexer group are searched by default. Because all searches that run on the monitoring console instance follow this behavior, non-monitoring console searches might have incomplete results." I'm not sure I really understand this, but as we only have 2 indexers and since they are the nodes that we want to put in the indexer group on the MC side, could it really leads to incomplete searchs ? It seems that this is the same advice given on dashboard, via the MC general setup page when trying to activate in distributed mode : "Do not configure the DMC in distributed mode if this is a production search head. Doing so can change the behavior of all searches on this instance. This is dangerous and unsupported." As already said, load consideration is secondary because we do not have a heavy searching activity. Thanks a lot. ... View more
Labels
Contact Me
Online Status
Offline
Date Last Visited
‎01-10-2024 11:16 AM
Karma given to