Hello, by default, DMA summaries are not replicated between nodes in indexer cluster (for warm and cold buckets). I wonder how command tstats with summariesonly=true behaves in case of failing one node in cluster. Imagine, I have 3-nodes, single-site IDX cluster in deafult setting. What happened, when one node fails (so summaries on that node are not available) and I run search using "|tstats summariesonly=true..." on this cluster? If search spans data from primary warm or cold buckets on failed node, will I get incomplete data, right? (I think so, because appropriate summaries are missing). And if so, will I get any error message on search page? And how it change in case of multi-site cluster? I assume in case of failing one node, I should get complete data, becuase AFAIK in multi-site cluster every site has primary copy of bucket with DMA summaries. Is it right or not? I need this info because of one project I am working on. Thank you for answers. Best regards Lukas Mecir ... View more

Hi all, I have one question: I upgraded my Splunk deployment from 8.1.6 to 9.0.4. Deployment is: 3-nodes SH cluster, 3-nodes IDX cluster, 2 x HF, MC, SHC-D, CM, LM, DS. After upgrade I noticed one thing about queues on Monitoring Console. Before upgrade, all queues on all IDXs have 0% fill: But after upgrade, there is small fill (average about 5%, up to 10%) on Typing an Indexing queue: From my point of view it is strange, because nothing changed during upgrade - HW is the same, amount of ingested data is the same, kind of data is the same, no new log source etc. I search through documentation, but did not find anything relevant. So I would like to ask: what happens? Can it be ignored safely or there is really something wrong inside Splunk? Some config changes required because of some internal changes in Splunk? Could you share your experience with that, if you have one? Thank you in advance for any hint or glue. Best regards Lukas Mecir ... View more

Hello, I have question about pipeline parallelization. From docu and other sources I find that is safe enable pipeline parallelization if I have plenty of free resources in Splunk deployment, particularly CPU cores. In other words, if CPU on indexers or heavy forwarders are "underutilized". But, my question is - what does it mean "underutilized" in numbers? Especially in distributed environment. Example: lets imagine I have IDX cluster. 8 nodes, 16 CPU cores each. I see in Monitoring console )historical charts) average CPU load 40%, median CPU load 40% and maximum CPU load between 70 - 100%. My opinion is it is not safe to enable parallelization in this environment, OK? But when it is safe - if maximum load is under 50% Or 25%? What factors I should take into calculations and what numbers are "safe"? Could you please share your experience or point me to some available guide? Thank you very much in advance. Best regards Lukas Mecir ... View more

Hello, I have distributed environment with IDX cluster and DS. DS is used for deploy config to IDX cluster Manager Node and from it to IDX cluster nodes then. It is working fine. I upgraded DS from 8.1.6 to 8.1.10.1 (yes, because SVD-2022-0608...). Manager Node is on 8.1.6. After upgrade I noticed this log messages on MN: 10.88.28.93 - - [13/Jul/2022:15:56:33.540 +0200] "GET /services/server/info HTTP/1.1" 401 130 "-" "Splunk/8.1.10.1 (Linux 3.10.0-1160.62.1.el7.x86_64; arch=x86_64)" - 0ms  10.88.28.93 is IP address of DS I checked Search peers config on DS and there was MN in "sick" state. I edited its config by re-enter Remote username and Remote password and then MN changed status to Healthy and everything is working fine. My question is: what happened during upgrade of DS? My idea is that new pair of private+public keys was generated on DS on first run after upgrade (and then I had to distribute new public key to MN by re-entering Remote username and password of course), but am I right? And if I am right, why this happened? I made many Splunk upgrades before and I experienced this never before... Any info/hint/clue will be highly appreciated. Thank you. Best regards Lukas Mecir ... View more

Hello, I have one question about monitoring HF. I found many ways how to get outgoing data rate on HF (how mach data HF sends to IDX), but how to find incoming data rate on HF? In other words, how to find how many data coming from sources to HF (in total is enough for me)? Is any metric like this in Splunk log? Thanks for help in advance. Best regards Lukas Mecir ... View more

Hello, I am solving following problem: HEC on HF is used for data receiving. In splunkd.log on Heavy Forwarder I found these error:   ERROR HttpInputDataHandler - Failed processing http input, token name=linux_rh, channel=n/a, source_IP=10.177.155.14, reply=9, events_processed=18, http_input_body_size=8405, parsing_err="Server is busy"   There was 7 messages of this kind during 10 minute interval. I found that "reply=9" means "server is busy" - this is a message for log source "stop sending data", because that HF is overloaded (log source really stopped sending data). At the same time parsing, aggregation, typing, httpinput and splunktcpin queues had 100% fill ratio, indexing queue has 0% fill ratio. At the same time, VMWare host on which HF is running, was probably overloaded - CPU frequency on this host in usually about 1GHz, but grew up to 4GHz shortly for this time (it was not caused by Splunk HF probably). At the same time, there is no ERROR messages in splunkd.log on IDX cluster, which is receiving data from concerned HF. Based on this information, I came to the following conclusion: Because the indexqueue on the HF was not full and there were no ERRORs on the IDX cluster, there was no problem on the IDX cluster or on the network between the HF and the IDX cluster. Due to VMWare host overload, the HF did not have sufficient resources to process messages, so the parsing, aggregation, and typing queues became full. As a result, the following occurred: to populate the httpinput and splunktcpin queues to generate ERROR error HttpInputDataHandler - Failed processing http input stop receiving data from the log source As soon as the VMWare host overload ended (after cca 10 minutes), data reception was resumed, no data was lost. Could you please review my conclusion and tell, if I am right? Or there is something more to investigate? And what to do to avoid this problem in future? Re-configure queue setting (set higher max_size_kb)? Or add some power to VMWare host? Or something else? Thank you very much in advance for any input. Best regards Lukas Mecir  ... View more

Hello, I would like to ask about win log in XML format: Using Splunk, we collect Windows logs in XML format, because before indexing on Splunk, we modify and reduce them on Cribl - according to this document: Reducing Windows XML Events  It works fine, but now I would like to do one thing - convert values that are expressed in XML using numeric code to expressions in text form, as in the standard Windows log format. For example: In XML is: <Task>12544</Task> and corresponding value in text format of log is: TaskCategory=Logon So I tried to find conversion tables between text and XML format and  for elements <Opcode> <Keywords> <Level> I found some. But I cannot find any for element <Task> Do you know anyone about some? (Or for other XML elements as well)? If so, you can share it with me? It will be really appreciated. Best regards Lukas Mecir ... View more