Hello, I have all-in-one Splunk instance with data already indexed. Now I want to add new Indexer (not-clustered, clean installation). I would like to move part of indexed data to new Indexer (to have cca the same amount of data on both instances). My idea of process is: Stop all-in-one instance  Create new index(es) on new indexer Stop new indexer Copy (what is best - rsync?) part of buckets in given index(es) from all-in-one instance to new indexer Start new indexer and all-in-one instance Configure outputs.conf on forwarders - add new indexer Add new indexer as search peer to all-in-one instance Would it work or I missed something? Thank you for help. Best regards Lukas Mecir ... View more

Hello, I have Splunk distributed deployment (cca 20 servers + cca 100 UFs). On servers, I configured SSL encryption of management traffic and TLS certificate host name validation: server.conf   [sslConfig] enableSplunkdSSL = true serverCert = <path_to_the_server_certificate> sslVerifyServerCert = true sslVerifyServerName = true sslRootCAPath = <path_to_the_CA_certificate>   Everything is working well - servers communicate each other. But my question is: I use Deployment server for pushing config to UFs and I am little bit surprised that management traffic between UFs and Deployment server is still flowing (I see all UFs phoning home, I can push config) even I did not configure encryption nor hostname validation on any UF. Is it OK? Does it mean that hostname validation for management traffic cannot be configured on UF? Or there is a way how to config hostname validation on UFs? I found only how to configure hostname validation on UF in outputs.conf for sending collected data to Indexer, but nothing about management traffic. Thank you for any hint. Best regards Lukas Mecir ... View more

Hello, by default, DMA summaries are not replicated between nodes in indexer cluster (for warm and cold buckets). I wonder how command tstats with summariesonly=true behaves in case of failing one node in cluster. Imagine, I have 3-nodes, single-site IDX cluster in deafult setting. What happened, when one node fails (so summaries on that node are not available) and I run search using "|tstats summariesonly=true..." on this cluster? If search spans data from primary warm or cold buckets on failed node, will I get incomplete data, right? (I think so, because appropriate summaries are missing). And if so, will I get any error message on search page? And how it change in case of multi-site cluster? I assume in case of failing one node, I should get complete data, becuase AFAIK in multi-site cluster every site has primary copy of bucket with DMA summaries. Is it right or not? I need this info because of one project I am working on. Thank you for answers. Best regards Lukas Mecir ... View more

Hi all, I have one question: I upgraded my Splunk deployment from 8.1.6 to 9.0.4. Deployment is: 3-nodes SH cluster, 3-nodes IDX cluster, 2 x HF, MC, SHC-D, CM, LM, DS. After upgrade I noticed one thing about queues on Monitoring Console. Before upgrade, all queues on all IDXs have 0% fill: But after upgrade, there is small fill (average about 5%, up to 10%) on Typing an Indexing queue: From my point of view it is strange, because nothing changed during upgrade - HW is the same, amount of ingested data is the same, kind of data is the same, no new log source etc. I search through documentation, but did not find anything relevant. So I would like to ask: what happens? Can it be ignored safely or there is really something wrong inside Splunk? Some config changes required because of some internal changes in Splunk? Could you share your experience with that, if you have one? Thank you in advance for any hint or glue. Best regards Lukas Mecir ... View more

Hello, I have question about pipeline parallelization. From docu and other sources I find that is safe enable pipeline parallelization if I have plenty of free resources in Splunk deployment, particularly CPU cores. In other words, if CPU on indexers or heavy forwarders are "underutilized". But, my question is - what does it mean "underutilized" in numbers? Especially in distributed environment. Example: lets imagine I have IDX cluster. 8 nodes, 16 CPU cores each. I see in Monitoring console )historical charts) average CPU load 40%, median CPU load 40% and maximum CPU load between 70 - 100%. My opinion is it is not safe to enable parallelization in this environment, OK? But when it is safe - if maximum load is under 50% Or 25%? What factors I should take into calculations and what numbers are "safe"? Could you please share your experience or point me to some available guide? Thank you very much in advance. Best regards Lukas Mecir ... View more

Hello, I have distributed environment with IDX cluster and DS. DS is used for deploy config to IDX cluster Manager Node and from it to IDX cluster nodes then. It is working fine. I upgraded DS from 8.1.6 to 8.1.10.1 (yes, because SVD-2022-0608...). Manager Node is on 8.1.6. After upgrade I noticed this log messages on MN: 10.88.28.93 - - [13/Jul/2022:15:56:33.540 +0200] "GET /services/server/info HTTP/1.1" 401 130 "-" "Splunk/8.1.10.1 (Linux 3.10.0-1160.62.1.el7.x86_64; arch=x86_64)" - 0ms  10.88.28.93 is IP address of DS I checked Search peers config on DS and there was MN in "sick" state. I edited its config by re-enter Remote username and Remote password and then MN changed status to Healthy and everything is working fine. My question is: what happened during upgrade of DS? My idea is that new pair of private+public keys was generated on DS on first run after upgrade (and then I had to distribute new public key to MN by re-entering Remote username and password of course), but am I right? And if I am right, why this happened? I made many Splunk upgrades before and I experienced this never before... Any info/hint/clue will be highly appreciated. Thank you. Best regards Lukas Mecir ... View more

Hello, I have one question about monitoring HF. I found many ways how to get outgoing data rate on HF (how mach data HF sends to IDX), but how to find incoming data rate on HF? In other words, how to find how many data coming from sources to HF (in total is enough for me)? Is any metric like this in Splunk log? Thanks for help in advance. Best regards Lukas Mecir ... View more

Hello, I am solving following problem: HEC on HF is used for data receiving. In splunkd.log on Heavy Forwarder I found these error:   ERROR HttpInputDataHandler - Failed processing http input, token name=linux_rh, channel=n/a, source_IP=10.177.155.14, reply=9, events_processed=18, http_input_body_size=8405, parsing_err="Server is busy"   There was 7 messages of this kind during 10 minute interval. I found that "reply=9" means "server is busy" - this is a message for log source "stop sending data", because that HF is overloaded (log source really stopped sending data). At the same time parsing, aggregation, typing, httpinput and splunktcpin queues had 100% fill ratio, indexing queue has 0% fill ratio. At the same time, VMWare host on which HF is running, was probably overloaded - CPU frequency on this host in usually about 1GHz, but grew up to 4GHz shortly for this time (it was not caused by Splunk HF probably). At the same time, there is no ERROR messages in splunkd.log on IDX cluster, which is receiving data from concerned HF. Based on this information, I came to the following conclusion: Because the indexqueue on the HF was not full and there were no ERRORs on the IDX cluster, there was no problem on the IDX cluster or on the network between the HF and the IDX cluster. Due to VMWare host overload, the HF did not have sufficient resources to process messages, so the parsing, aggregation, and typing queues became full. As a result, the following occurred: to populate the httpinput and splunktcpin queues to generate ERROR error HttpInputDataHandler - Failed processing http input stop receiving data from the log source As soon as the VMWare host overload ended (after cca 10 minutes), data reception was resumed, no data was lost. Could you please review my conclusion and tell, if I am right? Or there is something more to investigate? And what to do to avoid this problem in future? Re-configure queue setting (set higher max_size_kb)? Or add some power to VMWare host? Or something else? Thank you very much in advance for any input. Best regards Lukas Mecir  ... View more

Hello, I would like to ask about win log in XML format: Using Splunk, we collect Windows logs in XML format, because before indexing on Splunk, we modify and reduce them on Cribl - according to this document: Reducing Windows XML Events  It works fine, but now I would like to do one thing - convert values that are expressed in XML using numeric code to expressions in text form, as in the standard Windows log format. For example: In XML is: <Task>12544</Task> and corresponding value in text format of log is: TaskCategory=Logon So I tried to find conversion tables between text and XML format and  for elements <Opcode> <Keywords> <Level> I found some. But I cannot find any for element <Task> Do you know anyone about some? (Or for other XML elements as well)? If so, you can share it with me? It will be really appreciated. Best regards Lukas Mecir ... View more

Hello, I have question about [thruput] setting on UF and internal Splunk log: I did some tests with Splunk UF - I needed to simulate a problem with the tcpout queue and therefore I reduced the value of the parameter [thruput] maxKBps = <integer> in the limits.conf file to low KBps values (eg 3KBps). UF is set to send its internal logs to IDX. However, I noticed that with such a low value of this parameter, UF stopped sending its internal metric logs (ie the contents of the $ SPLUNK_HOME/var/log/splunk/metrics.log file) to IDX. Logs were further written to the $ SPLUNK_HOME/var/log/splunk/metrics.log file, but were not sent to IDX. Is this normal behavior? It looks as if there is a mechanism that prioritizes the data collected over internal Splunk logs and suppresses the sending of internal Splunk logs to IDX - is it really so, is there such a mechanism? I tried to find something about it in the documentation, but without success. Thank you in advance for any information. Best regards Lukas Mecir ... View more

Hi, I have question about search.log. I know I can find log records related to particular search in search.log using Job inspector (clicking on link to search.log in bottom of Job inspector). But my question is: is there any way how to get records related to particular search in past? Example: I made some search yesterday and today I would like to get all log records related to this search from search.log file. Is there any way how to do it? Thanks in advance for any info or hint. Best regards Lukas ... View more