About lenrigodoy

lenrigodoy · ‎02-28-2023

As in my answer to main question, maybe the problem in your case continues with queue resizing, and, moreover, you will have other issues with your instance. The best practice is not move the queues size, do it only in case of an emergency meanwhile you request a support checking to determine where is the real problem. In your case, maybe the problem is disk IOPS, maybe a networking problem (check if you are using the same network device for input and output, the network speed, and if there is not any blockage at network level), or the number of events incoming in your instance from other forwarders, the number of other forwarders sending data through this router forwarder. Check that the server is not having any other problems related with CPU or RAM, if is Linux, that the server is correctly setted up, disabling Transparent Hugepages, modifying ulimits, filesystems used are consistent with Splunk's recommendations (System requirements for use of Splunk Enterprise on-premises - Splunk Documentation), etc Hope you can get the real issue and solve it.

lenrigodoy · ‎02-28-2023

As others answered, your indexing problem will not be solved by modifying the size of the queues, since you will keep the same root problem, in this case, disk write speed problems. You can use the information from this post to calculate the IOPS of disk arrays used by the indexers. On the other hand, I recommend using Splunk Sizing so that you can get an estimated IOPS recommendation for the disks based on the type of RAID you are using. In case your Splunk instance is a standalone instance, maybe you need to change it to a distributed environment, using a dedicated server or dedicated cluster for each tier (search head, indexer, forwarder, license manager) instead of a single server. Best wishes LGS

lenrigodoy · ‎02-11-2022

The same message appears me today over one of the SHC members. To solve, I follow the recommendations over the member (I have 3 SH, the last one was the problematic). First of all, you need to backup the KVstore. Splunk service should be running over the SH to backup. Later, you need to restore the KVStore. Splunk service should be stopped before of the command execution. All of these commands should be executed over the affected member. If you have multiple members affected, you would execute it on each affected member, or in all SH if all are being affected with this replication error. Last Splunk version (8.2.4) recommends to move KVStore to a new technology. Currently, Splunk uses mmapv1 as Storage Engine, but it's recommended to move to wiredTiger. Follow the docs to migrate your KVStore to recommended engine: Migrate the KV store storage engine - Splunk Documentation

lenrigodoy · ‎02-02-2022

I've follow the Docs about route data https://docs.splunk.com/Documentation/Splunk/8.2.4/Forwarding/Routeandfilterdatad#Filter_data_by_target_index In this doc, it's recommended to do: "If you want to forward only the data targeted for a single index (for example, as specified in inputs.conf), and drop any data that is not a target for that index, configure outputs.conf in this way: [tcpout] #Disable the current filters from the defaults outputs.conf forwardedindex.0.whitelist = forwardedindex.1.blacklist = forwardedindex.2.whitelist = #Forward data for the "myindex" index forwardedindex.0.whitelist = myindex This first disables all filters from the default outputs.conf file. It then sets the filter for your own index. Be sure to start the filter numbering with 0: forwardedindex.0." Now, I'm testing your config, I'll update my answer in case of that config works. Otherwise, I will test other configs to find the working one.

lenrigodoy · ‎02-01-2022

I'm working on an indexer to try to forward all data ingested with IT Essentials Work + Splunk Add-on for Unix & Linux to a remote indexer cluster. Until now, that indexer is receiving events into all itsi_* indexes, but, when I try to setup the forwarding option into that indexer, I cannot set the forwardedindex.n.whitelist and blacklist to forward only the itsi_* indexes to the IDX Cluster. I've try to overwrite all default whitelists and blacklists on local and reset whitelists with itsi_* indexes, but, this still forwarding all indexes, nor only itsi_* indexes. My outputs.conf file is like following: [tcpout] defaultGroup = default-autolb-group forwardedindex.0.whitelist = forwardedindex.1.blacklist = forwardedindex.2.whitelist = forwardedindex.0.whitelist = (itsi_grouped_alerts|itsi_im_meta|itsi_im_metrics|itsi_import_objects|itsi_notable_archive|itsi_notable_audit|itsi_summary|itsi_summary_metrics|itsi_tracked_alerts) indexAndForward = 1 [tcpout:default-autolb-group] disabled = false server = HFtoIDXCluster:9997 useACK = true If I use a "default" config option, overwriting the lists not resetting (not declaring the default 3 lists empty on the tcpout stanza) I have the same behaviour. This is the first time I try to set forwarding options from an indexer. I need to forward this data because it's used for administration of each Splunk instances, and it's required to get into a specific Splunk Enterprise cluster, but, all other indexes it's not required to be forwarded. Have I miss something to specify into config files? Best regards

lenrigodoy · ‎12-16-2021

Thank you, that was I want to use to recognize some UF that we use to forward data to our ES environment and aren't under my administration.

Posts	6
Solutions	0
Karma Given	1
Karma Received	1
Member Since	‎05-24-2021

Cannot forward one specific index between indexers

Re: Impact of increasing the queue size?

Re: Impact of increasing the queue size?

Re: How to resolve error on a search head member i...

Re: Cannot forward one specific index between inde...

Cannot forward one specific index between indexers

Re: How to get the forwarder IP address reproting ...