Data lost - splunk reload deploy-server

[rlaan]

I noticed that when pushing configuration changed from the deployment-servers to addition compoents (U-forwarders, H-forwarders, indexers, search heads) via the "splunk reload deploy-server" command that there were small 20-25 sec periods of missing data ingest.

I was under the impression that after these configuration pushes that splunk would catch-up or re-index  the data during the change. It appears that this is not the case after having to explain some missing data gaps. 

How can i push configuration changes without causing a loss of data ingest (working with busy access_combined/apache access logs so a few hundred events are missed during a 20 second window) 

Additionally, i have the full log files, is there a way to only re-index non-repeat events, i feel trying to delete the existing logs via splunk search and re-indexing with a "oneshot" method would be very time consuming over many servers.

[richgalloway]

It sounds like your Splunk architecture could use some improvements.  While it can be done, it's not typical (or recommended) to use the DS to manage all instance types.  The DS is designed to manage forwarders.

The search heads probably should be in a cluster and managed by a SHC Deployer.

More importantly, however, the indexers should be clustered and managed by a Manager Node (MN).  The MN will push apps to the indexers and contol when they restart to ensure at least one is available to receive data at all times.

Make sure all forwarders have the names (or IP addresses) of all indexers.  That way the forwarder can send data to another indexer if one is unavailable.  Consider using the Indexer Discovery feature where the forwarders get the list of indexers from the MN.  That will avoid having to update the forwarders when an indexer is added or removed.

[rlaan]

Thank you for the suggestion, I will have to do some reading into these components and how to configure them, I currently do not have clustered search heads or indexers replicating the data so i did not think they were required.  It provides me with a direction to investigate, thank you!