Thank you for your suggestions. We do not have a test server to restore before restoring it on to prod server.    On a separate note, is it possible to schedule a report or a script to backup kvstore on a daily basis to avoid restoring from  backup of /opt/splunk/var/lib/splunk/backup  directory ... View more

Thank you for the suggestion.  But I have mentioned, The current Splunk has been setup such that it's not letting me to use ./splunk start to accept license.   The only way I can start the splunk using systemctl start Splunkd ... View more

Here is part of the 1st event in the exported file: testindex,"/opt/splunk/var/log/cdm/events/events.log","testindex:testsourcetype","1665554739.952,root_host=""test.server.domain.com"",comp_id=""1234444"",check_checklist_name=""test name"",check_disa_cki_id="""",...... ... View more

Here is part of the 1st event in the exported file: testindex,"/opt/splunk/var/log/cdm/events/events.log","testindex:testsourcetype","1665554739.952,root_host=""test.server.domain.com"",comp_id=""1234444"",check_checklist_name=""test name"",check_disa_cki_id="""",...... ... View more

Thank you again.   Where can I find the password sslPassword for OUTPUTS.CONF   [tcpout:sndIndexers] server = xx.xx.xx.xx:9996 sslPassword =  ... View more

Thank you for looking in to it. I furthur looked in to the logs and found errors from indexer(s) also:   On Heavy Forwarder: ==================== 06-22-2022 13:24:03.471 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer 06-22-2022 13:24:03.472 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer 06-22-2022 13:24:03.472 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer 06-22-2022 13:24:03.472 -0400 WARN AutoLoadBalancedConnectionStrategy [19320 TcpOutEloop] - Applying quarantine to ip=xx.xx.xx.xx port=9996 _numberOfFailures=2 06-22-2022 13:24:03.473 -0400 ERROR TcpOutputFd [19320 TcpOutEloop] - Read error. Connection reset by peer 06-22-2022 13:24:03.473 -0400 WARN AutoLoadBalancedConnectionStrategy [19320 TcpOutEloop] - Applying quarantine to ip=yy.yy.yy.yy port=9996 _numberOfFailures=2 On Indexers: ============== TcpInputProc [27164 FwdDataReceiverThread] - Error encountered for connection from src=xxxxx error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol ... View more

It's in TRANSFORMS.CONF: [xxxx] REGEX=xxxxxx DEST_KEY=_TCP_ROUTING FORMAT=sndIndexers [yyyy] REGEX=yyyyy DEST_KEY=_TCP_ROUTING FORMAT=tstIndexers ... View more

Thank you for looking in to. Here is OUTPUTS.CONF:   [tcpout:sndIndexers] server = indexer1:9996 sslPassword = <<Password>> [tcpout:tstIndexers] server = indexer2:9996 sslPassword = <<password2>> ... View more

Thank you for your response. If configure to send data to only one indexer it's working fine.  This is an issue only when HF outputs.conf to send data to multiple indexers based on a REGEX  ... View more

Thank you for the response.    Regardless of the REGEX  ins transforms.conf in HF, the data always sending to the index in inputs.conf   Seems like splunk not allowing to overide index specified in inpouts.conf   TRANSFORMS.conf ====== #[cdmops] #REGEX=.*service=cdmops2.* #DEST_KEY=_MetaData:Index #FORMAT=cdmops #[cdmops2] #REGEX=.*service=cdmops2.* #DEST_KEY=_MetaData:Index #FORMAT=cdmops2 Data: ===== 2022/06/01 10:45:50 service=cdmops server=node3 score=50 seq=55041 2022/06/01 10:45:50 service=cdmops2 server=node1 score=17 seq=55042     ... View more

Thank you for the response.  To make it clear, I am NOT trying to split the event. I am trying to send the events from the same source to same indexer but to a different INDEXE( index1 or index2 based on the regex ) ... View more

Here is my correct  INPUTS.conf.  Missing index in the earlier inouts.conf   INPUTS.CONF ====== [monitor:///opt/splunk/var/log/testData] disabled = false host = Haveyforwarder1 index = index1 ... View more