Some data are not being send to Splunk

[szukaczov]

Hi team,

We had some issues with the Splunk forwarder which was not sending data to Splunk. After restart of the service we started to see only part of the logs. 

Logs which we are able to see are from: DNS index

Logs which we cannot see are from MS_AD index and are related to Domain Controller logs. 

From the debug log I can see below lines:

06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - IndexKey: ms_ad shouldForwardIndex: 1
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Cache Hit - indexKey: ms_ad shouldForward: 1
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Pushed eventId=22656 on chanId=46 to back of tcp client (tcp output) queue. source:source::WinEventLog:Security|host::xxxxxxx|WinEventLog:Security|
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Pushed eventId=22656 on chanID=0 to back of tcp client (tcp output) queue
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Received ACK for : 21272-21272 idx=xxxxx:9997
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Pushed eventId=21352 on chanId=46 to back of tcp client (tcp output) queue. source:source::WinEventLog:Security|host::xxxxxx|WinEventLog:Security|
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Pushed eventId=21352 on chanID=0 to back of tcp client (tcp output) queue
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Cache Hit - indexKey: ms_ad shouldForward: 1
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - IndexKey: ms_ad shouldForwardIndex: 1
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Received ACK for : 21275-21275 idx=xxxxxxx:9997
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Cache Hit - indexKey: ms_ad shouldForward: 1
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Pushed eventId=22657 on chanId=46 to back of tcp client (tcp output) queue. source:source::WinEventLog:Security|host::xxxxxxx|WinEventLog:Security|
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Pushed eventId=22657 on chanID=0 to back of tcp client (tcp output) queue
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Received ACK for : 21277-21277 idx=xxxxxx:9997
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Pushed eventId=21353 on chanId=46 to back of tcp client (tcp output) queue. source:source::WinEventLog:Security|host::xxxxxxx|WinEventLog:Security|
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Pushed eventId=21353 on chanID=0 to back of tcp client (tcp output) queue
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Received ACK for : 21279-21279 idx=35.234.126.255:9997
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Cache Hit - indexKey: ms_ad shouldForward: 1
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - IndexKey: ms_ad shouldForwardIndex: 1
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Cache Hit - indexKey: ms_ad shouldForward: 1
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Pushed eventId=22658 on chanId=46 to back of tcp client (tcp output) queue. source:source::WinEventLog:Security|host::xxxxxx|WinEventLog:Security|
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Pushed eventId=22658 on chanID=0 to back of tcp client (tcp output) queue
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Received ACK for : 21281-21281 idx=xxxxx:9997
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Pushed eventId=21354 on chanId=46 to back of tcp client (tcp output) queue. source:source::WinEventLog:Security|host::xxxxx|WinEventLog:Security|
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Pushed eventId=21354 on chanID=0 to back of tcp client (tcp output) queue
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Received ACK for : 21283-21283 idx=35.234.126.255:9997
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Cache Hit - indexKey: ms_ad shouldForward: 1
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - IndexKey: ms_ad shouldForwardIndex: 1
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Cache Hit - indexKey: ms_ad shouldForward: 1
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Pushed eventId=22659 on chanId=46 to back of tcp client (tcp output) queue. source:source::WinEventLog:Security|host::xxxxxxx|WinEventLog:Security|
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Pushed eventId=22659 on chanID=0 to back of tcp client (tcp output) queue
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Received ACK for : 21285-21285 idx=xxxxx:9997
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Pushed eventId=21355 on chanId=46 to back of tcp client (tcp output) queue. source:source::WinEventLog:Security|host::xxxxxx|WinEventLog:Security|
06-09-2021 16:54:22.752 +0800 DEBUG TcpOutputProc - Pushed eventId=21355 on chanID=0 to back of tcp client (tcp output) queue

Does the above log means that the logs are indexed and will be shown soon in Splunk?