We are not getting any internal logs from one of our forwarder but its phoning home. we can also add or delete an app through deployment server remotely. The forwarder is ingesting logs to one of our index but its not continuous. this all happened after when we tried to ingest logs from a folder on that server. Let me know if anyone have any idea.
the only way to debug this situation is _internal.
At first see if your _internal logs arre distributed in time: see if they arrive late but arrive all or if they don't arrive.
After you can check if there's a delay in indexing
index=_internal | eval diff=_indextime-_time, indextime=strftime(_indextime,"%Y/%m/%d %H:%M:%S.%3N") | table _time indextime diff
Then see if you continously receive other logs from UF or not: if other logs can arrive see how many they are, because if you have a too many logs, _internal logs are sent after.
Then check if there are network problems or congestion between UF and Indexers.
Thanks for the reply. We were actually trying make new ingestion from the server which is already ingesting some logs to already existing index. when we tried to ingest the new logs to already existed index, all the logs stopped ingesting including internal logs. Then we rolled backed the new log ingestion before that we added few parameters in limits.conf on the forwarder because we saw some known issues for that version(7.1.2)
[inputproc]# Threshold size (in mb) to trigger fishbucket rolling to a new db.file_tracking_db_threshold_mb = 500[thruput]# Throughput limiting at index time.maxKBps = 0
when we came back today on monday, the log ingestion was good, we were getting internal logs as well. now we tried to ingest the new logs again but to a new index but as soon as we did this again the ingestion got stopped from the forwarder including the internal logs. these are windows servers. Don't what is the issue.
just an update we are doing this new ingestion from 4 servers but we are getting problem with only one server. when we restart the forwarder service after adding the app and serverclass only that one server is getting shutdown and not getting any data.
we started getting the new logs from new log ingestion path and the internal logs from the server which stopped. we did not do anything and even the application team did not reboot the servers or restart the forwarder service. we did not get anything in splunkd logs as well.