Getting Data In

not getting internal logs from forwarder

sathwikr076
Communicator

Hello,

We are not getting any internal logs from one of our forwarder but its phoning home. we can also add or delete an app through deployment server remotely. The forwarder is ingesting logs to one of our index but its not continuous. this all happened after when we tried to ingest logs from a folder on that server. Let me know if anyone have any idea.

Thanks.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi sathwikr076,
the only way to debug this situation is _internal.

At first see if your _internal logs arre distributed in time: see if they arrive late but arrive all or if they don't arrive.

After you can check if there's a delay in indexing

index=_internal 
| eval diff=_indextime-_time, indextime=strftime(_indextime,"%Y/%m/%d %H:%M:%S.%3N") 
| table _time indextime diff

Then see if you continously receive other logs from UF or not: if other logs can arrive see how many they are, because if you have a too many logs, _internal logs are sent after.

Then check if there are network problems or congestion between UF and Indexers.

Bye.
Giuseppe

sathwikr076
Communicator

Hello,

Thanks for the reply. We were actually trying make new ingestion from the server which is already ingesting some logs to already existing index. when we tried to ingest the new logs to already existed index, all the logs stopped ingesting including internal logs. Then we rolled backed the new log ingestion before that we added few parameters in limits.conf on the forwarder because we saw some known issues for that version(7.1.2)
[inputproc]# Threshold size (in mb) to trigger fishbucket rolling to a new db.file_tracking_db_threshold_mb = 500[thruput]# Throughput limiting at index time.maxKBps = 0
when we came back today on monday, the log ingestion was good, we were getting internal logs as well. now we tried to ingest the new logs again but to a new index but as soon as we did this again the ingestion got stopped from the forwarder including the internal logs. these are windows servers. Don't what is the issue.

just an update we are doing this new ingestion from 4 servers but we are getting problem with only one server. when we restart the forwarder service after adding the app and serverclass only that one server is getting shutdown and not getting any data.
Thanks.

0 Karma

sathwikr076
Communicator

we started getting the new logs from new log ingestion path and the internal logs from the server which stopped. we did not do anything and even the application team did not reboot the servers or restart the forwarder service. we did not get anything in splunkd logs as well.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...