Indexing or forwarding rate- How to fix errors that occurred when uploading directory from ubuntu to monitor?

[aad]

Hi Team, 

I 'm new to Splunk and need little guidance with fixing errors that occurred when I uploaded a directory < .var/log >--from ubuntu to monitor 

-------------------------------------------------------------------------------------------------------------------------------

Health Status of Splunkd

 

 Real-time Reader-0

• Root Cause(s):
  • The monitor input cannot produce data because splunkd's processing queues are full. This will be caused by inadequate indexing or forwarding rate, or a sudden burst of incoming data.

  • Generate Diag?More infoIf filing a support case, click here to generate a diag.

  • Last 50 related messages:
    • 02-04-2023 20:02:25.936 -0800 WARN TailReader [4979 tailreader0] - Could not send data to output queue (parsingQueue), retrying...
    • 02-04-2023 20:02:25.910 -0800 WARN TailReader [4980 batchreader0] - Could not send data to output queue (parsingQueue), retrying...
    • 02-04-2023 20:02:20.904 -0800 WARN TailReader [4979 tailreader0] - Enqueuing a very large file=/var/log/auth.log.1 in the batch reader, with bytes_to_read=9885261283, reading of other large files could be delayed
    • 02-04-2023 20:02:20.875 -0800 INFO TailReader [4979 tailreader0] - Ignoring file '/var/log/wtmp' due to: binary
    • 02-04-2023 20:02:19.846 -0800 INFO TailReader [4966 MainTailingThread] - State transitioning from 1 to 0 (initOrResume).
    • 02-04-2023 20:02:19.846 -0800 INFO TailReader [4966 MainTailingThread] - State transitioning from 1 to 0 (initOrResume).
    • 02-04-2023 20:02:19.844 -0800 INFO TailReader [4980 batchreader0] - batchreader0 waiting to be un-paused
    • 02-04-2023 20:02:19.844 -0800 INFO TailReader [4980 batchreader0] - Starting batchreader0 thread
    • 02-04-2023 20:02:19.844 -0800 INFO TailReader [4980 batchreader0] - Registering metrics callback for: batchreader0
    • 02-04-2023 20:02:19.844 -0800 INFO TailReader [4979 tailreader0] - tailreader0 waiting to be un-paused
    • 02-04-2023 20:02:19.844 -0800 INFO TailReader [4979 tailreader0] - Starting tailreader0 thread
    • 02-04-2023 20:02:19.844 -0800 INFO TailReader [4979 tailreader0] - Registering metrics callback for: tailreader0
      splunkd

      Data Forwarding

       

      File Monitor Input

      Forwarder Ingestion Latency

      Ingestion Latency

      Large and Archive File Reader-0

      Real-time Reader-0

      Index Processor

       

      Resource Usage

       

      Workload Management

[gcusello]

Hi @aad,

this message should mean that you have a congestion problem on your Indexer that block indexing of data from the input.

So, what's the hardware resources of your Indexer? Splunk requests at least 12 CPUs and 12 GB RAM.

Then what are the performances of your storage? Splunk requires at least 800 IOPS, better 1200.

Then what's the your network performances?

This means that you need to re-design your architecture starting from requirements definition.

My hint is to give this assignment to a Splunk Architect.

Ciao.

Giuseppe