Indexing or forwarding rate- How to fix errors that occurred when uploading directory from ubuntu to monitor?

aad — Mon, 06 Feb 2023 00:16:05 GMT

Hi Team,

I 'm new to Splunk and need little guidance with fixing errors that occurred when I uploaded a directory < .var/log >--from ubuntu to monitor

-------------------------------------------------------------------------------------------------------------------------------

Health Status of Splunkd

Real-time Reader-0

Root Cause(s):
- The monitor input cannot produce data because splunkd's processing queues are full. This will be caused by inadequate indexing or forwarding rate, or a sudden burst of incoming data.
- Generate Diag?More infoIf filing a support case, click here to generate a diag.
- Last 50 related messages:
  - 02-04-2023 20:02:25.936 -0800 WARN TailReader [4979 tailreader0] - Could not send data to output queue (parsingQueue), retrying...
  - 02-04-2023 20:02:25.910 -0800 WARN TailReader [4980 batchreader0] - Could not send data to output queue (parsingQueue), retrying...
  - 02-04-2023 20:02:20.904 -0800 WARN TailReader [4979 tailreader0] - Enqueuing a very large file=/var/log/auth.log.1 in the batch reader, with bytes_to_read=9885261283, reading of other large files could be delayed
  - 02-04-2023 20:02:20.875 -0800 INFO TailReader [4979 tailreader0] - Ignoring file '/var/log/wtmp' due to: binary
  - 02-04-2023 20:02:19.846 -0800 INFO TailReader [4966 MainTailingThread] - State transitioning from 1 to 0 (initOrResume).
  - 02-04-2023 20:02:19.846 -0800 INFO TailReader [4966 MainTailingThread] - State transitioning from 1 to 0 (initOrResume).
  - 02-04-2023 20:02:19.844 -0800 INFO TailReader [4980 batchreader0] - batchreader0 waiting to be un-paused
  - 02-04-2023 20:02:19.844 -0800 INFO TailReader [4980 batchreader0] - Starting batchreader0 thread
  - 02-04-2023 20:02:19.844 -0800 INFO TailReader [4980 batchreader0] - Registering metrics callback for: batchreader0
  - 02-04-2023 20:02:19.844 -0800 INFO TailReader [4979 tailreader0] - tailreader0 waiting to be un-paused
  - 02-04-2023 20:02:19.844 -0800 INFO TailReader [4979 tailreader0] - Starting tailreader0 thread
  - 02-04-2023 20:02:19.844 -0800 INFO TailReader [4979 tailreader0] - Registering metrics callback for: tailreader0
    
    splunkd
    Data Forwarding
    
    File Monitor Input
    
    Forwarder Ingestion Latency
    Ingestion Latency
    Large and Archive File Reader-0
    Real-time Reader-0
    Index Processor
    
    Resource Usage
    
    Workload Management

Re: Indexing or forwarding rate

gcusello — Sun, 05 Feb 2023 06:40:10 GMT

Hi @aad,

this message should mean that you have a congestion problem on your Indexer that block indexing of data from the input.

So, what's the hardware resources of your Indexer? Splunk requests at least 12 CPUs and 12 GB RAM.

Then what are the performances of your storage? Splunk requires at least 800 IOPS, better 1200.

Then what's the your network performances?

This means that you need to re-design your architecture starting from requirements definition.

My hint is to give this assignment to a Splunk Architect.

Ciao.

Giuseppe

topic Indexing or forwarding rate- How to fix errors that occurred when uploading directory from ubuntu to monitor? in Monitoring Splunk

Indexing or forwarding rate- How to fix errors that occurred when uploading directory from ubuntu to monitor?

Real-time Reader-0

Re: Indexing or forwarding rate