Index bz2 files take too much time

[indeed_2000]

Hi

I have a directory that contain 60 bz2 files. Totally 27 GB

After 24 hours still index processing not completed!

How can I check index status of this directory? (How much remain? How much pass?)

How can I tune splunk to index compress files more quickly?

FYI: there is no issue about license limitation.

FYI: I have enough disk space.

any idea?

Thanks

[codebuilder]

To ingest these files Splunk first has to decompress them and that is a single threaded process (and consumes a lot of memory).

You'll see better performance by ingesting them before they are compressed, or decompress them prior to ingestion.

[venkatasri · Accepted solution]

@indeed_2000 

The thruput is limited to 256 Kbps in limit.conf on UF,  it depends on network speed you have to calculate how much data can be ingested.

Issue following command under $SPLUNK_HOME/bin and when prompt provide the username/password. This will provide the list of the files being read by UF and their current stage/progress.

./splunk list inputstatus

----

An upvote would be appreciated and Accept solution if it helps!