topic Index bz2 files take too much time in Splunk Search

Index bz2 files take too much time

indeed_2000 — Sun, 04 Jul 2021 07:57:04 GMT

I have a directory that contain 60 bz2 files. Totally 27 GB

After 24 hours still index processing not completed!

How can I check index status of this directory? (How much remain? How much pass?)

How can I tune splunk to index compress files more quickly?

FYI: there is no issue about license limitation.

FYI: I have enough disk space.

any idea?

Thanks

Re: Index bz2 files take too much time

venkatasri — Mon, 05 Jul 2021 02:25:39 GMT

@indeed_2000

The thruput is limited to 256 Kbps in limit.conf on UF, it depends on network speed you have to calculate how much data can be ingested.

Issue following command under $SPLUNK_HOME/bin and when prompt provide the username/password. This will provide the list of the files being read by UF and their current stage/progress.

./splunk list inputstatus

----

An upvote would be appreciated and Accept solution if it helps!

Re: Index bz2 files take too much time

codebuilder — Tue, 06 Jul 2021 14:30:28 GMT

To ingest these files Splunk first has to decompress them and that is a single threaded process (and consumes a lot of memory).

You'll see better performance by ingesting them before they are compressed, or decompress them prior to ingestion.