topic Re: Indexing all text log files in a directory in Getting Data In

Indexing all text log files in a directory

ankit — Sun, 09 May 2021 19:24:57 GMT

I have a directory with about 750 log files. The files are all text files and the total size of this directory is 117 GB. I need to index the files once (not continuously).

Would the best option to index all the files in the directory be to copy the files to my Splunk instance and then use the directory input option from Splunk Web i.e. Settings > Data Inputs > Files & Directories ?

Any other recommended options ?

Re: Indexing all text log files in a directory

aasabatini — Mon, 10 May 2021 06:07:55 GMT

Hi @ankit

you can collect all the logs based on your directory.

you need to go

Settings > Data Inputs > Files & Directories

click on

add your path on the form and select index once

Example path: /var/log

Considerations

You can collect more than 100 Gb if you have a properly license.

In another case if you violates the license several time, splunk will be blocked.

Re: Indexing all text log files in a directory

gcusello — Mon, 10 May 2021 06:08:59 GMT

Hi @ankit,

if you already have a Forwarder on the system where files are stored you can input them there, in this way you optimize the process using the compressing features of Splunk and leave Splunk work for you.

As you can think it will tale a long time!

If you haven't a Forwarder on that system, I hint to compress your files and copy rhem on a folder of the indexer, then uncompress and index as you said.

Probably the second way is the quicker, but the first is easier.

Ciao.

Giuseppe

Re: Indexing all text log files in a directory

ankit — Mon, 10 May 2021 12:39:09 GMT

Thanks @gcusello @aasabatini . I will be copying the compressed directory to the server, uncompress them and selecting the Directory input method from Splunk Web.

Re: Indexing all text log files in a directory

ankit — Tue, 11 May 2021 14:50:54 GMT

I went ahead and tried a different approach. I had each of the 749 files compressed as log.gz files beforehand. So just copied those compressed files to a directory on my Splunk instance and added a directory input. Worked like a charm !