Using Universal forwarder to send data from files in a folder to Splunk Enterprise

[neltonk]

Hi,
I have no experience with Splunk, learning the ABCs of splunk so please be patient... I am configuring the universal forwarder on a Windows Server to monitor a folder that contains log files (existing data size: 5 GB) from an application. I have created an index in Splunk Enterprise and then added a monitor in the splunk universal forwarder:

splunk add monitor D:\SplunkForwarderCache\TimeSyncLogs\Windows -index timesyncWindowslogs

Everyday this folder gets updated, log files of size in total of 1.5 GB. Does the index get recreated / rebuilt everyday copying the entire folder each day?

Is there a quick way to upload a large sized folder to splunk as I see the index creation is very slow?

[mayurr98 · Accepted solution]

No, it won't index the entire file again. Rather it will index only updated files. For the first time, I am assuming that you are monitoring historical data as well, so that could be the reason why your indexing is quite slow but eventually it will settle down.
Give some time to Splunk after all it is a machine!

If you have older version of splunk
It is normally limited by the throughput of network bandwidth.

[thruput]
maxKBps = 256

Please try to use the command ./splunk cmd btool limits list thruput to check if the maxKBps is set to 0. Otherwise, indexing rate of an indexer will be limited and not fully utilized the IO rate of the disk system.

You may also refer the link below showing that index_thruput is controlled by thruput value.
http://wiki.splunk.com/Community:HowIndexingWorks

If you have newer version then by default it is 0. so do not worry about it.

Also in order to check the rate of indexing..go to monitoring console > indexing > Indexing Performance: Instance
and troubleshoot.

Let me know if this helps!

[neltonk]

Thanks again Mayur for your help