Upload large file

[mwdbhyat]

Hi,

How does one upload files larger than 500mb? I get an error "File too large. The file selected is 996Mb. Maximum file size is 500Mb" Is this due to using the trial ?

Thanks

[AceAxis]

The default upload size for files in Splunk is 500Mb to consider the browser memory usage. The good part is that this can be altered, as admins usually upload files 10 to 50 Gb in size using CLI ingestion. 

The command below can be used to change the upload size to 2Gb. Note - run the CMD as administrator.

echo [settings] > "C:\Program Files\Splunk\etc\system\local\web.conf"
echo max_upload_size = 2048 >> "C:\Program Files\Splunk\etc\system\local\web.conf"

size can be increased by increasing the limit. Post this restart the Splunk. 

cd "C:\Program Files\Splunk\bin"
splunk restart

[PickleRick]

Don't do that!

Firstly, it's not a good practice to put settings into system/local except for very few settings which are really, really local.

But most importantly, if you run the supplied commands, they will overwrite any settings you might have already had there! (and while it's not the best practice, people often put some stuff there; like cert definitions).

[marnall]

For people finding this question in the years after 2016, you can set the max_upload_size setting in web.conf

[settings]
# set to the MB max
max_upload_size = 500
# you can also set a larger splunkdConnectionTimeout value so it wont timeout when uploading
splunkdConnectionTimeout=600

ref: https://docs.splunk.com/Documentation/Splunk/9.2.0/Admin/Webconf

[Rizqi_Iskandar]

btw, where can i find web.conf in windows?

Because i cant find the right one to edit this file

[inventsekar]

You can find it here..

For list of all config files

https://docs.splunk.com/Documentation/Splunk/9.2.2/Admin/Listofconfigurationfiles

[Rizqi_Iskandar]

I have edit the max_upload size from 500 to 8000 but still it cant upload enterprise security.

I try this way repeatly and restart splunk everytime i save this configuration, but nothing happen

Do you have other way to install splunk ES on windows? 

[marnall]

What happens if you run btool on the settings stanza and grep for max_upload_size?

e.g.

/opt/splunk/bin/splunk btool web list settings | grep max_upload

If it shows a value other than 8000, then likely your web.conf file is in the wrong place, or being overridden by another.

[Rizqi_Iskandar]

I think this is happen because its run on Windows

I will try to install it on Linux first and i will let you know if the problem fixed

thanks for the help dude

[marnall]

Sorry, I am too used to Linux. I believe the equivalent btool command on windows is:

$splunkhome$/bin/splunk.exe btool web list settings | FINDSTR max_upload

[dbcase]

I've had the same problem. You can use the Linux SPLIT command to break the file up into smaller parts

http://askubuntu.com/questions/54579/how-to-split-larger-files-into-smaller-parts

[ChrisG]

Yes, both the trial and free licenses have an indexing limit of 500MB a day. For file upload on any license, the maximum file size is 500 MB, as the UI indicates.

[somesoni2 · Accepted solution]

The 500MB limit is for uploading file from Splunk Web, regardless of the license type.

http://docs.splunk.com/Documentation/Splunk/6.4.3/Data/Uploaddata

[mwdbhyat]

Thanks - Is there a way around this without using the Web, not splitting the file ?

[dbcase]

I don't think so but you could try using the universal forwarder instead of uploading the file

[dbcase]

Is it possible to zip the file so it is less than 500MB?

https://answers.splunk.com/answers/279/does-splunk-index-gzip-files.html

[mwdbhyat]

Figured it out with the CLI.. did a oneshot monitor. http://docs.splunk.com/Documentation/Splunk/6.4.2/Data/MonitorfilesanddirectoriesusingtheCLI

[dbcase]

Ah good to know, haven't used that before