Is there any way in which we can download the apps from splunk base without having to manually download the tar file?

[meoo]

Hi

We are planning to automate the Splunk application installation and configuration process for quicker provisioning.

In this scenario, our first step is to install the splunk app from CLI, for which we use this command:

./splunk install app https://splunkbase.splunk.com/app/1274/

However it gives an error saying:

Error during app install: failed to extract app from /opt/splunk/var/run/87b95d9a426d8ebd.tar.gz to /opt/splunk/var/run/splunk/bundle_tmp/91801e5fc0eab8b4: No such file or directory

Is there any way in which we can download the apps from splunkbase without having to manually download the tar file.

[gotarr]

Hi, 

I script something by myself and I want to share it with you.

https://github.com/Gotarr/Splunkbase-Download-script 

(python-script)

My Inspiration is from @tfrederick74656  but his script dosnt work for me very well.

Happy splunking and let me know if something dosnt work.

[dbizzle]

This is how it works if you want to script around it.

You'll need 'jq' ... apt-get install jq. 

Get a session key, sessionid is its name.

curl -k -XPOST -D - https://splunkbase.splunk.com/api/account:login/ -d 'username=username&password=password'

View all releases for app

curl -sS -H "X-Auth-Token: token" https://splunkbase.splunk.com/api/v1/app/3110/release/ | jq '.[] | {name}'

View latest release version. Pretty sure index 0 is always the latest

curl -sS -H "X-Auth-Token: token" https://splunkbase.splunk.com/api/v1/app/3110/release/ | jq '.[0] | {name}'

Request version. Note the ?origin=sb query string. Also note, different hostname and different api version.

curl -sS -H "X-Auth-Token: token” -D - https://api.splunkbase.splunk.com/api/v2/apps/3110/releases/4.5.2/download/\?origin\=sb

http/302 is returned linking to file location. Grab that url with wget. 

https://cdn.splunkbase.splunk.com/media/private/signed_3110_28526_1677516671.tgz?response-content-disposition=attachment%3Bfilename%3D%22splunk-add-on-for-microsoft-cloud-services_452.tgz

Fun fact, if you don't know what the appid is for a particular addon and its on splunkbase, you can make a request like so, using the app title (eg folder name) 

curl -k -D - https://apps.splunk.com/apps/id/Splunk_TA_microsoft-cloudservices

This will (mostly) return a 302 to the splunkbase url

HTTP/2 302 
content-type: text/html; charset=utf-8
location: http://splunkbase.splunk.com/app/3110

neat.

[tfrederick74656]

Building on @mabrafoo's answer, I wrote a standalone script to do this. It allows you to authenticate to Splunkbase and download an app without the need for a separate web browser. Once you have the app.tgz, you can use the standard ./splunk install app <filename> syntax.

https://github.com/tfrederick74656/splunkbase-download

[Luis_Torres]

Hi 

@tfrederick74656 

I tried the script and got this:

0Warning: Remote filename has no length!

curl: (23) Failed writing body (0 != 16195)

I also tried forcing the specific URL of an app and got the same result.

Any ideas?

Thank you.

[tfrederick74656]

Hi @Luis_Torres,

First, are you specifying the sid and SSOID arguments (example values show below) when running download? You'll get an error message just like this if you don't specify them, if the values are incorrect, or if the session they refer to is expired. All Splunkbase downloads are authenticated, so it's mandatory to supply these. The sid value is case-sensitive alphanumeric, so it can be easy to mistake "0" "o" and "O", for example. The SSOID value should be all hexadecimal (0-9, a-f).

It's also worth double-checking that you actually have permissions to write to the directory where you're saving to. By default, the script will write to your current directory.

If none of that works, can you let me know if this is happening for all apps, or only a single app? If the latter, can you let me know the App ID and App Version you're trying to download?

Thanks,
Tyler

[ww9rivers]

Seeing these posts being so recent, I was hoping to get this to work. I like this better than the curlfire suggestion as this is possible to completely automate the downloads. So if possible I would like to get this working.

@tfrederick74656 I would like to help if there is still a chance of getting it to work.

[net_id]

I tried this and had problems.

I experienced the same issue of remote filename has no length.
Where did you find the documentation for the okta/auth endpoint?

When I plugged in the sessionid from my browser and added the cookie 'splunkbase_cookied_policy_accepted=true'
it then worked.

But the sid and SSOID combination with the additional cookie would not work.

Also the script needed if  [ "$1" == "1" ] on line 53 to work correctly.

[mabrafoo]

Here is one way to do it. Use at your own risk.

• Download https://github.com/talwrii/curlfire
• Log into splunkbase.splunk.com with firefox

curlfire will access the firefox cookies so that we can avoid the "please log in to download" message that curl would get.

For this example the app is the splunk add-on for Unix and Linux. The url says it is app 833. These instructions assume we know that the app id number is 833.

After running this command
username@computername:~/Downloads/splunk/curlfire-master$ ./curlfire "https://splunkbase.splunk.com/app/833/" | grep 833 | grep download | grep release

The output is
sb-href="/app/833/release/6.0.0/download/">
sb-href="/app/833/release/5.2.4/download/">

Now we know the Download URL for the latest version is
https://splunkbase.splunk.com/app/833/release/6.0.0/download/

Download the file using curlfire (see notes for curlfire chanages to make it work better below)
username@computername:~/Downloads/splunk/curlfire-master$ ./curlfire "https://splunkbase.splunk.com/app/833/release/6.0.0/download"

Output
curl: Saved to filename 'splunk-add-on-for-unix-and-linux_600.tgz'

Notes
In order to get the download to work properly, 3 flags were changed when curl was run in the curlfire bash file.

Before
curl -b "$curlcookies" "${args[@]}" ;
After
curl -O -J -L -b "$curlcookies" "${args[@]}" ;

option -O Write output to a local file name like the remote file
option -J Tell the -O option to use the filename found in the http header
option -L Follow redirects

Also, after looking at a random script that I downloaded from github I usually will change this.

!/bin/bash

to this.

!/bin/bash -x

in order to display all of the bash scripts commands and their expanded arguments.

And obviously don't forget to change your curl user agent to something common like "I ❤️ splunk."

[martin_mueller]

In order to download Apps from Splunkbase you need to be signed on to Splunkbase. Are you doing anything to sign on?

Personally, I wouldn't recommend automatically installing things that are downloaded fresh off the internet. How do you know it doesn't break your environment?
I'd keep a local repository of known good / fixed versions and install automatically from there.