Okay, I love it when stupid simple things don't work right out of the gate.
I signed up for Splunk Light Cloud Trial. I want to send logs written to a file on Ubuntu to Splunk. I followed instructions here, per the email I got: http://docs.splunk.com/Documentation/SplunkLight/6.4.2/Cloud/GettingdataintoSplunkLightcloudserviceu...
Below is the complete (sniped) output (and yes I used something for username:password below that's valid). So, can someone explain what stupid simple steps I missed that 'are not' in the documentation?
~$ sudo tar -xvzf splunkforwarder-6.4.2-00f5bb3fa822-Linux-x86_64.tgz -C /opt ..snip list of files from tar... ~$ cd /opt/splunkforwarder/bin :/opt/splunkforwarder/bin$ ll total 35792 drwxr-xr-x 3 506 506 4096 Jun 28 11:13 ./ drwxr-xr-x 8 506 506 4096 Jun 28 11:13 ../ -r-xr-xr-x 1 506 506 55600 Jun 28 11:13 btool* -r-xr-xr-x 1 506 506 55600 Jun 28 11:13 btprobe* -r-xr-xr-x 1 506 506 38208 Jun 28 11:13 bzip2* -r-xr-xr-x 1 506 506 55600 Jun 28 11:13 classify* -r--r--r-- 1 506 506 57 Jun 28 09:59 copyright.txt -r-xr-xr-x 1 506 506 2367 Jun 28 09:59 genRootCA.sh* -r-xr-xr-x 1 506 506 206 Jun 28 09:59 genSignedServerCert.sh* -r-xr-xr-x 1 506 506 144 Jun 28 09:59 genWebCert.sh* -r-xr-xr-x 1 506 506 586632 Jun 28 11:13 openssl* -r-xr-xr-x 1 506 506 7390 Jun 28 09:59 pid_check.sh* drwxr-xr-x 2 506 506 4096 Jun 28 11:09 scripts/ -r--r--r-- 1 506 506 1360 Jun 28 09:59 setSplunkEnv -r-xr-xr-x 1 506 506 461032 Jun 28 11:13 splunk* -r-xr-xr-x 1 506 506 35290832 Jun 28 11:13 splunkd* -r-xr-xr-x 1 506 506 465 Jun 28 09:59 splunkdj* -r-xr-xr-x 1 506 506 16072 Jun 28 11:13 splunkmon* -r-xr-xr-x 1 506 506 28872 Jun 28 11:13 srm* :/opt/splunkforwarder/bin$ sudo ./splunk install app /home/devadmin/splunkclouduf.spl -auth username:password ...snip... This appears to be your first time running this version of Splunk. This command [POST /services/apps/local/] needs splunkd to be up, and splunkd is down. :/opt/splunkforwarder/bin$
The first time you install and run the Splunk UF/HF, you need to accept the license before it will start correctly.
You can do..
/opt/splunkforwarder/bin/splunk start --accept-license
That and check your permissions, confirm that the user you are wanting to run Splunk as owns everything.
What the heck?
devadmin@ip-10-98-86-21:/opt/splunkforwarder/bin$ sudo ./splunk install app /home/devadmin/splunkclouduf.spl -auth admin:changeme
App '/home/devadmin/splunkclouduf.spl' installed
I'm using SplunkLight Cloud service and the 'admin' password is 'changeme'? Where do I change this ridiculous password? Should forwarding clients being using some sort of account called 'admin'?
Okay, so I do see admin:changeme in the documentation, but this really doesn't make any sense. If that account exists on the machine and not in the cloud, step 1 should be 'change the password'.
There are two sets of credentials here, one for Splunk Light Cloud, and the other for the Universal Forwarder, and maybe that's what's causing the confusion.
The universal forwarder software comes with its own set of credentials, as @andrewb described above. These are different from the credentials that you log into Splunk Light Cloud with.
It is good practice to change the credentials for the universal forwarder, and you can do that if you want to. But that change does not affect your Splunk Light Cloud credentials, and vice versa.
I hope this helps clarify things.
Believe me, I appreciate the 'quick start' documentation that doesn't go through all the onerous in's and out's of everything before you can get something up and running. The issue is as I stated, there are a few things missing, such as ./splunk start, and the password issue. If there is a default password of 'changeme' then obviously it should be changed as a step 1.
In San Diego, a TV station didn't change their default password for their emergency alert SCADA system, and the next thing they knew there was an interruption of local TV news from the emergency broadcast system describing a zombie outbreak and advising residents to stay indoors! In best Sys Admin practice world it should be well known by now, one does not change passwords like 'admin:changeme' if they want to, rather that should be the very first thing you do and documentation from software vendors should support that regardless of what they're protecting.
At first I thought it was my splunk login I should have been using. It's still not clear to me if that admin account exists locally on the machine or in the Splunk server/cloud account. It's just some mysterious account called admin that's securing I have no idea what. As a new user, I shouldn't have to start at the beginning of the full documentation to figure out how to change a password for something that's so painfully obvious.
Thanks for considering a documentation update for this.
Thanks bandries for your input and suggestions about improving the documentation for this issue. I will update the documentation to add clarity about the username and password when installing the universal forwarder credentials, add information about changing the username and password, and more information about the admin account.
Also, our instructions indicate performing a ./splunk restart on the universal forwarder after setting the deploy-poll command, but I'll also add information about ./splunk start in case there is an issue.
Thanks again for your input for documentation improvements. We appreciate it!
admin/changeme is the default login for the Universal Forwarder, not for your Splunk Light cloud account. After your first login to the UF, you should change that to something secure.
You access your Splunk Light cloud trial via single sign-on after logging into your Splunk.com user account. The username and password is the one you chose when you signed up.
We will take another look at the documentation to make sure that this is clearer.
Makes sense. Tried that then the splunk install command and got "Login failed" which makes no sense. I then took the username and password I'm using and successfully logged into the splunk console. Unless this is a different username and password?
Console output below:
devadmin@ip-10-98-86-21:/opt/splunkforwarder/bin$ sudo ./splunk start
Checking mgmt port : open
New certs have been generated in '/opt/splunkforwarder/etc/auth'.
Checking conf files for problems...
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-6.4.2-00f5bb3fa822-linux-2.6-x86_64-manifest'
All installed files intact.
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Generating a 1024 bit RSA private key
Getting CA Private Key
writing RSA key
devadmin@ip-10-98-86-21:/opt/splunkforwarder/bin$ sudo ./splunk install app /home/devadmin/splunkclouduf.spl -auth username:password