topic Re: First time Splunk Light Cloud setup: Why is splunkd down? in Splunk Enterprise

First time Splunk Light Cloud setup: Why is splunkd down?

bandries — Fri, 19 Aug 2016 22:51:32 GMT

Okay, I love it when stupid simple things don't work right out of the gate.

I signed up for Splunk Light Cloud Trial. I want to send logs written to a file on Ubuntu to Splunk. I followed instructions here, per the email I got: http://docs.splunk.com/Documentation/SplunkLight/6.4.2/Cloud/GettingdataintoSplunkLightcloudserviceusingLinux

Below is the complete (sniped) output (and yes I used something for username:password below that's valid). So, can someone explain what stupid simple steps I missed that 'are not' in the documentation?

~$ sudo tar -xvzf splunkforwarder-6.4.2-00f5bb3fa822-Linux-x86_64.tgz -C /opt
..snip list of files from tar...
~$ cd /opt/splunkforwarder/bin
:/opt/splunkforwarder/bin$ ll
total 35792
drwxr-xr-x 3 506 506     4096 Jun 28 11:13 ./
drwxr-xr-x 8 506 506     4096 Jun 28 11:13 ../
-r-xr-xr-x 1 506 506    55600 Jun 28 11:13 btool*
-r-xr-xr-x 1 506 506    55600 Jun 28 11:13 btprobe*
-r-xr-xr-x 1 506 506    38208 Jun 28 11:13 bzip2*
-r-xr-xr-x 1 506 506    55600 Jun 28 11:13 classify*
-r--r--r-- 1 506 506       57 Jun 28 09:59 copyright.txt
-r-xr-xr-x 1 506 506     2367 Jun 28 09:59 genRootCA.sh*
-r-xr-xr-x 1 506 506      206 Jun 28 09:59 genSignedServerCert.sh*
-r-xr-xr-x 1 506 506      144 Jun 28 09:59 genWebCert.sh*
-r-xr-xr-x 1 506 506   586632 Jun 28 11:13 openssl*
-r-xr-xr-x 1 506 506     7390 Jun 28 09:59 pid_check.sh*
drwxr-xr-x 2 506 506     4096 Jun 28 11:09 scripts/
-r--r--r-- 1 506 506     1360 Jun 28 09:59 setSplunkEnv
-r-xr-xr-x 1 506 506   461032 Jun 28 11:13 splunk*
-r-xr-xr-x 1 506 506 35290832 Jun 28 11:13 splunkd*
-r-xr-xr-x 1 506 506      465 Jun 28 09:59 splunkdj*
-r-xr-xr-x 1 506 506    16072 Jun 28 11:13 splunkmon*
-r-xr-xr-x 1 506 506    28872 Jun 28 11:13 srm*

:/opt/splunkforwarder/bin$ sudo ./splunk install app /home/devadmin/splunkclouduf.spl -auth username:password
...snip...

This appears to be your first time running this version of Splunk.
This command [POST /services/apps/local/] needs splunkd to be up, and splunkd is down.
:/opt/splunkforwarder/bin$

Re: First time Splunk Light Cloud setup: Why is splunkd down?

andrewb_splunk — Sat, 20 Aug 2016 00:23:32 GMT

I haven't installed a *nix Universal Forwarder, but after initial install you might need to explicitly start it:

$SPLUNK_HOME/bin/splunk start

Re: First time Splunk Light Cloud setup: Why is splunkd down?

bandries — Sat, 20 Aug 2016 00:56:31 GMT

Makes sense. Tried that then the splunk install command and got "Login failed" which makes no sense. I then took the username and password I'm using and successfully logged into the splunk console. Unless this is a different username and password?

Console output below:

devadmin@ip-10-98-86-21:/opt/splunkforwarder/bin$ sudo ./splunk start

Splunk> 4TW

Checking prerequisites...
Checking mgmt port [8089]: open
Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n
Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css
Creating: /opt/splunkforwarder/var/run/splunk/upload
Creating: /opt/splunkforwarder/var/spool/splunk
Creating: /opt/splunkforwarder/var/spool/dirmoncache
Creating: /opt/splunkforwarder/var/lib/splunk/authDb
Creating: /opt/splunkforwarder/var/lib/splunk/hashDb
New certs have been generated in '/opt/splunkforwarder/etc/auth'.
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-6.4.2-00f5bb3fa822-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Generating a 1024 bit RSA private key
...++++++
............++++++

writing new private key to 'privKeySecure.pem'

Signature ok
subject=/CN=ip-10-98-86-21/O=SplunkUser
Getting CA Private Key
writing RSA key
Done

devadmin@ip-10-98-86-21:/opt/splunkforwarder/bin$ sudo ./splunk install app /home/devadmin/splunkclouduf.spl -auth username:password
Login failed
:/opt/splunkforwarder/bin$

Re: First time Splunk Light Cloud setup: Why is splunkd down?

esix_splunk — Sat, 20 Aug 2016 11:44:40 GMT

The first time you install and run the Splunk UF/HF, you need to accept the license before it will start correctly.

You can do..

/opt/splunkforwarder/bin/splunk start --accept-license

That and check your permissions, confirm that the user you are wanting to run Splunk as owns everything.

Re: First time Splunk Light Cloud setup: Why is splunkd down?

malmoore — Mon, 22 Aug 2016 16:56:18 GMT

The default credentials are 'admin' and 'changeme'.

Re: First time Splunk Light Cloud setup: Why is splunkd down?

bandries — Mon, 22 Aug 2016 17:34:11 GMT

What the heck?

devadmin@ip-10-98-86-21:/opt/splunkforwarder/bin$ sudo ./splunk install app /home/devadmin/splunkclouduf.spl -auth admin:changeme
App '/home/devadmin/splunkclouduf.spl' installed

I'm using SplunkLight Cloud service and the 'admin' password is 'changeme'? Where do I change this ridiculous password? Should forwarding clients being using some sort of account called 'admin'?

Doesn't someone think this might need to be in the documentation?
Doesn't someone think this sort of default credentials is a bad idea?

Re: First time Splunk Light Cloud setup: Why is splunkd down?

andrewb_splunk — Mon, 22 Aug 2016 17:37:45 GMT

admin/changeme is the default login for the Universal Forwarder, not for your Splunk Light cloud account. After your first login to the UF, you should change that to something secure.

You access your Splunk Light cloud trial via single sign-on after logging into your Splunk.com user account. The username and password is the one you chose when you signed up.

We will take another look at the documentation to make sure that this is clearer.

Re: First time Splunk Light Cloud setup: Why is splunkd down?

bandries — Mon, 22 Aug 2016 17:40:01 GMT

Okay, so I do see admin:changeme in the documentation, but this really doesn't make any sense. If that account exists on the machine and not in the cloud, step 1 should be 'change the password'.

Re: First time Splunk Light Cloud setup: Why is splunkd down?

malmoore — Mon, 22 Aug 2016 19:20:31 GMT

Hi,

There are two sets of credentials here, one for Splunk Light Cloud, and the other for the Universal Forwarder, and maybe that's what's causing the confusion.

The universal forwarder software comes with its own set of credentials, as @andrewb described above. These are different from the credentials that you log into Splunk Light Cloud with.

It is good practice to change the credentials for the universal forwarder, and you can do that if you want to. But that change does not affect your Splunk Light Cloud credentials, and vice versa.

I hope this helps clarify things.

Re: First time Splunk Light Cloud setup: Why is splunkd down?

bandries — Tue, 23 Aug 2016 19:17:05 GMT

Believe me, I appreciate the 'quick start' documentation that doesn't go through all the onerous in's and out's of everything before you can get something up and running. The issue is as I stated, there are a few things missing, such as ./splunk start, and the password issue. If there is a default password of 'changeme' then obviously it should be changed as a step 1.

In San Diego, a TV station didn't change their default password for their emergency alert SCADA system, and the next thing they knew there was an interruption of local TV news from the emergency broadcast system describing a zombie outbreak and advising residents to stay indoors! In best Sys Admin practice world it should be well known by now, one does not change passwords like 'admin:changeme' if they want to, rather that should be the very first thing you do and documentation from software vendors should support that regardless of what they're protecting.

At first I thought it was my splunk login I should have been using. It's still not clear to me if that admin account exists locally on the machine or in the Splunk server/cloud account. It's just some mysterious account called admin that's securing I have no idea what. As a new user, I shouldn't have to start at the beginning of the full documentation to figure out how to change a password for something that's so painfully obvious.

Thanks for considering a documentation update for this.

Re: First time Splunk Light Cloud setup: Why is splunkd down?

gneumann_splunk — Tue, 23 Aug 2016 20:15:30 GMT

Thanks bandries for your input and suggestions about improving the documentation for this issue. I will update the documentation to add clarity about the username and password when installing the universal forwarder credentials, add information about changing the username and password, and more information about the admin account.

Also, our instructions indicate performing a ./splunk restart on the universal forwarder after setting the deploy-poll command, but I'll also add information about ./splunk start in case there is an issue.

Thanks again for your input for documentation improvements. We appreciate it!