Why is data not being forwarded to my Splunk Light Cloud?

[brianackermann]

I fear I'm suffering from a number of interrelated issues. The top most issue is that no data is coming through from my forwarder to my Splunk Light Cloud instance.

My setup is as basic as I can imagine:

• I am demo-ing a Splunk Light Cloud instance.
• I have created a single account.
  
• I have downloaded the universal forwarders for windows, and installed it on my local machine.
  
• I have downloaded the credentials file

From within Splunk, I can see that my forwarder is "phoning home", so at least that much is working. But there isn't any data coming through.

When I try to install the credentials,

splunk install app "C:\Users\brian.TREES\Downloads\splunkclouduf.spl" -auth user:pass

I get the error Login Failed . I have triple checked that the user:pass I am using works, by logging into the portal again. I don't see anywhere where this might be configured. There is only the one user that I can see.

From somewhere else on this forum, I saw a possible answer to my main problem (that no info is being forwarded) here: https://answers.splunk.com/answers/400954/how-to-troubleshoot-why-a-universal-forwarder-is-n-2.html
But unfortunately, it suffers from the same problem.. I can't login

splunk add forward-server  -auth user:pass

So I guess the main question is, what do I do about this Login Failed problem? Is this NOT the credentials I use to log into the Splunk cloud instance? If not, where do I set-up new users in the interface ?

Once this Login hurdle is passed, am I on the right track, for my most basic situation?

Thanks

Edit 10-13-2016 9:45am

All issues with UniversalForwarder authenticcation have been resolved, per my comment below. However, the issue remains that no data is being sent to the Cloud Light instance. I have added an outputs.conf file (as an experiment) with the content:

[tcpout:group1]
server=prd-p-{REDACTED}.cloud.splunk.com:9997

I'm not the least bit certain about the port number, but that seems to be what the examples show. Its not working, though, because I'm getting this error in my splunkd.log :

10-13-2016 09:38:39.279 -0500 WARN TcpOutputProc - Cooked connection to ip=XX.YY.ZZ.29:9997 timed out

Edit 10-13-2016 9:50am

Well, this is new. Probably new since I fixed the credentials problem yestreday afternoon, but, when i look at Manage Indexes, I now see that some data is being loaded...

asdf Edit Delete Disable 3 MB 5 GB 15K 13 days ago 25 minutes ago 5 days

I've got some 15K events in my index! But when I head to the search tab....it still tells me:

No data has been added. Please add data.

[brianackermann · Accepted solution]

It appears that correctly credentialing the UniversalForwarder app was sufficient to get everything working. My remaining problems seem to be related to the fact that I decided to use a different index to house this data, and because of that, nothing shows up in the search, unless I manually specify it (which is probably a situation covered in the documentation somewhere, but haven't yet seen/read)

At any rate, I am successfully adding at least SOME events to the logger via the UF, so I am satisfied.

[gneumann_splunk]

Glad you resolved the issues you were having. I reviewed our basic universal forwarder installation instructions with our QA, and they seem fine. Our QA person also reviewed the problems you were having, and he stated to be sure to search index=new index name. Basically, does your search contain a directive to look in the new index. It seems you also came to this conclusion.

[brianackermann]

I'm working too fast. I missed this Note on Step 4 of the document Forward data to Splunk Light cloud service using Microsoft Windows

Note: When you install the credentials file into the universal forwarder, note that the default username and password for a first-time installation of the universal forwarder is admin:changeme. To change the admin password, run the edit user command. For example: splunk edit user admin -password foo -auth admin:changeme.

I've changed the password, and updated my efforts to use the admin:newpassword auth ... I'm now past the failed login issue, but I'm still not sending data. I then tried the add forward-server idea, but I guess I don't have a clue what url to provide it... and I don't see it documented anywhere (though I'm probably missing it 😄 )

[gneumann_splunk]

What operating system are you using? Windows?

[gneumann_splunk]

Sorry..yes, looks like you're using Windows

[gneumann_splunk]

checking on some options to try and resolve your issue...

[gneumann_splunk]

I've been reviewing the issue about your data not being forwarded, and will continue my review/testing tomorrow with our QA. I'll get back to you with any further suggestions to resolve this issue. In the interim, a few questions/suggestions:
- Did you "uncheck" the box at the beginning of the Windows universal forwarder installation wizard to indicate you want this forwarder to contact a cloud instance?

- The forward-server command is only configured for on-premises instances.
- The deploy-poll command is configured for both on-premises and cloud instances, but the Windows installer configures this as long as you add your cloud service hostname, such as "input-abc-d-12abcdefghij.cloud.splunk.com" during the wizard steps. For Mac and Linux installs, you have to to manually configure the deploy-poll command.
- For cloud instances, it does take some time, such as 15 minutes or so, to have your forwarders and data display in your instance due to the instance talking to the cloud.
- You can delete the forwarder and credentials and reinstall. If you do so, be sure to empty your trash and restart your system for a clean install, and follow each step of the installation instructions.

[brianackermann]

yes, I unchecked that box.
yes, I added "input-" prefix to my cloud service hostname.
It has been several hours, and still no data.
I have uninstalled and reinstalled several times (but not since worked out that bit about the UF password being different from my account password.