splunkforwarder not forwarding to sandbox with TcpOutputFd

[will_paxata]

I am having trouble getting started with a sandbox. I would love some help so I can start getting value out of splunk and become a paying customer, etc.

In my sandbox dashboard at https://prd-p-rnfbdk7swh3x.cloud.splunk.com/en-US/app/search/search, I see no data has been received. The host with the splunkforwarder shows this in its splunkd.log:

INFO  TcpOutputProc - Connected to idx=54.86.164.71:9997 using ACK.
ERROR TcpOutputFd - Read error. Connection reset by peer
ERROR TcpOutputFd - Read error. Connection reset by peer
... repeating ...

I believe the forward-server is correctly configured:

[root@qa-c1-ps etc]# /opt/splunkforwarder/bin/splunk list forward-server
Active forwards:
    input-prd-p-rnfbdk7swh3x.cloud.splunk.com:9997 (ssl)
Configured but inactive forwards:
    None

My splunkforwarder/etc/system/local/inputs.conf looks like this:

[default]
host = qa-c1-ps.paxatadev.com

and my splunkforwarder/etc/system/local/outputs.conf looks like this:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = input-prd-p-rnfbdk7swh3x.cloud.splunk.com:9997

[tcpout-server://input-prd-p-rnfbdk7swh3x.cloud.splunk.com:9997]

I have my monitored files configured also, and I have made sure that the qa-c1-ps host can access the sandbox at port 9997 via ssl. I appreciate any help anyone can provide.

[chanfoli · Accepted solution]

If you are getting connection reset errors like I am from my Raspberry Pi Universal Forwarder, it would appear that there have been some changes made involving authenticating external inputs. I noticed a lot of similar questions being posted so I decided to try sandbox and set up some inputs from a Pi I have at home. It looks like the steps required to add data from forwarders are much different than the simple process you would use on a normal splunk installation and they are not clear nor intuitive even to an experienced splunk user. I found this new and possibly relevant info by digging around and trying different options and not getting my connection to work, then finally seeing the last comment on this answers post:

http://answers.splunk.com/answers/147295/how-do-i-send-my-own-data-into-a-splunk-cloud-sandbox-trial...

[excerpt]

"The answers given above were valid at the time of writing but recently we secured all data inputs with a unique SSL certificate and key for each instance so you cannot just manually add the config files and make it work anymore. You have to download the universal forwarder app which has the required credentials embedded."

EDIT: The following helped get this working!

1. Log into your sandbox instance and click on Universal Forwarder from your launch page.
2. Click on the button to download the cloud credentials.
3. Install this as an app on your forwarder ( /opt/splunkforwarder/bin/splunk install app /PATH/TO/splunkcouduf.spl )
4. Make sure your output is named splunkcloud in your outputs.conf - mine is below

5. Restart splunk

   [tcpout]
   defaultGroup = splunkcloud

   [tcpout:splunkcloud]
   server = input-prd-p-MYSERVERID.cloud.splunk.com:9997

[chanfoli]

Please note my edit at the end of my answer, it may help you.