Active forwards don't forward data to Splunk Cloud instance

[johnwl]

I use username: admin and password: changeme to log in to my Splunk universal forwarder. I am trying to forward logs from my Ubuntu server that's running on Vagrant VM. I know that the forwarder is active because:

---

root@vagrant-ubuntu-trusty-64:/opt/splunkforwarder/bin# ./splunk list forward-server
Your session is invalid.  Please login.
Splunk username: admin
Password: 
Active forwards:
    192.168.33.10:9997
Configured but inactive forwards:
    None

---

[monitor://$SPLUNK_HOME/var/log/splunk]
index = _internal

[monitor://$SPLUNK_HOME/etc/splunk.version]
_TCP_ROUTING = *
index = _internal
sourcetype=splunk_version

[monitor:///var/log/upstart/docker.log]

[monitor:///var/log/upstart/]

[monitor:///var/log/]

---

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 192.168.33.10:9997

[tcpout-server://192.168.33.10:9997]

---

I configured the guest and host ports on the Vagrant VM as 9997. But nothing at all is being sent to my Splunk Cloud. Any help?!!! Thanks

[johnwl · Accepted solution]

I solved the problem. I had to put the following into the outputs.conf file:

[tcpout]
defaultGroup = default-autolb-group,splunkcloud

[esix_splunk]

One thing to note about this configuration, is that if one of the group defined is down or blocked, both tcpoutputs will halt. If you need to dual stream to your cloud instance, and your local instance, its recommended to install another UF on your host and route data to your cloud instance from a different UF.

[johnwl]

I just installed the security file splunkclouduf.spl and I have some progress: The spunk cloud instance is finally listed as a forward input-prd-p-m56vqljf2w93.cloud.splunk.com:9997 (ssl) .

---

Active forwards:
192.168.33.10:9997
input-prd-p-m56vqljf2w93.cloud.splunk.com:9997 (ssl)
Configured but inactive forwards:
None

---

The data still doesn't forward though.

[esix_splunk]

If you are seeing the cloud indexers in your forwarder, its most likely working. On your cloud SH, run a search on the _internal index and see if you can see your forwarder.

Aside from that, what does your outputs look like? What sourcetype and index are you sending to?

[johnwl]

Thanks for helping me. I have not specified any specific index for indexing. I am getting hundreds of thousands of events, but they are all from the spunk log, and not from the path that I wanted them to come from (/var/log/mylogs/). This is an example of what an entry looks like in my Splunk Cloud interface:

6/22/15 6:04:21.485 PM

2015-06-22 18:04:21,485 WARNING Generator Queue Full, looping
host = ip-192-168-106-249 source = /opt/splunk/var/log/splunk/eventgen.log sourcetype = eventgen

Any idea on what I can do to get them to come from the path I specified?

[ChrisG]

Are you using the Splunk Cloud universal forwarder app for the certificate and credentials?

[johnwl]

Yes, from the page where all of this is listed:

---

Use this app to set up the universal forwarder. After you download the universal forwarder, follow the steps to install and start it, then specify the data you want to send to Splunk Cloud.

To set up the Universal Forwarder:
Download the universal forwarder from splunk.com to the /opt directory on the machine that will send data to Splunk Cloud.

Download the universal forwarder credentials to the /opt directory of the machine that will send data to Splunk Cloud.

Install the universal forwarder on your operating system by following the Splunk Enterprise installation instructions.
Install the universal forwarder credentials by entering the following command:

/opt/splunkforwarder/bin/splunk install app /opt/splunkclouduf.spl -auth admin:changeme
Add data to Splunk Cloud using the command line interface (CLI).
For example, add application logs to Splunk Cloud using the following command:

/opt/splunkforwarder/bin/splunk add monitor -auth admin:changeme /path/to/app/logs/
Where /path/to/app/logs/ is the path to application logs that you want to add to Splunk Cloud.

---

I downloaded the 64-bit linux distribution .deb and saw the following when I executed it yesterday:

---

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ sudo dpkg -i splunkforwarder-6.2.3-264376-linux-2.6-amd64.deb
Selecting previously unselected package splunkforwarder.
(Reading database ... 97090 files and directories currently installed.)
Preparing to unpack splunkforwarder-6.2.3-264376-linux-2.6-amd64.deb ...
Unpacking splunkforwarder (6.2.3) ...
Setting up splunkforwarder (6.2.3) ...
complete