topic Re: What am I missing to get a successful connection between my Universal Forwarder and the sandbox? in Getting Data In

What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

appzen — Fri, 09 Jan 2015 22:38:28 GMT

I followed the tutorial very carefully on setting up the forwarder on my two Tomcat servers. Now I am trying to verify that I can actually receive data from my catalina logs to my sandbox. When I go to 'Add Data', and click on 'forward' it gives me the notice: "There are currently no forwarders configured as deployment clients to this instance." But at the top of my screen I get another notice stating that: "Forwarding to indexer group default-autolb-group blocked for 1200 seconds.", which 'default-autolb-group' is the defaultGroup in my /opt/splunkforwarder/etc/system/local/output.conf file. I think that I am close on getting a connection but I am missing some step to complete it. Can someone help me on what I missing to verify a successful connection?

Also, my inputs.conf file only has the ip address of my server; do I need to put information about my catalina log file and if so what is the format, thanks!

Re: What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

chanfoli — Sat, 10 Jan 2015 18:42:47 GMT

Please post your inputs.conf and outputs.conf files. In a simple setup on your forwarder you should have your sandbox set up as a forward server and your inputs should be defined.

For tomcat, you would want monitor stanza(s) specifying the files you want to start indexing. I just answered another question (here: http://answers.splunk.com/answers/207373/why-am-getting-error-there-are-currently-no-forwar.html ) with regards to the "deployment clients" error. It seems that some information about setting up deployment clients has been left out here for the way sandbox "wizards" are designed. I am thinking that you are pretty close and perhaps seeing the conf files will help get it straightened out.

Re: What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

chanfoli — Sat, 10 Jan 2015 20:20:40 GMT

I've also tried to get this going myself since I am seeing a lot of similar questions from folks having problems. For one thing, I learned that the sandbox server needs to have input- appended to the hostname in order to actually connect to the correct IP. After you get this far, you will probably see as I did that your connection to sandbox gets reset, this appears to be because splunk has made some changes to make this "easier". There are apparently some embedded credentials in a special forwarder package which need to be used. I guess this is not going to work for the universal forwarder that I installed on my Raspberry Pi. Hopefully they will improve the documentation as there is nothing to guide even experienced splunk users to getting this connection to work manually. See the last comment on this question for a clue about why so many might be having issues with sandbox trial inputs:

http://answers.splunk.com/answers/147295/how-do-i-send-my-own-data-into-a-splunk-cloud-sandbox-trial.html

Re: What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

chanfoli — Sat, 10 Jan 2015 20:25:48 GMT

Paraphrasing my above comment as an answer: If you are getting connection reset errors like I am from my Raspberry Pi Universal Forwarder, it would appear that there have been some changes made involving authenticating external inputs. I found this by digging around and trying different options and not getting my connection to work, then seeing the last comment on this answers post:

http://answers.splunk.com/answers/147295/how-do-i-send-my-own-data-into-a-splunk-cloud-sandbox-trial.html

[excerpt]

"The answers given above were valid at the time of writing but recently we secured all data inputs with a unique SSL certificate and key for each instance so you cannot just manually add the config files and make it work anymore. You have to download the universal forwarder app which has the required credentials embedded."

EDIT: The following helped get this working!

Log into your sandbox instance and click on Universal Forwarder from your launch page.
Click on the button to download the cloud credentials.
Install this as an app on your forwarder ( /opt/splunkforwarder/bin/splunk install app /PATH/TO/splunkcouduf.spl )
Make sure your output is named splunkcloud in your outputs.conf - mine is below
Restart splunk

[tcpout]
defaultGroup = splunkcloud

[tcpout:splunkcloud]
server = input-prd-p-MYSERVERID.cloud.splunk.com:9997

Re: What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

chanfoli — Sun, 11 Jan 2015 04:09:12 GMT

Please note my edit at the end of my answer, it may help you.

Re: What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

appzen — Mon, 12 Jan 2015 18:45:44 GMT

I followed your last comment and my outputs.conf is:

[tcpout-server://input-prd-p-c325dgfktbm7.cloud.splunk.com:9997]

[tcpout:splunkcloud]
disabled = false
server = input-prd-p-c325dgfktbm7.cloud.splunk.com:9997

[tcpout]
defaultGroup = splunkcloud

and my inputs.conf is:

[default]
host = ip-172-31-35-141

I have only made changes to my outputs.conf and I am not sure on what to change for inputs

Re: What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

chanfoli — Mon, 12 Jan 2015 19:09:59 GMT

You will need to have appropriate monitor stanzas on the forwarder for the tomcat logs you want to start indexing, ideally these will also need to be assigned an appropriate sourcetype.

Have a look at this:
http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitorfilesanddirectories

Here is another answer which should get you in the right direction on inputs. This person appears to have set up different sourcetypes for the different logs:
http://answers.splunk.com/answers/135355/proper-input-conf-setup-apache-tomcat.html

My procedure is to load an example file on a splunk instance through add data and use the "data preview" functionality it to make sure timestamps and event breaks are getting parsed and what sourcetype settings are needed to make this happen for each sourcetype.

BTW, I removed tcpout-server stanza from my outputs.conf before my remote forwarder actually connected to the sandbox and forwarded events.

Re: What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

appzen — Mon, 12 Jan 2015 19:12:21 GMT

I also realized that I am changing my files from /opt/splunkforwarder/etc/system/local/outputs.conf but should it be from /opt/splunkforwarder/etc/apps/search/local?

Re: What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

chanfoli — Mon, 12 Jan 2015 19:14:52 GMT

In my opinion, no. The configs under SPLUNKHOME/etc/apps/search for the search app, which is not relevant on a Universal Forwarder system.

Re: What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

appzen — Mon, 12 Jan 2015 19:18:46 GMT

thanks for your help, quick question about the monitor, i cant simply just do

/opt/splunkforwarder/bin/splunk add monitor /var/lib/tomcat7/logs

to add a monitor?

Re: What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

appzen — Mon, 12 Jan 2015 19:20:42 GMT

and if my inputs.conf file isn't correctly setup with a monitor, would that be the reason why I am still not picking up the forwarder?

Re: What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

appzen — Mon, 12 Jan 2015 19:23:50 GMT

I just changed my inputs.conf to:

[default]
host = ip-172-31-35-141

[monitor:/var/lib/tomcat7/logs/catalina.*]

disabled = false
index = test
sourcetype = catalina

Re: What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

chanfoli — Mon, 12 Jan 2015 19:31:40 GMT

At this point I would check the splunkd.logs on your fowarder and run the following search on your sandbox:
index=_internal xx.xx.xx.xx

where xx.xx.xx.xx is your forwarder's outside IP address.

This might provide some clues about connection status.

Re: What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

appzen — Mon, 12 Jan 2015 19:34:59 GMT

where are the splunkd.logs located?

Re: What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

chanfoli — Mon, 12 Jan 2015 19:40:59 GMT

SPLUNKHOME/var/log/splunk/splunkd.logs

On *nix home is usually /opt/splunkforwarder and on windows it would be under Program Files/splunkforwarder

Re: What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

appzen — Mon, 12 Jan 2015 19:42:02 GMT

i did this command: index="_internal" 54.174.120.69 source="/opt/splunk/var/log/splunk/splunkd.log" and I get this error:

1/12/15
6:30:44.095 PM

01-12-2015 18:30:44.095 +0000 ERROR TcpInputProc - Error encountered for connection from src=54.174.120.69:60649. Local side shutting down
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/12/15
6:30:44.095 PM

01-12-2015 18:30:44.095 +0000 ERROR TcpInputProc - Error encountered for connection from src=54.174.120.69:60648. Local side shutting down
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/12/15
6:30:44.045 PM

01-12-2015 18:30:44.045 +0000 ERROR TcpInputProc - Error encountered for connection from src=54.174.120.69:60546. Local side shutting down
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/10/15
12:28:46.502 AM
01-10-2015 00:28:46.502 +0000 WARN TcpOutputProc - Cooked connection to ip=54.174.120.69:9997 timed out
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/10/15
12:28:16.500 AM
01-10-2015 00:28:16.500 +0000 WARN TcpOutputProc - Cooked connection to ip=54.174.120.69:9997 timed out
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd

Re: What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

appzen — Mon, 28 Sep 2020 18:37:12 GMT

and with the first command, index=internal xx.xx.xx.xx, i get:
1/12/15
7:43:52.260 PM

192.168.48.247 - admin [12/Jan/2015:19:43:52.260 +0000] "GET /en-US/api/shelper?snippet=true&snippetEmbedJS=false&namespace=search&search=search+index%3D%22_audi%22+54.174.120.69&useTypeahead=true&useAssistant=true&showCommandHelp=true&showCommandHistory=true&showFieldInfo=false&=1421091322744 HTTP/1.0" 200 641 "https://prd-p-c325dgfktbm7.cloud.splunk.com/en-US/app/search/search?q=search%20index%3D%22_audit%22%2054.174.120.69&earliest=&latest=&display.page.search.tab=events&sid=1421091825.12991" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36" - 54b423f8427f421431a250 20ms
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/web_access.log sourcetype = splunk_web_access
1/12/15
7:43:46.729 PM
01-12-2015 19:43:46.729 +0000 INFO StatusMgr - destPort=9997, eventType=connect_close, group=tcpin_connections, sourceHost=54.174.120.69, sourceIp=54.174.120.69, sourcePort=33886, statusee=TcpInputProcessor
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/metrics.log sourcetype = splunkd
1/12/15
7:43:46.707 PM
01-12-2015 19:43:46.707 +0000 INFO StatusMgr - destPort=9997, eventType=connect_done, group=tcpin_connections, sourceHost=54.174.120.69, sourceIp=54.174.120.69, sourcePort=33886, statusee=TcpInputProcessor
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/metrics.log sourcetype = splunkd

Re: What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

appzen — Mon, 12 Jan 2015 19:51:29 GMT

and it appears that I cant access my splunkd.logs from my forwarder

Re: What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

chanfoli — Mon, 12 Jan 2015 20:02:29 GMT

Okay a couple of things here. Is the 54.x.x.69 IP your universal forwarder? A couple of log entries indicate that something was trying to forward logs TO this IP which makes me think that this is your sandbox IP or there was some other misconfiguration.... Also, the local side shutting down errors might be missed heartbeats and could simply be when splunk is being restarted.

please execute the following on your forwarder to check connectivity:

splunk list forward-server and again checking the splunkd.log from the forwarder might help.

Re: What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

chanfoli — Mon, 12 Jan 2015 20:04:39 GMT

Sorry missed this. Do not have administrative access to this system or are you just not finding the log?