I followed the tutorial very carefully on setting up the forwarder on my two Tomcat servers. Now I am trying to verify that I can actually receive data from my catalina logs to my sandbox. When I go to 'Add Data', and click on 'forward' it gives me the notice: "There are currently no forwarders configured as deployment clients to this instance." But at the top of my screen I get another notice stating that: "Forwarding to indexer group default-autolb-group blocked for 1200 seconds.", which 'default-autolb-group' is the defaultGroup in my /opt/splunkforwarder/etc/system/local/output.conf file. I think that I am close on getting a connection but I am missing some step to complete it. Can someone help me on what I missing to verify a successful connection?
Also, my inputs.conf file only has the ip address of my server; do I need to put information about my catalina log file and if so what is the format, thanks!
Paraphrasing my above comment as an answer: If you are getting connection reset errors like I am from my Raspberry Pi Universal Forwarder, it would appear that there have been some changes made involving authenticating external inputs. I found this by digging around and trying different options and not getting my connection to work, then seeing the last comment on this answers post:
[excerpt]
"The answers given above were valid at the time of writing but recently we secured all data inputs with a unique SSL certificate and key for each instance so you cannot just manually add the config files and make it work anymore. You have to download the universal forwarder app which has the required credentials embedded."
EDIT: The following helped get this working!
Restart splunk
[tcpout]
defaultGroup = splunkcloud
[tcpout:splunkcloud]
server = input-prd-p-MYSERVERID.cloud.splunk.com:9997
Please note my edit at the end of my answer, it may help you.
I've also tried to get this going myself since I am seeing a lot of similar questions from folks having problems. For one thing, I learned that the sandbox server needs to have input- appended to the hostname in order to actually connect to the correct IP. After you get this far, you will probably see as I did that your connection to sandbox gets reset, this appears to be because splunk has made some changes to make this "easier". There are apparently some embedded credentials in a special forwarder package which need to be used. I guess this is not going to work for the universal forwarder that I installed on my Raspberry Pi. Hopefully they will improve the documentation as there is nothing to guide even experienced splunk users to getting this connection to work manually. See the last comment on this question for a clue about why so many might be having issues with sandbox trial inputs:
Please post your inputs.conf and outputs.conf files. In a simple setup on your forwarder you should have your sandbox set up as a forward server and your inputs should be defined.
For tomcat, you would want monitor stanza(s) specifying the files you want to start indexing. I just answered another question (here: http://answers.splunk.com/answers/207373/why-am-getting-error-there-are-currently-no-forwar.html ) with regards to the "deployment clients" error. It seems that some information about setting up deployment clients has been left out here for the way sandbox "wizards" are designed. I am thinking that you are pretty close and perhaps seeing the conf files will help get it straightened out.
I followed your last comment and my outputs.conf is:
[tcpout-server://input-prd-p-c325dgfktbm7.cloud.splunk.com:9997]
[tcpout:splunkcloud]
disabled = false
server = input-prd-p-c325dgfktbm7.cloud.splunk.com:9997
[tcpout]
defaultGroup = splunkcloud
and my inputs.conf is:
[default]
host = ip-172-31-35-141
I have only made changes to my outputs.conf and I am not sure on what to change for inputs
You will need to have appropriate monitor stanzas on the forwarder for the tomcat logs you want to start indexing, ideally these will also need to be assigned an appropriate sourcetype.
Have a look at this:
http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitorfilesanddirectories
Here is another answer which should get you in the right direction on inputs. This person appears to have set up different sourcetypes for the different logs:
http://answers.splunk.com/answers/135355/proper-input-conf-setup-apache-tomcat.html
My procedure is to load an example file on a splunk instance through add data and use the "data preview" functionality it to make sure timestamps and event breaks are getting parsed and what sourcetype settings are needed to make this happen for each sourcetype.
BTW, I removed tcpout-server stanza from my outputs.conf before my remote forwarder actually connected to the sandbox and forwarded events.
I also realized that I am changing my files from /opt/splunkforwarder/etc/system/local/outputs.conf but should it be from /opt/splunkforwarder/etc/apps/search/local?
In my opinion, no. The configs under SPLUNKHOME/etc/apps/search for the search app, which is not relevant on a Universal Forwarder system.
thanks for your help, quick question about the monitor, i cant simply just do
/opt/splunkforwarder/bin/splunk add monitor /var/lib/tomcat7/logs
to add a monitor?
I just changed my inputs.conf to:
[default]
host = ip-172-31-35-141
[monitor:/var/lib/tomcat7/logs/catalina.*]
disabled = false
index = test
sourcetype = catalina
At this point I would check the splunkd.logs on your fowarder and run the following search on your sandbox:
index=_internal xx.xx.xx.xx
where xx.xx.xx.xx is your forwarder's outside IP address.
This might provide some clues about connection status.
where are the splunkd.logs located?
i did this command: index="_internal" 54.174.120.69 source="/opt/splunk/var/log/splunk/splunkd.log" and I get this error:
1/12/15
6:30:44.095 PM
01-12-2015 18:30:44.095 +0000 ERROR TcpInputProc - Error encountered for connection from src=54.174.120.69:60649. Local side shutting down
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/12/15
6:30:44.095 PM
01-12-2015 18:30:44.095 +0000 ERROR TcpInputProc - Error encountered for connection from src=54.174.120.69:60648. Local side shutting down
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/12/15
6:30:44.045 PM
01-12-2015 18:30:44.045 +0000 ERROR TcpInputProc - Error encountered for connection from src=54.174.120.69:60546. Local side shutting down
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/10/15
12:28:46.502 AM
01-10-2015 00:28:46.502 +0000 WARN TcpOutputProc - Cooked connection to ip=54.174.120.69:9997 timed out
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/10/15
12:28:16.500 AM
01-10-2015 00:28:16.500 +0000 WARN TcpOutputProc - Cooked connection to ip=54.174.120.69:9997 timed out
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
and with the first command, index=internal xx.xx.xx.xx, i get:
1/12/15
7:43:52.260 PM
192.168.48.247 - admin [12/Jan/2015:19:43:52.260 +0000] "GET /en-US/api/shelper?snippet=true&snippetEmbedJS=false&namespace=search&search=search+index%3D%22_audi%22+54.174.120.69&useTypeahead=true&useAssistant=true&showCommandHelp=true&showCommandHistory=true&showFieldInfo=false&=1421091322744 HTTP/1.0" 200 641 "https://prd-p-c325dgfktbm7.cloud.splunk.com/en-US/app/search/search?q=search%20index%3D%22_audit%22%..." "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36" - 54b423f8427f421431a250 20ms
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/web_access.log sourcetype = splunk_web_access
1/12/15
7:43:46.729 PM
01-12-2015 19:43:46.729 +0000 INFO StatusMgr - destPort=9997, eventType=connect_close, group=tcpin_connections, sourceHost=54.174.120.69, sourceIp=54.174.120.69, sourcePort=33886, statusee=TcpInputProcessor
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/metrics.log sourcetype = splunkd
1/12/15
7:43:46.707 PM
01-12-2015 19:43:46.707 +0000 INFO StatusMgr - destPort=9997, eventType=connect_done, group=tcpin_connections, sourceHost=54.174.120.69, sourceIp=54.174.120.69, sourcePort=33886, statusee=TcpInputProcessor
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/metrics.log sourcetype = splunkd
Okay a couple of things here. Is the 54.x.x.69 IP your universal forwarder? A couple of log entries indicate that something was trying to forward logs TO this IP which makes me think that this is your sandbox IP or there was some other misconfiguration.... Also, the local side shutting down errors might be missed heartbeats and could simply be when splunk is being restarted.
please execute the following on your forwarder to check connectivity:
splunk list forward-server and again checking the splunkd.log from the forwarder might help.
54.x.x.69 is the IP where the universal forwarder was dled. After running the command I get:
Active forwards:
input-prd-p-c325dgfktbm7.cloud.splunk.com:9997 (ssl)
Configured but inactive forwards:
None
and Im trying to check the logs from the forwarder but I don't think any exists, but Ill try again
okay, so it looks like your connection is up. Since you are setting host to ip-172-31-35-141 in your inputs you should be able to search for host=ip-172-31-35-141 ( i would use all-time for troubleshooting in this case, just in case there are timestamp discrepancies). You also need to make sure that the user the splunk forwarder is running as has read permissions on the logs you have added.
Also, based on the above config, you will need to have created a "test" index on your sandbox as well, and depending on user/role you may need to use index=test in your search. Also, did you create a catalina sourcetype on your sandbox instance? If your data is not one of the built in types, this should be done.
when i do this command, host="ip-172-31-35-141" source="/var/lib/tomcat7/logs/catalina.out", I get a bunch of logs, is this the results that I should be getting?
It sounds like your monitor input is getting indexed on your sandbox instance. Yay! You did it!