Getting Data In

What am I missing to get a successful connection between my Universal Forwarder and the sandbox?

appzen
Path Finder

I followed the tutorial very carefully on setting up the forwarder on my two Tomcat servers. Now I am trying to verify that I can actually receive data from my catalina logs to my sandbox. When I go to 'Add Data', and click on 'forward' it gives me the notice: "There are currently no forwarders configured as deployment clients to this instance." But at the top of my screen I get another notice stating that: "Forwarding to indexer group default-autolb-group blocked for 1200 seconds.", which 'default-autolb-group' is the defaultGroup in my /opt/splunkforwarder/etc/system/local/output.conf file. I think that I am close on getting a connection but I am missing some step to complete it. Can someone help me on what I missing to verify a successful connection?

Also, my inputs.conf file only has the ip address of my server; do I need to put information about my catalina log file and if so what is the format, thanks!

chanfoli
Builder

Paraphrasing my above comment as an answer: If you are getting connection reset errors like I am from my Raspberry Pi Universal Forwarder, it would appear that there have been some changes made involving authenticating external inputs. I found this by digging around and trying different options and not getting my connection to work, then seeing the last comment on this answers post:

http://answers.splunk.com/answers/147295/how-do-i-send-my-own-data-into-a-splunk-cloud-sandbox-trial...

[excerpt]

"The answers given above were valid at the time of writing but recently we secured all data inputs with a unique SSL certificate and key for each instance so you cannot just manually add the config files and make it work anymore. You have to download the universal forwarder app which has the required credentials embedded."

EDIT: The following helped get this working!

  1. Log into your sandbox instance and click on Universal Forwarder from your launch page.
  2. Click on the button to download the cloud credentials.
  3. Install this as an app on your forwarder ( /opt/splunkforwarder/bin/splunk install app /PATH/TO/splunkcouduf.spl )
  4. Make sure your output is named splunkcloud in your outputs.conf - mine is below
  5. Restart splunk

    [tcpout]
    defaultGroup = splunkcloud

    [tcpout:splunkcloud]
    server = input-prd-p-MYSERVERID.cloud.splunk.com:9997

chanfoli
Builder

Please note my edit at the end of my answer, it may help you.

0 Karma

chanfoli
Builder

I've also tried to get this going myself since I am seeing a lot of similar questions from folks having problems. For one thing, I learned that the sandbox server needs to have input- appended to the hostname in order to actually connect to the correct IP. After you get this far, you will probably see as I did that your connection to sandbox gets reset, this appears to be because splunk has made some changes to make this "easier". There are apparently some embedded credentials in a special forwarder package which need to be used. I guess this is not going to work for the universal forwarder that I installed on my Raspberry Pi. Hopefully they will improve the documentation as there is nothing to guide even experienced splunk users to getting this connection to work manually. See the last comment on this question for a clue about why so many might be having issues with sandbox trial inputs:

http://answers.splunk.com/answers/147295/how-do-i-send-my-own-data-into-a-splunk-cloud-sandbox-trial...

chanfoli
Builder

Please post your inputs.conf and outputs.conf files. In a simple setup on your forwarder you should have your sandbox set up as a forward server and your inputs should be defined.

For tomcat, you would want monitor stanza(s) specifying the files you want to start indexing. I just answered another question (here: http://answers.splunk.com/answers/207373/why-am-getting-error-there-are-currently-no-forwar.html ) with regards to the "deployment clients" error. It seems that some information about setting up deployment clients has been left out here for the way sandbox "wizards" are designed. I am thinking that you are pretty close and perhaps seeing the conf files will help get it straightened out.

0 Karma

appzen
Path Finder

I followed your last comment and my outputs.conf is:

[tcpout-server://input-prd-p-c325dgfktbm7.cloud.splunk.com:9997]

[tcpout:splunkcloud]
disabled = false
server = input-prd-p-c325dgfktbm7.cloud.splunk.com:9997

[tcpout]
defaultGroup = splunkcloud

and my inputs.conf is:

[default]
host = ip-172-31-35-141

I have only made changes to my outputs.conf and I am not sure on what to change for inputs

0 Karma

chanfoli
Builder

You will need to have appropriate monitor stanzas on the forwarder for the tomcat logs you want to start indexing, ideally these will also need to be assigned an appropriate sourcetype.

Have a look at this:
http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitorfilesanddirectories

Here is another answer which should get you in the right direction on inputs. This person appears to have set up different sourcetypes for the different logs:
http://answers.splunk.com/answers/135355/proper-input-conf-setup-apache-tomcat.html

My procedure is to load an example file on a splunk instance through add data and use the "data preview" functionality it to make sure timestamps and event breaks are getting parsed and what sourcetype settings are needed to make this happen for each sourcetype.

BTW, I removed tcpout-server stanza from my outputs.conf before my remote forwarder actually connected to the sandbox and forwarded events.

0 Karma

appzen
Path Finder

I also realized that I am changing my files from /opt/splunkforwarder/etc/system/local/outputs.conf but should it be from /opt/splunkforwarder/etc/apps/search/local?

0 Karma

chanfoli
Builder

In my opinion, no. The configs under SPLUNKHOME/etc/apps/search for the search app, which is not relevant on a Universal Forwarder system.

0 Karma

appzen
Path Finder

thanks for your help, quick question about the monitor, i cant simply just do

/opt/splunkforwarder/bin/splunk add monitor /var/lib/tomcat7/logs

to add a monitor?

0 Karma

appzen
Path Finder

I just changed my inputs.conf to:

[default]
host = ip-172-31-35-141

[monitor:/var/lib/tomcat7/logs/catalina.*]

disabled = false
index = test
sourcetype = catalina

0 Karma

chanfoli
Builder

At this point I would check the splunkd.logs on your fowarder and run the following search on your sandbox:
index=_internal xx.xx.xx.xx

where xx.xx.xx.xx is your forwarder's outside IP address.

This might provide some clues about connection status.

0 Karma

appzen
Path Finder

where are the splunkd.logs located?

0 Karma

appzen
Path Finder

i did this command: index="_internal" 54.174.120.69 source="/opt/splunk/var/log/splunk/splunkd.log" and I get this error:

1/12/15
6:30:44.095 PM

01-12-2015 18:30:44.095 +0000 ERROR TcpInputProc - Error encountered for connection from src=54.174.120.69:60649. Local side shutting down
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/12/15
6:30:44.095 PM

01-12-2015 18:30:44.095 +0000 ERROR TcpInputProc - Error encountered for connection from src=54.174.120.69:60648. Local side shutting down
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/12/15
6:30:44.045 PM

01-12-2015 18:30:44.045 +0000 ERROR TcpInputProc - Error encountered for connection from src=54.174.120.69:60546. Local side shutting down
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/10/15
12:28:46.502 AM
01-10-2015 00:28:46.502 +0000 WARN TcpOutputProc - Cooked connection to ip=54.174.120.69:9997 timed out
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/10/15
12:28:16.500 AM
01-10-2015 00:28:16.500 +0000 WARN TcpOutputProc - Cooked connection to ip=54.174.120.69:9997 timed out
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd

0 Karma

appzen
Path Finder

and with the first command, index=internal xx.xx.xx.xx, i get:
1/12/15
7:43:52.260 PM

192.168.48.247 - admin [12/Jan/2015:19:43:52.260 +0000] "GET /en-US/api/shelper?snippet=true&snippetEmbedJS=false&namespace=search&search=search+index%3D%22_audi%22+54.174.120.69&useTypeahead=true&useAssistant=true&showCommandHelp=true&showCommandHistory=true&showFieldInfo=false&
=1421091322744 HTTP/1.0" 200 641 "https://prd-p-c325dgfktbm7.cloud.splunk.com/en-US/app/search/search?q=search%20index%3D%22_audit%22%..." "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36" - 54b423f8427f421431a250 20ms
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/web_access.log sourcetype = splunk_web_access
1/12/15
7:43:46.729 PM
01-12-2015 19:43:46.729 +0000 INFO StatusMgr - destPort=9997, eventType=connect_close, group=tcpin_connections, sourceHost=54.174.120.69, sourceIp=54.174.120.69, sourcePort=33886, statusee=TcpInputProcessor
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/metrics.log sourcetype = splunkd
1/12/15
7:43:46.707 PM
01-12-2015 19:43:46.707 +0000 INFO StatusMgr - destPort=9997, eventType=connect_done, group=tcpin_connections, sourceHost=54.174.120.69, sourceIp=54.174.120.69, sourcePort=33886, statusee=TcpInputProcessor
host = ip-192-168-16-190 source = /opt/splunk/var/log/splunk/metrics.log sourcetype = splunkd

0 Karma

chanfoli
Builder

Okay a couple of things here. Is the 54.x.x.69 IP your universal forwarder? A couple of log entries indicate that something was trying to forward logs TO this IP which makes me think that this is your sandbox IP or there was some other misconfiguration.... Also, the local side shutting down errors might be missed heartbeats and could simply be when splunk is being restarted.

please execute the following on your forwarder to check connectivity:

splunk list forward-server and again checking the splunkd.log from the forwarder might help.

0 Karma

appzen
Path Finder

54.x.x.69 is the IP where the universal forwarder was dled. After running the command I get:
Active forwards:
input-prd-p-c325dgfktbm7.cloud.splunk.com:9997 (ssl)
Configured but inactive forwards:
None

and Im trying to check the logs from the forwarder but I don't think any exists, but Ill try again

0 Karma

chanfoli
Builder

okay, so it looks like your connection is up. Since you are setting host to ip-172-31-35-141 in your inputs you should be able to search for host=ip-172-31-35-141 ( i would use all-time for troubleshooting in this case, just in case there are timestamp discrepancies). You also need to make sure that the user the splunk forwarder is running as has read permissions on the logs you have added.

0 Karma

chanfoli
Builder

Also, based on the above config, you will need to have created a "test" index on your sandbox as well, and depending on user/role you may need to use index=test in your search. Also, did you create a catalina sourcetype on your sandbox instance? If your data is not one of the built in types, this should be done.

0 Karma

appzen
Path Finder

when i do this command, host="ip-172-31-35-141" source="/var/lib/tomcat7/logs/catalina.out", I get a bunch of logs, is this the results that I should be getting?

0 Karma

chanfoli
Builder

It sounds like your monitor input is getting indexed on your sandbox instance. Yay! You did it!

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...