Yeah, looks to be that way, having messed with it a couple times. The alternative name was what was causing the issues. I have accepted my own answer for that reason and will make a comment on the docs page. ... View more

Thank you, yes, I was using his SSL slides to assist with this. This is actually the last thing I think I needed to get working before I am completed with getting everything switched to custom certs. So the certificates aren't the issue, per se, as they work with everything else (forwarding and web connections). I also could totally pull the certificates using the s_client command over port 8089 and it lists the common name as I expected it to. ... View more

As of version 6.3: It would appear that they have removed "This feature does not work with the deployment server and client communication over SSL" from the specification file as it relates to sslCommonNameToCheck, so this should be fine to use from now on at least according to the documentation. ... View more

Had read all through it, and I think it was throwing me off since I was afraid that the "tcpout-server://" setting would maybe somehow keep it from load balancing correctly. by instead duplicating all the data equally to the servers. I will give this a go and see if that works. I am just not sure the best way to validate that it is load balancing and not sending the data out to all of my indexers at the same time (e.g. instead of it logging something once, I would guess to find my data copied 4 times over on my 4 indexers.) ... View more

yes, I commonly am seeing that same error briefly when I have to bring down the license master for maintenance. So it would seem that for some reason those slaves cannot connect over 8089. Either your username/password is incorrect or it is an open port issue. If you are on linux, try running a telnet to your master node over port 8089 and see if it connects. Alternatively you can do a wget/curl via https://masternode:8089 and see what comes back. You can even pass through curl/wget the username/password combination you want to see if that works as well. (This is basically accessing the REST API yourself to test connectivity). Depending on what message you get back as to what the next step would be to troubleshoot the issue. Since some of them work and some of them don't I am going to guess you either fat-fingered the username/password on it, or more likely, there is some kind of firewall/port block stopping the communication. ... View more

Oh, sorry, for some reason I didn't register that Logstash was another product (which is basically a fork of ELK, right?). What you may want to do is run a Heavy Forwarder so you can collect the data and then forward it on via syslog. Which I don't think you can do from a UF. You should be able to register the Enterprise instance as either the free license or the forwarding license since I assume you are not paying for Splunk since you are using ELK. Just make sure to turn off indexing on your Heavy Forwarder and have it dump the data via syslog to ELK. Otherwise I think I am getting outside my depth here since I am not familiar enough with ELK to help further. ... View more

All your indexers (really all of your servers) should be showing up in your license master under whatever pool (default pool is the catch all) you have specified. For example I have 14 servers registering under my default license pool, although I only have 4 indexers. As long as you turn off indexing on everything else this should be fine and you will only see hits against your license from your indexers. I would validate the settings on your servers that are not showing up in the pool to see if maybe you typo'd a setting. From the GUI go to Settings - Licensing - Switch to Slave. If you are already set as a slave, change yourself to a local master, and then reset it back as a slave making sure your settings are correct. (alternatively, these settings are stored in your server.conf file). As for the licensing itself, if things are not balanced across your indexers evenly I would check your outputs files across the deployment to ensure that you are appropriately set to auto load balance across all four indexers. It could be that the events are going to one server over another because of this. If these settings are valid and you are still seeing it unevenly balanced, check your splunkd.log and verify that you are not showing errors in TCPOut / TCPIn that says you were unable to connect to X server. Sorry this one is a bit all over the place, since it could be a couple of different issues. Now if you mean you are actually going over your license, you might want to talk with your Splunk Account rep about pulling your events under the license or getting a larger license. Hope this helps! ... View more

I think you want the props.conf on the receiving server (not the forwarder), to specify your newline character as the end of the log. The default for this is a regex pattern on "\r\n", but depending on how the log comes in this may not be the case. Below is a snippet from the props.conf specification file: http://docs.splunk.com/Documentation/Splunk/6.3.1/admin/Propsconf LINE_BREAKER = <regular expression> * Specifies a regex that determines how the raw text stream is broken into initial events, before line merging takes place. (See the SHOULD_LINEMERGE attribute, below) * Defaults to ([\r\n]+), meaning data is broken into an event for each line, delimited by any number of carriage return or newline characters. You might also be able to get the data to cooperate on the forwarder itself by specifying that the type of data is "JSON" see this section in the props.conf: * This feature and all of its settings apply at input time, when data is first read by Splunk. The setting is used on a Splunk system that has configured inputs acquiring the data. # Special characters for Structured Data Header Extraction: # Some unprintable characters can be described with escape sequences. The # attributes that can use these characters specifically mention that # capability in their descriptions below. # \f : form feed byte: 0x0c # \s : space byte: 0x20 # \t : horizontal tab byte: 0x09 # \v : vertical tab byte: 0x0b INDEXED_EXTRACTIONS = < CSV|W3C|TSV|PSV|JSON > * Tells Splunk the type of file and the extraction and/or parsing method Splunk should use on the file. CSV - Comma separated value format TSV - Tab-separated value format PSV - pipe "|" separated value format W3C - W3C Extended Extended Log File Format JSON - JavaScript Object Notation format * These settings default the values of the remaining settings to the appropriate values for these known formats. * Defaults to unset. So you can either fix this on the receiving side or the sending side. ... View more

But yes, I had to have a conversation with our Linux engineer that excessive permissions is not the way to go. It is general all around bad security practices anyway. Depending on the security requirements in your organization this might even be a policy violation since you would have effectively set global read/execute permissions to your collected log data (which is bad). That was why I suggested raking back the global permissions across your install. Linux, unlike Windows, will get quite upset with you for doing bad permissions. ... View more

Sorry, correction on this, mongodb should have a 400 permission. I would specifically fix this: chmod 400 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key See if that clears the error. I would also check out the following Answer as it might help pinpoint if this is indeed your issue, since it is also suggested in there that an upgrade to 6.3 didn't refresh one user's SSL certificates which also was causing an issue with the KVStore: https://answers.splunk.com/answers/236495/splunk-kv-store-does-not-start.html ... View more

When you have more than one Splunk device you need to set up a license master and point everything to that device. Just about anything can work as your license master, I personally put it on my Deployment Server. Check out the documentation on this here: http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/Configurealicensemaster The documentation walks you through doing it through either the Web GUI or the CLI, what isn't obvious from the documentation is if you wanted to do this through a configuration file. You would want to edit your server.conf file with the following entry: [license] master_uri = https://yourserver.splunk.com:8089 There should already be an auto-license pool for your enterprise key with a * on the slaves so anything that checks in with your license master will go into that default pool. I would reference the documentation above if you need to take this further. Hope this helps! ... View more

mogodb does not like excessive permissions. If you did a 755 to your entire installation folder I suggest you retract this back. chmod o-rwx /opt/splunk -R Assuming splunk is installed in /opt/splunk This should fix your KVStore issues. Not sure if it will fix your overall issues though, post back your findings after fixing mongodb. ... View more

You have it typed as "tes*tt*essinstall" there is just one "t". Otherwise no, the username/password prompt should be normal (although you could also do -auth admin -password "changme" on the end of it. Just make sure that you have the single quotes around the search command as I listed above and it should work fine. The only output I got after running it (and this took like a good 30 minutes to complete mind you) was the following: ERROR: command="testessinstall", Initialization complete, please restart Splunk You should also see various messages showing up in your logs, as well as messages in the GUI next to your Display Name. Since you commented earlier about getting stuck on the installing Apps piece you may be experiencing a slightly different problem and it might be worth opening a support ticket with Splunk if you haven't already. ... View more

I have a support ticket opened with Splunk on the issue, if and when I get a resolution I'll certainly post it, maybe it will help others! ... View more

(Sorry it wouldn't let me edit my original post... something about not enough reputation points...) Update 1: As I have been digging through the logs, there are two log sets that are relevant to this: essinstaller2.log and es_installer_controller.log Inside the es_installer_controller.log I am getting essentially a timeout error through the different run attempts (just pasting those errors here) 2015-11-04 11:54:29,549 INFO file=<string>:stage:61 | stage="disable_apps" status="failed" exc="SplunkdConnectionException(u"Error connecting to /services/apps/local/SA-ThreatIntelligence/disable: ('The read operation timed out',)",) is not JSON serializable" 2015-11-04 11:54:29,549 INFO stage="disable_apps" status="failed" exc="SplunkdConnectionException(u"Error connecting to /services/apps/local/SA-ThreatIntelligence/disable: ('The read operation timed out',)",) is not JSON serializable" 2015-11-04 13:14:41,319 INFO stage="disable_apps" status="failed" exc="SplunkdConnectionException(u"Error connecting to /services/apps/local/SA-NetworkProtection/disable: ('The read operation timed out',)",) is not JSON serializable" 2015-11-04 13:41:57,632 INFO stage="disable_apps" status="failed" exc="SplunkdConnectionException(u"Error connecting to /services/apps/local/DA-ESS-AccessProtection/disable: ('The read operation timed out',)",) is not JSON serializable" and inside the essinstaller2.log at the end of each run when it "hangs" it just does this: 2015-11-04 13:41:57,632 INFO stage="disable_apps" app="TA-tippingpoint" status="True" msg="disable" 2015-11-04 13:41:57,632 INFO STAGE COMPLETE: "disable_apps" back in the middle of the log you will get an error like the below, which seems to correspond with the later thrown error inside the "controller" log: 2015-11-04 13:40:45,209 INFO stage="disable_apps" app="DA-ESS-AccessProtection" status="False" msg="Splunkd daemon is not responding: (u"Error connecting to /services/apps/local/DA-ESS-AccessProtection/disable: ('The read operation timed out',)",)" What this all means I couldn't tell you. What I will say is I did have what appeared to be one "clean" run at 12:13 - blah which showed none of the above errors, yet still just decided it would hang and do nothing... 2015-11-04 12:13:30,833 INFO STAGE STARTING: "refresh" 2015-11-04 12:13:30,834 INFO stage="refresh" msg="Installation record found" 2015-11-04 12:13:30,834 INFO stage="refresh" msg="State data loaded." 2015-11-04 12:13:30,834 INFO stage="refresh" app="SA-EndpointProtection" status="False" msg="currently installed version: 3.3.1" ........ 2015-11-04 12:13:30,842 INFO STAGE COMPLETE: "refresh" 2015-11-04 12:13:30,866 INFO STAGE STARTING: "deprecate_apps" 2015-11-04 12:13:30,866 INFO stage="deprecate_apps" msg="Installation record found" 2015-11-04 12:13:30,866 INFO stage="deprecate_apps" msg="State data loaded." 2015-11-04 12:13:31,755 INFO stage="deprecate_apps" app="TA-juniper" status="True" msg="deprecated" ........ 2015-11-04 12:13:36,200 INFO STAGE COMPLETE: "deprecate_apps" 2015-11-04 12:13:36,235 INFO STAGE STARTING: "disable_apps" 2015-11-04 12:13:36,235 INFO stage="disable_apps" msg="Installation record found" 2015-11-04 12:13:36,235 INFO stage="disable_apps" msg="State data loaded." 2015-11-04 12:13:37,647 INFO stage="disable_apps" app="Splunk_TA_windows" status="True" msg="disable" ........ 2015-11-04 12:16:14,137 INFO STAGE COMPLETE: "disable_apps" and the comparable lines from the "controller" log: 2015-11-04 12:09:24,802 INFO Entering method: order 2015-11-04 12:09:24,802 INFO Installation stage order retrieved: [{'name': 'refresh', 'description': 'Refreshing app information'}, {'name': 'deprecate_apps', 'description': 'Deprecating old apps'}, {'name': 'disable_apps', 'description': 'Disabling apps prior to installation'}, {'name': 'install_apps', 'description': 'Installing new apps'}, {'name': 'reenable_apps', 'description': 'Re-enabling apps'}, {'name': 'postinstall', 'description': 'Conducting postinstall actions'}, {'name': 'finalize', 'description': 'Finalizing installation'}] 2015-11-04 12:13:29,527 INFO Entering method: order 2015-11-04 12:13:29,527 INFO Installation stage order retrieved: [{'name': 'refresh', 'description': 'Refreshing app information'}, {'name': 'deprecate_apps', 'description': 'Deprecating old apps'}, {'name': 'disable_apps', 'description': 'Disabling apps prior to installation'}, {'name': 'install_apps', 'description': 'Installing new apps'}, {'name': 'reenable_apps', 'description': 'Re-enabling apps'}, {'name': 'postinstall', 'description': 'Conducting postinstall actions'}, {'name': 'finalize', 'description': 'Finalizing installation'}] 2015-11-04 12:13:30,833 INFO Entering method: stage 2015-11-04 12:13:30,833 INFO stage="refresh" status="starting" 2015-11-04 12:13:30,843 INFO stage="refresh" status="completed" 2015-11-04 12:13:30,866 INFO Entering method: stage 2015-11-04 12:13:30,866 INFO stage="deprecate_apps" status="starting" 2015-11-04 12:13:36,200 INFO stage="deprecate_apps" status="completed" 2015-11-04 12:13:36,234 INFO Entering method: stage 2015-11-04 12:13:36,234 INFO stage="disable_apps" status="starting" 2015-11-04 12:16:14,137 INFO stage="disable_apps" status="completed" So I am still over here scratching my head if there was a "clean" run through with no errors why did it still hit the "disable_apps" completed, yet never actually PROGRESS the web interface... The solution may be in the splunkd.log, as it has the following errors right after I get my last error from the "controller" log. 11-04-2015 13:41:57.633 -0500 WARN HttpListener - Socket error from 127.0.0.1 while accessing /en-US/custom/SplunkEnterpriseSecuritySuite/installer_controller/disable_apps: Broken pipe (web_service.log) "2015-11-04 13:41:57,633 INFO [563a50edc37f76689ceb10] root:129 - ENGINE: Bus STOPPING" So that was after the last run (with the other errors) so lets look at the one that "seemingly" should have succeeded... 11-04-2015 12:16:14.138 -0500 ERROR HttpListener - Exception while processing request from 127.0.0.1 for /en-US/custom/SplunkEnterpriseSecuritySuite/installer_controller/disable_apps: Connection closed by peer 11-04-2015 12:16:14.139 -0500 ERROR HttpListener - Handler for /en-US/custom/SplunkEnterpriseSecuritySuite/installer_controller/disable_apps sent a 0 byte response after earlier claiming a Content-Length of 3055! (web_service.log) 2015-11-04 12:16:14,139 INFO [563a3b87897facc4ad6ad0] root:129 - ENGINE: Bus STOPPING Anyway, I am at a loss here, still... It looks like there is just something wrong with the way the installer is coded, but I am not getting clear enough error messages to know what to fix. Why can't we have a manual installation route? ... View more

Have tried that a few times and no luck 😞 ... View more

I am currently in the middle of an upgrade to 4.0, however I had 3.3.1 installed and verified that it was working correctly on 6.3 before I started the upgrade. I know you are looking for 3.3.0, but I would assume that if 3.3.1 and 3.3.2 work that it is really 3.3.x that works. Hope this helps! ... View more

I know this is an older question but as I was recently tackling this issue since splunk was randomly crashing on me (I still haven't got to the root of that issue) I needed some way to force splunk to start back up again after it died so I didn't potentially start losing logs as this is one of my heavy forwarders that keeps crashing. #!/bin/sh # ## Splunk Health Checking Script to run hourly ## # This will run some basic checks to ensure ## # splunk is running and restart those services ## # if it fails a check. ## ## ############################################# ## service=splunk # Error handling function function errorCheck { if [ $? -ne 0 ] ; then echo "Error occurred connecting on port 8089 for $service" /etc/init.d/$service start fi } # check for the processes to be running if (( $(ps -ef | grep -v grep | grep $service | wc -l) > 0 )); then echo "$service is running!!!" else /etc/init.d/$service start fi # check for the service itself to be running # sometimes the service can crash leaving stale PID's running if (( $(/etc/init.d/$service status | grep "splunkd is running" | wc -l) > 0 )) ; then echo "$service is running!!!" else /etc/init.d/$service start fi # check if we can connect locally on port 8089 /usr/bin/curl -s -k -o "/dev/null" https://127.0.0.1:8089 errorCheck Note that because I am running this on a heavy forwarder, it is not running splunk web. So I haven't tested this yet on something running splunk web to see the impact of this or if it will need to be tweaked. Also, standard disclosure, please test this script in a safe way as I take no responsibility for any impact of using this script. It worked for me, but YMMV. So just to break it down, I took the recommendations from other people's answers here to go through the checks that would be needed to ensure it is fully working. First: $(ps -ef | grep -v grep | grep $service | wc -l) > 0 This runs the command recommended above with a twist, I pipe into wc and have it just tell me how many lines are returned, if there are no running splunk processes this will be 0, so if it is 0 we need to start splunk. Second: $(/etc/init.d/$service status | grep "splunkd is running" | wc -l) > 0 Same concept, this time doing a service status check. Note that you need this registered with your system as a service. This is done as part of the "enable boot-start" command on the splunk CLI, you could replace this with pointing to the splunk binary directly instead. If it isn't running it should return a 0. Lastly: /usr/bin/curl -s -k -o "/dev/null" https://127.0.0.1:8089 errorCheck This uses curl, you could also do a wget instead. I opted for curl because it was installed by default on my RHEL image whereas wget is not so to avoid adding extra software to the system I opted for curl. Essentially the principle here is that you are running it silently, I added the "-k" option because you are calling it by the loopback IP and your certificates will not like that and error out on you. We aren't trying to validate the certificates, just see if it is running so therefore I threw in the "-k" (again wget has a similar option if you need to switch to that). lastly I am outputing the results of this call to null so it doesn't get saved anywhere. I don't need the page itself it is the error codes I care about. Which brings me to the "errorCheck" function. This is looking for the results to have come back as a 0, meaning success. You can go find what the other codes mean, I didn't care since anything other than 0 is a problem which means we should start the splunk service. Hope this helps! ... View more

According to: http://docs.splunk.com/Documentation/Splunk/6.2.5/Data/Specifyinputpathswithwildcards It needs to have a wildcard of some kind in that section of the stanza in order to activate the usage of regex. I have tried this (before I went and actually read the linked document) and it didn't work, unfortunately. Why, oh why, can't they just use full on regex in this field instead of making it so complicated -_- ... View more