Good morning, I am trying to figure out an error 500 error on 6 new Centos 7 servers. Here is my scenario: Splunk Web comes up without an issue. I can connect to the URL and get a login screen. When I put the admin/password in, it immediately pops a "500 Internal Server Error" error screen. An interesting side note is that if I put the initial password in wrong it keeps prompting me for the correct password, so Splunk appears to be resolving the password. I have tried changing the password using " splunk edit user admin -password newPassowrd -auth admin:changeme" and also the method of using user-seed.conf to no avail. I am getting the following error in the web_service.log: 2019-11-07 14:47:28,051 INFO [5dc490f00a7f11b0320f50] error:323 - GET /en-US/ 127.0.0.1 8065 2019-11-07 14:47:28,051 INFO [5dc490f00a7f11b0320f50] error:324 - 500 Internal Server Error The server encountered an unexpected condition which prevented it from fulfilling the request. 2019-11-07 14:47:28,051 ERROR [5dc490f00a7f11b0320f50] error:325 - Traceback (most recent call last): File "/opt/splunk/lib/python2.7/site-packages/cherrypy/cprequest.py", line 606, in respond cherrypy.response.body = self.handler() File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/htmlinjectiontoolfactory.py", line 72, in wrapper resp = handler(*args, **kwargs) File "/opt/splunk/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 25, in __call_ return self.callable(*self.args, **self.kwargs) File "<string>", line 1, in <lambda> File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 38, in rundecs return fn(*a, **kw) File "<string>", line 1, in <lambda> File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 119, in check return fn(self, *a, **kw) File "<string>", line 1, in <lambda> File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 167, in validate_ip return fn(self, *a, **kw) File "<string>", line 1, in <lambda> File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 246, in preform_sso_check update_session_user(sessionKey, remote_user) File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 189, in update_session_user en = splunk.entity.getEntity('authentication/users', user, sessionKey=sessionKey) File "/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 265, in getEntity serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True) File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/init.py", line 500, in simpleRequest raise splunk.SplunkdConnectionException('Error connecting to %s: %s' % (path, str(e))) SplunkdConnectionException: Splunkd daemon is not responding: ('Error connecting to /services/authentication/users/admin: [Errno 111] Connection refused',) My splunk-launch.conf file: # Version 7.3.2 # Modify the following line to suit the location of your Splunk install. # If unset, Splunk will use the parent of the directory containing the splunk # CLI executable. # SPLUNK_HOME=/opt/splunk # By default, Splunk stores its indexes under SPLUNK_HOME in the # var/lib/splunk subdirectory. This can be overridden # here: # # SPLUNK_DB=/opt/splunk-home/var/lib/splunk SPLUNK_DB=/indexes/splunk # Splunkd daemon name SPLUNK_SERVER_NAME=splunkd # Splunkweb daemon name SPLUNK_WEB_NAME=splunkweb # Changing the default bind from 127.0.0.1 to actual IP SPLUNK_BINDIP=10.72.10.XXX # If SPLUNK_OS_USER is set, then Splunk service will only start # if the 'splunk [re]start [splunkd]' command is invoked by a user who # is, or can effectively become via setuid(2), $SPLUNK_OS_USER. # (This setting can be specified as username or as UID.) # SPLUNK_OS_USER=splunk My web.conf file: # location of splunkd; don't include http[s]:// in this anymore. mgmtHostPort = 10.72.10.XXX:8089 NOTE: I tried adding "server.socket_host = 10.72.10.106" as one Answers response suggested for a similar problem - it has no effect. My server.conf: [general] serverName = splunk-idx1.FQDN.com pass4SymmKey = $7$qKGYODrBHTX/iU48JAtICIiYzLSTkJ/cJV0WReqxkB== [sslConfig] sslPassword = $7$yq5VWVVe1wXyurxFgpnBOQZx8XjykeVyg4xmW4Q9KOK8== [lmpool:auto_generated_pool_download-trial] description = auto_generated_pool_download-trial quota = MAX slaves = * stack_id = download-trial [lmpool:auto_generated_pool_forwarder] description = auto_generated_pool_forwarder quota = MAX slaves = * stack_id = forwarder [lmpool:auto_generated_pool_free] description = auto_generated_pool_free quota = MAX slaves = * stack_id = free It seems like it is having an authentication issue within Splunk itself, but I am not sure where to look at this point: SplunkdConnectionException: Splunkd daemon is not responding: ('Error connecting to /services/authentication/users/admin: [Errno 111] Connection refused',) Any help would be greatly appreciated! ... View more

My question focuses around using one search to create a list of files I don’t want to consider in the primary search. Here is an overview of the problem I am trying to solve. I have a directory that fills with small files, one file per transaction series for an application we run. 1) Each file is a source. 2) If the file contains “Transaction Rolled Back" OR "Process FNXML Complete”, I do not want to have the filename (source) in the final list of sources. I am looking for transactions that did not complete. 3) I would like the final output to be a list of sources (log files) that do not have the “Transaction Rolled Back" OR "Process FNXML Complete” messages within the last three file lines. Data examples are: Action 1/9/2018 3:08:33 PM: XML Processing is complete. Action 1/9/2018 3:08:33 PM: *******Commit Transaction ******* Action 1/9/2018 3:08:33 PM: Not writing to if_header - No transaction was set in the XML. Action 1/9/2018 3:08:33 PM: ProcessNode Complete Action 1/9/2018 3:08:33 PM: Process FNXML Complete. OR Action 1/9/2018 3:05:37 PM: ******** Transaction Rolled Back ********** Action 1/9/2018 3:05:37 PM: ErrorDesc=The current status of the shop order [RUN] is not in the list of old shop order oper status values. Action 1/9/2018 3:05:37 PM: Not writing to if_header - No transaction was set in the XML. When I run the following command, I will get a list of sources that looks like something like this: (host=vel*) index=velocity sourcetype="velocity:icim" source="E:\\logs\\icim\\*" | stats count by source OUTPUT: source count (This is linecount) E:\logs\Intercim\10076010920181646645312.log 17 E:\logs\Intercim\10076010920181646746093.log 34 E:\logs\Intercim\10076010920181646750905.log 20 I can create the lookup table successfully, and create a separate search to compare the search against the lookup table, but I can’t seem to create a combined search. Part of the issue is that the second half, as I get a “Mismatched ‘]’ error. I think the issue is a subsearch within a subsearch, but am not sure how to get past it. Any help would be greatly appreciated. Works to create lookup table: (host=vel*) index=velocity sourcetype="velocity:icim" source="E:\\logs\\icim\\*" ("Transaction Rolled Back" OR "Process FNXML Complete") | dedup source | stats count by source | fields source | outputlookup icimlogs.csv Works to search with lookup table: (host=vel*) index=velocity sourcetype="velocity:icim" (source="E:\\logs\\icim\\*" NOT source="E:\\logs\\icim\\2018*") Action NOT [| inputlookup icimlogs.csv] | dedup source | table source Combined search that fails (This is just the latest iteration): (host=vel*) index=velocity sourcetype="velocity:icim" source="E:\\logs\\icim\\*" ("Transaction Rolled Back" OR "Process FNXML Complete") | dedup source | stats count by source | fields source | outputlookup icimlogs.csv | append [ search (host=vel*) index=velocity sourcetype="velocity:icim" (source="E:\\logs\\icim\\*" NOT source="E:\\logs\\icim\\2018*") Action NOT [| inputlookup icimlogs.csv]] | dedup source | table source] Thanks for your help, everyone. ... View more