We are using Splunk 6.2.4 build 271043 on Ubuntu and we are seeing a couple of pages in the Lookups section that are giving 500 internal errors.
When clicking on the Lookup Definitions link (see directly below):
We get the 500 internal server error shown directly below:
Note that in the example of the error, I am also showing Google Chrome developer tools to show the network information. I do not see any stack traces or clues in Google Chrome developer tools traffic (it's probably logging these locally to ensure security).
When I click on the "View more information about your request" link shown on the error page, it shows me a Splunk _internal search with no results. It seems strange to me that the error page is saying "at https://127.0.0.1:8089" rather than an external IP address. It leads me to believe that the server may be misconfigured. I am new to Splunk and have inherited the system, so any suggestions are welcome.
Additionally, the "Add new" link to the right of the "Automatic lookups" section also generates a 500 internal error. None of the non-Lookups pages on our Splunk web site give errors like these.
I have searched answers.splunk.com as well as Google searching but have not been able to find what the issue could be. Please feel free to refer me to any articles that could be helpful or any logs on the server that may shed some light on the issue. I do have admin access to the machine and can see any logs or .conf's that are relevant.
Your browser is connected to splunkweb at splunk.censored.local:80 which in turn is connected to splunkd at 127.0.0.1:8089 - splunkweb and splunkd run on the same machine, so localhost makes sense.
Not seeing errors in Chrome is expected, 500 is a server error. From Chrome's point of view everything's fine.
index=_internal for error messages happening at the time you see a 500. Maybe that link saying "view more information" on the error page itself has more information as well.
Thank you for your response and for the insight on splunkd and 127.0.0.1. The "view more information" link does not show any events for the query of 'index=internal host="censored" source=*webservice.log log_level=ERROR requestid=571c37fea37fd5745f3250'
I have done some Splunk searches on 'index=internal' and have seen two entries, one for webaccess.log and the other splunkduiaccess.log. I looked in these log files for the timestamp and found for web_access.log:
127.0.0.1 - merpenbeck [23/Apr/2016:23:27:37.020 -0400] "GET /en-US/manager/search/data/transforms/lookups HTTP/1.1" 500 3070 "http://splunk.mk6.local/en-US/manager/search/lookups" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36" - 571c3d29057fd5b443a410 310ms
And in the splunkduiaccess.log file:
172.18.20.15 - merpenbeck [23/Apr/2016:23:27:37.019 -0400] "GET /en-US/manager/search/data/transforms/lookups HTTP/1.1" 500 3070 "http://splunk.mk6.local/en-US/manager/search/lookups" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36" - afbdfc0c38412f8496710fc57c992ff9 311ms
But I do not see any detail on why the 500 error is occurring. Does Splunk have a .log file that contains internal server error data?
I ended up resolving the issue of the "Lookup Definition" 500 error.
I found that the path did not contain $SPLUNKHOME\bin. It's just a theory, but I'm thinking that something in that page needs $SPLUNKHOME\bin. Then again, maybe just the act of rebooting the server fixed the issue. 😉
I did the following:
1. modify /opt/splunk/etc/splunk-launch.conf to include the $SPLUNKHOME variable
2. modify /etc/environment to include $SPLUNKHOME/bin
3. reboot the server
Now I am able to see the "Lookup Definition" page without the 500 error.
I hope that this helps someone else.
Hi I'm experiencing the same error too. May I know how do you modify the .conf and /etc environment? Where are those files located? Sorry i'm quite new to Splunk.
I had this exact error on this exact page, but it had nothing to do with setting SPLUNK_HOME. It was data related. A user had a private lookup that was pointing to a non-existent .swap file. Once I backed up and nuked the user's entire ./etc/users/[myuser]/SplunkEnterpriseSecuritySuite folder, and did a debug/refresh, the lookup definitions form rendered fine.