Splunk Search

500 internal server errors with /search/data/transforms/lookups page

Erpenbeck
Path Finder

We are using Splunk 6.2.4 build 271043 on Ubuntu and we are seeing a couple of pages in the Lookups section that are giving 500 internal errors.

When clicking on the Lookup Definitions link (see directly below):
alt text

We get the 500 internal server error shown directly below:
alt text

Note that in the example of the error, I am also showing Google Chrome developer tools to show the network information. I do not see any stack traces or clues in Google Chrome developer tools traffic (it's probably logging these locally to ensure security).

When I click on the "View more information about your request" link shown on the error page, it shows me a Splunk _internal search with no results. It seems strange to me that the error page is saying "at https://127.0.0.1:8089" rather than an external IP address. It leads me to believe that the server may be misconfigured. I am new to Splunk and have inherited the system, so any suggestions are welcome.

Additionally, the "Add new" link to the right of the "Automatic lookups" section also generates a 500 internal error. None of the non-Lookups pages on our Splunk web site give errors like these.

I have searched answers.splunk.com as well as Google searching but have not been able to find what the issue could be. Please feel free to refer me to any articles that could be helpful or any logs on the server that may shed some light on the issue. I do have admin access to the machine and can see any logs or .conf's that are relevant.

Thank you.

0 Karma
1 Solution

Erpenbeck
Path Finder

I ended up resolving the issue of the "Lookup Definition" 500 error.

I found that the path did not contain $SPLUNK_HOME\bin. It's just a theory, but I'm thinking that something in that page needs $SPLUNK_HOME\bin. Then again, maybe just the act of rebooting the server fixed the issue. 😉

I did the following:
1. modify /opt/splunk/etc/splunk-launch.conf to include the $SPLUNK_HOME variable
2. modify /etc/environment to include $SPLUNK_HOME/bin
3. reboot the server

Now I am able to see the "Lookup Definition" page without the 500 error.

I hope that this helps someone else.

Best regards,

Michael

View solution in original post

0 Karma

jclyde_splunk
Splunk Employee
Splunk Employee

I had this exact error on this exact page, but it had nothing to do with setting SPLUNK_HOME. It was data related. A user had a private lookup that was pointing to a non-existent .swap file. Once I backed up and nuked the user's entire ./etc/users/[myuser]/SplunkEnterpriseSecuritySuite folder, and did a debug/refresh, the lookup definitions form rendered fine.

Erpenbeck
Path Finder

I ended up resolving the issue of the "Lookup Definition" 500 error.

I found that the path did not contain $SPLUNK_HOME\bin. It's just a theory, but I'm thinking that something in that page needs $SPLUNK_HOME\bin. Then again, maybe just the act of rebooting the server fixed the issue. 😉

I did the following:
1. modify /opt/splunk/etc/splunk-launch.conf to include the $SPLUNK_HOME variable
2. modify /etc/environment to include $SPLUNK_HOME/bin
3. reboot the server

Now I am able to see the "Lookup Definition" page without the 500 error.

I hope that this helps someone else.

Best regards,

Michael

0 Karma

qiaojing
Path Finder

Hi I'm experiencing the same error too. May I know how do you modify the .conf and /etc environment? Where are those files located? Sorry i'm quite new to Splunk.

Thank you.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Your browser is connected to splunkweb at splunk.censored.local:80 which in turn is connected to splunkd at 127.0.0.1:8089 - splunkweb and splunkd run on the same machine, so localhost makes sense.

Not seeing errors in Chrome is expected, 500 is a server error. From Chrome's point of view everything's fine.
Search index=_internal for error messages happening at the time you see a 500. Maybe that link saying "view more information" on the error page itself has more information as well.

0 Karma

Erpenbeck
Path Finder

Martin:
Thank you for your response and for the insight on splunkd and 127.0.0.1. The "view more information" link does not show any events for the query of 'index=_internal host="censored" source=*web_service.log log_level=ERROR requestid=571c37fea37fd5745f3250'

I have done some Splunk searches on 'index=_internal' and have seen two entries, one for web_access.log and the other splunkd_ui_access.log. I looked in these log files for the timestamp and found for web_access.log:
127.0.0.1 - merpenbeck [23/Apr/2016:23:27:37.020 -0400] "GET /en-US/manager/search/data/transforms/lookups HTTP/1.1" 500 3070 "http://splunk.mk6.local/en-US/manager/search/lookups" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36" - 571c3d29057fd5b443a410 310ms

And in the splunkd_ui_access.log file:
172.18.20.15 - merpenbeck [23/Apr/2016:23:27:37.019 -0400] "GET /en-US/manager/search/data/transforms/lookups HTTP/1.1" 500 3070 "http://splunk.mk6.local/en-US/manager/search/lookups" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36" - afbdfc0c38412f8496710fc57c992ff9 311ms

But I do not see any detail on why the 500 error is occurring. Does Splunk have a .log file that contains internal server error data?

Thank you,
Michael

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...