Installation

500 Internal Server Error After Reconfigure

zebulajams
Explorer

Hello all,

I'm am new to Splunk and installed the free Enterprise version to start learning to expand my skill set. I am able to install Splunk locally and monitor files on the computer it is installed on. However I am now wanting to try to monitor a remote computer. I have set up a test VM and was going to install the Universal Forwarder when it asked me for my Receiving Indexer. Obviously I cannot input the 127.0.0.1 for the IP, so I tried changing the IP where the Splunk server is running. Per the Splunk documentation, I changed the mgmtHostPort line in the web.conf from 127.0.0.1:8089 to 10.xx.xx.xx:8089. I also added the SPLUNK_BINDIP=10.xx.xx.xx to the splunk-launch.conf file. After doing this, I tried to restart Splunk and it timed out due with a entry in the log, "Could not bind to ip 10.xx.xx.xx port 8089". Ok - so I reverted all my changes to their default configuration and now when I try to log into Splunk, I get "500 Internal Server Error". Everything is as it was when it was first installed and I could log in, and I've also tried 3-4 times restarting the Splunk service on my PC.

This is a Windows installation p.s.

Any ideas? This happened last week and the only thing I could do to fix it was uninstall and reinstall Splunk. Is that the only fix for when Splunk acts up?

Thanks!

Labels (3)
Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

You should have been able to undo the changes you made to restore normal function, but when that fails re-installing often is the best way to go.

Splunk automatically listens on all NICs so there's usually no need to set a bind address.  Just tell the forwarder to send to the server's public IP address (every Windows box on a network should have one).  Then tell the server to listen on port 9997 (Settings->Forwarding and Receiving).

---
If this reply helps you, Karma would be appreciated.
0 Karma

zebulajams
Explorer

Oh. So if I understand correctly: even though the Splunk server URL is http://127.0.0.1:8000/en-US/, I can keep it that way without modifying those files and just tell the forwarder to look for my machine's IP instead and use port 9997? Then just set up the receiving port on the Splunk server to be 9997.

Is that right?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

That's right.  The URL http://127.0.0.1:8000 is for the user interface.  Other interfaces will use other ports.

---
If this reply helps you, Karma would be appreciated.
0 Karma

zebulajams
Explorer

Ok. I reinstalled Splunk on my local machine and then installed the Universal Forwarder on my VM and set the IP to my local machine and the port to the one I set within the Receiving settings. However if I do a search for anything it does not return any results. Is there something I missed in the Forwarder setup? I checked all the boxes that it offered and also told it to monitor a directory I created on the VM.

Thanks for your help so far!

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Have you checked your firewalls?

---
If this reply helps you, Karma would be appreciated.
0 Karma

zebulajams
Explorer

I double checked and followed the Splunk Troubleshooting Universal Forwarder On Windows .pdf and set the inbound and outbound rules to accept the connections from the desired ports. However now when I restart Splunk from the CLI I get, "ERROR: http port [8000] - port is already bount. Splunk needs to use this port. Would you like to change ports? [y/n]:" Is there a better port I can switch it to?

0 Karma

SinghK
Builder

Btw you can do netstat -nabo |findstr 8000 to find nd which application is using 8000.

0 Karma

zebulajams
Explorer

Looks like it does not return anything when I run that command. If I try to restart it again it is now telling me that mgmt port [8089] is already bound and says http port [8000] is open. If I do that same netstat command for 8089 it does not return anything either.

0 Karma

SinghK
Builder

Just change web port to 8001 and mgmt port to 8090 or some thing unique that is not in use for sure will be easier

0 Karma

zebulajams
Explorer

I have changed the ports but now it's not restarting the service in a timely fashion, nor is it sending any data to my Splunk server from the Forwarder. I am also receiving errors about "The TCP output processor has paused the data flow." Seems like installing/configuring Splunk is a difficult and long process. I've been using the Splunk documentation to try to install, is there a video reference for this I can use too? Otherwise I think I might give up on learning Splunk lol.

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Splunk is easy to install and run.  However, it's easier on Linux than on Windows.

If you get an error about port 8000 already in use when Splunk starts then it usually means Splunk is already running.  Try using splunk stop followed by splunk start rather than splunk restart.  Avoid changing ports as that can open a can of worms that is hard to contain.

The "The TCP output processor has paused the data flow" message means the forwarder has been unable to connect to a Splunk server for a while.  Once you resolve the connectivity problem the message will stop.

Make sure you have the right IP address in the forwarder.  Double-check the firewalls.  Ensure the VM has network access.

When the connection works, you should see results returned from the search index=_internal host=<<your VM name>>*

---
If this reply helps you, Karma would be appreciated.
0 Karma

zebulajams
Explorer

The only other thing I can think that is wrong is that the VM resides on vSphere on our domain, but the VM itself is not on the domain whereas the Splunk server (my machine) is on the domain. Is that an issue? I haven't read anywhere where they need to be on the same domain.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Splunk doesn't care about domains, but your network might.  Talk to your network admin.

---
If this reply helps you, Karma would be appreciated.
0 Karma

zebulajams
Explorer

After spinning up a new VM and starting from scratch, creating the 9997 listening port in Splunk, enabling the firewall rules, and keeping all the default port configurations, it still does not communicate with Splunk Enterprise. I found this link which I've been reading through and following but it does not seem to help. I have already configured it as the documentation says.

Not sure what else to try even after watching a LinkedIn Learning beginners guide on installing Splunk and the UF and not working from there either.

Thanks to everyone for the help. Looks like this one is left a mystery on how to get Splunk to work!

Thanks again!

0 Karma

zebulajams
Explorer

Interesting. Maybe I'll create a separate VM and install Splunk Enterprise there to see if it will communicate with that one first. I'm baffled why it won't accept anything from the Universal Forwarder machine, even after completely disabling the firewall. I verified it can ping other machines too, but it can't ping my local machine which might have something to do with it (no idea why it doesn't like my machine even though other VM's can ping it just fine).

I guess I'll try that and go from there. Not much else I can do.

0 Karma

SinghK
Builder

Switch to 443 enable https l.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...