Hello everyone, I need your help and guidance.
Since I configured the universal forwarder on my Splunk Enterprise and in a forwarder, I have been having issues logging on my server.
I saw several topics that discussed about this issue but anything solve it.
What I discovered:
I can login on the same server but not from a different computer.
I can't login through Splunk mobile.
After serveral server restart / and service restart I am able to login again.
If I delete the password file and restart the Splunk Service I manage to login again but I have to create again the users.
I am looking for a permanent solution, due I am evaluating Splunk but this issue is about to end with my patience.
Can anyone help me to figure out what's wrong and how to solve it.
I know that I should look in the logs but there is to much files and I don't know where to begin.
If you can login sometimes but not others, the problem may be that you have 2 machines with the same identity and sometimes you are hitting one and sometimes you are hitting the other. This could be caused deliberately by using a load-balancer/VIP or accidentally by using the same IP Address on 2 different servers/NICs.
Since you can login from the same server, and not another, and you're running on Windows, could a problem be you have Windows Firewall running and blocking inbound connections?
So in general if you're having connectivity issues from outside the server, you probably have something blocking 8000, 8089, and 9997.
Here is more info on some of the typical/default Splunk ports that are used:
As far as the password issues, my only thought is a typo was made typing in the password when changing from the default of admin/changeme when installed. Then, when you login again you aren't replicating that typo. Plus, I can't think of a normal situation where I've had this happen. I think the only time I've ever manually touched the Splunk internal passwd file is when I forgot the admin password and had to reset it to changeme. In this case I followed the steps outlined in one of your links - backup file, delete original, restart, new password, copy everything from backup file back into the new passwd EXCEPT for the admin user. Also, long term, if you do not hook Splunk up to an LDAP provider you will hate managing users/roles/etc. Not because Splunk has any issues...just because that's what LDAP is for and you want to do Splunk, not user-management.
The only other thing I can think of that happened with the passwd file is if after the fact the Splunk secret got messed up...this is generated on install, and is used for hashing things from then on. If you changed the password, messed with the secret, then nothing can hash the same that is bad. And if you chose to mess with the secret on purpose I would highly suggest rethinking that because you enter dangerous territory and really need to know what you're doing it for (e.g. have a shared hash-result across all of your Splunk instances). I've never messed with this personally...
So overall I am going to lean towards some sort of environment issue right now, because in my experience doing Splunk Administration/Architecture I haven't seen these things under normal circumstances, e.g. not, "Whoops I broke it by messing around just to see what happens..."
What OS are you running on? Is this Windows? (sounds like it from your description of stopping/starting a service, but maybe it is AIX...)
Do you have just one server in all of this? You mention Splunk Enterprise and a Universal Forwarder (UF). If you have just one server and you're trying out Splunk, you just need to install the "regular" Splunk Enterprise download. This would allow you to index log files locally on that server - either continuously, or via one-time scenarios where you just have a log file you want to test out.
Unless you actually have several other servers that you want to send data from to this test instance, then don't worry about the Universal Forwarder installation for now. Let's try and just get your Splunk Enterprise instance working and understand what your overall test is, then start hooking up other things using Universal Forwarders.
Can you provide links to the other topics you found here on SplunkAnswers? That might help us understand what scenarios you have tried and lead us in the right direction of where you would like to go.
the Splunk Enterprise Server is on a Windows server 2012 X64 and the UF is on a Windows 7 x86 computer, I need to continue with this structure due I plan to connect at least 20 more UF.
the topics that I read:
the common work around solution was to rename the Password file to gain access back on the app but the problem is that I cant do this every day.
thank you so much.
I tested on the same network and outside.
When I am inside the same network I can login on my Splunk without a problem, but when I am trying outside the network I cant reach the server.
Only after I remove the Password file on my server, is that I can login again.
Thanks for all the help.
With this post in mind, I went and reviewed the firewall request for the last day and finally discover that some ISP Blocked my request (so I wasn't able to reach the splunk server).
So I modified the web port from 8000 to 8080.
Thanks to everyone that replied my post.