Why is Splunk 6.2.2 unable to search logs from my Linux server with the universal forwarder installed?

[Ealderiso]

Hello,

I am having an issue with logs coming into my instance of Splunk Enterprise (version 6.2.2) through a Linux server with the universal forwarder installed.

I have the server properly whitelisted in my serverclass.conf, ports 9997 and 8089 are also allowed through the firewall between the forwarder and the indexer, the server is able to phone home in my server class, and I can see in metrics.log that my address is connected and is sending events: connect_close and connect_done to my Splunk server.

Despite all of this, I cannot search through any of the logs in the Search & Reporting app. I made sure I have the right location for the logs in the server class and in the server itself. Everything should be fine and logs should be coming in normally (like my other servers) but this one is still not working correctly.

Does anyone have any ideas as to why this is happening and have any suggestions for some troubleshooting steps?

[nychawk · Accepted solution]

What user is the Splunk user running as?

If you su'd to that user, does that user have permission to read those files? If not, a change is in order, best performed on the directory and/or file you wish to have read.

If this does not work, try looking inside your /opt/splunkuniversalforwarder/var/log/splunk/splunkd.log for some clues.

HTH

[Ealderiso]

Are you talking about on the forwarder server? If so - we installed Splunk as root for now. So it should not be a permissions issue.

[nychawk]

Yes, I was referring to the UF on the forwarder. Your DS sees a check-in, but you are not getting data, right?
Does your serverclass that this machine belongs to have an outputs.conf assigned?

Next option would be to look inside of /opt/splunkforwarder/var/log/splunk/splunkd.log for some tell-tale signs.

[Ealderiso]

Yes it does - we define an outputs.conf globally for all apps. That is correct it is able to check-in to the deployment server but I just cannot search any data. In splunk I see that the server is connected and is deployed to the appropriate app. But I cannot search any of the logs.

I will try looking in the splunk.d log as well.

[twinspop]

If you're seeing no internal logs from the host, it's not connected to the INDEXER. Connecting to the DS is only half the battle. Can you telnet to the indexer from the forwarder on the port you're using for splunktcp?

[Ealderiso]

I can telnet to the IP address on ports 9997 and 8089. The only difference is that in the outputs.conf on the DS it has in internal DNS entry defined. This remote server does not use our internal DNS - so it does not know where that address is located. I am going to try editing the hosts file to see if that works.

[nychawk]

Can you telnet to port 9997 on your indexer from your UF?

[Ealderiso]

Okay so thanks to your help I was able to discover that the issue was that the outputs.conf on the indexer pointed to an internal DNS entry that the UF located outside our network did not know about. So now the forwarder is connected and the apache logs are searchable in splunk.

However, one of my apps is working properly and the other is not (I cannot see the logs on one of them). I can get logs from Apache but not a custom logging location. Are there are any troubleshooting steps you would recommend to see why an app is not able send logs to the splunk indexer. I have everything defined properly on the DS and Indexing server.

[esix_splunk]

First try searching _internal to see if your host is actually sending its internal logs.

index=_internal earliest=-15m@m | stats count by host

That will tell you all the hosts sending internal logs in the last 15minutes.

If your server is listed in there, then it is working correctly. Most likely you dont have inputs setup on your host to collect its logs. Splunk doesnt automagically do this.

If its not showing your expected hosts, then you should check your outputs and make sure your indexers are listed.

Also, on your deployment server, is it showing the hosts as connecting and apps being deployed?

[Ealderiso]

Okay Awesome - so it is not listed there which makes sense why it is not working.

How do I setup inputs on my forwarding host to allow it to collect logs?
So when you say I should check my outputs to see if my indexer is listed... where do I do that?

It is actually showing that the host is connected in my deployment server. It is able to Phone Home and it has 3 Apps Deployed. I cannot search for my host in any of those apps though 😕

It is just a little odd because it seems as if my host is able to connect - but there is something wrong in the configuration which is causing the logs to not be searchable.

[esix_splunk]

Your outputs.conf on the host in question should point to your indexers. Most likely if you compare to a host that is working you will find that its not setup properly. Or perhaps there is a network (Firewall) issue blocking connectivity..

./splunk btool outputs list

Run that on the broken host and one a working host...

[Ealderiso]

Okay thank you. The command you gave me - btool is not working properly. It says that outputs is an invalid command.

[esix_splunk]

Had it backwards, should be btool outputs list

Updated the comment.

[Ealderiso]

Thank you! So that command worked and I will compare with my other forwarder. My only strange concern is that on my splunk forwarder in: $SPLUNK_HOME/etc/system/local there is no outputs.conf - only deploymentclients.conf inputs.conf and server.conf

However, that forwarder still works and is configured properly in my indexing server so I am able to search through the logs properly.

Should I create an outputs.conf there?

[esix_splunk]

If there is no outputs.conf on your host, then it wont send to your indexers. Which might be why you cannot see it. As @twinspop says, create an app on your DS with the outputs in it, and deploy that since the host is already a member of the DS. Thats the easiest way to do it.

[Ealderiso]

Well actually on my forwarder outputs.conf is defined in $splunk/etc/system/default

So I think I am okay. I do not want to make that change on the deployment server because there are other forwarders that are working properly the way it is configured now.

The only thing that is different about this particular forwarder is that the traffic is going through a Site-Site VPN tunnel - but we have all the appropriate traffic allowed. From the forwarder server I can access TCP ports 9997 and 8089 on the deployment server.

[twinspop]

I'd recommend defining outputs.conf in a new app on your deployment server.

$splunk/etc/deployment-apps/fwd_outputs/local/outputs.conf

Edit the app in the DS GUI to "restart splunkd" and include this new app in your server class. That way you'll be able to update it easily if needed in the future.

[sk314]

Try searching across "All Time" (also "All Time Real Time" if you are continuously sending data) to check if it's a timestamp issue. This might sound trivial...however I have found this to be the issue many times in my experience.

[Ealderiso]

No this was not it unfortunately 😞 they are unsearchable across all time. I was never able to search the logs when I added this forwarder

[skoelpin]

Check out this article which covers a lot of common things that could go wrong

http://docs.splunk.com/Documentation/Splunk/6.4.3/Troubleshooting/Cantfinddata