Another consideration could be if Splunk is being redirected through a Gateway, Load Balancer, etc. Busy Gateways can timeout depending on several variables... For the purposes of this question, this possibility should be ruled out if it is not a factor. If it is a factor, as mentioned above, switch to Developer mode in your browser (F12), and that should display all communications, good and bad, from client to server. ... View more

Another consideration could be if Splunk is being redirected through a Gateway, Load Balancer, etc. Busy Gateways can timeout depending on several variables... For the purposes of this question, this possibility should be ruled out if it is not a factor. If it is a factor, switch to Developer mode in your browser (F12), and that should display all communications, good and bad, from client to server. ... View more

Another consideration could be if Splunk is being redirected through a Gateway, Load Balancer, etc. Busy Gateways can timeout depending on several variables... For the purposes of this question, this possibility should be ruled out if it is not a factor. If it is a factor, switch to Developer mode in your browser (F12), and that should display all communications, good and bad, from client to server. ... View more

Looks like the link may have been changed, and, the latest is showing 6.4.10 https://www.splunk.com/page/previous_releases/universalforwarder ... View more

I realize that this is a very late answer... but, I would like to add that, in addition to the above, if you are running Linux<=6.9 (just not sure about 7.x)... AND you "enable boot-start"... Splunk UF and Splunk Enterprise have the same name. In other words , I had to modify the name of the forwarder script in init.d and manually add the script to chkconfig. ... View more

I thought of another thing to check... If the user running splunk is not root... what permissions does the user have? I ran into a scenario where I deployed my forwarders as root, but, my SH, IDX, and HF's as splunk.... so, while all of the other boxes were reporting their/var/log/*... my splunk infrastructure was not sending logs due to permissions and not just inputs.conf sequencing. Sorry to muddy the waters... but, it's a variable. ... View more

In addition to the above suggestion, I would recommend using the btool to re-evaluate inputs.conf. The order of inputs stanzas matter. In my case, I uploaded an add-on for Unix and Linux, and due to the way Splunk aggregates the inputs, it didn't reach the stanzas I put in because it was hitting an earlier stanza and putting the data into another index. In the end, just make sure your aggregated stanza order, for inputs.conf, is not interfering with your intentions, if you can't find your data or you find it in the wrong index. ... View more

Some may come across this problem, if the .conf file is not in the expected location. For example, I created an app, that I checked in triplicate... only to figure out that, I was putting the .conf in the wrong location... so, Splunk couldn't pick it up. I received the same message as above. To test connectivity, I also reinstalled the UF and used the GUI to see if it would, "pop," in the deployment server... SUCCESS!?!? So, I later realized that Splunk couldn't see the file because I didn't put the .conf file in the local folder inside of the app name folder. The file belongs here on the UF: %SplunkHome%/etc/apps//local/<.conf file> Not: %SplunkHome%/etc/apps//<.conf file> As mentioned above, I have seen varying results from different syntax. My observation is that spaces sometimes matter after the equals sign. Going directly from docs.splunk.com, it seems that there are very subtle inconsistencies in syntax. Spacing the argument from the variable seems to work most consistently for me. Best Wishes. ... View more

I will look into this method. Thank you for the suggestion! ... View more

Perhaps, I wasn't very clear, my apologies, but, when I spoke of stovepipes, it should have read as single Splunk standalone instances (SIs). So, because each SI is not a source for information across all environments... this flattened way of doing business is being changed. My initial proposal was to turn the SI's into Heavy Forwarders, and build a proper indexing tier and search head. But, it seems that may be more than was originally anticipated by the customer. So, the traditional Splunk architecture appears to have a few road blocks. Bringing this full circle, I proposed the question above. The idea is to keep the single instances in place because they are already collecting the proper data. However, if we can, "cherry pick," events that we will trigger across all environments, we could send these alerts to a brand new SH that would be dedicated to only triggered alerts. Here's a crude ASCII art sketch: A SI B SI |________________| C SI - | ALERTS SH | - D SI E SI - |________________| - X SI Let me know if this clears things up. Thank you taking the time to help. ... View more

ALCON, This question registers, but, the answer doesn't really stand out. So, here it is: Check the roles and indexes available to whichever user is trying to see data. For example, my admin user didn't see the windows data on my recent install and constantly said, "waiting for data." When I checked the role permissions to the index that I created... admin didn't have access by default. After adding the new index to the admin user, I then saw the number of indexed events on the search app "home," page. Happy Splunking! ... View more

If you know your forwarders are checking in, then you can simply run a search... with an eye on timestamps. First, check to see that your specific forwarders are checking in using the monitoring console (MC). Second, search for specific data using the Searching and Reporting app. ... View more

Interesting... my recall may be failing me, but, I think I was on an indexer (creating an index via GUI... don't judge me too badly... :D) and trying to save a colddb to a location inside of $SPLUNK_HOME. It's possible that it may have been one directory higher... Hmmm... maybe I didn't apply the correct permissions after after expanding the .tgz? Unfortunately, there is no way for me to check, the system I built was only temporary. Do you have any information on the /udev/random permission? And, thank you for taking time to help me understand. ... View more