Logs aren't coming in, forward servers are listed as inactive

happycaptain — Fri, 07 Feb 2020 02:07:08 GMT

I have very little experience with splunk, and am on a time crunch, so a bit of patience for my ignorance would be awesome. So today I was setting up an enterprise splunk solution for logs. I set up the universal forwarders on a few devices, and set up my indexer on a centos server. I set the receiving port (the default of 9997), set up a new index to sort my data out, and added from the indexer section, which seemed to work, except that I don't actually see any logs. When I get into those operating systems and run a list forward-server command (on linux) it comes back with inactive: ipaddress:port. I tried to see if there was something wrong with my firewall, but everything seems to be open for the 9997 port, I can ping back and forth between systems, I checked my outputs.conf file to make sure that there was the right server address there, and my inputs.conf seem right. I'm beyond clueless after reading all kinds of forums.

I also am having a bit of an issue with space on the system. Splunk tells me that my disk space is at the minimum under opt/splunk8 to deployment, but I don't know what is taking that space. Maybe it's the logs that were sent but never indexed? Where would those end up? (I made the mistake of not setting an index for the monitors that I set up earlier.)

Any help is appreciated, and again, I don't know a whole lot about splunk, so I'm just trying to get it to work... I had plans on integrating splunk into splunk phantom, but that's not happening until splunk works lol.

Thanks!

Re: Logs aren't coming in, forward servers are listed as inactive

manjunathmeti — Fri, 07 Feb 2020 15:48:11 GMT

Indexing stops once minimum free disk space is reached for the directories where indexed data is stored. Check if you are storing your indexes in default splunk indexes path (/opt/splunk/var/lib/splunk). If yes, you need to change the path to some other path with sufficient disk space available.

And the default minimum free disk space required is 5000MB.

Re: Logs aren't coming in, forward servers are listed as inactive

happycaptain — Fri, 07 Feb 2020 20:48:35 GMT

I set the minimum down to 500mb (I know its not suggested) and the same thing..

topic Re: Logs aren't coming in, forward servers are listed as inactive in Splunk Search

Logs aren't coming in, forward servers are listed as inactive

Re: Logs aren't coming in, forward servers are listed as inactive

Re: Logs aren't coming in, forward servers are listed as inactive