Splunk Search

Logs aren't coming in, forward servers are listed as inactive

happycaptain
Loves-to-Learn

I have very little experience with splunk, and am on a time crunch, so a bit of patience for my ignorance would be awesome. So today I was setting up an enterprise splunk solution for logs. I set up the universal forwarders on a few devices, and set up my indexer on a centos server. I set the receiving port (the default of 9997), set up a new index to sort my data out, and added from the indexer section, which seemed to work, except that I don't actually see any logs. When I get into those operating systems and run a list forward-server command (on linux) it comes back with inactive: ipaddress:port. I tried to see if there was something wrong with my firewall, but everything seems to be open for the 9997 port, I can ping back and forth between systems, I checked my outputs.conf file to make sure that there was the right server address there, and my inputs.conf seem right. I'm beyond clueless after reading all kinds of forums.

I also am having a bit of an issue with space on the system. Splunk tells me that my disk space is at the minimum under opt/splunk8 to deployment, but I don't know what is taking that space. Maybe it's the logs that were sent but never indexed? Where would those end up? (I made the mistake of not setting an index for the monitors that I set up earlier.)

Any help is appreciated, and again, I don't know a whole lot about splunk, so I'm just trying to get it to work... I had plans on integrating splunk into splunk phantom, but that's not happening until splunk works lol.

Thanks!

0 Karma

happycaptain
Loves-to-Learn

I set the minimum down to 500mb (I know its not suggested) and the same thing..

0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

Indexing stops once minimum free disk space is reached for the directories where indexed data is stored. Check if you are storing your indexes in default splunk indexes path (/opt/splunk/var/lib/splunk). If yes, you need to change the path to some other path with sufficient disk space available.

And the default minimum free disk space required is 5000MB.

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...