So I have been using the "free" 500mb version of Splunk at home for about 6 months now and I have had to reinstall Splunk at least 5 times. The reason....it disappears! I know this sounds crazy, I will qualify and explain as best I can.
I install Splunk onto a fresh Ubuntu VM and follow the instructions (which have changed over the past 6 months or so) provided on the website. After installation, everything works perfectly....I access the web interface, I add all my stuff and it works brilliantly. I monitor my logs for a month or so and then I leave it. It is just my home lab, so I do not check it daily (as I know I should), but if I leave it alone for too long, (I have not been able to pinpoint a time frame, but I went away camping for a little over a week, figure I didn't check 5 ish days on either end, so I'd say about 2 weeks at least) it stops working.
By not working / disappeared, I mean that the Splunk Web interface no longer answers. Splunk is installed on the VM, but there is no splunk service. I cannot run service splunk start and I can see very little traces of splunk other than it's installed (apt-get install splunk comes back already newest) and the only fix I have found is a reinstall (which loses all the data and configs). I have also tried the VM on multiple servers with various configurations in case it was some sort of hardware related issue...same result each time.
Is there something I am missing? I do not even know where to begin troubleshooting because it just seems that my problem is, Splunk just freaking disappeared on me, it got up, took half its stuff and vacated my server.