I am just starting out to learn Splunk, and have just attempted the Module 4 Lab from Splunk 7.x Fundamentals Part 1. I have installed the free trial Splunk Enterprise on a cloud server employing Ubuntu 18.04.
As part of the Module 4 Lab, in the course of executing "Add Data" and "Upload files from my computer" and selecting the lab-provided file "access_30Day.log" from my laptop, I ended up with a process that took about an hour and a half, and resulted in 29,173,335 indexed events from 36 sources.
According to the tutorial, after loading the three files, I should have 239,625 indexed events. Clearly something went awry. I will speculate on what, at the close of this post. Meanwhile...
(1) How do I delete or remove the events, NONE of which bear any association with the lab's data file?
(2) How do I prevent the inadvertent, ongoing collection of this data? I wish to have NOTHING incoming that is not explicitly part of the "Fundamentals" course.
I have looked through answers provided in this forum, but not knowing how to properly ask, have been unsuccessful at zeroing in on what to do. Among other sources of confusion, many referred to functionality of older versions of Splunk. Others referred to operations that take one outside of the Web Console or into a context I could not identify.
I did take a look at splunk's /etc/system/local/inputs.conf file. The contents (in contrast to the default version) are minimal:
[default] host = cnit-ubuntu18
The Sources of the indexed Events aligns with this, as they consist primarily of log files on my Ubuntu cloud server. Is the phrase [default] in the the local file basically a call to include the official "default" inputs.conf?
Speculation on what happened: I had an assignment in another class that had us install Splunk several weeks back, and it ran us through a couple of operations. That assignment was completed without incident, via rote execution and with little comprehension of the whys and wherefores on my part. I assume that either the lab script either wasn't conscientious about "cleaning up" or that I may have overlooked something that led to the current state of affairs.
Regardless, I'd just like a clean slate now, and it seems to me removing inputs and data should be easy to accomplish. I just am unable to figure out how to do so.
How is this for a plan of action?
1) stop Splunk
2) remove the local/inputs.conf file
My assumption is that the default will run in its place.
3) run the following command:
sudo ./splunk clean eventdata
4) restart Splunk
Based on my reading of the documentation topic "Managing Indexers and Clusters of Indexers -> Remove indexes and indexed data" I took a look at Settings -> Indexes and Settings -> Inputs. I can't tell if some of them are things that should be left alone.
@philfrei 's comment addresses how to delete the events using
splunk clean. I would not bother removing local/inputs.conf, however. Don't use sudo, either, as Splunk shouldn't be running as root.
[default] in inputs.conf is a stanza name, not an invocation. This stanza groups settings that apply to all inputs, unless overridden by the individual inputs. The default/inputs.conf file is always invoked with settings in local/inputs.conf overriding those in default.
Thanks for this.
I can't vouch that the answer solved the issue. I'm sure everything you wrote is correct, and am upvoting accordingly. I guessing I ran into the 5000MB limit and this is preventing an upload of even the smallest of the three sample files, despite having executed the clean command.
The error given for the upload attempt was the following, for a 5MB linux log file:
Upload failed with ERROR : Read Timeout
But I also have received the message
The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch.
Also, don't know if this is worth mentioning, but when I ran the cleanevents, I got the following message (after a number of "cleaning database dbname" messages)
Disabled database 'splunklogger': will not clean.
Maybe the thing to do is to remove this installation altogether and start over with a new test install. We shall see if Splunk allows that.
You can ignore the "disabled database" message.
The "minimum free disk space" message can be cleared by freeing some disk space, increasing the disk size, or by changing
minFreeSpace to a smaller number in server.conf.
Yes, Splunk allows you do delete and re-install the software.