tl;dr: SplunkTrust Nominations and Applications are open now through Aug 25th! Nominate a peer for the SplunkTrust with this form ; and/or apply for your own spot with this form.
It’s That Time of Year!
Knowledge is great... but helping is even better! Are you a rock-star-helper-outer in our community? Or have you seen a few exemplary members helping others? Now is the time to recognize and reward our most helpful and contributing community members by considering them for our SplunkTrust MVP program.
Nominating Someone for the SplunkTrust...
Annually since 2015, Splunk has selected exemplary members of the Splunk community to join the SplunkTrust. Any community member can nominate their peers for selection in the program, and community members may also apply for a spot themselves.
What makes a great submission? We’re not looking for people who know the most about Splunk; we already have training and certifications to reflect that knowledge. We look for people who are passionate about helping their peers be successful using whatever Splunk & domain knowledge they have.
If you know of someone who represents these values, use this form to nominate them . We’ll make sure any nominees have an opportunity to apply before the deadline. In your nomination, be sure to tell us how they have helped you, or others, to be successful, and other community-supporting accomplishments.
Applying for the SplunkTrust...
Do you feel you fit these values? Or were you recently nominated and want to formally apply? Let us know by using this form to apply yourself . Tell us about yourself, your community-focused accomplishments, and encourage your peers to submit nominations on your behalf. We love celebrating the exceptional members of our Splunk community.
About the Selection Process
This year's SplunkTrust selection process will include four phases: 1. Submission, 2. Analysis, 3. Hammer Fight, and 4. Induction.
Each candidate, whether nominated by someone else or applying themselves, must complete their own application. We plan to accept applications through Wednesday, August 25th, 2021. There are no limits to how many nominations you can submit, and there are no limits on the quantity of applications we will consider. So nominate often!
We'll compile all applications and enrich them with data from various community platforms and programs (Splunk Answers, User Groups, Community Slack, and more). We also ask existing members to help rate applicants, and all of this information is provided to the members involved in Hammer Fight so they may review applicants prior to the final selection.
3. Hammer Fight
Over a dozen Splunk employees gather for an hours-long meeting to discuss all of the applicants and determine the new SplunkTrust cohorts. Hammer Fight is the colloquial name given to this final selection process. We don’t recall exactly when the name Hammer Fight became associated with this, and there are no hammers or fighting actually involved in the process 😀 .
Last, and most excitedly, is notifying and welcoming next year's cohorts. Selected members will be notified mid-September of their status. At .conf21 we will have an induction ceremony to welcome the members of the SplunkTrust and provide them with a fez. Selected members do not need to attend .conf21 in-person or virtually.
So... you have a few short weeks to nominate your peers or apply yourself, and I really hope you do both! Because the only thing better than being an exemplary, passionate, and helpful member of our community is doing it in a stylish new fez!
In Fez We Trust,
Jason Hupka, Sr. Manager and Principal Trust Wrangler
P.S. Questions are welcome! Just comment below! Here are a few of the more common questions I hear...
I’m a Splunk expert, so can I be on the SplunkTrust?
Remember, selection is not based on what you know, but how you help others with whatever you know.
I am already a SplunkTrust member. Do I need to re-apply?
Yes - the SplunkTrust cohort is selected annually so you must re-apply
Do I need to attend .conf21 if I am selected for the SplunkTrust?
No - you do not need to attend .conf21 if selected. We do have an induction ceremony at the conference, but you do not need to be present to accept the honor.
What is the airspeed velocity of an unladen swallow?
African or European?
How many SplunkTrust members do you select?
We do not have a set number of membership slots for the program.
When will I know if I have been selected for the SplunkTrust?
We plan to notify all applicants the week of September 16th, 2021 of their status.
... View more
Has anyone ran Splunk on an Intel Compute Stick like the CS125? How does it perform versus the CS325 or CS525?
I'm curious to hear any stories from the trenches on how a Search Head or Universal Forwarder worked on them.
... View more
Since you can login from the same server, and not another, and you're running on Windows, could a problem be you have Windows Firewall running and blocking inbound connections?
By default Splunk will install with the web-gui listening on Port 8000.
Port 8089 will be the splunkd admin port - normally you do not need to worry about the admin port at this point because you probably aren't using the REST interface from the outside, running a Deployment Server, and some of the more "enterprise" features.
If you are wanting Universal Forwarders to connect and send data in, and you've been following some of the Splunk Documentation, I am going to assume you created a TCP Input on Port 9997
So in general if you're having connectivity issues from outside the server, you probably have something blocking 8000, 8089, and 9997.
Here is more info on some of the typical/default Splunk ports that are used:
As far as the password issues, my only thought is a typo was made typing in the password when changing from the default of admin/changeme when installed. Then, when you login again you aren't replicating that typo. Plus, I can't think of a normal situation where I've had this happen. I think the only time I've ever manually touched the Splunk internal passwd file is when I forgot the admin password and had to reset it to changeme. In this case I followed the steps outlined in one of your links - backup file, delete original, restart, new password, copy everything from backup file back into the new passwd EXCEPT for the admin user. Also, long term, if you do not hook Splunk up to an LDAP provider you will hate managing users/roles/etc. Not because Splunk has any issues...just because that's what LDAP is for and you want to do Splunk, not user-management.
The only other thing I can think of that happened with the passwd file is if after the fact the Splunk secret got messed up...this is generated on install, and is used for hashing things from then on. If you changed the password, messed with the secret, then nothing can hash the same that is bad. And if you chose to mess with the secret on purpose I would highly suggest rethinking that because you enter dangerous territory and really need to know what you're doing it for (e.g. have a shared hash-result across all of your Splunk instances). I've never messed with this personally...
So overall I am going to lean towards some sort of environment issue right now, because in my experience doing Splunk Administration/Architecture I haven't seen these things under normal circumstances, e.g. not, "Whoops I broke it by messing around just to see what happens..."
... View more
What OS are you running on? Is this Windows? (sounds like it from your description of stopping/starting a service, but maybe it is AIX...)
Do you have just one server in all of this? You mention Splunk Enterprise and a Universal Forwarder (UF). If you have just one server and you're trying out Splunk, you just need to install the "regular" Splunk Enterprise download. This would allow you to index log files locally on that server - either continuously, or via one-time scenarios where you just have a log file you want to test out.
Unless you actually have several other servers that you want to send data from to this test instance, then don't worry about the Universal Forwarder installation for now. Let's try and just get your Splunk Enterprise instance working and understand what your overall test is, then start hooking up other things using Universal Forwarders.
Can you provide links to the other topics you found here on SplunkAnswers? That might help us understand what scenarios you have tried and lead us in the right direction of where you would like to go.
... View more
There's a few options available. With the built-in Email Alert Action you can control some of the formatting, add your own links (albeit hard to make dynamic), and change what is in the body with the default options.
On the other hand writing a Custom Alert Action in Splunk is fairly straight forward and plugs into an existing interface to make it easily configurable via the Splunk Web GUI:
Also, there are many Custom Alert Action apps out on splunkbase that could fulfill your needs:
For example, if you do need more of an alert/incident workflow, there's an app for that:
Finally, from my experience I was never a fan of sending a basic email alert that just dumped people into Search with the raw results. I always preferred to use the alert as the notification of the problem, then the email would link them to a relevant dashboard that searched not only the data that might have triggered the alert, but also correlating data to immediately help with root-cause analysis. For example, if an order management system had orders failing, the dashboard would not only display what has failed and easily let them select criteria to group by server/customer/etc to see who is being affected with failures, but would also display *nix data of CPU, memory, and disk usage for the servers processing orders so common failure-modes are already being displayed to the person handling the alert to begin investigation.
... View more
If you are wanting to generate your own link directly to the results you'll need to find out the Search ID (SID) of the query you just ran. The easiest way to do this is via the addinfo command. For example:
index=awesome_data earliest=-60m | ... | addinfo
This will add a few fields to all of your event results, but the one you care about is info_sid. You can use this field to build your own reference to the results of the search. The search page within Splunk can take a form field named sid with the value of the Search ID like the following:
So if you wanted to do this within the search you could do something like this:
index=awesome_data earliest=-60m | ... | addinfo | eval results_url="https://splunk.deshpandevikasv.com/en-US/app/search/search?sid=" . info_sid
When you go to the link with a provided sid then Splunk will find the results from that prior query from the dispatch directory.
A couple notes:
Keep in mind the default retention of search results. You don't want Splunk to delete the results after 10m if you want to access the data via URL for a couple days. If you're doing this via a Scheduled Search then make sure you set the TTL of the results appropriately. This will also impact disk space if you are saving lots of results for a long time.
You might not want to go directly to the Search page - but obtaining that sid using the addinfo command is the key take-away. That sid value can be used other places once you have it (e.g. using the loadjob command)
... View more
Like people have mentioned, there are several free (as in free-beer) ways to get a perpetual license that is sufficient for personal use and learning. There are also several free courses you can take through Splunk Education here: https://www.splunk.com/en_us/view/education/SP-CAAAAH9
Overall the free license options are a balance - just like the free-samples at the store are for a taste...not a whole meal.
... View more
It is also important to consider that if you are often hitting the maxTotalDataSizeMB before the age specified in frozenTimePeriodInSecs, then you can potentially confusing "holes" in your data when people search. This especially becomes apparent with multiple Indexers because Indexer 1 might start running out of disk and culling buckets, but Indexer 2 is fine on space and has older data spanning the time period of the deleted buckets from Indexer 1. When you search over the time period, there might be end-user confusion if they are expecting 100% of the results for that time period but due to size half were deleted on Indexer 1.
... View more