TL;DR - We are recognizing your growing contributions and leadership in our community with a new MVP program alongside the SplunkTrust. We are delighted to announce the launch of our new Splunk Community MVPs program, which will sit alongside our existing SplunkTrust program. The SplunkTrust will always recognize the best, set the highest standard, and be our most elite MVP-style program, recognizing our top contributors across all community areas; our new Splunk Community MVPs will join our MVP programs to recognize and reward a broader collection of top contributors in each community area. Once upon a time, there was a fez. And the fez was brilliant; the fez was inspiring; the fez represented our SplunkTrust… our most esteemed and most contributing community leaders. The first cohort of 22 members were inducted at .conf2015 , and the most recent cohort of 69 members were inducted at .conf21 . We are proud to continue to recognize and reward our highest and most esteemed community of valuable contributors with our SplunkTrust program, and the fez will always signify these members across our community spaces. With all respect and praise to the fez… when we look across these programs and spaces, we can’t help but notice hundreds of community members standing out and stepping up. The fact is, our Splunk Community has grown and flourished in so many fabulous ways. In just the last two years, we’ve launched a new technology platform for Community , a new platform for User Groups , and welcomed the Community Slack workspace (finally!) into the fold of official programs. Splunk has also continued its own meteoric rise and growth with new products, new solutions, and even new companies as part of the growing family. And so it makes sense that we’ve seen thousands of new customers and new community members join our programs in recent years. Amidst all of the growth - across Splunk and across our Splunk Community - we wanted to say that… WE SEE YOU! We see SO many of you posting, engaging, solving, and contributing in even more diverse and numerous ways. In fact, YOUR growth has been so tremendous that we feel we simply must recognize and reward even more of you for this! So while our SplunkTrust will always remain as our most elite program, we’re so stoked to have it be joined by this new one, welcoming and recognizing so many more. How do I become a Splunk Community MVP? Chances are, you’re already well on your way! This program looks across all areas of community contributions, prioritizing engagement and support, in whatever areas you might be most passionate about. Do you love answering questions about Splunk SOAR in Splunk Answers? That’s one way! Driving fabulous engagement in your Splunk User Group? That’s another way! Rockin’ out the Slack contributions? Yep, that could definitely score you an invite! Are you contributing some amazing technical content in our blogs and Tech Talks? Yep, even that. The beauty of this tier is that it can help recognize stand-out contributions in even a single area, and it will recognize hundreds of members overall. Twice each year, we’ll review these kinds of contributions alongside specific selection criteria (sorry, the specifics are secret to help avert gaming the system!), and we’ll send special invitations to community members that qualify. What are the perks and responsibilities as a Splunk Community MVP? Why should I say “yes”? As an MVP, the expectations are few, and the rewards are many. First, if invited, we hope you’ll continue engaging in the ways you have been or look to grow them even more. We want to provide you the limelight and recognition with a special community badge, upgraded permissions, featured content contributions, and of course… some rad swag. We ask that MVPs continue to be culture carriers of our Splunk community ideals, lead by example to bolster the community, and participate in advocacy opportunities to highlight their successes with Splunk. Benefits Asks Public recognition with your MVP badge Upgraded community permissions Inclusion in our private MVP community areas Featured contributions in community blogs Educational perks and learning opportunities MVP sneak peeks for product news & more And of course... AWESOME SWAG! Keep doing what you're doing! (Contribute and lead by example!) Attend at least one quarterly MVP meeting (or watch the recap) every six months Participate in one Splunk Love advocacy activity every six months Keep engaging and keep an eye on your inbox! Again, participation is totally optional, but if you meet the criteria and receive an invite, and if you want to join the program, we’d love to have you! So keep up the great community engagement, and you might receive that special invitation very soon. We can’t wait to introduce you to our first group of Splunk Community MVPs ! -- Jason Hupka (a.k.a. "snooplogg"), Sr. Community Trust & MVP Manager
... View more
tl;dr: SplunkTrust Nominations and Applications are open now through Aug 25th! Nominate a peer for the SplunkTrust with this form ; and/or apply for your own spot with this form.
It’s That Time of Year!
Knowledge is great... but helping is even better! Are you a rock-star-helper-outer in our community? Or have you seen a few exemplary members helping others? Now is the time to recognize and reward our most helpful and contributing community members by considering them for our SplunkTrust MVP program.
Nominating Someone for the SplunkTrust...
Annually since 2015, Splunk has selected exemplary members of the Splunk community to join the SplunkTrust. Any community member can nominate their peers for selection in the program, and community members may also apply for a spot themselves.
What makes a great submission? We’re not looking for people who know the most about Splunk; we already have training and certifications to reflect that knowledge. We look for people who are passionate about helping their peers be successful using whatever Splunk & domain knowledge they have.
If you know of someone who represents these values, use this form to nominate them . We’ll make sure any nominees have an opportunity to apply before the deadline. In your nomination, be sure to tell us how they have helped you, or others, to be successful, and other community-supporting accomplishments.
Applying for the SplunkTrust...
Do you feel you fit these values? Or were you recently nominated and want to formally apply? Let us know by using this form to apply yourself . Tell us about yourself, your community-focused accomplishments, and encourage your peers to submit nominations on your behalf. We love celebrating the exceptional members of our Splunk community.
About the Selection Process
This year's SplunkTrust selection process will include four phases: 1. Submission, 2. Analysis, 3. Hammer Fight, and 4. Induction.
Each candidate, whether nominated by someone else or applying themselves, must complete their own application. We plan to accept applications through Wednesday, August 25th, 2021. There are no limits to how many nominations you can submit, and there are no limits on the quantity of applications we will consider. So nominate often!
We'll compile all applications and enrich them with data from various community platforms and programs (Splunk Answers, User Groups, Community Slack, and more). We also ask existing members to help rate applicants, and all of this information is provided to the members involved in Hammer Fight so they may review applicants prior to the final selection.
3. Hammer Fight
Over a dozen Splunk employees gather for an hours-long meeting to discuss all of the applicants and determine the new SplunkTrust cohorts. Hammer Fight is the colloquial name given to this final selection process. We don’t recall exactly when the name Hammer Fight became associated with this, and there are no hammers or fighting actually involved in the process 😀 .
Last, and most excitedly, is notifying and welcoming next year's cohorts. Selected members will be notified mid-September of their status. At .conf21 we will have an induction ceremony to welcome the members of the SplunkTrust and provide them with a fez. Selected members do not need to attend .conf21 in-person or virtually.
So... you have a few short weeks to nominate your peers or apply yourself, and I really hope you do both! Because the only thing better than being an exemplary, passionate, and helpful member of our community is doing it in a stylish new fez!
In Fez We Trust,
Jason Hupka, Sr. Manager and Principal Trust Wrangler
P.S. Questions are welcome! Just comment below! Here are a few of the more common questions I hear...
I’m a Splunk expert, so can I be on the SplunkTrust?
Remember, selection is not based on what you know, but how you help others with whatever you know.
I am already a SplunkTrust member. Do I need to re-apply?
Yes - the SplunkTrust cohort is selected annually so you must re-apply
Do I need to attend .conf21 if I am selected for the SplunkTrust?
No - you do not need to attend .conf21 if selected. We do have an induction ceremony at the conference, but you do not need to be present to accept the honor.
What is the airspeed velocity of an unladen swallow?
African or European?
How many SplunkTrust members do you select?
We do not have a set number of membership slots for the program.
When will I know if I have been selected for the SplunkTrust?
We plan to notify all applicants the week of September 16th, 2021 of their status.
... View more
Has anyone ran Splunk on an Intel Compute Stick like the CS125? How does it perform versus the CS325 or CS525?
I'm curious to hear any stories from the trenches on how a Search Head or Universal Forwarder worked on them.
... View more
Since you can login from the same server, and not another, and you're running on Windows, could a problem be you have Windows Firewall running and blocking inbound connections?
By default Splunk will install with the web-gui listening on Port 8000.
Port 8089 will be the splunkd admin port - normally you do not need to worry about the admin port at this point because you probably aren't using the REST interface from the outside, running a Deployment Server, and some of the more "enterprise" features.
If you are wanting Universal Forwarders to connect and send data in, and you've been following some of the Splunk Documentation, I am going to assume you created a TCP Input on Port 9997
So in general if you're having connectivity issues from outside the server, you probably have something blocking 8000, 8089, and 9997.
Here is more info on some of the typical/default Splunk ports that are used:
As far as the password issues, my only thought is a typo was made typing in the password when changing from the default of admin/changeme when installed. Then, when you login again you aren't replicating that typo. Plus, I can't think of a normal situation where I've had this happen. I think the only time I've ever manually touched the Splunk internal passwd file is when I forgot the admin password and had to reset it to changeme. In this case I followed the steps outlined in one of your links - backup file, delete original, restart, new password, copy everything from backup file back into the new passwd EXCEPT for the admin user. Also, long term, if you do not hook Splunk up to an LDAP provider you will hate managing users/roles/etc. Not because Splunk has any issues...just because that's what LDAP is for and you want to do Splunk, not user-management.
The only other thing I can think of that happened with the passwd file is if after the fact the Splunk secret got messed up...this is generated on install, and is used for hashing things from then on. If you changed the password, messed with the secret, then nothing can hash the same that is bad. And if you chose to mess with the secret on purpose I would highly suggest rethinking that because you enter dangerous territory and really need to know what you're doing it for (e.g. have a shared hash-result across all of your Splunk instances). I've never messed with this personally...
So overall I am going to lean towards some sort of environment issue right now, because in my experience doing Splunk Administration/Architecture I haven't seen these things under normal circumstances, e.g. not, "Whoops I broke it by messing around just to see what happens..."
... View more
What OS are you running on? Is this Windows? (sounds like it from your description of stopping/starting a service, but maybe it is AIX...)
Do you have just one server in all of this? You mention Splunk Enterprise and a Universal Forwarder (UF). If you have just one server and you're trying out Splunk, you just need to install the "regular" Splunk Enterprise download. This would allow you to index log files locally on that server - either continuously, or via one-time scenarios where you just have a log file you want to test out.
Unless you actually have several other servers that you want to send data from to this test instance, then don't worry about the Universal Forwarder installation for now. Let's try and just get your Splunk Enterprise instance working and understand what your overall test is, then start hooking up other things using Universal Forwarders.
Can you provide links to the other topics you found here on SplunkAnswers? That might help us understand what scenarios you have tried and lead us in the right direction of where you would like to go.
... View more
There's a few options available. With the built-in Email Alert Action you can control some of the formatting, add your own links (albeit hard to make dynamic), and change what is in the body with the default options.
On the other hand writing a Custom Alert Action in Splunk is fairly straight forward and plugs into an existing interface to make it easily configurable via the Splunk Web GUI:
Also, there are many Custom Alert Action apps out on splunkbase that could fulfill your needs:
For example, if you do need more of an alert/incident workflow, there's an app for that:
Finally, from my experience I was never a fan of sending a basic email alert that just dumped people into Search with the raw results. I always preferred to use the alert as the notification of the problem, then the email would link them to a relevant dashboard that searched not only the data that might have triggered the alert, but also correlating data to immediately help with root-cause analysis. For example, if an order management system had orders failing, the dashboard would not only display what has failed and easily let them select criteria to group by server/customer/etc to see who is being affected with failures, but would also display *nix data of CPU, memory, and disk usage for the servers processing orders so common failure-modes are already being displayed to the person handling the alert to begin investigation.
... View more
If you are wanting to generate your own link directly to the results you'll need to find out the Search ID (SID) of the query you just ran. The easiest way to do this is via the addinfo command. For example:
index=awesome_data earliest=-60m | ... | addinfo
This will add a few fields to all of your event results, but the one you care about is info_sid. You can use this field to build your own reference to the results of the search. The search page within Splunk can take a form field named sid with the value of the Search ID like the following:
So if you wanted to do this within the search you could do something like this:
index=awesome_data earliest=-60m | ... | addinfo | eval results_url="https://splunk.deshpandevikasv.com/en-US/app/search/search?sid=" . info_sid
When you go to the link with a provided sid then Splunk will find the results from that prior query from the dispatch directory.
A couple notes:
Keep in mind the default retention of search results. You don't want Splunk to delete the results after 10m if you want to access the data via URL for a couple days. If you're doing this via a Scheduled Search then make sure you set the TTL of the results appropriately. This will also impact disk space if you are saving lots of results for a long time.
You might not want to go directly to the Search page - but obtaining that sid using the addinfo command is the key take-away. That sid value can be used other places once you have it (e.g. using the loadjob command)
... View more
Like people have mentioned, there are several free (as in free-beer) ways to get a perpetual license that is sufficient for personal use and learning. There are also several free courses you can take through Splunk Education here: https://www.splunk.com/en_us/view/education/SP-CAAAAH9
Overall the free license options are a balance - just like the free-samples at the store are for a taste...not a whole meal.
... View more
It is also important to consider that if you are often hitting the maxTotalDataSizeMB before the age specified in frozenTimePeriodInSecs, then you can potentially confusing "holes" in your data when people search. This especially becomes apparent with multiple Indexers because Indexer 1 might start running out of disk and culling buckets, but Indexer 2 is fine on space and has older data spanning the time period of the deleted buckets from Indexer 1. When you search over the time period, there might be end-user confusion if they are expecting 100% of the results for that time period but due to size half were deleted on Indexer 1.
... View more