Awesome Admins: Managing Your Hybrid/On-Prem Deployment - Wed 6/28/23

Community Office Hours
×

Are you a member of the Splunk Community?

Sign in or Register with your Splunk account to get your questions answered, access valuable resources and connect with experts!
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Ask a Question

Awesome Admins: Managing Your Hybrid/On-Prem Deployment - Wed 6/28/23

Awesome Admins: Managing Your Hybrid/On-Prem Deployment - Wed 6/28/23

3 Comments
Cover Images - Office Hours.png
Published on ‎05-03-2023 04:44 PM by Splunk Employee | Updated on ‎07-25-2023 12:09 PM

This thread is for the Office Hours session Awesome Admins: Managing Your Hybrid/On-Prem Deployment on Wed, June 28, 2023 at 1pm PT / 4pm ET.

 

Register Here to level up your Admin Chops! Join our new bi-weekly Office Hour zoom series where technical Splunk experts answer questions and provide how-to guidance on a different topic every month! This session is dedicated to Splunk hybrid/on-prem admins and will cover any topics or questions related to: 

  • Managing your hybrid or on-prem Splunk deployment
  • Optimizing performance 
  • Intermediate forwarding tiers
  • Federated search
  • Anything else you'd like to learn!

 

Please submit your questions below as comments in advance. You can also head to the #office-hours user Slack channel to ask questions (request access here). 

 

Pre-submitted questions will be prioritized. After that, we will go in order of the questions posted below, then will open the floor up to live Q&A with meeting participants. If there’s a quick answer available, we’ll post as a direct reply.

 

Look forward to connecting!


Labels (2)
0 Karma
adepp
Splunk Employee
Splunk Employee
‎05-31-2023 06:12 PM
‎05-31-2023 06:12 PM

Hey Everyone!

Please add your questions/comments here for any topics you'd like to see discussed in the Community Office Hours session (you can also head to the #office-hours user Slack channel to ask questions and join the discussion - request access here).

0 Karma
adepp
Splunk Employee
Splunk Employee
‎07-10-2023 08:48 AM
‎07-10-2023 08:48 AM

Here are some questions from the session (to view the live recording, join the #office-hours user Slack channel):

 

Q1: What do you find is the easiest way to alert on hosts that stop sending a particular sourcetype when events of that sourcetype may not appear every day?

  • A:  | metadata type=sourcetypes index=<name or * for all> 

|convert ctime(lastTime) AS "last Indexed Time" 

|table "last Indexed Time" sourcetype totalCount 

|sort -totalCount

 

Q2: What are the advantages of using federated search as opposed to just pointing the search head to each indexer/cluster master directly?

  • A: The main benefit of Fed Search is the ability to give access to a remote Splunk dataset by using a secured search head that acts as a proxy layer between both deployments.
  • This can simplify environment interconnectivity requirements by utilizing communication between search heads on the splunk management port interface versus having to peer search heads to all known indexers for search access; which may require admin level access or networking changes.
  • Using RBAC and service accounts Federated Providers can have greater control of which Federated Search Heads have access to datasets and can appropriately set resource capacity limits.
  • Federated Search Transparent Mode and Standard Mode both give more options for security and access control when compared with distributed/hybrid search. 

 

Q3: I have/am an on-premise or BYOL cloud customer who is experiencing performance issues and/or skipped searches. How can Splunk help?

  • A: Pre-Flight Check (PFC): PFC is a new Splunk process that’s designed to help the customer and Splunk to perform a remote Splunk deployment-level analysis for any On-Premise & BYOL Public Cloud Splunk deployments from where diags can be collected and uploaded to a low priority Support ticket using our Customer Portal.
  • Splunk resources can then use PFC to help check and provide prescriptive feedback on:
    • Splunk versioning (core components & UFs)
    • Infrastructure CPU/Memory and Storage layout info
    • SmartStore & Health Status 
    • Resource Utilization overview & Health.log information
    • Splunk Search concurrency metrics and full listing of Splunk Apps installed 
    • BTOOL Navigator - a BTOOl UI to help analyze set configurations vs Splunk defaults

Other Tools/Apps discussed 

 

  • Hipster (available in Kate’s Public repo): A tool to help identify some of these similar constraints. 
    • Disclaimer from Kate: Use at your own risk! Not a Splunk-supported product. It is only meant for testing & is considered experimental.
  • Splunk Cloud Migration Assessment (SCMA) app: Splunkbase app to assess your current deployment - designed to be put on your monitoring console and to run against a live environment. 

 

Q4: How does a deployment server work and what are some best practices? 

  • A: The Splunk Deployment Server is a Splunk Enterprise feature that when enabled allows you to manage the app configuration update process across sets of other Splunk Enterprise & Universal Forwarder hosts.
  • Deployment Servers have 2 primary components: deployment-apps and serverclass.conf 
  • deployment-apps can either be customer-built or downloaded from SplunkBase. Apps are collections of .conf files that are distributed to the hosts for where the data is to be collected.
  • The serverclass.conf is the main configuration file that controls the distribution of the deployment-apps throughout the Splunk Enterprise & Universal Forwarder hosts by the use of whitelist and blacklist filters. 
  • Additional Resources: Deployment Server Update - 2023

 

0 Karma
adepp
Splunk Employee
Splunk Employee
In response to adepp
‎07-10-2023 08:49 AM
‎07-10-2023 08:49 AM

Live Q&A 

Q1: I have an issue where the update.sh script is continuously checking for package updates and holding a lock on the yum repo on some of our machines. How do we go about troubleshooting, something like that? Is it an issue? A configuration problem?

  • A:  Yea that's not us. We wouldn't necessarily deploy it like that. So we'd want to look at what that script is doing, to validate that. Checking for package updates is what the deployment server is supposed to do. What you should be trying to do is set them all to the deployment server and update them from there through a push process, because you don't necessarily want things going out and grabbing stuff randomly

 

Follow-up dialogue: 

  • So inside the Splunk_TA_nix addon, we do see the update there, it’s trying to update the regular Linux packages.
    • A: Just comment that out then, because that doesn’t need to happen.
  • The problem is here: let’s say we get a new updated version of our TA, then again, it breaks us again, we have to go back there and update that. So what would be our permanent solution? 
    • A: The permanent solution would be to download it to the deployment server and then deploy it from there. Download the app with the configuration that has the setting you want (with possibly a local override) and do it that way.

 

Q2: We write our own security content for clients, so we have a development environment (Splunk Enterprise), and then the production environment (Splunk Cloud). The initial issue we've been having when writing our own custom content or custom apps, is that sometimes they'll pass vetting, but there are some features that may stop working and we think it might be due to some security limitations spunk have put in place on Cloud, but we are not sure about this. And is there a best practice way to kind of get around that or develop things?

  • A: There’s an application validation check that has to happen for any app before it goes into Splunk Cloud. Because we don’t want nutty things going into Cloud. We’ll connect you with the right people to talk to on the Splunk Cloud Developer Edition team to discuss this!
0 Karma