Getting Data In: Splunk Platform - Wed 9/11/24

1 Comment
Cover Images - Office Hours (7).png
Published on ‎06-12-2024 09:37 AM by Splunk Employee | Updated on ‎09-20-2024 12:43 PM

Register here. This thread is for the Community Office Hours session on Getting Data In (GDI) to Splunk Platform on Wed, September 11, 2024 at 1pm PT / 4pm ET.

 

This is your opportunity to ask questions related to your specific GDI challenge or use case, including:

  • Onboarding common data sources (AWS, Azure, Windows, *nix, etc.)
  • Forwarder troubleshooting, connectivity issues, blocked queues, etc.
  • Apps and add-ons to get data in
  • Processing data (filter, mask, enrich, route) with Edge Processor, Ingest Processor, or Ingest Actions
  • Archiving your data
  • Anything else you’d like to learn!

 

Please submit your questions at registration or as comments below. You can also head to the #office-hours user Slack channel to ask questions (request access here)

 

Pre-submitted questions will be prioritized. After that, we will open the floor up to live Q&A with meeting participants.

 

Look forward to connecting!



0 Karma
adepp
Splunk Employee

Here are a few questions from the session (get the full Q&A deck and live recording in the #office-hours Slack channel):

 

Q1: What are the optimal forwarder configurations forgetting data into Splunk Cloud with Forwarders (e.g., best practices architectures, with a more technical how to)

  • Use Universal Forwarders, when possible (unless an HF is necessary - for transforms, routing, or to run some apps such as DB Connect)
    • Include EVENT_BREAKER in props.conf
  • Avoid intermediate forwarders
    • Layers add complexity, increase failure points, and may lead to uneven distribution of events across indexers.
  • Use volume-based load balancing over time-based load balancing
  • Consider using HEC
  • Documentation: Splunk Validated Architectures

Q2: What are some new features/solutions for getting data into Splunk?

  • Edge Processor
    • Available to all, but must also be a Splunk Cloud customer
    • Processes data locally, before it is sent to indexers
  • Ingest Processor
    • Available in Splunk Cloud only
    • Processes data in Splunk Cloud,  before shipping to destination(s)
  • Ingest Actions
    • Available to all
    • Unlike transforms, can process cooked data
  • OpenTelemetry Collector (CNCF, second largest project after Kubernetes)
  •  
    • Open source
    • Supports HEC, OTLP

Documentation: 

Q3: I am very new to Edge and Ingest Processor. Could you please walk-through with an example from a beginner's perspective?

Resources:

Documentation: 

Other Questions/Topics (check the #office-hours Slack channel for responses):

  • How can I start routing and filtering with the ingest processor after it has been enabled in the Cloud Stack?
  • Splunk licence saving tips
  • AWS S3 data ingestion through SQS
  • Let's talk about Hybrid landscape
  • How to efficiently GDI Commvault storage data?
  • How to efficiently GDI Dell Unity data?
  • How to efficiently GDI Microsoft System Center VM Manager (SCVMM) and System Center Operations Manager (SCOM) data?
  • Best services to use for best practice for S3 data lakes and Splunk?  Security, anomaly detection, scanning of logs, etc.
  • GDI and Auto-discovery of entities from BMC Helix/ADMM and auto creation and dependency mapping of related services, entities in ITSI
  • Testing custom apps with splunk cloud. IE testing in prod because it’s cloud.
  • I am sending data via HEC to a stand alone Indexer, how can I send data via HEC to an Indexer Cluster instead? Best practice ?
  • When Edge processor and Ingest Processor will be available for Splunk onprem ?
  • I have an indexer cluster where an IDX is getting more activity that others? how can i fix it ?