Getting Data In: Splunk Platform - Wed 9/11/24

adepp — Fri, 20 Sep 2024 19:43:37 GMT

Register here. This thread is for the Community Office Hours session on Getting Data In (GDI) to Splunk Platform on Wed, September 11, 2024 at 1pm PT / 4pm ET.

This is your opportunity to ask questions related to your specific GDI challenge or use case, including:

Onboarding common data sources (AWS, Azure, Windows, *nix, etc.)
Forwarder troubleshooting, connectivity issues, blocked queues, etc.
Apps and add-ons to get data in
Processing data (filter, mask, enrich, route) with Edge Processor, Ingest Processor, or Ingest Actions
Archiving your data
Anything else you’d like to learn!

Please submit your questions at registration or as comments below. You can also head to the #office-hours user Slack channel to ask questions (request access here).

Pre-submitted questions will be prioritized. After that, we will open the floor up to live Q&A with meeting participants.

Look forward to connecting!

Re: Getting Data In: Splunk Platform - Wed 9/11/24

adepp — Fri, 20 Sep 2024 19:43:11 GMT

Here are a few questions from the session (get the full Q&A deck and live recording in the #office-hours Slack channel):

Q1: What are the optimal forwarder configurations forgetting data into Splunk Cloud with Forwarders (e.g., best practices architectures, with a more technical how to)

Use Universal Forwarders, when possible (unless an HF is necessary - for transforms, routing, or to run some apps such as DB Connect)
- Include EVENT_BREAKER in props.conf
Avoid intermediate forwarders
- Layers add complexity, increase failure points, and may lead to uneven distribution of events across indexers.
Use volume-based load balancing over time-based load balancing
Consider using HEC
Documentation: Splunk Validated Architectures

Q2: What are some new features/solutions for getting data into Splunk?

Edge Processor

Available to all, but must also be a Splunk Cloud customer
Processes data locally, before it is sent to indexers

Ingest Processor

Available in Splunk Cloud only
Processes data in Splunk Cloud, before shipping to destination(s)

Ingest Actions

Available to all
Unlike transforms, can process cooked data

OpenTelemetry Collector (CNCF, second largest project after Kubernetes)

Open source
Supports HEC, OTLP

Documentation:

Q3: I am very new to Edge and Ingest Processor. Could you please walk-through with an example from a beginner's perspective?

Resources:

Edge Processor Demo video - live example of how to create a pipeline to filter, enrich, and route data.
Data Management Pipeline Builders Resource Hub

Documentation:

Lantern step-by-step guide: Getting started with Splunk Data Management Pipeline Builders
First-time setup instructions for:

Other Questions/Topics (check the #office-hours Slack channel for responses):

How can I start routing and filtering with the ingest processor after it has been enabled in the Cloud Stack?
Splunk licence saving tips
AWS S3 data ingestion through SQS
Let's talk about Hybrid landscape
How to efficiently GDI Commvault storage data?
How to efficiently GDI Dell Unity data?
How to efficiently GDI Microsoft System Center VM Manager (SCVMM) and System Center Operations Manager (SCOM) data?
Best services to use for best practice for S3 data lakes and Splunk? Security, anomaly detection, scanning of logs, etc.
GDI and Auto-discovery of entities from BMC Helix/ADMM and auto creation and dependency mapping of related services, entities in ITSI
Testing custom apps with splunk cloud. IE testing in prod because it’s cloud.
I am sending data via HEC to a stand alone Indexer, how can I send data via HEC to an Indexer Cluster instead? Best practice ?
When Edge processor and Ingest Processor will be available for Splunk onprem ?
I have an indexer cluster where an IDX is getting more activity that others? how can i fix it ?

rss.livelink.thread@place:occasion

Getting Data In: Splunk Platform - Wed 9/11/24

Re: Getting Data In: Splunk Platform - Wed 9/11/24