Getting Data In: Platform - Wed 5/8/24

adepp — Thu, 16 May 2024 18:13:24 GMT

Register here. This thread is for the Community Office Hours session on Getting Data In (GDI) to Splunk Platform on Wed, May 8, 2024 at 1pm PT / 4pm ET.

Join our Office Hour series where technical Splunk experts answer questions and provide how-to guidance on a different topic every month! This is your opportunity to ask questions related to your specific GDI challenge or use case, including:

How to onboard common data sources (AWS, Azure, Windows, *nix, etc.)
Using forwarders
Apps and add-ons to get data in
Processing data with Edge Processor, Ingest Processor, and Ingest Actions
Archiving your data
Anything else you’d like to learn!

Please submit your questions at registration or as comments below. You can also head to the #office-hours user Slack channel to ask questions (request access here).

Pre-submitted questions will be prioritized. After that, we will open the floor up to live Q&A with meeting participants.

Look forward to connecting!

Re: Getting Data In: Platform - Wed 5/8/24

adepp — Thu, 16 May 2024 18:13:02 GMT

Here are a few questions from the session (get the full Q&A deck and live recording in the #office-hours Slack channel):

Q1: Seeing bottlenecks in forwarder getting data into Splunk Cloud from syslog server, should output be pointed to multiple ports?

Check maxKBps setting in limits.conf
Bottlenecks can sometimes be an indication of a need for an interim layer of forwarders (UF or HF) to help balance the load, especially if it fluctuates. This also will differ if you’re using (or not using) Splunk Connect for Syslog.
parallelIngestionPipelines could be leverage if output is the bottleneck. For inputs, additional ports OR leveraging the forwarder reading local syslogs stored on the host can be leveraged for increasing throughput
https://lantern.splunk.com/Data_Descriptors/Syslog
https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Pipelinesets#Forwarders_and_multiple_pipeline_sets

Q2: How can I extract additional fields from the "properties.log" field from AKS events sent to an EH (Azure Event Hub?) being ingested via MSCS app?

For individual fields, you can use the rex command or EXTRACT in props.conf.
To extract all fields, use the spath command.
https://docs.splunk.com/Documentation/Splunk/9.2.1/SearchReference/Rex
EXTRACT
https://docs.splunk.com/Documentation/Splunk/9.2.1/SearchReference/Spath

Q3: How do I get Syslog, SNMP traps, Streaming Telemetry, and non-standard formats in?

See the SNMP Modular Input app on splunkbase
Common GDI methods: UF, API, DB Connect, or HTTP Event Collector
Custom modular input or dedicated receiver
You probably will have to write your own props.conf settings
Getting Data In manual
Send syslog via UF: https://www.youtube.com/watch?v=XnCEZTKOm90
SC4S overview: https://www.youtube.com/watch?v=7ZmVgy9NL3U
Splunk Connect for syslog (SC4S): https://www.youtube.com/watch?v=iJ1iBZdXt2o
How to connect SC4S in 5 mins: https://www.youtube.com/watch?v=1Ur3xDNaE4s

Other Questions (check the #office-hours Slack channel for responses):

Preferred Getting Data In (GDI) method recommended by Splunk
Can we have master and slave Splunk Enterprise instances? Slave is connected always but master is connected only sometimes.
Syslog forwarder setup
Splunk license saving tips
Splunk in 2030: Getting Data In (GDI) experience
I’d like to hear/watch how to ingest logs from Cisco devices switches/routers with IOS, usage of sc4s with IOS or maybe not using sc4s?
I would like to hear your thoughts on potential root cause for duplicate data coming from a single endpoint however each duplicate event has a different timestamp. Using TA-microsoft-graph-security-add-on-for-splunk
How do you charge based on resources if it is 100% on prem owned by the customer?
Splunk docs talk about Hybrid-Cloud to mean Splunk manages infrastructure and application at the indexer and above level. What is the definition of a full cloud environment (not Hybrid). Can everything from the UF all the way be managed in the cloud by Splunk for large org?
What is the definition of a full cloud environment (not Hybrid). Can everything from the UF all the way be managed in the cloud by Splunk for large org?

rss.livelink.thread@place:occasion

Getting Data In: Platform - Wed 5/8/24

Re: Getting Data In: Platform - Wed 5/8/24