rss.livelink.thread@place:occasion

Getting Data In: Forwarders & Edge Processor - Wed 8/23/23

adepp — Thu, 24 Aug 2023 21:12:58 GMT

[1pm PT / 4pm ET] - Register here and ask questions below. This thread is for the Community Office Hours session on Getting Data In (GDI): Forwarders & Edge Processor on Wed, August 23, 2023 at 1pm PT / 4pm ET.

This is your opportunity to ask questions related to getting data into Splunk Platform using forwarders or Splunk Edge Processor. Including:

Universal forwarder or heavy forwarder setup and troubleshooting
Forwarder connectivity issues, blocked queues, and tuning
Using Edge Processor
Forwarders vs. Edge Processor vs. Ingest Actions
Anything else you’d like to learn!

Please submit your questions at registration or as comments below. You can also head to the #office-hours user Slack channel to ask questions (request access here).

Pre-submitted questions will be prioritized. After that, we will go in order of the questions posted below, then will open the floor up to live Q&A with meeting participants. If there’s a quick answer available, we’ll post as a direct reply.

Look forward to connecting!

Re: Getting Data In: Forwarders & Edge Processor - Wed 8/23/23

adepp — Mon, 21 Aug 2023 18:01:17 GMT

Hey Everyone,

Don't forget to submit your questions at registration! You can also post a comment here for any topics you'd like to see discussed in the Community Office Hours session, or head to the #office-hours user Slack channel to ask questions and join the discussion (request access here).

Re: Getting Data In: Forwarders & Edge Processor - Wed 8/23/23

splunkzilla — Tue, 22 Aug 2023 22:00:48 GMT

First off, thanks so much for putting this together!

My main issue so far with the EP is configuring TLS to work when performing the initial configuration of a processor. According to the documentation I can use the Splunk Forwarder certs that I downloaded from the Splunk Cloud platform and place them in the Server Key/Server Cert/CA Cert sections of the Edge Processor config. There are two problems I run into when I do this:

1. I get the error -

There was a problem creating your new Edge Processor.

INVALID_SERVER_PK_PEM_FORMAT (I can confirm that the files are in PEM format)

2. The other issue is (I think) that the outputs.conf would need to be updated to reflect the architecture change (UF/HEC -> Splunk Cloud to UF/HEC -> EP -> Splunk Cloud). Would that be done after the cert config is completed?

Re: Getting Data In: Forwarders & Edge Processor - Wed 8/23/23

adepp — Wed, 23 Aug 2023 22:05:14 GMT

Expert Solution:

Occurs because the private key is formatted incorrectly for a private key PEM. The file must be in correct PEM format and correctly tagged as a private key in the header and footer.
Yes, certs need to be updated in the output.conf file. This can be done before or after EP has be updated. Data will not be received until both are updated.

Re: Getting Data In: Forwarders & Edge Processor - Wed 8/23/23

splunkzilla — Wed, 23 Aug 2023 22:08:52 GMT

Thank you! I did discover before the session began that the issue was related to a number of intermediate certs contained in the PEM file. Once I removed those the import worked properly.

Re: Getting Data In: Forwarders & Edge Processor - Wed 8/23/23

adepp — Thu, 24 Aug 2023 20:57:39 GMT

Great! Glad you figured it out. I also posted the session recording in the #office-hours if you'd like to rewatch the experts talk through it.

Cheers,

Baylie

Re: Getting Data In: Forwarders & Edge Processor - Wed 8/23/23

adepp — Thu, 24 Aug 2023 21:12:22 GMT

Here are a few questions from the session (get the full Q&A deck and live recording in the #office-hours Slack channel):

Q1: Can Forwarder be on the same server as Splunk? Do you need forwarders for security monitoring?

Yes, you can multi-tenant a UF with another instance, but it's not advised due to resource contention.
Strictly speaking, no, you don't need forwarders for security monitoring. While not strictly required, forwarders improve real-time data collection, helping in timely security monitoring.

Q2: How do you ingest logs from linux auditd into your Splunk app for processing?

The simplest solution is often the best! https://splunkbase.splunk.com/app/833 (or Splunk Add-On for Unix/Linux)
followed by… https://docs.splunk.com/Documentation/AddOns/released/Linux/Configure4
- If you do the above but don’t see data, check the troubleshooting step in the same doc for a helpful tip when using TCP protocol: https://docs.splunk.com/Documentation/AddOns/released/Linux/Troubleshoot#Audit_data_not_collected
- There are 3rd party solutions available as well though we cannot endorse them. As with all things, test in a small isolated instance before deploying any new configuration to a larger environment!

Q3: Why am I having blocked queues? How do I troubleshoot blocked queues that are preventing data from being indexed?

The short answer is usually either a bottleneck or a failure in the pipeline. The longer answer can be a bit more complicated.
First you want to identify where the blockage originates!
- Like a river with a dam, always follow it downstream until you find the obstruction. Using ‘metrics.log (blocked=true)’ as your guide, this means start from the Indexer’s queues and work your way backwards first. Then once the instance is identified, narrow it down to a specific queue.
Once you find the queue that is affected, the troubleshooting steps can vary particularly heavily. Here’s some examples by blocked queue type:
- Parsing: Check for new data that may not have proper encoding, headers or linebreak rules. splunkd.log can help!
- AggQueue: Check for bad date/time or bad extractions for date/time and any MUST_BREAK rules.
- Typing: Regex, regex, regex….usually. New data with new extractions? Too many .*’s in that regex pattern? Regex Profile can help here too!
- Index: Usually throughput or network/OS configuration at play here but could be any number of possibilities. splunkd.log is your best friend with this one!

Q4: What’s the difference between "search in" vs "search IN"?

Short answer: they are the same thing but made distinct because of which command precedes them. This is referenced in the following sections of our documentation:
- “in(<field>,<list>)” refers to its use (in lowercase) as part of the ‘where’ or ‘eval’ commands, for example.
- “Multiple field-value comparisons with the IN operator” refers to its use (in uppercase) in the ‘search’ or ‘tstats’ command function as a comparison operator.
Syntactically, SPL will alert you to the validity of your case choice based on the command within which it is contained and the attached modifiers and lists though, typically, these are the rules:
- IN (“field1”,”field2”) is valid for search and tstats
- in(field1,field2) is valid for where, eval and fieldstats

Q5: How do you configure Edge Processor servers and where? In cloud or in our HF(LAN)?

https://docs.splunk.com/Documentation/SplunkCloud/latest/EdgeProcessor/CreateNode
OR https://docs.splunk.com/Documentation/SplunkCloud/latest/EdgeProcessor/QuickStart
The configuration of each node happens on your environment locally but can be managed all from the Cloud Platform!

Q6: How can I set up Edge Processor as a service?

We’ve got you covered with instructions for configuring systemd upon installation here: https://docs.splunk.com/Documentation/SplunkCloud/9.0.2305/EdgeProcessor/CreateNode#Install_an_instance_and_configure_systemd

Other Questions (check the #office-hours Slack channel for responses):

Is Edge Processor available today and who can use it?
How do I access Edge Processor?
Why do I need a Forwarder?
How can I get Windows internal logging (wineventlog) only starting from the time that I startup the Windows UF and NOT all historical, so I don’t exceed licensing?
Installing a universal forwarder that doesn't have connectivity to a deployment server. What do I need to copy from a host that was deployed by the deployment server? This is Splunk Cloud
Edge Processor overview
Edge Processor demo videos

Re: Getting Data In: Forwarders & Edge Processor - Wed 8/23/23

adrifesa95 — Wed, 21 Feb 2024 11:59:05 GMT

Hello, I would like to have access to the video in https://splunk-usergroups.slack.com/