Here are a few questions from the session (get the full Q&A deck and live recording in the #office-hours Slack channel):

Q1: Can Forwarder be on the same server as Splunk?  Do you need forwarders for security monitoring?

• Yes, you can multi-tenant a UF with another instance, but it's not advised due to resource contention. 
• Strictly speaking, no, you don't need forwarders for security monitoring. While not strictly required, forwarders improve real-time data collection, helping in timely security monitoring.

Q2: How do you ingest logs from linux auditd into your Splunk app for processing?

• The simplest solution is often the best! https://splunkbase.splunk.com/app/833 (or Splunk Add-On for Unix/Linux)
• followed by…  https://docs.splunk.com/Documentation/AddOns/released/Linux/Configure4
  • If you do the above but don’t see data, check the troubleshooting step in the same doc for a helpful tip when using TCP protocol: https://docs.splunk.com/Documentation/AddOns/released/Linux/Troubleshoot#Audit_data_not_collected 
  • There are 3rd party solutions available as well though we cannot endorse them. As with all things, test in a small isolated instance before deploying any new configuration to a larger environment!

Q3: Why am I having blocked queues? How do I troubleshoot blocked queues that are preventing data from being indexed?

• The short answer is usually either a bottleneck or a failure in the pipeline. The longer answer can be a bit more complicated.
  First you want to identify where the blockage originates!
  
  • Like a river with a dam, always follow it downstream until you find the obstruction. Using ‘metrics.log (blocked=true)’ as your guide, this means start from the Indexer’s queues and work your way backwards first. Then once the instance is identified, narrow it down to a specific queue.
• Once you find the queue that is affected, the troubleshooting steps can vary particularly heavily. Here’s some examples by blocked queue type:
  
  • Parsing: Check for new data that may not have proper encoding, headers or linebreak rules. splunkd.log can help!
  • AggQueue: Check for bad date/time or bad extractions for date/time and any MUST_BREAK rules.
  • Typing: Regex, regex, regex….usually. New data with new extractions? Too many .*’s in that regex pattern? Regex Profile can help here too!
  • Index: Usually throughput or network/OS configuration at play here but could be any number of possibilities. splunkd.log is your best friend with this one!

Q4: What’s the difference between "search in" vs "search IN"?

• Short answer: they are the same thing but made distinct because of which command precedes them. This is referenced in the following sections of our documentation:
  • “in(<field>,<list>)” refers to its use (in lowercase) as part of the ‘where’ or ‘eval’ commands, for example.
  • “Multiple field-value comparisons with the IN operator” refers to its use (in uppercase) in the ‘search’ or ‘tstats’ command function as a comparison operator.
• Syntactically, SPL will alert you to the validity of your case choice based on the command within which it is contained and the attached modifiers and lists though, typically, these are the rules:
  • IN (“field1”,”field2”) is valid for search and tstats
  • in(field1,field2) is valid for where, eval and fieldstats

Q5: How do you configure Edge Processor servers and where? In cloud or in our HF(LAN)?

• https://docs.splunk.com/Documentation/SplunkCloud/latest/EdgeProcessor/CreateNode
• OR https://docs.splunk.com/Documentation/SplunkCloud/latest/EdgeProcessor/QuickStart
  
• The configuration of each node happens on your environment locally but can be managed all from the Cloud Platform!

Q6: How can I set up Edge Processor as a service?

• We’ve got you covered with instructions for configuring systemd upon installation here: https://docs.splunk.com/Documentation/SplunkCloud/9.0.2305/EdgeProcessor/CreateNode#Install_an_insta... 

Other Questions (check the #office-hours Slack channel for responses):

• Is Edge Processor available today and who can use it?
• How do I access Edge Processor?
• Why do I need a Forwarder?
• How can I get Windows internal logging (wineventlog) only starting from the time that I startup the Windows UF and NOT all historical, so I don’t exceed licensing?
• Installing a universal forwarder that doesn't have connectivity to a deployment server. What do I need to copy from a host that was deployed by the deployment server? This is Splunk Cloud
• Edge Processor overview
• Edge Processor demo videos

Hello, I would like to have access to the video in https://splunk-usergroups.slack.com/