Hello everyone. We are experiencing download and a few upload failures from our Indexers to SmartStore in AWS S3. Graph for the last 24 hours. I previously increased the Cache Manager limits from the default of 8 to 128 with a custom server.conf: [cachemanager] max_concurrent_downloads = 128 max_concurrent_uploads = 128 An example of an upload faliure from the splunkd.log file (sourcetype=splunkd source="/opt/splunk/var/log/splunk/splunkd.log" component=CacheManager log_level=ERROR): 08-12-2022 03:37:28.565 +0000 ERROR CacheManager [950069 cachemanagerUploadExecutorWorker-0] - action=upload, cache_id="dma|<INDEX>~925~054DE1B7-4619-4FBC-B159-D4013D4C30AE|C1AB9688-CBC2-428C-99F5-027FA469269D_DM_Splunk_SA_CIM_Network_Sessions", status=failed, reason="Unknown", elapsed_ms=12050 08-12-2022 03:37:28.484 +0000 ERROR CacheManager [950069 cachemanagerUploadExecutorWorker-0] - action=upload, cache_id="dma|<INDEX>~925~054DE1B7-4619-4FBC-B159-D4013D4C30AE|C1AB9688-CBC2-428C-99F5-027FA469269D_DM_Splunk_SA_CIM_Network_Sessions", status=failed, unable to check if receipt exists at path=<INDEX>/dma/de/07/925~054DE1B7-4619-4FBC-B159-D4013D4C30AE/C1AB9688-CBC2-428C-99F5-027FA469269D_DM_Splunk_SA_CIM_Network_Sessions/receipt.json(0,-1,), error="network error" An example of a download failure: 08-12-2022 09:06:44.488 +0000 ERROR CacheManager [1951184 cachemanagerDownloadExecutorWorker-113] - action=download, cache_id="dma|<INDEX>~204~431C8F6B-2313-4365-942D-09051BE286B8|C1AB9688-CBC2-428C-99F5-027FA469269D_DM_Splunk_SA_CIM_Performance", status=failed, reason="Unknown", elapsed_ms=483 We previously had an issue with NACLs in AWS where the S3 IP ranges had been updated but the NACLs were out of date. We have allowed access to all S3 IP ranges in our region. Does anyone have an idea of how I can troubleshoot this so we can reduce, or eliminate the failures? Anyone else had any experience with this?     ... View more

Hello everyone. I'm looking for some assistance with a problem where I get differing search results from what should be the same search. Backstory I’m testing changes to the “ESCU - Malicious PowerShell Process - Execution Policy Bypass – Rule” so that I can filter out known PowerShell events. Using the same search head, user,  date and time range, and what should be two identical macros, I get different search results.   The original search uses this macro: “malicious_powershell_process___execution_policy_bypass_filter” The original search is: | tstats `security_content_summariesonly` values(Processes.process_id) as process_id, values(Processes.parent_process_id) as parent_process_id values(Processes.process) as process min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes where Processes.process_name=powershell.exe (Processes.process="* -ex*" OR Processes.process="* bypass *") by Processes.process_id, Processes.user, Processes.dest | `drop_dm_object_name(Processes)` | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)` | `malicious_powershell_process___execution_policy_bypass_filter`   The test search uses this macro: “malicious_powershell_process___execution_policy_bypass_filter-test” The test search is: | tstats `security_content_summariesonly` values(Processes.process_id) as process_id, values(Processes.parent_process_id) as parent_process_id values(Processes.process) as process min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes where Processes.process_name=powershell.exe (Processes.process="* -ex*" OR Processes.process="* bypass *") by Processes.process_id, Processes.user, Processes.dest | `drop_dm_object_name(Processes)` | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)` | `malicious_powershell_process___execution_policy_bypass_filter-test` Both macros contain the same content to exclude Splunk Universal Forwarder PowerShell scripts: search (process!="C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe -executionPolicy RemoteSigned -command \". 'C:\\Program Files\\SplunkUniversalForwarder\\etc\\apps\\Splunk_TA_windows\\bin\\powershell\\nt6-health.ps1'\"" AND process!="C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe -executionPolicy RemoteSigned -command \". 'c:\\Program Files\\SplunkUniversalForwarder\\etc\\apps\\Splunk_TA_windows\\bin\\powershell\\nt6-repl-stat.ps1'\"" AND process!="C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe -executionPolicy RemoteSigned -command \". 'c:\\Program Files\\SplunkUniversalForwarder\\etc\\apps\\Splunk_TA_windows\\bin\\powershell\\nt6-siteinfo.ps1'\"" AND process!="C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe -executionPolicy RemoteSigned -command \". 'C:\\Program Files\\SplunkUniversalForwarder\\etc\\apps\\Splunk_TA_windows\\bin\\powershell\\dns-zoneinfo.ps1'\"" AND process!="C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe -executionPolicy RemoteSigned -command \". 'C:\\Program Files\\SplunkUniversalForwarder\\etc\\apps\\Splunk_TA_windows\\bin\\powershell\\dns-health.ps1'\"")     When I run both searches I get different results and I’m unsure why. The macro appended -test works fine. When I copy its contents to the original macro, that search does not seem to use the new contents. I made these changes last week and today get the same results. Any ideas as to what might be causing this? ... View more

Hello! Can anyone please lend a hand with this issue? I'm still fairly new to this and am working my way through Fundamentals 2. Scenario: Sophos Central antivirus clients installed on Linux and Windows. Using a Heavy Forwarder to pull Sophos Central logs via an API into a dedicated antivirus index. These logs lack the product_version needed to populate the "Malware Operations - Clients By Product Version" dashboard panel. I've found the data I need in two places. A log entry in /var/log/messages which is being ingested to the unix index and looks like this: Jul 13 03:59:37 server-name savd[5860]: update.updated: Updated to versions - SAV: 10.5.2, Engine: 3.79.0, Data: 5.85 And in a file /opt/sophos-av/engine/suiteVersion which is readable by the splunk user and contains: 10.5.2.3790.203   I used the field extractor to create a regex string that matches the log entry and extracts the product_version. I've created a custom app with these files in it, pushed from the Deployment Server onto one host, and pushed from the Deployer to our Enterprise Security Search Head Cluster: props.conf [syslog] EXTRACT-date,time,host,process_name,product_version = ^(?P<date>\w+\s+\d+)\s+(?P<time>[^ ]+)\s+(?P<host>[^ ]+)\s+(?P<process_name>\w+)(?:[^ \n]* ){6}(?P<product_version>\SAV:\s\d*\.\d*\.\d*,\sEngine:\s\d*\.\d*\.\d*,\sData:\s\d*.\d*.+) eventtypes.conf [product_version] search=product_version=* tags.conf [eventtype=product_version] malware = enabled operations = enabled When I search the unix index I can see the product_version field in the search results and the tags. Questions: 1.) How do I copy these events into the antivirus index and is this necessary? (I'm thinking of efficiency as the cim_Malware_indexes macro contains the antivirus and firewall indexes which are much smaller than the unix index). 2.) How do I get the product_version to show in the "Clients By Product Version" panel which uses this search? | `malware_operations_tracker(time_product_version)` | search | stats dc(dest) by product_version | sort 10 - dc(dest) 3.) Is there a better way to do this? Any help is appreciated. ... View more