Hi, Thank you for your aswer. I'm trying to prevent SC4S to send via HEC syslog-ng logs, metrics and any other traffic besides the actual logs becase we have a low ressources environment. In /opt/sc4s/local/config/destinations/block_me.conf: destination d_block_metrics { file("/dev/null"); }; And in /opt/sc4s/local/config/log_paths/block_me.conf: log { source(s_internal); source(s_system); #destination(d_hec_debug); destination(d_block_metrics); flags(final); }; I guess I'm doing something wrong because even with flags(final); all metrics and errors are still being sent to Splunk. I just need to restrict  ressources because used memory grows uncontrollably until it reaches the 256Mb allocated to the container. Thank's a lot Daniel

As far as I know, any index that receives results of a scheduled report is considered a summary index (i.e. using the collect command in a search or configuration of the "action.summary_index" parameter in savedsearches.conf. To look for savedsearches using either one of these methods you can search the rest endpoint like this.             | rest splunk_server=local /servicesNS/-/-/saved/searches | fields + title, qualifiedSearch, "action.summary_index", "action.summary_index.*" | where match(qualifiedSearch, "(?i)\|(?:\s|\n)*collect") OR ('action.summary_index'=="1" OR match('action.summary_index', "(?i)true")) | rename title as savedsearch_name | rex field=qualifiedSearch max_match=0 "(?<collect_spl>\|\s*collect\s+[^\n]+)" | fields + savedsearch_name, collect_spl, "action.summary_index", "action.summary_index.*"             From here you could set up regex to extract index/sourcetype from the "collect_spl" field or use the "action.summary_index.*" values to gather that info. Its possible for the "collect_spl" field to contain only index and even then, that index specification could be stored in a macro, so those situations may be a bit more tricky. It is also possible for a parameter called "output_format=hec" to be used along with the collect command and if this is the case then, sourcetype and source will not be specified with the collect command and are rather defined in the SPL itself. You can see examples of these scenarios here To use this method to the end result of a report listing index/sourcetypes that are being utilized as a summary index you can use SPL like this. (Note: there is a custom splunk command being used in this code that expands macros all the way down before we attempt to do any extractions of collect metadata. You can DM me if you would want me to share the script to do this) | rest splunk_server=local /servicesNS/-/-/saved/searches | fields + title, qualifiedSearch, "action.summary_index", "action.summary_index.*" | where match(qualifiedSearch, "(?i)\|(?:\s|\n)*collect") OR ('action.summary_index'=="1" OR match('action.summary_index', "(?i)true")) | rename title as savedsearch_name ``` this is a splunk custom command I created, reach out to me through DM and I can share the code ``` | expandmacros input_field=qualifiedSearch output_field=expanded_spl | rex field=expanded_spl max_match=0 "(?<collect_spl>\|\s*collect\s+[^\n]+)" | where isnotnull(collect_spl) OR ('action.summary_index'=="1" OR match('action.summary_index', "(?i)true")) | fields + savedsearch_name, collect_spl, expanded_spl, "action.summary_index", "action.summary_index.*" | rex field=expanded_spl max_match=0 "(?i)\|\s*(?<eval_spl>eval\s+[^\|]+)" | eval eval_spl=mvfilter(match(eval_spl, "\s+source(?:type)?\"?\s*\=\s*\"")) | rex field=eval_spl max_match=0 "\s+sourcetype\"?\s*\=\s*\"(?<inline_set_sourcetype>[^\"]+)" | rex field=eval_spl max_match=0 "\s+source\"?\s*\=\s*\"(?<inline_set_source>[^\"]+)" | rex field=collect_spl max_match=0 "index\s*\=\s*\"?(?<summary_index>[a-zA-Z0-9\-\_]+)" | rex field=collect_spl max_match=0 "sourcetype\s*\=\s*\"?(?<summary_sourcetype>[a-zA-Z0-9\-\_]+)" | rex field=collect_spl max_match=0 "source\s*\=\s*\"?(?<summary_source>[a-zA-Z0-9\-\_]+)" | fields + savedsearch_name, collect_spl, summary_index, summary_sourcetype, summary_source, inline_set_sourcetype, inline_set_source, "action.summary_index", "action.summary_index.*" | eval summary_index=mvdedup( mvappend( 'summary_index', 'action.summary_index._name' ) ), summary_sourcetype=mvdedup( mvappend( summary_sourcetype, inline_set_sourcetype ) ), summary_source=mvdedup( mvappend( summary_source, inline_set_source ) ) | fillnull value="stash" summary_sourcetype | fields - inline_* | stats dc(savedsearch_name) as dc_savedsearches by summary_index, summary_sourcetype | sort 0 -dc_savedsearches  Final output would look something like this. (screenshot has been redacted)  

@gcusello , appreciate the response.   Yes, this is a DMC that has been operational for ~ 3years, and suddenly started doing this.  All sources are connected without error.  I even removed a few and added back to see what happened.  No immediate change.  The interface works..... some of the time, about 20% but cannot figure out why it suddenly works, and without any change, refresh the screen and it is broken AGAIN.   I did update as soon as 9.1.2 appeared hoping that would fix it (updated the entire architecture).  No luck, still broken About to the point what I am simply going to delete it, and start over

Hi @tlmayes, at first: did youconfigured yur DMC to take logs from the other servers? To do this you have to connect the DMC to all the server to monitor on the 8089 port as a Search Head in Distributed Search. Then, have you the message "N.A." or what else? if you have "N.A.", are you using the last Splunk version (9.1.2)? because the previous version had a bug. Ciao. Giuseppe

I have a couple of questions: - Are you trying to get rid of the the metrics data from Splunk's metrics.log? - Can you post the props and transforms config that you tried? What you're trying to do may not "fix" the memory utilization on your container.

Getting a bit more complex but I think something like this will get you started. | rex field=_raw "\]\s+(?<log_level>[A-Z]+)\s+(?<log_type>[^\s]+)\s+\-" | rex field=_raw "\-\s+\([^\)]+\)\s+(?<message_detail>(?:.*(?:[\r\n]+)*)*)" | rex field=_raw "user\s+\'(?<user>[^\']+)\'" | rex field=_raw "client\s+(?<client>\d{1,3}(?:\.\d{1,3}){3})\:(?<port>\d+)" | rex field=_raw "\[TID\:(?<tid>[^\]]+)\]" | rex field=_raw "\]\s+[A-Z]+\s+[a-zA-Z0-9]+(?:\.[a-zA-Z0-9]+)*\s+\-\s+(?<message>.*)" | rex field=_raw "(?i)cannot\s+open\s+(?<service>[^\s]+)\s+service\s+on\s+computer\s+\'(?<computer>[^\']+)\'" ``` log_type="Thycotic.Discovery.Sources.Scanners.Dependency.ApplicationPoolScanner" ``` | rex field=_raw "(?i)\s+\-\s+(?<app>.*?)\s+unable\s+to\s+connect\s+to\s+(?<hostname>[^\s]+)\s+with\s+exception\s+(?<exception_type>[^\:]+)\:\s+(?<exception_message>.*)" ``` log_type="Thycotic.Discovery.Sources.Scanners.Dependency.ApplicationPoolScanner" error_message ``` | rex field=_raw "(?i)\s+\-\s+retrieving\s+the\s+(?<class>[^\s]+)\s+class\s+factory\s+for\s+remote\s+component\s+with\s+clsid\s+\{(?<clsid>[^\}]+)\}\s+from\s+machine\s+(?<hostname>[^\s]+)\s+failed\s+due\s+to\s+the\s+following\s+error\:\s+(?<error_code>[^\s]+)" | rex field=_raw "(?i)(?<exception_type>\w+(\.\w+)*exception)\:\s+(?<exception_message>.*)"   Since the evals to standardize messages probably isn't all that scalable with these logs, it may make sense to set up eventtyping around them to assist with any sort of analysis. Once you have eventtypes set up that can assist with targeting specific type of events and finding patterns that you can develop regex for and pull out all the necessary fields. screenshot below for reference of what the above regex should do.  

Hello, I am having the same issue, and I understood that it could be due to a license violation. I just found out that even when a license peer can't connect to license manager for 72 hours it will cause warnings (I m not sure yet if for a single error message = a warning or not!!) check your internal logs (if you still have them) for this: index=_internal component=LMTracker ("failed to send rows" OR "unable to connect")   then i understood that the warning/violation will be reset after 30days (this period could be different too!), so I am assuming that after 30days of the violation, the issue will disappear!  The other possibility is to change the file: /opt/splunk/share/splunk/search_mrsparkle/templates/licensing/overview.html and comment the section where it checks for the pool quota violation (I haven't tried that yet).  

Thanks @dtburrows3 , I have another use case is discovery process from AD to Secret server like scan AD and finds the local id creates the id/password into secret. 1. Sample event: 2024-01-02 11:18:28,597 [CID:] [C:] [TID:131] ERROR Thycotic.Discovery.Sources.Scanners.Dependency.ApplicationPoolScanner - WMI (IIS) Unable to connect to WINDOWS1.mf.win.cisco.com with Exception System.Management.ManagementException: Invalid namespace at System.Management.ThreadDispatch.Start() at System.Management.ManagementScope.Initialize() at Thycotic.Discovery.Sources.Scanners.Dependency.ApplicationPoolScanner.<>c__DisplayClass10_0.<IsIisRunningWmi>b__0(Object x) - (null) 2. Sample event: 2024-01-02 11:18:14,330 [CID:] [C:] [TID:PriorityScheduler Elastic Thread @ Normal] ERROR Thycotic.Discovery.Sources.Scanners.Dependency.ApplicationPoolScanner - Service Controller (IIS) Unable to connect to LINUX.mf.win.cisco.com with Exception System.InvalidOperationException: Cannot open W3SVC service on computer 'LINUX.mf.win.cisco.com'. ---> System.ComponentModel.Win32Exception: Access is denied --- End of inner exception stack trace --- at System.ServiceProcess.ServiceController.GetServiceHandle(Int32 desiredAccess) at System.ServiceProcess.ServiceController.GenerateStatus() at System.ServiceProcess.ServiceController.get_Status() at Thycotic.Discovery.Sources.Scanners.Dependency.ApplicationPoolScanner.IsIisRunningServiceController() - (null) 3. Sample Event : 2024-01-02 11:18:06,544 [CID:] [C:] [TID:PriorityScheduler Elastic Thread @ Normal] ERROR Thycotic.Discovery.Sources.Scanners.Dependency.ApplicationPoolScanner - Retrieving the COM class factory for remote component with CLSID {45RFGHDH-FFSG-6474-HDHDH-1454TGGD} from machine UBUNTU.mf.win.cisco.com failed due to the following error: 80070005 UBUNTU.mf.win.cisco.com. - (null) System.UnauthorizedAccessException: Retrieving the COM class factory for remote component with CLSID {2YHFDO-3443-4FGDH-5353-RFRRE6} from machine UBUNTU.mf.win.cisco.com failed due to the following error: 80070005 UBUNTU.mf.win.cisco.com. at Thycotic.Discovery.Sources.Scanners.Dependency.ApplicationPoolScanner.ScanIIS7AndAbove() 4. Sample event : 2024-01-02 11:18:03,664 [CID:] [C:] [TID:PriorityScheduler Elastic Thread @ Normal] ERROR Thycotic.Discovery.Sources.Scanners.Dependency.ApplicationPoolScanner - Service Controller (IIS) Unable to connect to IHUIHD13.mf.win.cisco.com with Exception System.InvalidOperationException: Cannot open W3SVC service on computer 'YHGVDJ.mf.win.cisco.com'. ---> System.ComponentModel.Win32Exception: Access is denied --- End of inner exception stack trace --- at System.ServiceProcess.ServiceController.GetServiceHandle(Int32 desiredAccess) at System.ServiceProcess.ServiceController.GenerateStatus() at System.ServiceProcess.ServiceController.get_Status() at Thycotic.Discovery.Sources.Scanners.Dependency.ApplicationPoolScanner.IsIisRunningServiceController() - (null) Thanks.

Something like this maybe?   <base_search> | rex field=_raw "\]\s+(?<log_level>[A-Z]+)\s+(?<log_type>[^\s]+)\s+\-\s+(?<message>.*?)\s+\-" | rex field=_raw "\-\s+\([^\)]+\)\s+(?<message_detail>(?:.*(?:[\r\n]+)*)*)" | rex field=_raw "user\s+\'(?<user>[^\']+)\'" | rex field=_raw "client\s+(?<client>\d{1,3}(?:\.\d{1,3}){3})\:(?<port>\d+)" | rex field=_raw "\[TID\:(?<tid>[^\]]+)\]"   Not familiar with the data but based on the 2 sample logs you provided was able to identify a few fields that I think would be useful. You can see from the screenshot below that these fields appear to be successfully extracted. And to standardize the messages (at least for these examples) you can add an eval like this after the field extractions. | eval standardized_message=replace(replace(message, "client\s+(\d{1,3}(?:\.\d{1,3}){3})\:(\d+)", "client"), "user\s+\'([^\']+)\'", "user")  You can see in the screenshot that the field 'standardized_message' removes any variable data so you can do some sort of aggregate analysis against the standardized message itself if needed.