Hello FelixL, I have the same problem as you. Did you find out why it happened and how to fix it? If you change the locale in the url, it will sometimes start working. For example, en-US to en-GB ... View more

I found out that the message about duplicated configuration is from ES (Enterprise Security). This check has a period of 10 minutes. apps/SplunkEnterpriseSecuritySuite/bin/configuration_checks/confcheck_es_correlationmigration.py:MSG_DUPLICATED_STANZA = 'Configuration file settings can be duplicated in multiple applications: stanza="%s" conf_type="%s" apps="%s"' I tested the above scenario on a clean Splunk Enterprise without ES and the behavior matches the documentation, but btool not. [splunk@siemsearch01 apps]$ cat ATest_app/default/savedsearches.conf [ss] search = `super_macro` atest [splunk@siemsearch01 apps]$ cat Test_app/default/savedsearches.conf [ss] search = `super_macro` test request.ui_dispatch_app = search disabled = 0 alert.track = 0 cron_schedule = */2 * * * * dispatch.earliest_time = -4m dispatch.latest_time = -2m enableSched = 1 [splunk@siemsearch01 apps]$ cat ZTest_app/default/savedsearches.conf [ss] search = `super_macro` ztest [splunk@siemsearch01 apps]$ [splunk@siemsearch01 apps]$ /opt/splunk/bin/splunk btool savedsearches list --debug ss | grep "search =" /opt/splunk/etc/apps/ATest_app/default/savedsearches.conf search = `super_macro` atest btool returns:  search = `super_macro` atest Gui returns:  index=ztest ztest, i.e. `super_macro` ztest     ... View more

I have my reasons. I don't want to impose changes on the local. I need to use the original addon and add my correctly named addon to it, which would override the search= parameter in original one. Orig add-on is Splunk_TA_openldap default/savedsearches.conf [Update openldap_user_lookup KV Store collection] request.ui_dispatch_app = search disabled = 0 alert.track = 0 cron_schedule = */2 * * * * dispatch.earliest_time = -4m dispatch.latest_time = -2m enableSched = 1 search = sourcetype="openldap:access" operation="BIND" | dedup conn cn | table conn op cn | rename cn as user | lookup openldap_user_lookup conn, op OUTPUTNEW _key AS _key | outputlookup append=t openldap_user_lookup My append is A10_aaa_ta_openldap default/savedsearches.conf [Update openldap_user_lookup KV Store collection] search = `openldap_index` sourcetype="openldap:access" operation="BIND" | dedup conn cn | table conn op cn | rename cn as user | lookup openldap_user_lookup conn, op OUTPUTNEW _key AS _key | outputlookup append=t openldap_user_lookup I know btool and I am using it.  There are more problems. One is that according to btool, the savedsearch.conf precedence does not behave as documented, i.e. app/user context with reverse reverse-lexicographic order. The second is that Splunk reports a problem with duplicate configuration. So far I haven't found any information in the documentation that savedsearches.conf should behave differently than for example macros, props etc. ... View more

I did not find anything weird about the interface stats. Similar problem occurs in all Linux nodes, but differs in period/delay. There is btool output configuration      ... View more

I tried setting parallelIngestionPipelines = 2 in server.conf and the behavior did not change.  I also tried stopping sysmon deamon and disabling sysmon journald input. It had no effect on the above behavior. ... View more

I got a direct access to the sever again and I checked OS version. It is Red Hat Enterprise Linux release 9.4 (Plow). I will try to add pipeline and I will check if it helps. I am going to check if there is not something connected with sysmon.  It was right. There were only few log entries in audit.log during the period. I checked it on filesystem. After my ssh connection there is more log entrie.  Last 90 minuts /opt/splunkforwarder/var/log/splunk/audit.log 2 /opt/splunkforwarder/var/log/splunk/conf.log 1 /opt/splunkforwarder/var/log/splunk/configuration_change.log 3 /opt/splunkforwarder/var/log/splunk/health.log 26 /opt/splunkforwarder/var/log/splunk/metrics.log 8975 /opt/splunkforwarder/var/log/splunk/splunkd-utility.log 10 /opt/splunkforwarder/var/log/splunk/splunkd.log 1055 /opt/splunkforwarder/var/log/watchdog/watchdog.log 3 /var/log/audit/audit.log 1337 /var/log/messages 9418 /var/log/secure 543 journald://sysmon 6482   I revealed an interesting correlation. You can see a "gap" or change in behavior in the graph. It starts after the UF is restarted. There are messages "Found currently active indexer. Connected to idx=X.X.X.X:9992:0, reuse=1." before UF restart. After 20 minutes from restart they are back. ... View more

I am collecting logs from some files from /var/log and sysmon from journald. last 90 minutes /opt/splunkforwarder/var/log/splunk/audit.log 41 /opt/splunkforwarder/var/log/splunk/health.log 39 /opt/splunkforwarder/var/log/splunk/metrics.log 8911 /opt/splunkforwarder/var/log/splunk/splunkd.log 598 /var/log/audit/audit.log 7 /var/log/messages 936 /var/log/secure 10 journald://sysmon 919   inputs.conf [monitor:///var/log/syslog] disabled = 0 sourcetype = syslog index = linux [monitor:///var/log/messages] disabled = 0 sourcetype = syslog index = linux [monitor:///var/log/secure] disabled = 0 sourcetype = linux_secure index = linux [monitor:///var/log/auth.log] disabled = 0 sourcetype = linux_secure index = linux [monitor:///var/log/audit/audit.log] disabled = 0 sourcetype = linux_audit index = linux [journald://sysmon] interval = 5 journalctl-quiet = true journalctl-include-fields = PRIORITY,_SYSTEMD_UNIT,_SYSTEMD_CGROUP,_TRANSPORT,_PID,_UID,_MACHINE_ID,_GID,_COMM,_EXE journalctl-exclude-fields = __MONOTONIC_TIMESTAMP,__SOURCE_REALTIME_TIMESTAMP journalctl-filter = _SYSTEMD_UNIT=sysmon.service sourcetype = sysmon:linux index = linux   I did not change number of pipelines. I thing that default count is 1. I will find out the OS version later. I do not have direct access to the OS. I thing it is CentOS/Redhat 8 or 9, but I may be wrong.   ... View more

I looked at the events for the component you mentioned and found that there is only one type of log entry. I also tried it for the "last 7 days" time range.   ... View more

I have two weeks off, so I'll continue troubleshooting after that. In my opinion there are not any interesting stuff in _internal log. You can see it on the screenshot. I used cluster command to reduce log number. There is component != metric in SPL.     ... View more

I know that these errors are unrelated. I tried to show that internal log are not full of "error" messages. Situation is Thruput is not limited (thruput se to 10240) Number of logs is low logs in files are generated fluently, i checked by "tail -f" during aprox 20 minutes after UF restart there is no problem after this time  problem  appears the problem is Data are buffered somewhere in front of indexer server, it is buffered aprox 9 minutes. After I restarted UF or droped TCP session, data were suddenly sent to the indexer. I belive It must be buffered on UF side. I saw no dat period and then data burst on indexer site. Shape of the grahp is saying the same thing. Data are somehere for some period of time and then are flushed to indexer. Older data are bigger diff a newer data are lower diff. Index time   SendQ TCPout   Queues internal messages (clustered) ... View more

UF host for last 60 minutes with now errors and warnings   IDX side    Still a problem here. This morning we had to reboot from the Splunk servers due to a security patch of the operating system. You can see it at the beginning of the graph. This meant that the connection between UF and IDX had to be re-established, i.e. when IDX or UF restarts, about 20 minutes yesterday and today 10 minutes is not the delay or batch processing. ... View more

I tried the change and It does not help. You can se on first screenshot. You were right that the trigger could be the colon. If I remove the ":" from field and calculate the stat then all mvfield displayed the same way. It doesn't completely solve my problem.  ... View more

Thank you for your reply. I think it is not for first time when I have seen it. There must be some small bug. I know that field is still mvfield and it behaves like that. ... View more

I tested it on artificial data and I used a field "msg" in rex command. I thing you have data in the field "_raw".    You should use | rex field=_raw "\w+:\s+\S+\s+(\/[^\/]+)*\/(?<filename>[^\s\/]+)" or | rex "\w+:\s+\S+\s+(\/[^\/]+)*\/(?<filename>[^\s\/]+)" note: _raw field is default field for rex command     ... View more

Or something like this | makeresults | eval msg="MDTM|07/02/2023 23:58:59.007|[SFTP:3460819_0:eftpos:10.18.168.158] READ: *MDTM /eftpos/prod/AR-100-01_20230702_PAY.zip 16883063270" | rex field=msg "\w+:\s+\S+\s+(\/[^\/]+)*\/(?<filename>[^\s\/]+)" ... View more