Here are the error messages
2022-09-26 12:38:02,976 ERROR [itsi_re(reId=cRdG)] [main] RulesEngineSearch:75 - RulesEngineTask=RealTimeSearch, Status=Stopped, FunctionMessage="java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonParser.getReadCapabilities()Lcom/fasterxml/jackson/core/util/JacksonFeatureSet;"
host = myhost = _internalsource = /opt/splunk/var/log/splunk/itsi_rules_engine.log sourcetype = itsi_internal_log
2022-09-26 12:38:02,976 ERROR [itsi_re(reId=cRdG)] [main] RulesEngineSearch:74 - RulesEngineTask=RulesEngineJob, Status=Stopped
host = myhost = _internalsource = /opt/splunk/var/log/splunk/itsi_rules_engine.log sourcetype = itsi_internal_log
2022-09-26 12:38:02,902 DEBUG [itsi_re(reId=cRdG)] [main] PropertyLoader:209 - itsiRulesEngine.localConfigurationFile properties file is not defined.
host = myhost = _internalsource = /opt/splunk/var/log/splunk/itsi_rules_engine.log sourcetype = itsi_internal_log
All the SH are on the same lan/network, no firewall.
The ERROR [itsi_re(reId=yVNs)] [main] RulesEngineSearch:75 - RulesEngineTask=RealTimeSearch, Status=Stopped, FunctionMessage="java.lang.NoSuchMethodError: 'com.fasterxml.jackson.core.util.JacksonFeatureSet com.fasterxml.jackson.core.JsonParser.getReadCapabilities()'" is logged every minute.
This is a know issue documented here
https://docs.splunk.com/Documentation/ITSI/4.13.1/ReleaseNotes/Knownissues#Notable_Events
Bug ITSI-24488
Rules engine search fails to start after upgrade to ITSI 4.13.0
Workaround:
Move the jackson-core-2.10.0.jar and jackson-annotations-2.10.0.jar to the .bkup folder under $SPLUNK_HOME/etc/apps/SA-ITOA/lib/java/event_management/libs directory.
This is a know issue documented here
https://docs.splunk.com/Documentation/ITSI/4.13.1/ReleaseNotes/Knownissues#Notable_Events
Bug ITSI-24488
Rules engine search fails to start after upgrade to ITSI 4.13.0
Workaround:
Move the jackson-core-2.10.0.jar and jackson-annotations-2.10.0.jar to the .bkup folder under $SPLUNK_HOME/etc/apps/SA-ITOA/lib/java/event_management/libs directory.