Looks like you have two savedsearches.conf for a similar saved search on your ES search head. You cannot have multiple saved searches that carry the same name.
The configuration checker script that runs in the background will flag this as a warning message to the UI as you're observing.
The saved search "Endpoint - Outbreak Observed - Rule" is present in the below two savedsearches.conf which is causing a conflict here.
find . -name savedsearches.conf | xargs grep -i "Endpoint - Outbreak Observed - Rule"
./apps/SA-EndpointProtection/default/savedsearches.conf:[Endpoint - Outbreak Observed - Rule]
./apps/SA-EndpointProtection/local/savedsearches.conf:[Endpoint - Outbreak Observed - Rule]
./users/kquesada/SplunkEnterpriseSecuritySuite/local/savedsearches.conf:[Endpoint - Outbreak Observed - Rule]
It's doubtful a user had cloned this search and saved it with the same name as the Splunk UI will report the error:
"A saved search with that name already exists"
In this case, I'm assuming that the savedsearches.conf file for "Endpoint - Outbreak Observed - Rule" may have been manually copied to the user's folder in $SPLUNK_etc/users/~
To fix this, you'll need to remove either reference to this search or rename the one in the user directory.
You will observe the below logs when you run into this issue.
2019-03-18 17:22:38,988+0000 WARNING pid=36415 tid=MainThread file=configuration_check.py:run:165 | status="completed" task="confcheck_es_correlationmigration" message="Configuration file settings may be duplicated in multiple apps: stanza="Endpoint - Outbreak Observed - Rule" conf_type="savedsearches" apps="SplunkEnterpriseSecuritySuite,SA-EndpointProtection""
host = secsplunk-sc2-es1.vmware.com source = /opt/splunkcoreengine/ce_customers/0014000000KBwJnAAL/1319898/secsplunk-sc2-es1.vmware.com-sh_idx_uf_hf_lf_dplyr_cm_ds_ls_-20190319-073940/log/configuration_check.log sourcetype = configuration_check
2019-03-18 17:12:38,472+0000 WARNING pid=29372 tid=MainThread file=configuration_check.py:run:165 | status="completed" task="confcheck_es_correlationmigration" message="Configuration file settings may be duplicated in multiple apps: stanza="Endpoint - Outbreak Observed - Rule" conf_type="savedsearches" apps="SplunkEnterpriseSecuritySuite,SA-EndpointProtection""
Hope this helps!
... View more
I found a similar bug regarding this. Below are the details,
Bug ID: SOLNESS-17285
It appears the ES app is not handling multi-level inheritance for roles well for status transitions (meaning if we have roles A->B-C), transitions for out of the box statuses are only implicitly enabled for B, for C status transitions are not enabled. The problem is with the transitioners rest handler.
If we have the following role hierarchy test1->test2 (and test1 inherits ess_analyst, ess_user, and user) – we only show test1 in the imported roles for test2 (and not the full expanded list – test1, ess_analyst, ess_user, user).
Looks like this is not an issue with the upgrade but rather the issue with the ES version, where it cannot handle multi-level inheritance as expected.
The bug will be fixed in the next release 5.3 which is targeted to be released around March 2019 timeframe.
From the configurations, it looks like the affected user is importing other roles (ess_analyst; user) listed in the reviewstatues.conf. It appears that the role inheritance works one time but not the next.
The workaround for this is to assign the affected user all roles (default role) / ess_analyst / user vs just the one role that inherits the other two.
Hope this helps. Cheers!
... View more
Made the below changes on the UF and restarted. Post which the Security logs were read from the oldest logs and the UF took some time to catch up with the latest logs.
Under Program Files/SplunkUniversalforwarder/etc/system/local/inputs.conf, add
evt_ad_cache_exp = 1200
evt_ad_cache_exp_neg = 1200
evt_ad_cache_max_entries = 40000
evt_sid_cache_exp = 300
evt_sid_cache_exp_neg = 300
evt_sid_cache_max_entries = 4000
use_old_eventlog_api = 1
evt_dc_name = localhost
Under Program Files/SplunkUniversalforwarder/etc/system/local/outputs.conf, add
tcpSendBufSz = 512000
The above are the configurations suggested by the Splunk engineering team for the delay in indexing Windows security logs. The issue has been resolved after the changes.
... View more
You might have received this warning message while upgrading the Splunk forwarder, and this warning message indicates that your Splunk instance is using its own default Certificate Authority and suggesting you use either commercial-CA-signed or self-CA-signed certificates in order to establish SSL connections between your Splunk servers.
This does not have an impact on your environment and it is not necessary to use commercial-CA-signed or self-CA-signed certificates and you can continue using the default Splunk certificates as per your requirement.
This is default notification you will receive while upgrading UF to Splunk 7.x
"It seems that the Splunk default certificates are being used. If certificate validation is turned on using the default certificates (not-recommended), this may result in loss of communication in mixed-version Splunk environments after an upgrade."
This is a message added in Splunk UF 7.x to notify that the Splunk default certificates are being used. I have received the same message after the upgradation but the connection between the UF and indexer seems to be normal and working as expected.
I can still see that my UF is forwarding data to the indexer as expected and I believe there should not be any issues with the above warning message that you have received while upgrading your UF.
In case if you would like to use self-signed or third-party certificates instead of Splunk default certificates, please refer the below links to know more details regarding this.
1.) How to get certificates signed by a third-party
2.) How to prepare your signed certificates for Splunk authentication
3.) Configure Splunk forwarding to use your own certificates
... View more
Disable all other stanzas. Leave only the affected stanza enabled.
Run the input from the command line to see if it can read events.
$ splunk cmd splunkd print-modinput-config WinEventLog | splunk-WinEvtLog.exe
Remove the checkpoint file (make a copy of it first) and restart Splunk service.
Run the input again to see if it can read events.
If this is because of the checkpoint file, step 2 will not produce events. Step 4 should produce events.
On the UF, run command prompt as administrator
Navigate to $SPLUNK_HOME\bin
Run the below two commands,
$ set SPLUNK_HOME="c:\program files\SplunkUniversalForwarder"
$ splunk cmd splunkd print-modinput-config WinEventLog
You can consider upgrading the affected Splunk UF's as well.
... View more