Looks like you have two savedsearches.conf for a similar saved search on your ES search head. You cannot have multiple saved searches that carry the same name. The configuration checker script that runs in the background will flag this as a warning message to the UI as you're observing. The saved search "Endpoint - Outbreak Observed - Rule" is present in the below two savedsearches.conf which is causing a conflict here. find . -name savedsearches.conf | xargs grep -i "Endpoint - Outbreak Observed - Rule" ./apps/SA-EndpointProtection/default/savedsearches.conf:[Endpoint - Outbreak Observed - Rule] ./apps/SA-EndpointProtection/local/savedsearches.conf:[Endpoint - Outbreak Observed - Rule] ./users/kquesada/SplunkEnterpriseSecuritySuite/local/savedsearches.conf:[Endpoint - Outbreak Observed - Rule] It's doubtful a user had cloned this search and saved it with the same name as the Splunk UI will report the error: "A saved search with that name already exists" In this case, I'm assuming that the savedsearches.conf file for "Endpoint - Outbreak Observed - Rule" may have been manually copied to the user's folder in $SPLUNK_etc/users/~ To fix this, you'll need to remove either reference to this search or rename the one in the user directory. You will observe the below logs when you run into this issue. REF: 2019-03-18 17:22:38,988+0000 WARNING pid=36415 tid=MainThread file=configuration_check.py:run:165 | status="completed" task="confcheck_es_correlationmigration" message="Configuration file settings may be duplicated in multiple apps: stanza="Endpoint - Outbreak Observed - Rule" conf_type="savedsearches" apps="SplunkEnterpriseSecuritySuite,SA-EndpointProtection"" host = secsplunk-sc2-es1.vmware.com source = /opt/splunkcoreengine/ce_customers/0014000000KBwJnAAL/1319898/secsplunk-sc2-es1.vmware.com-sh_idx_uf_hf_lf_dplyr_cm_ds_ls_-20190319-073940/log/configuration_check.log sourcetype = configuration_check 2019-03-18 17:12:38,472+0000 WARNING pid=29372 tid=MainThread file=configuration_check.py:run:165 | status="completed" task="confcheck_es_correlationmigration" message="Configuration file settings may be duplicated in multiple apps: stanza="Endpoint - Outbreak Observed - Rule" conf_type="savedsearches" apps="SplunkEnterpriseSecuritySuite,SA-EndpointProtection"" Hope this helps! ... View more

Splunk-wmi.path script contains the below configuration. $SPLUNK_HOME\bin\splunk-wmi.exe As for the script location goes, ideally, it should be placed under "$SPLUNK_HOME\bin\scripts" by default. This is a default Splunk configuration and placing this in a different location might not work always. Hope the above helps! ... View more

I see that this is a known issue which was reported for Splunk 6.6.x version and is fixed in Splunk 7.1.0. Below is the reference number for the same. SPL-140507 The above reference number wasn't publicly documented. It appears that is was only reported as an internal defect. I could recreate the 404 error in my test environment and I then looked to compare permissions and capabilities my instances but there wasn't really anything of note. But it might not be the case always. Looks like when the Splunk GUI on the search head is hitting a forbidden page this known issue is being hit on the search head. Hope the above helps! ... View more

Candidate bundles are created every replication_period_sec by: 1.Iterating through $SPLUNK_HOME/etc 2.Checking each item against the blacklist 3.Checking each item against the whitelist 4.No match? Drop it. Candidate bundle is compared to the existing bundle and distributed if necessary, MD5 of candidate and previous bundle is compared Hope the above helps! Cheers. ... View more

Hey Siva, You can run the btool command to identify the locations of the SSL certificates (for both splunkweb & encryption between forwarders & indexers) and replace them with the new ones. $ splunk btool web list --debug $ splunk btool server list --debug You need to find the path of the certificates from the above command's output and replace them with the new certificates. You need to generate new Selft-signed certificates by following the below splunk doc. For SplunkWeb: https://docs.splunk.com/Documentation/Splunk/6.5.2/Security/Self-signcertificatesforSplunkWeb For Data encryption: https://docs.splunk.com/Documentation/Splunk/6.5.2/Security/ConfigureSplunkforwardingtousesignedcertificates Hope the above helps! Cheers. ... View more

The search query of alert "[DMC Alert - Search Peer Not Responding]" hitting the rest point on "splunk_server=local" where "local" refers to the search head. Below is the base search, | rest splunk_server=local /services/search/distributed/peers/ | where status!="Up" | fields peerName, status | rename peerName as Instance, status as Status Looks like the DMC is timing out when search peer is not available to reach and throws the inconsistent results to the search query. Below is the workaround to fix this: On the DMC please add the below parameters under $SPLUNK_HOME/etc/system/local/distsearch.conf, [distributedSearch] statusTimeout = 120 connectionTimeout = 120 serverTimeout = 120 sendTimeout = 120 receiveTimeout= 120 authTokenConnectionTimeout = 120 authTokenSendTimeout = 120 authTokenReceiveTimeout = 120 Restart Splunk service on DMC for the changes to take effect. ==== On indexers, please add the below parameters under $SPLUNK_HOME/etc/system/local/distsearch.conf, [replicationSettings] connectionTimeout = 120 sendRcvTimeout = 120 Restart Splunk service on indexers for the changes to take effect. Hope it helps! Cheers. ... View more

Disable all other stanzas. Leave only the affected stanza enabled. Run the input from the command line to see if it can read events. $ splunk cmd splunkd print-modinput-config WinEventLog | splunk-WinEvtLog.exe Remove the checkpoint file (make a copy of it first) and restart Splunk service. Run the input again to see if it can read events. If this is because of the checkpoint file, step 2 will not produce events. Step 4 should produce events. On the UF, run command prompt as administrator Navigate to $SPLUNK_HOME\bin Run the below two commands, $ set SPLUNK_HOME="c:\program files\SplunkUniversalForwarder" $ splunk cmd splunkd print-modinput-config WinEventLog You can consider upgrading the affected Splunk UF's as well. Cheers! ... View more

Thanks, mate. It was a typo. ... View more

Hi, If the search job hasn't been touched by any requests in x seconds we'll cancel the job. One common use case is when users fire off a job in the search bar ... then modify the search and hit enter before the old search finishes. Nobody is going to poll the status of the old job and nobody cares about it .. so we auto-cancel it. On the jobs endpoint, there's an auto_cancel option you can use to control this timeout. Setting to 0 should disable it. You can check the settings for the job in a file called runtime.csv in the dispatch directory. If you are facing this issue with any of the saved searches, then configure the saved search for archiving with "dispatch.auto_cancel = 0" successfully run the job without auto canceled. Refer the below Splunk doc for the pause and cancel options. (runtime.csv) is the filename. https://docs.splunk.com/Documentation/Splunk/7.1.3/Search/Dispatchdirectoryandsearchartifacts#File_descriptions_for_search-specific_directories Check "Phased execution settings" in the below limits.conf to know more details about the execution process. http://docs.splunk.com/Documentation/Splunk/7.1.2/Admin/Limitsconf Workaround for saved searches: dispatch.auto_cancel = * If specified, the job automatically cancels after this many seconds of inactivity. (0 means never auto-cancel) * Default is 0. http://docs.splunk.com/Documentation/Splunk/7.1.2/Admin/Savedsearchesconf#dispatch_search_options Hope this helps!! Cheers. ... View more

This error means that the search artifact (the file package containing the search results) requested by Splunk Web could not be found in $SPLUNK_HOME/var/run/splunk/dispatch for the search that was just dispatched. This problem commonly happens when the $SPLUNK_HOME/var/run/dispatch directory is hosted on a network device with a time setting behind the system clock of the operating system where splunkd is running. What happens then is that the search artifacts are created in the dispatch directory with a modification time behind the system time known to splunkd. Verify the time difference between your indexers/search head and check your system clocks and make sure they are all in sync. (Using NTP) This is a known issue for some of the Splunk versions, and below is the workaround to fix this issue. Under $SPLUNK_HOME/etc/system/local/limits.conf, add [search] min_settings_period = 60 Note: This is in seconds. Defaults to 1 second. ... View more

Hi, Splunk built-in "rebuild" command is for single bucket. The indexer automatically deletes the old index and metadata files and rebuilds them. You don't need to delete any files yourself. You can use the below command to rebuild a single bucket splunk rebuild (name of the bucket directory) Refer the below link for more details regarding the same. https://docs.splunk.com/Documentation/Splunk/7.0.0/Indexer/Bucketissues Cheers, Meeran. ... View more

Hey, I'd like to help with your question. Where did you change the permissions? Was it for the dashboard and/or the data model? My first thought is that both objects might need permissions updates. Important: When you share a data model the knowledge objects associated with that data model (such as lookups or field extractions) must have the same permissions. Otherwise you run the risk of running into errors when other people try to use the data model. REF : http://docs.splunk.com/Documentation/Splunk/6.2.4/Knowledge/Managedatamodels#About_data_model_permissions You can check the permissions on the data model to make sure that they also are the same for the other user you're trying to share with. That way, that other user and you have the same access to the dashboard and to the data model that's driving the dashboard. Cheers, Meeran. ... View more

Hi, By default only admin and power roles have write permission in the search app. So any user with read permission will be able to access search & reporting app and will be able to create reports and dashboards and can create private knowledge objects but they will not be able to share those to others. If you remove read access for search & reporting app, then those users will not be able to see search & reporting and also settings will be hidden for them. To force your users to use particular app you could create roles for each department, map those users via LDAP or local auth and then set the default app context to their departmental app. If that's not feasible then you have to do some user education to get people out of the search app and into their departmental app if they are going to save knowledge objects. You could also change the default app that the user sees upon logging into splunk by role or user: Settings --> Access Controls --> Roles|Users--> select the desired role --> Select a default app from the drop-down list under Default app. ... View more

Warnings and violations occur when you exceed the maximum indexing volume allowed for your license. If you exceed your licensed daily volume on any one calendar day, you get a violation warning. If you have 5 or more warnings on an enforced Enterprise license, or 3 warnings on a Free license, in a rolling 30-day period, you are in violation of your license. If you get a license warning, you have until midnight (going by the time on the license master) to resolve it before it counts against the total number of warnings within the rolling 30 day period. Splunk does not stop indexing your data. Splunk only blocks search while you exceed your license. Search capabilities return when you have fewer than 5 (Enterprise) or 3 (Free) warnings in the previous 30 days. Please refer the below link in order to know more details about license violations. http://docs.splunk.com/Documentation/Splunk/6.6.2/Admin/Aboutlicenseviolations Hope the above addresses your queries. Cheers, Meeran. ... View more