This is a known issue, currently we are working to address it. In the meantime you can suppress it by creating a user, "system". https://docs.splunk.com/Documentation/Splunk/7.0.2/Security/ConfigureuserswiththeCLI If it is still the same then you may need to log a support case. Make sure to provide the below; - Splunk Deployment architecture. - Enable DEBUG and have it run for a few mins - depends on the frequency of the log messages. $ ./splunk set log-level UiSAML -level DEBUG $ ./splunk set log-level Saml -level DEBUG $ ./splunk set log-level AuthenticationManagerSAML -level DEBUG $ ./splunk set log-level AttrQueryRequestJob -level DEBUG Or if you can, try to disable apps one by one and see which app is causing this error and go from there. ... View more

This turned out to be a case of one of java script file missing for unknown reasons. HAR dump showed us some hints about the issue. There was a 404 error for a request as below; After recovering common.js which was producing error in the screenshot, the issue has gone away. For some reason the file disappeared. Try to find 4xx error in HAR dump same thing could happen if some of files were deleted accidentally. ... View more

As crossServerChecksum = true is configured, splunk will not consider timestamp or permissions of files in calculating checksums, there is no manual way of simulating the checksum calculation exactly as splunk does as of this writing but you can briefly check using md5 as below; splunk cmd openssl md5 "tarball of the app" Also make sure there is no hidden files or directories which can cause different checksums. Below has .git which caused different checksums. -rw-r--r-- 1 support support 0 Jan 31 14:07 README.md drwxr-xr-x 7 support support 4096 Feb 26 13:59 .git drwxr-xr-x 2 support support 4096 Feb 26 13:59 auth drwxr-xr-x 2 support support 4096 Feb 26 13:59 local ... View more

Search.log shows strange error suggesting that lookup filename is not recognized and going to implicit filename. This could have been worked around by matching the lookup name and its file name. ---> Search.log under dispatch/"SID" --- 02-12-2018 06:00:05.056 INFO SearchProcessor - Building search filter 02-12-2018 06:00:05.059 WARN LookupOperator - Unable to find property=filename for lookup=ABC will attempt to use implicit filename. 02-12-2018 06:00:05.059 WARN LookupOperator - No valid lookup found for lookup=ABC This is just for an immediate workaround. To root cause it please contact splunk support with DEBUG enabled diag, which is taken after a repro - make sure to provide SID. To enable DEBUG for search; - Set rootCategory=DEBUG in $SPLUNK_HOME/etc/log-searchprocess.cfg rootCategory=DEBUG,searchprocessAppender appender.searchprocessAppender.maxBackupIndex=50 ... View more

Below worked for many cases ; Make sure to add it to props.conf on the server where monitor stanza is configured - mostly UF. ["source or sourcetype "] NO_BINARY_CHECK=true CHARSET=AUTO If it doesn't work then enable DEBUG and open a support ticket along with a diag with DEBUG messages. - Below are the categories to be set to DEBUG ./splunk set log-level FileClassifierManager -level DEBUG ./splunk set log-level FileClassifier -level DEBUG ./splunk set log-level WatchedFile -level DEBUG ./splunk set log-level TailingProcessor -level DEBUG ./splunk set log-level TailReader -level DEBUG ... View more

If you find error messages from the SID, like below in search.log, then you are hitting a known issue which can be worked around by i) timezone reset to default ( Web UI > userName > settings ) OR ii) Add shebang to command.sh - i.e below is for db connect running on linux system. vi splunk_app_db_connect/linux_x86_64/bin/command.sh #!/bin/bash <--- Add this to the 1st line of the file. Error messages in search.log 10-02-2017 17:21:42.843 INFO ChunkedExternProcessor - Running process: /opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/command.sh -Dlogback.configurationFile=../config/command_logback.xml -DDBX_COMMAND_LOG_LEVEL=ERROR -cp ../jars/command.jar com.splunk.dbx.command.DbxQueryCommand 10-02-2017 17:21:42.844 ERROR ChunkedExternProcessor - Failure starting process 10-02-2017 17:21:42.844 ERROR ChunkedExternProcessor - Error in 'dbxquery' command: Invalid message received from external search command during setup, see search.log. If none of above are working then please open a support case with a diag which has dispatch directory of the SID for further investigation. ... View more

As you can see, those entries are from search optimization feature which was just introduced to 6.5. This just kicks in before it actually start search. The symptom disappears after it disabled as suggested in the link below and performance wise it worked well as expected. It's not something we should disable but it worked for the case. In $SPLUNK_HOME/etc/apps/splunk_app_db_connect/local/limits.conf [search_optimization] enabled=false You can find more information from the documentation; http://docs.splunk.com/Documentation/Splunk/latest/Search/Built-inoptimization#Turn_off_optimization_for_a_specific_search ... View more

This happens as the field, "myType" is not part of the raw data and index time field extraction is applied. By default the fields in search is extracted from raw data but this myType=Find_me is not found in raw events - that is the reason for the symptom. To make it work for the case use fields.conf - https://docs.splunk.com/Documentation/Splunk/latest/Admin/Fieldsconf - http://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Aboutindexedfieldextraction i.e) [myType] INDEXED=true ... View more

- If you still have primary or secondary and only some of them went out of sync, then use the method in the doc: https://docs.splunk.com/Documentation/Splunk/latest/Admin/ResyncKVstore - If you think the KVStore cluster is broken, such as no primary or secondary at all and need recovery then follow the below; This could happen because you didn't have shcluster captain when the search was started. That's why the KVStore is in starting, not able to make it to "Ready" because SHC captain is the one should tell KVStore which members are available for ReplicaSet. Follow the steps below to correct the situation: 1. Do backup $SPLUNK_HOME from all members!!! 2. Stop all SHC instances. 3. Run the command from all members      $ rm -rf $SPLUNK_HOME/var/run/splunk/_raft/*      $ splunk clean kvstore --cluster      > This is not deleting database but deleting cluster info. 4. Choose one member, where you think KVStore worked before. Edit as below,     in "$SPLUNK_HOME/etc/system/local/server.conf"     [shclustering]     replication_factor=1 5. Start this member 6. Bootstrap SHC with just this member      $ ./bin/splunk bootstrap shcluster-captain -servers_list "https://THIS_MEMBER_URL:8089" -auth admin:password 7. Verify SHC status      $ ./splunk show shcluster-status 8. Verify KVStore status, shoudl see 'ready' ,      $  curl -k -s https://localhost:8089/services/server/info | grep kvStore     OR $./splunk show kvstore-status 9. Stop this instance and change replication_factor to whatever it was before.       in "$SPLUNK_HOME/etc/system/local/server.conf"       [shclustering]       replication_factor=what it was before 10. Clean folder $SPLUNK_HOME/var/run/splunk/_raft/ on this instance.         Or run, "$ splunk clean raft" 11. Start ALL instances. 12. Bootstrap SHC with all members.       $ ./bin/splunk bootstrap shcluster-captain -servers_list "https://server1:8089,https://server2:8089,https://server3:8089,https://server4:8089" -auth admin:changeme        > replace the server1~4 with your member info. 13. Verify SHC status        $ ./splunk show shcluster-status 14. Verify KVStore status from all members        $ curl -k -s https://localhost:8089/services/server/info | grep kvStore        OR $./splunk show kvstore-status   15. If any data or collections are found to be missing then restore it from any latest good backup by using restore command, details found in the doc below; https://docs.splunk.com/Documentation/Splunk/8.0.5/Admin/BackupKVstore   ... View more

This turned out to be a defect, of which fix is soon to be released - check with splunk support. Workaround : Restart the search-head which is issuing failed checksum warnings during searches. Saved jobs (ie. results from searches stored for further usage) that showed those warnings will not be repaired, they need to be re-ran. Fixed versions : 6.3.11, 6.4.8, 6.5.5, 6.6.3, 7.0.0 ... View more