This is an issue reported to Splunk, found to be outstanding in 7.2 & 7.3. The workaround is , go and edit the xml view file, such as $SPLUNK_HOME/etc/users//local/data/ui/views/.xml You can use this to get encoding values for characters - https://www.urlencoder.org. I.e) ? = %3F ... View more

Here's what I found: i) The search below shows that notables are not created intermittently and happening when it were assigned to one search head. index=_internal source=/scheduler.log status=success result_count > 0 alert_action="" savedsearch_name=A savedsearch_name=B OR savedsearch_name=C | stats count by host This is found to be happening 7days ago from the SH. ii) There are also errors in splunkd.log ERROR ModularInputs - Unable to initialize modular input "itsi_entity_exchange_consumer" Timechart suggested that this error started to happen from a specific time, i.e: Oct 26 05:49 and persisting since then. Also the splunkd.log shows it stopped and started back. Checking logs in /var/log/messages, kernel complaint about out of memory and OOM killer killed kvstore and splunkd at Oct 26 05:46. Then splunk might have been restarted at 05:49. iii) Ever since then it fails to load python libraries. 10-29-2019 00:12:49.313 -0700 ERROR sendmodalert - action=itsi_event_generator STDERR - Traceback (most recent call last): 10-29-2019 00:12:49.313 -0700 ERROR sendmodalert - action=itsi_event_generator STDERR - File "/users/splunk/etc/apps/SA-ITOA/bin/itsi_event_generator.py", line 8, in 10-29-2019 00:12:49.313 -0700 ERROR sendmodalert - action=itsi_event_generator STDERR - from ITOA.setup_logging import getLogger 10-29-2019 00:12:49.313 -0700 ERROR sendmodalert - action=itsi_event_generator STDERR - File "/users/splunk/etc/apps/SA-ITOA/lib/ITOA/setup_logging.py", line 9, in 10-29-2019 00:12:49.313 -0700 ERROR sendmodalert - action=itsi_event_generator STDERR - from splunk.appserver.mrsparkle.lib import i18n SNIP 10-29-2019 00:12:49.313 -0700 ERROR sendmodalert - action=itsi_event_generator STDERR - _install_highlighting() 10-29-2019 00:12:49.314 -0700 ERROR sendmodalert - action=itsi_event_generator STDERR - File "/users/splunk/lib/python2.7/site-packages/mako/exceptions.py", line 252, in _install_highlighting 10-29-2019 00:12:49.314 -0700 ERROR sendmodalert - action=itsi_event_generator STDERR - _install_fallback() 10-29-2019 00:12:49.314 -0700 ERROR sendmodalert - action=itsi_event_generator STDERR - File "/users/splunk/lib/python2.7/site-packages/mako/exceptions.py", line 243, in _install_fallback 10-29-2019 00:12:49.314 -0700 ERROR sendmodalert - action=itsi_event_generator STDERR - from mako.filters import html_escape 10-29-2019 00:12:49.314 -0700 ERROR sendmodalert - action=itsi_event_generator STDERR - ImportError: cannot import name html_escape iv) The error with ImportError suggests it fails to load python lib and pyc file corruption is suspected due to the OOM situation. On the SH, we followed the steps below so that pyc file should be auto-generated by the interpreter when splunk starts and imports modules. $ splunk stop $ cd $SPLUNK_HOME/lib/python2.7/site-packages $ find . -name "*.pyc" -exec rm -f {} \; $ splunk start After this step the issue never came back again. ... View more

Further investigation, the kernel message has this; Dec 01 10:15:29 idx15 kernel: [72693.445279] Task in /system.slice/splunk.service killed as a result of limit of /system.slice/splunk.service Dec 01 10:15:29 idx15 kernel: [72693.445891] memory: usage 104857600kB, limit 104857600kB, failcnt 1977805211 According to the log it hits the max usage defined by splunk.service of systemd unit, which is limited by "MemoryLimit=100G" - this appears to be static regardless of the memory available on the server. We decide to increase the value to the 90% of the memory installed on the server. If you see the similar symptoms you may need to check the value for the param and adjust it accordingly. ... View more

This has been fixed in the version 7.3.3+ which is already available for downloading. You can also implement the workaround below until the upgrade; ** Workaround Open to edit $SPLUNK_HOME/share/splunk/search_mrsparkle/exposed/build/pages/enterprise/authorization_roles.js Find the line below; this.indexListSearchJob.getResults().subscribe(function(results) Add the following between the parentheses which will allow 1000 indexes to be displayed. { count: 1000 } Should be like this: this.indexListSearchJob.getResults({ count: 1000 }).subscribe(function(results) Save file and hit the _bump endpoint. Click the "Bump version" button to force browser caches to make the change take effect. http(s)://yourSplunk:8000/en-US/_bump We have planned to fix this permanantly by implementing pagination for the list instead of listing up all in one. ... View more

Search.log in Search job inspector shows that searches were only dispatched to the 2 existing cluster indexers but not to the new indexers. The search GUI shows the field name, splunk_server_group field shows that dmc_xxx groups. This means the deployment monitoring console, DMC is enabled on the search head. With DMC enabled on SH, only search peers that are members of the indexer group are searched by default. Because all searches that run on the monitoring console instance follow this behavior, non-monitoring console searches might have incomplete results. "Do Not Use Production Search Head for DMC" which is also documented in the link below - Resolution: Switching to "standalone" next to "distributed" fixed the issue. https://docs.splunk.com/Documentation/Splunk/7.2.4/DMC/WheretohostDMC ... View more

This is caused by some post-process searches, such as if a post-process search doesn't contain <query> element, then dashboard is not loading. This is due to JavaScript undefined error. And can be worked around by adding empty query element for the post-process searches without query. This is just a temp workaround and may not be needed once fixed (7.3.4+, 8.0.1+). for example, below is Okay <search base="callsA" id="myTable"> <query> | search call_id="$token1$" ...<SNIP> </query> </search> But if a post-process search doesn't have any query but only has some options, which will cause the error/symptoms (SPL-178012). <search base="callB" id="yourTable"></search> It can be worked around by adding empty query element as below. <search base="callB" id="yourTable"> <query></query> </search> ... View more

You may need to use, curl with options, like '-L' and headers, X-Splunk-Form-Key and X-Requested-With: XMLHttpRequest.. Here's the article for it, https://docs.splunk.com/Documentation/StreamApp/7.1.3/DeployStreamApp/SplunkAppforStreamRESTAPI $ curl -Lk -u admin http://localhost:8000/en-GB/splunkd/__raw/services/myRESTCall -H 'Cookie: splunkd_8000=QHASm07YxvF4C8642PFHMRAS0PxRXLie7THbp261otk3pTmWJ2QVn4twqheZM^3_KZ6DDPzenJYszELi8VwvJPrKyJrK_hM75x45XjuzknW1Y7CGH6J4Kp6wakjj9MNAWF; splunkweb_csrf_token_8000=5089449155758749034; session_id_8000=b970b2c4c0b38eb917d980c5b6384f5152a8ad72; login=; splunkweb_uid=' -H 'X-Splunk-Form-Key: 5089449155758749034' -H 'X-Requested-With: XMLHttpRequest' -X POST Enter host password for user 'admin': session: PIPsP72nJ7Ha4Z_KEypI4TKtkSiIUQqlqkYmLYVp6PweuwSMRUF0hgHxPx8p_L1GDQQ7UpyU2t8Tk9QQm9fjDrAIIIEioNl5nquFPB1NYwD4Pn9MKWBmBpUwJ12kdxVMksZGfOxkqQ0 host: 127.0.0.1:8089 cookie: splunkd_8000=QHASm07YxvF4C8642PFHMRAS0PxRXLie7THbp261otk3pTmWJ2QVn4twqheZM^3_KZ6DDPzenJYszELi8VwvJPrKyJrK_hM75x45XjuzknW1Y7CGH6J4Kp6wakjj9MNAWF; splunkweb_csrf_token_8000=5089449155758749034; session_id_8000=b970b2c4c0b38eb917d980c5b6384f5152a8ad72; login=; splunkweb_uid= x-requested-with: XMLHttpRequest user-agent: curl/7.54.0 authorization: Basic YWRtaW46V2VsY29tZTA= accept: */ x-splunk-form-key: 5089449155758749034* Please make sure to include those headers for your testing. ... View more

I found error messages repeating as below, which suggests it has been failing for days. 09-18-2019 18:35:58.803 +0000 ERROR ConfReplication - Error creating snapshot: /opt/splunk/var/run/splunk/snapshot/15831677-5b6c4f95a711c6431341ba397e4c6b012a.bundle.f3effb6944a1e.tmp; Configurations changed while generating snapshot, original_latest_change=5b6c4f95a711c6431341ba397e4c6b012a, new_latest_change=2f2baeb33f5867261227d7636d5c7ed3b0d38749; consecutiveRejectionFromNewChanges=336;* Check conf.log to see if any app or client is making frequent configuration changes; Continuous snapshot creation failures can lead to configuration replication issues if this member becomes the captain* As it suggests in the message above the conf.log shows a lot of changes "addCommit" from ES import, due to this it updates local.meta and interrupts the creation of snapshot. == Use the below searches to identify the changes that interrupts the operation == index=_internal source=*/splunkd.log consecutiveRejectionFromNewChanges  earliest=-1d latest=now Index=_internal source=/conf.log source=*/conf.log* data.task=addCommit| timechart span=5m count by data.optype_desc Especially this issue was caused by the ES import modular input which updates several 100s of apps and add-ons installed on the SH. The import operation is only needed when new apps/add-ons installed on the server, without it ES will not recognize the data to be monitored. This has been worked around by increasing the interval to, like 2hrs, for ES import mod input, which is in inputs.conf of /etc/apps/SplunkEnterpriseSecuritySuite, this import has been removed in the latest version of ESS 5.3.1. It depends on the deployments environment - this time it was caused by ES import but there could be some other apps/add-on which could frequently update the configs. ... View more

According to the documentation below, it doesn't appear to accept "empty" string for the value and has to be prefixed by "$app_root$". https://docs.splunk.com/Documentation/Splunk/7.3.1/Admin/Serverclassconf *excludeFromUpdate = [,]... Specifies paths to one or more top-level files or directories (and their contents) to exclude from being touched during app update. Note that each comma-separated entry MUST be prefixed by "$app_root$/" to avoid warning messages. * Can be overridden at the serverClass level. * Can be overridden at the app level. * Requires version 6.2.x or higher for both the deployment server and client.* Instead, you can work it around by adding non-existing directory, such as "$app_root$/dummy . ... View more