Splunk Enterprise Security: Why can I not create notable events and get "Splunk cannot find "data/inputs/threatlist""?

[meirwah]

I installed Splunk Enterprise Security, but nothing seems to function (cannot create notable events for instance, getting 500 error in many steps)
When I look at the web_service.log I see : Splunk cannot find "data/inputs/threatlist"

I'm using win server 2012 , ES 4.0.0, latest Splunk version 6.3.3

Any idea?

[mdessus_splunk]

So you first need to check if the KVstore is running (a mongodb process). Look also for error messages concerning kvstore or mongodb in internal logs (index=_*).
Is it a fresh new Search Head ? Is it a 64 bit version ? Was it used before ?

[meirwah]

Fixed the mongod issue (needed to delete the mongod.lock file and restart)
Some of the dashboards are ok now..

But I still cannot create a notable event , in the logs(web_service.log) I see :

2016-02-11 17:26:17,592 ERROR   [56bcc43938836e8eec50] __init__:321 - Unable to obtain template "dashboard.html":

 Traceback (most recent call last):
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\controllers\__init__.py", line 316, in render_template
    templateInstance = mako_lookup.get_template(template_name)
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\controllers\__init__.py", line 199, in get_template
    raise exceptions.TopLevelLookupException(_("Splunk has failed to locate the template for uri '%s'." % uri))
TopLevelLookupException: Splunk has failed to locate the template for uri 'dashboard.html'.

any idea?

[meirwah]

Issue I see now in web_service.log:

2016-02-15 16:58:28,367 ERROR   [56c203b3dd836e2840f0] __init__:340 - Mako failed to render: 

Traceback (most recent call last):
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\controllers\__init__.py", line 336, in render_template
    return templateInstance.render(**template_args)
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\mako\template.py", line 443, in render
    return runtime._render(self, self.callable_, args, data)
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\mako\runtime.py", line 803, in _render
    **_kwargs_for_callable(callable_, data))
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\mako\runtime.py", line 835, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\mako\runtime.py", line 860, in _exec_template
    callable_(context, *args, **kwargs)
  File "C:\Program Files\Splunk\share\splunk\search_mrsparkle\templates/layout/base.html", line 22, in render_body
    <%self:render/>
  File "C:\Program Files\Splunk\share\splunk\search_mrsparkle\templates/layout/base.html", line 28, in render_render
    <%self:pagedoc/>
  File "C:\Program Files\Splunk\share\splunk\search_mrsparkle\templates/layout/base.html", line 102, in render_pagedoc
    <%next:body/>
  File "C:\Program Files\Splunk\share\splunk\search_mrsparkle\templates/layout/view.html", line 24, in render_body
    ${next.body()}
  File "C:\Program Files\Splunk\share\splunk\search_mrsparkle\templates/view/dashboard.html", line 124, in render_body
    <%call expr="parent.getFloatLayoutRow(modules, rowNumber)"></%call>
  File "C:\Program Files\Splunk\share\splunk\search_mrsparkle\templates/view/dashboard.html", line 327, in render_getFloatLayoutRow
    <%call expr="next.getDashboardPanel(modules, panelNamesByColumn[col])"></%call>
  File "C:\Program Files\Splunk\share\splunk\search_mrsparkle\templates/view/dashboard.html", line 84, in render_getDashboardPanel
    <%call expr="parent.buildPanelContents(modules, groupName)"></%call>
  File "C:\Program Files\Splunk\share\splunk\search_mrsparkle\templates/view/dashboard.html", line 231, in render_buildPanelContents
    <%call expr="buildModule(module)"></%call>
  File "C:\Program Files\Splunk\share\splunk\search_mrsparkle\templates/view/dashboard.html", line 195, in buildModule
    <%def name="buildPanelContents(modules, panelName)"><%
  File "C:\Program Files\Splunk\share\splunk\search_mrsparkle\templates/view/dashboard.html", line 189, in render_buildModule
    <%include file="${module['templatePath']}" args="module=module"/>\
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\mako\runtime.py", line 730, in _include_file
    callable_(ctx, **_kwargs_for_include(callable_, context._data, **kwargs))
  File "C:\Program Files\Splunk\etc\apps\SA-ThreatIntelligence\appserver\modules\NotableEventCreator\NotableEventCreator.html", line 1, in render_body
    <%# Copyright (C) 2009-2012 Splunk Inc. All Rights Reserved.
  File "C:\Program Files\Splunk\etc\apps\SA-ThreatIntelligence\bin\shortcuts\__init__.py", line 162, in getOwners
    unused_response, content = KvStoreHandler.get(None, session_key, options)
  File "C:\Program Files\Splunk\etc\apps\SA-Utils\lib\SolnCommon\kvstore.py", line 37, in get
    response, content = splunk.rest.simpleRequest(uri, sessionKey=session_key)
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\rest\__init__.py", line 529, in simpleRequest
    raise splunk.ResourceNotFound(uri, extendedMessages=extractMessages(body))
ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/servicesNS/nobody/SA-ThreatIntelligence/storage/collections/data/notable_owne...; [{'type': 'ERROR', 'text': 'Application is disabled: SA-ThreatIntelligence', 'code': None}]

2016-02-15 16:58:28,401 ERROR   [56c203b3dd836e2840f0] __init__:321 - Unable to obtain template "dashboard.html": 

Traceback (most recent call last):
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\controllers\__init__.py", line 316, in render_template
    templateInstance = mako_lookup.get_template(template_name)
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\controllers\__init__.py", line 199, in get_template
    raise exceptions.TopLevelLookupException(_("Splunk has failed to locate the template for uri '%s'." % uri))
TopLevelLookupException: Splunk has failed to locate the template for uri 'dashboard.html'.

[meirwah]

I have access to the dashboards, but but they have errors like:

like under the "Notable Events Over Time" dashboard:
Error in 'inputlookup' command: External command based lookup 'es_notable_events' is not available because KV Store initialization has failed. Please contact your system administrator.

[mdessus_splunk]

What do you mean by "nothing seems to function" ? Do you have access to domain dashbaords for ex. (even if they are empty) ?
The threatlist message is not important.