After a clean install of Splunk IT Service Intelligence, why are we getting error "HttpListener...Could not find object id=dummy_collection..."?

[Julieda]

Hi,

After a clean install of Splunk IT Service Intelligence, the following error message appears over and over again in the splunkd.log:

ERROR HttpListener - Exception while processing request from 127.0.0.1 for /servicesNS/nobody/SA-ITOA/storage/collections/data/dummy_collection_nvfjdnvjkfdnvjkfnvjkfnvernvjfnvjkfsdnvuenvkjfnvjka?output_mode=json: Could not find object id=dummy_collection_nvfjdnvjkfdnvjkfnvjkfnvernvjfnvjkfsdnvuenvkjfnvjka

At this point, nothing has been configured in ITSI, no KPIs, no entities or services.
Is this normal, or does it have to be dealt with before continuing with configurations?
ITSI is running on Splunk 6.3.1 on CentOS 7. 

[jonathon]

It may not be the most elegant solution, but I changed the log level for HttpListener to provide less clutter in my splunkd.log using the splunk set log-level HttpListener -level FATAL -auth admin:admn_pw command

[laytonj76]

I'm seeing the same behavior; however, I'm running 6.6.0. Was your issue resolved in time as tfletcher suggested?

[tfletcher_splun]

Short Answer
That error message is actually expected and it is harmless.

Long Answer
That error message is a result of some of the background management processes in ITSI starting up. They validate the availability of the KV store by trying to access a non-existent collection and expecting a 404 response. KV Store is started up independently of the splunkd and splunkweb appserver start up, so there is a window of time where ITSI cannot perform its normal functions until that KV store responds appropriately. In certain environments this can take a few minutes. In most cases there should only be a few of these messages every so often after the KV store has properly initialized. You will likely see more of them with nothing configured in ITSI as the background management processes have nothing to manage. So they will wake up check availability check for things to do find nothing and go back down for a few minutes.

[EDIT] As an aside, you may want to validate that splunk 6.3 is supported for your version of ITSI. The currently available versions of ITSI require at least splunk 6.4