Seeing a bunch of there errors:
htmlinjectiontoolfactory:81 - HTML injection tool factory error: Failed to inject hook in the response.
Best I can tell, it has something to do with DB Connect.
There is an associated error in splunk_app_db_connect_dbx.log:
[htmlinjectiontoolfactory.py], line 81 : HTML injection tool factory error: Failed to inject hook in the response.
And appears to be associated with embed requests, based on searching for requestid which is present in the first error.
I see that this is a known issue which was reported for Splunk 6.6.x version and is fixed in Splunk 7.1.0.
Below is the reference number for the same.
The above reference number wasn't publicly documented. It appears that is was only reported as an internal defect.
I could recreate the 404 error in my test environment and I then looked to compare permissions and capabilities my instances but there wasn't really anything of note.
But it might not be the case always. Looks like when the Splunk GUI on the search head is hitting a forbidden page this known issue is being hit on the search head.
Hope the above helps!
Thanks. We're on 7.0.2, so I'll let you know if going to 7.2.4 will fix it.