After upgrading Splunk to latest version(7.0.1), ES dashboard for "Notable" & "Incident Review" not displaying any new data/events.

[att35]

Hi,

We recently upgraded to latest Splunk version 7.0.1 but it seems that since that day, ES is not able to populate anything under "Notables" or "Incident Review" as if ES doesn't have access to indexes anymore.

Verified that all correlation and searches related to notable are running.

We did come to know about a specific Bug in UI which causes all the assigned indexes to disappear from Roles. SPL-145546. Fix was applied to the search heads after which we were able to re-assign indexes to roles. could this bug be responsible for the ES issues ?

Noticed that for all ES specific roles(ess_admin, ess_analyst, ess_user), the assigned index section were blank. Added all indexes and restarted Splunk but we still dont see anything under Notables(Securtiy Posture) and for incident review, when search for "all time", the last event is from the day we did the upgrade.

When I check for internal indexes on the SH, latest event timestamp is 25 days ago, which matches exactly the day Splunk got upgraded to 7.0.1

I should mention here that we are still able to search all events outside of ES App. Also, within ES, dashboards like Access Center or Traffic Center do show current data. It's just the notable and Incident review that are completely blank.

Has anyone else seen such issue? Anything else I can check to isolate whatever is causing this issue?

Splunk Version - 7.0.1
Splunk ES Version - 4.7.4

Many Thanks,

~ Abhi

[Mohsin123]

Okay.
Even we faced with this issue after upgrading splunk to 7.0.1

PFA a similar link where a workaround is provided :

https://answers.splunk.com/answers/114064/indexes-from-peer-nodes-not-visible-in-role-creation-on-se...

Please note, the issue has reoccurred in Splunk 7.0 and the following bug has been raised for this matter:

SPL-145546 - in 7.x in Roles admin Indexes are for local search head only

Workaround:

Step 1) Create a local directory in the search app on the SH with the correct permissions for splunkd to access i.e.

$SPLUNK_HOME/etc/apps/search/local/data/ui/manager

Step 2) Copy an old "authentication_roles.xml" file from "$SPLUNK_HOME/etc/apps/search/default/data/ui/manager" in any 6.x version or simply download a new 6.x version of Splunk and extract the file there, then place it into the folder created in step 1.

Step 3) Refresh the SH configuration with debug refresh via the web browser:

http://:8000/en-US/debug/refresh

Step 4) Create a new role on the SH and you should see all your indexes configured on the index cluster.

Note: In the workaround provided above, there is a known issue (SPL-146171) where only 1000 indexes is displayed in the UI. If you have more than 1000 indexes, you should modify authorize.conf to add the index(es) to role(s) instead

[att35 · Accepted solution]

We were able to resolve this with the help of Support.

Turns out, log forwarding was enabled on the ES search-heads but the respective indexes, e.g. notables, were not present on the Indexer and hence the logs were just getting dropped. Once we exported the "Splunk_TA_ForIndexers" app to each of the indexers, all notable related dashboards started getting data.

Thanks,

~ Abhi

[sk314]

Since this was an ES upgrade, didn't you already have the Splunk_TA_ForIndexers on your indexers? I do have it and the notable index was present. Interestingly, the notable index has no data after the upgrade!

[sk314]

were you able to fix this?