Splunk Enterprise security Incident Review table not showing.

[jadengoho]

Hi All,
Would like to know what causes this issue , please see screenshot attached.
There's an event "42" showing and time range is showing , but the table is not showing.
SplunkEnterpriseSecuritySuite = version :5.3.0

[jgbricker]

Check your kvstore status. mongod.log generally has messages that can give you more information if it is a kvstore startup issue. Usually if you do a recursive chmod you can give too much access to the relevant key file. That's usually what gets me. The splunk.key file for mongo needs to be owner read-only 'chmod 400 ./splunk.key'

https://docs.splunk.com/Documentation/Splunk/8.0.3/Admin/TroubleshootKVstore

Did you just upgrade or make UI changes?
Sometimes there can be compatibility issues with custom view settings.

[jgbricker]

https://answers.splunk.com/answers/326863/kvstore-initialization.html is related to the permissions issue with splunk.key

[DavidHourani]

Hi @jadengoho,

Did you try clearing your browser cache or connect using another browser ? Seems like broken or cached CSS

Cheers,
David

[jwelch_splunk]

My guess is that you are hitting a bug:

SPL-189084 " /services/search/jobs returns empty results"
SOLNESS-22669 "Incident Review Filters causing results table to break"

Your best option here is file a support case and reference this number so we can do some research.

Okie

[jadengoho]

Hi @DavidHourani - i tried restarting my laptop and reconnecting to the internet.Will try clearing browser cache when it happend again.

[DavidHourani]

@jadengoho, that's great to hear ! Please accept the answer if your problem is solved 🙂

[jadengoho]

@DavidHourani - i tried clearing cache, changing browser and restarting my device but still issue occur.

[DavidHourani]

Ummm.. could you please check what you get if you run index=notable from the search interface ? That will help you make sure that those notables are actually populated and not empty events.
If index=notable is working then try this to ensure that events from incident review are there : |incident_review
Also play around with the time picker to see if you can see older events on both searches and on the incident review page.

[jadengoho]

@DavidHourani - search is returning results, and when i change the time picker = time range shows the event count per day - but the table is not showing anything.
Also the pagination is showing. tried to change page still not showing.

[DavidHourani]

@jadengoho, have you done any upgrades for ES recently ? Or have you changed any permissions for your user recently ?
There is a known issue for ES 5.3.0 where mis-configured roles might lead to the incident review page not loading :
https://docs.splunk.com/Documentation/ES/5.3.0/RN/KnownIssues
Issue : SOLNESS-21783

[jadengoho]

@DavidHourani - i do have splunk admin access but issue still occur.
sometime the table shows but most of the time it's not showing.

[DavidHourani]

@jadengoho, that's weird ... this "sometime the table shows but most of the time it's not showing" is most of the times due to cache... What browser are you using ? And could you try to change it ?

[jadengoho]

@DavidHourani - we are using Internet Explorer version 11.09. We can't use other browser [IT setup that way].

@MuS hahahha now i get it 42 is the real deal
i do audits on ES but nothing really .

[DavidHourani]

@jadengoho, then in that case next time you face the issue please try hitting the _bump or refresh endpoint:
mysplunkhost:8000/en-US/debug/refresh or mysplunkhost:8000/en-US/_bump

And make sure you didn't set up splunk on deep thought. This could be why you're getting 42. @MuS can confirm.

[jadengoho]

@DavidHourani - sure will notify you once issue occurred again. Also, thanks for the notif whahaha. 🙂

[jadengoho]

Hi All, Issue still exist- and we are looking at the internet connection using vpn might be the issue

[MuS]

This ^^^ or permission issues or ... anything else that could cause an error in ES.

Did you check all the internal logs of Splunk to see if you get errors when opening the 'Incident Review'?
Any other error in any other log files?
As @DavidHourani has asked, did you recently upgraded and did you restart Splunk after that?
Have to tried to _bump the Splunk instance?
I could add so many things to this list, but without more details we will never be able to help.

cheers, MuS

PS: It looks like you did not get my previous joke about 42 😉

[jadengoho]

@MuS
Did you check all the internal logs of Splunk to see if you get errors when opening the 'Incident Review'? Yes i investigated it , think all Error and Warn are really not related to the issue like

• truncating lines because of limits.
• Socket Errors from.
• Asynchronous bundle replication to
• An error occurred during the last operation ('deleteData'
• Error checking for update, URL=https://apps.splunk.com
• Received fatal SSL* alert. ssl_state='SSLv*
• Racing between mark job dispatched and heartbeats
• No response received from IMonitoredThread
• Missing a search command before
• The instance is approaching the maximum number of historical searches
• We recommed using RSA-SHA* for 'inboundSignatureAlgorithm'

did you recently upgraded and did you restart Splunk after that? im not the one who upgrade it 1yr ago, but i saw in the process that it has a restart.

Have to tried to _bump the Splunk instance? Not yet , will this once the issue occur again

It looks like you did not get my previous joke about 42 😉 - HAHAHA still didn't get it.

[MuS]

In regards of 42 ... https://www.independent.co.uk/life-style/history/42-the-answer-to-life-the-universe-and-everything-2...

And you have some search errors there; on ES that could indicate a problem - just saying ...

[MuS]

There are 42 matching events ... I reckon this ES has found the answer to everything 😉

cheers, MuS