Splunk Enterprise Security

Splunk Enterprise security Incident Review table not showing.

jadengoho
Builder

Hi All,
Would like to know what causes this issue , please see screenshot attached.
There's an event "42" showing and time range is showing , but the table is not showing.
SplunkEnterpriseSecuritySuite = version :5.3.0

alt text

Labels (2)

jgbricker
Contributor

Check your kvstore status. mongod.log generally has messages that can give you more information if it is a kvstore startup issue. Usually if you do a recursive chmod you can give too much access to the relevant key file. That's usually what gets me. The splunk.key file for mongo needs to be owner read-only 'chmod 400 ./splunk.key'

https://docs.splunk.com/Documentation/Splunk/8.0.3/Admin/TroubleshootKVstore

Did you just upgrade or make UI changes?
Sometimes there can be compatibility issues with custom view settings.

0 Karma

jgbricker
Contributor

https://answers.splunk.com/answers/326863/kvstore-initialization.html is related to the permissions issue with splunk.key

0 Karma

DavidHourani
Super Champion

Hi @jadengoho,

Did you try clearing your browser cache or connect using another browser ? Seems like broken or cached CSS

Cheers,
David

jwelch_splunk
Splunk Employee
Splunk Employee

My guess is that you are hitting a bug:

SPL-189084 " /services/search/jobs returns empty results"
SOLNESS-22669 "Incident Review Filters causing results table to break"

Your best option here is file a support case and reference this number so we can do some research.

Okie

jadengoho
Builder

Hi @DavidHourani - i tried restarting my laptop and reconnecting to the internet.Will try clearing browser cache when it happend again.

0 Karma

DavidHourani
Super Champion

@jadengoho, that's great to hear ! Please accept the answer if your problem is solved 🙂

0 Karma

jadengoho
Builder

@DavidHourani - i tried clearing cache, changing browser and restarting my device but still issue occur.

0 Karma

DavidHourani
Super Champion

Ummm.. could you please check what you get if you run index=notable from the search interface ? That will help you make sure that those notables are actually populated and not empty events.
If index=notable is working then try this to ensure that events from incident review are there : |incident_review
Also play around with the time picker to see if you can see older events on both searches and on the incident review page.

jadengoho
Builder

@DavidHourani - search is returning results, and when i change the time picker = time range shows the event count per day - but the table is not showing anything.
Also the pagination is showing. tried to change page still not showing.

0 Karma

DavidHourani
Super Champion

@jadengoho, have you done any upgrades for ES recently ? Or have you changed any permissions for your user recently ?
There is a known issue for ES 5.3.0 where mis-configured roles might lead to the incident review page not loading :
https://docs.splunk.com/Documentation/ES/5.3.0/RN/KnownIssues
Issue : SOLNESS-21783

jadengoho
Builder

@DavidHourani - i do have splunk admin access but issue still occur.
sometime the table shows but most of the time it's not showing.

0 Karma

DavidHourani
Super Champion

@jadengoho, that's weird ... this "sometime the table shows but most of the time it's not showing" is most of the times due to cache... What browser are you using ? And could you try to change it ?

0 Karma

jadengoho
Builder

@DavidHourani - we are using Internet Explorer version 11.09. We can't use other browser [IT setup that way].

@MuS hahahha now i get it 42 is the real deal
i do audits on ES but nothing really .

0 Karma

DavidHourani
Super Champion

@jadengoho, then in that case next time you face the issue please try hitting the _bump or refresh endpoint:
mysplunkhost:8000/en-US/debug/refresh or mysplunkhost:8000/en-US/_bump

And make sure you didn't set up splunk on deep thought. This could be why you're getting 42. @MuS can confirm.

0 Karma

jadengoho
Builder

@DavidHourani - sure will notify you once issue occurred again. Also, thanks for the notif whahaha. 🙂

0 Karma

jadengoho
Builder

Hi All, Issue still exist- and we are looking at the internet connection using vpn might be the issue

0 Karma

MuS
SplunkTrust
SplunkTrust

This ^^^ or permission issues or ... anything else that could cause an error in ES.

Did you check all the internal logs of Splunk to see if you get errors when opening the 'Incident Review'?
Any other error in any other log files?
As @DavidHourani has asked, did you recently upgraded and did you restart Splunk after that?
Have to tried to _bump the Splunk instance?
I could add so many things to this list, but without more details we will never be able to help.

cheers, MuS

PS: It looks like you did not get my previous joke about 42 😉

jadengoho
Builder

@MuS
Did you check all the internal logs of Splunk to see if you get errors when opening the 'Incident Review'? Yes i investigated it , think all Error and Warn are really not related to the issue like

  • truncating lines because of limits.
  • Socket Errors from.
  • Asynchronous bundle replication to
  • An error occurred during the last operation ('deleteData'
  • Error checking for update, URL=https://apps.splunk.com
  • Received fatal SSL* alert. ssl_state='SSLv*
  • Racing between mark job dispatched and heartbeats
  • No response received from IMonitoredThread
  • Missing a search command before
  • The instance is approaching the maximum number of historical searches
  • We recommed using RSA-SHA* for 'inboundSignatureAlgorithm'

did you recently upgraded and did you restart Splunk after that? im not the one who upgrade it 1yr ago, but i saw in the process that it has a restart.

Have to tried to _bump the Splunk instance? Not yet , will this once the issue occur again

It looks like you did not get my previous joke about 42 😉 - HAHAHA still didn't get it.

0 Karma

MuS
SplunkTrust
SplunkTrust

In regards of 42 ... https://www.independent.co.uk/life-style/history/42-the-answer-to-life-the-universe-and-everything-2...

And you have some search errors there; on ES that could indicate a problem - just saying ...

0 Karma

MuS
SplunkTrust
SplunkTrust

There are 42 matching events ... I reckon this ES has found the answer to everything 😉

cheers, MuS

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...