topic Re: Cannot get data Index in Datamodel in Deployment Architecture

Cannot get data Index in Datamodel

zksvc — Mon, 23 Sep 2024 03:14:45 GMT

Hi there,

I was ingest new server to new index (Ubuntu with UF)

Let say my index is index=ABC

I want to connect it in Datamodel, unfortunately im not the first who was create it. And when i check it i got error "This object has no explicit index constraint. Consider adding one for better performance."

And when i check it in macros `cim_Endpoint_indexes` it only show ()

When i want to add my new index in that macros i got this 500 server error

According from this question : https://community.splunk.com/t5/Knowledge-Management/Adding-index-to-accelerated-CIM-datamodel/m-p/586847#M8722 it said 2 solution :

if you don't rebuild the DataModel, Splunk will start to add logs from that index when you save the macro and old events aren't added to the Datamodel, only the new ones,
if you rebuild the DataModel, Splunk will add to the DataModel all the events in all indexes contained in the macro until the retention period (e.g. Network Traffic 1month, Authentication 1 year, and so on).

Since i know it cannot add from macros, i create new Eventtype and Tag for my new index.

And that Eventtype also in Tag like this

Eventtype	Tag
eventtype=ABC_endpoint_event	tag=endpoint, tag=asset, tag=network
eventtype=ABC_process_event	tag=process, tag=endpoint
eventtype=ABC_network_event	tag=network, tag=communication
eventtype=ABC_security_event	tag=security, tag=endpoint

One from base search in Datamodel Endpoint is using tag=process

(`cim_Endpoint_indexes`) tag=process tag=report | eval process_integrity_level=lower(process_integrity_level)

From that query it calling tag=process

But when i try to running it, it don't show my new index.

Anyone can help me to solving this issue ?

~Danke

Re: Cannot get data Index in Datamodel

marnall — Wed, 25 Sep 2024 20:48:54 GMT

If you do not specify an index= filter in your search, then Splunk will search your role's default indexes, which can be toggled in the role settings. If you have no default indexes or no data in your default indexes, then no results will appear.

The purpose of the cim_Endpoint_indexes macro is to list the indexes from which to find data to populate the data model, so you /should/ be able to list your index filters in there. E.g.
index=ABC
or
index IN (ABC,DEF)

The problem is that your Splunk instance is returning a 500 Internal Server Error when you try to edit the macro. In a working system it would not do that. Can you check the web_service.log to see what is causing the problem?

If you can access the shell of your splunk search head and it is Linux, then the log should be findable at:

/opt/splunk/var/log/splunk/web_service.log

Re: Cannot get data Index in Datamodel

zksvc — Thu, 26 Sep 2024 11:16:35 GMT

i was download log file in `/opt/splunk/var/log/splunk/web_service.log` and i open with Notepad++ like this

When i search 500 ERROR it showed too much data, could you please give me specify keyword? Because when i want to search macros it not show anything.

Sorry very confuse about it

Re: Cannot get data Index in Datamodel

PickleRick — Thu, 26 Sep 2024 12:14:01 GMT

Small correction. If you don't define the cim_* macros their contents will of course be empty and while searching without using the accelerated summaries in your ad-hoc or scheduled searches it will use indeed your user's role's default indexes, the datamodel acceleration summary building searches will be spawned with the system user's default indexes which is an empty list. You need to have explicitly defined list of indexes to have CIM acceleration built properly.

Re: Cannot get data Index in Datamodel

marnall — Thu, 26 Sep 2024 18:23:53 GMT

Are you able to go to one of those 500 Error lines, then post 10-20 of the previous lines? If it contains sensitive data then you should sanitize it before posting.

Re: Cannot get data Index in Datamodel

zksvc — Fri, 27 Sep 2024 01:43:51 GMT

I think there is no sensitive data, so here the full error message

2024-09-23 21:06:30,760 INFO [66f175e6c17fc8d479ab10] error:337 - 500 Internal Server Error The server encountered an unexpected condition which prevented it from fulfilling the request. 2024-09-23 21:06:30,760 ERROR [66f175e6c17fc8d479ab10] error:338 - Traceback (most recent call last): File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 628, in respond self._do_respond(path_info) File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 687, in _do_respond response.body = self.handler() File "/opt/splunk/lib/python3.7/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/htmlinjectiontoolfactory.py", line 75, in wrapper resp = handler(*args, **kwargs) File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-2042>", line 2, in help File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 41, in rundecs return fn(*a, **kw) File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-2040>", line 2, in help File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 102, in check if verify_session and not util.checkRequestForValidFormKey(requireValidFormKey=must_login): File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/util.py", line 1481, in checkRequestForValidFormKey if not isValidFormKey(form_key) or not doesFormKeyMatchCookie(form_key): File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/util.py", line 1452, in isValidFormKey logger.warn('CSRF form_key mismatch received=%s expected[redacted]=%s' % (key, getFormKey()[-4:])) TypeError: 'NoneType' object is not subscriptable

Before that i would to say thankyou for reply this post and want to help me

Re: Cannot get data Index in Datamodel

PickleRick — Fri, 27 Sep 2024 07:38:28 GMT

OK. It seems to be CSRF-prevention cookie related. Try clearing your browser cache and cookies. Maybe your browser has some invalid cookie stored which it supplies with your requests.

Re: Cannot get data Index in Datamodel

marnall — Sat, 28 Sep 2024 19:27:20 GMT

As Picklerick suggests, this may resolve if you clear your cookies and/or cache.

Re: Cannot get data Index in Datamodel

zksvc — Sun, 29 Sep 2024 03:33:06 GMT

Hi @PickleRick @marnall
Thankyou for your advice, but unfortunately i still can't change it even after i clear my cookies and/or cache.

Can this issue solved using another method ?

Re: Cannot get data Index in Datamodel

PickleRick — Sun, 29 Sep 2024 05:23:25 GMT

Well, not everything can be solved just by joined community wisdom. Sometimes you simply have to raise support case.

Re: Cannot get data Index in Datamodel

marnall — Sun, 29 Sep 2024 12:05:53 GMT

There are some troubleshooting steps you could try:

1. Use a different browser
2. Try to edit other macros
3. Try to add a new macro
4. Try to edit other knowledge objects, like field extractions, dashboards, etc
5. Make a new user with very high permissions (e.g. admin) and try editing the macro with it
6. Install a new search head, connect it to your indexers, then edit the macro