All Apps and Add-ons

Installation cacti splunk cluster

aecruzp
Path Finder

regards

    Currently, you try to install the app in a cluster environment, 3 search head and 6 indexer, but at the time of deploy and bundle, from the search head the following message is displayed:

[splunk-indexer-01-cnt] Streamed search execute failed because: Error in 'SearchParser': The search specifies a macro 'cacti_index' that can not be found. Reasons include: the macro name is misspelled, you do not have "read" permission for the macro, or the macro has not been shared with this application. Click Settings, Advanced search, Search Macros to view macro information.

    This message appears for each indexer.

    Is there any recommendation and configuration in the search head and indexer? All permissions have been enabled for cacti macros and we have not had favorable answers.

    I hope you can help me.

0 Karma

mattymo
Splunk Employee
Splunk Employee

I would imagine that the macro doesn't exist on the indexers, or the permissions on the macro might be wrong?

Was the the app pushed to the indexer cluster?

When you are running your search, which app are you in?? Check if the macro permissions are global:

alt text

Its appears global in my instance.

the macro is technically not mandatory for your searches either, was just bet practice to allow the users to change it, so you could technically just not use it as well.

hit me up on slack if you are in the chat (splk.it/slack to sign up), I'm @mattymo

aecruzp
Path Finder

Thanks for the prompt response.

The macro currently exists in each indexer and the write permissions have been modified in the same way, the write and read permissions of the .conf have also been modified, thinking about the possibility that these files could be interfering

alt text

alt text

0 Karma

aecruzp
Path Finder

the app is in the indexer and deployed in the search head, the strange thing is that in the index cacti there is information and from the search head there are errors but in the indexer no, but there is no information

indexer
alt text

Search head
alt text

0 Karma

mattymo
Splunk Employee
Splunk Employee

so, you have GUI access to your indexer cluster members?

What happens when you search with the macro in search and reporting app on the indexer?

what happens when you search index=cacti on the indexer?

can you get cli on the indexer and double check $SPLUNK_HOME/etc/apps/Splunk_TA_Cacti/default/macros.conf and see what is in there?

Also be sure that the cacti index you created in present on the indexers too. As per our splunkbase docs:

This TA can be deployed to the indexer without any further changes.
It is recommended to create a new index called cacti. An indexes.conf file is not included with this TA.

The mirage_lookup_build search relies on the the scripted input that needs to run and send us sourcetype=cacti:lookup:mirage. Are you seeing any of that? If not, go to the forwarder running on Cacti and check $SPLUNK_HOME/etc/apps/Splunk_TA_Cacti/default/inputs

[script://./bin/cacti_lookup_mirage.py /usr/share/cacti]
source = cacti_lookup_mirage.py
disabled = true
index = cacti
sourcetype = cacti:lookup:mirage
interval = 0 6 * * *
#interval = 86400
0 Karma

aecruzp
Path Finder

What happens when you search with the macro in search and reporting app on the indexer?

  A: do not know the use of macros but I think the query would be like this:


sourcetype = cacti: lookup: mirage | cacti_index

warning Search Factory: Unknown search command 'index'.


what happens when you search index = cacti on the indexer?

A: There are currently events and can be viewed from the indexers like the search head index = cacti

can you get cli on the indexer and double check $ SPLUNK_HOME / etc / apps / Splunk_TA_Cacti / default / macros.conf and see what is in there?

R: being a cluster distribution, I only see that the following files are inside the route

$ SPLUNK_HOME / etc / apps / Splunk_TA_Cacti /
    local
    > eventtypes.conf
    metadata
    > local.meta

0 Karma

mattymo
Splunk Employee
Splunk Employee

you search the with the marco with backticks around the marco name - `cacti_index`- simply put this in your search bar, not after any pipes etc.

sounds like your index is good to go and you are receiving the mirage_poller_output.log properly. so that is good.

on your cluster members you should have $SPLUNK_HOME/etc/apps/Splunk_TA_Cacti/defaultthat contains the macro definition. it can be in local too if you want, but if you dont have a macros.conf in the Splunk_TA_Cacti app on the clustered indexers, that is your problem

0 Karma

aecruzp
Path Finder

with respect to just putting cacti_index in the search, splunk shows the information.

with respect to which the files are not in the indexer, but is it necessary that they be there? the app was deployed just so that the index = cacti was replicated to the 6 indexer ... or should only be deployed in the search head ?, the documentation only says that certain configurations must be created or modified depending on the need or UF or SH ..

with respect to the configurations from the Universal Forwarder, they are fine since the information is being indexed.

0 Karma

aecruzp
Path Finder

to perform a further test, the entire default folder is copied to an index and the error message continues to appear.

0 Karma

mattymo
Splunk Employee
Splunk Employee

so if you run the search using the macro, directly on an indexer, it works???

if the macro doesn't exist on the indexers, that seems to line up with the errors you are seeing, right??

I am not clear. Do you, or do you not have the macro defined on the indexers? based on the errors you showed, i have to assume you do not. And that needs to be fixed.

When you say you copied the default folder...you mean pushed it to all indexers from the master and did a rolling restart right??? The app, and splunk, are built in such away that most times you can copy the whole app and the pieces that need to work will work where they need to.

The UF is not fine if you dont have cacti:lookup:mirage events. you need to enable the input to get the lookup to build. (separate issue from your macro item)

Happy to get you sorted in a live chat if that would help. hit me up on slack (splk.it/slack)

0 Karma

aecruzp
Path Finder

Thanks for your reply.

   With respect to all the folders and distribution, it is known that to deploy configurations only must enter the master server path and apply bundle, we have not had problems with any other app, in the same way with the app in the deployer.

   With respect to the configurations of the macros are currently in all indexers, via web but not physically.

   The search head report errors of a file and missing configuration, which is in each machine as it was attached in the previous images.

   It should be noted that the installation and configuration work without any adjustment in a standalone server, not in a cluster.

   The version of splunk that we currently use is 6.6.1

0 Karma

mattymo
Splunk Employee
Splunk Employee

please run btool on your search head as well as your indexers and provide the output:

[splunker@n00bserver bin]$ ./splunk btool macros list cacti_index --debug
/home/splunker/splunk/etc/apps/Splunk_TA_Cacti/default/macros.conf [cacti_index]
/home/splunker/splunk/etc/apps/Splunk_TA_Cacti/default/macros.conf definition = index=cacti
/home/splunker/splunk/etc/apps/Splunk_TA_Cacti/default/macros.conf iseval = 0

In the meantime, as I advised, the macro is not mandatory. You can workaround it easily by updating the searches in the app. In fact, taking a quick look, only the "Cacti Spine Events" and "Cacti Spine Errors" panels on the "Polling & Lookup" status dashboard use them, so you can easily update those searches.

The macro was put in as a best practice, but most of the searches rely on eventtypes instead.

Regardless, I'd still like to help you find root cause. I will try and spin up a cluster and test.

I still think your cluster members are misconfigured somehow.

0 Karma

aecruzp
Path Finder

As requested, we have the following:

[root@splunk_searchhead03_cdlv bin]# ./splunk btool macros list cacti_index --debug
/home/splunk/splunk/etc/apps/Splunk_TA_Cacti/default/macros.conf [cacti_index]
/home/splunk/splunk/etc/apps/Splunk_TA_Cacti/default/macros.conf definition = index=cacti
/home/splunk/splunk/etc/apps/Splunk_TA_Cacti/default/macros.conf iseval = 0


with respect to the queries ... and if they are modified and reference is not made to the macro or eventype?

index = cacti <<<<<<<<<<<<<<<<<< show results
sourcetype = "cacti: lookup: mirage" <<<<<< shows results
| eval cacti = host
| dedup cacti, local_data_id
| rename hostname as ip
| rename description as hostname
| table cacti, local_data_id, name_cache, host_id, hostname, ip, data_source_type_id
| dedup cacti, local_data_id
| outputlookup cacti_lookup_mirage


eventtype = cacti: lookup: mirage <<<<<<<<< Errors 0 results
| eval cacti = host
| dedup cacti, local_data_id
| rename hostname as ip
| rename description as hostname
| table cacti, local_data_id, name_cache, host_id, hostname, ip, data_source_type_id
| dedup cacti, local_data_id
| outputlookup cacti_lookup_mirage

0 Karma

mattymo
Splunk Employee
Splunk Employee

where is your btool for the indexers??

yes you can change the searches like that and be totally fine, you need to update the eventtype to remove the macro from it. That used to work but doesnt anymore.

I will look at updating this app very soon and that will def be removed

Just update your eventtypes to be "index=cacti sourcetype=cacti: lookup: mirage" etc.

0 Karma

aecruzp
Path Finder

query in indexer

[root@splunk-indexer-01-cnt bin]# ./splunk btool macros list cacti_index --debug
/local1/index/etc/slave-apps/Splunk_TA_Cacti/default/macros.conf [cacti_index]
/local1/index/etc/slave-apps/Splunk_TA_Cacti/default/macros.conf definition = index=cacti
/local1/index/etc/slave-apps/Splunk_TA_Cacti/default/macros.conf iseval = 0

0 Karma

mattymo
Splunk Employee
Splunk Employee

hmm, looks good to me. not sure why you would be getting those errors only when sending a search from the SH to the indexers...that might be worth a case.

0 Karma

aecruzp
Path Finder

modifications were made to cacti_lookup_mirage_build and cacti: mirage events by cacti server, we will review the information with the requesting user and verify that the data shown is complete.

0 Karma

mattymo
Splunk Employee
Splunk Employee

yeah, as long as your cacti index contains the 3 sourcetypes:

cacti:mirage
cacti:system
cacti:lookup:mirage

then you have all the data you'll need.

mattymo
Splunk Employee
Splunk Employee

Hey @aecruzp any update on this??

0 Karma

aecruzp
Path Finder

hi.

    Well, in the end only the macro was modified referring to the index and everything is working well.

    Thanks for the help.

0 Karma

aecruzp
Path Finder

alt text

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!