Why can't I see the indexes I created in search results?

[Lorenzo1]

hi pls am having problem viewing the indexes i created in my clustered environment. They were all created on the cluster manager ..._cluster and also same on the deployer but when i try to search them i don't get to see any of them. When i tried to check the indexer GUI i see them under indexes but not on the seacrhhead GUI. What am i doing wrong?

Also i installed a TA (add-on for unix and linux) and tried to use one of the monitor stanza as a input on the DS; yet still not working. My serverclasses are fine. Below is the stanza i copied from the TA which i used in my inputs.conf in the local folder of the TA under deployment apps. Kindly assist. Thanxx 

[PickleRick]

As a general rule - you don't see indexes on the search-heads. And you shouldn't.

If you have a multi-layer architecture (separate SHs, separate indexers), search-heads don't typically hold any indexes locally (they might have _internal index but in a well-architected environment the events destined for _internal should be pushed to the indexer layer). So you don't see any indexes in the webUI.

It makes even more sense if you take into account that a single SH (or SH cluster) can search multiple indexers (or indexer clusters) holding different sets of indexes.

So managing indexes from the webUI makes sense only in case of an all-in-one installation.

With more advanced architecture you need to use rest, tstats or dbinspect, depending on the information you want to get.

[Lorenzo1]

@PickleRickso what i meant was that i don't see those indexes i created on the cluster master when i try to search for them using any of the SH. Also can you explain to me (am confused at this concept) why when configuring input on the deployment server, how and why they copy input monitor stanza from somewhere. I simply fail to understand that from the videos i've watched on youtube. Will appreciate pls. Thanxx

[PickleRick]

OK. There are several separate things here 🙂

1. You don't see the indexes from the indexers on the search-heads in configuration part. You simply don't. Since you can't manage them from indexers, it doesn't make sense to show them there. And since you can have single search head (or search head cluster) search across many separate indexers and/or indexer clusters, those indexers can have different indexes defined on them.

2. Just because an index is defined, it doesn't mean there's anything searchable in that index. So if you create and index called - for example - indexXYZ and do a search with

index=indexXYZ earliest=0

will still show 0 results unless you sent something to that index.

3. There are three separate deployment mechanisms within splunk environment.

- deployer - pushes configuration to search-head clusters

- cluster manager (formerly cluster master) - pushes configuration bundle to indexer cluster members

- deployment server - serves apps to deployment clients which connect to it and pull their associated apps - typically it's used for distributing configuration to forwarders but can also be used for stand-alone indexers and search-head as well as cluster managers and deployers in more advanced configurations.

[Lorenzo1]

thanks alot @PickleRick . i fully understand all you wrote. Now for no. 2, what of a situation whereby am not able to see "indexXYZ" at all when trying to search for it even though am aware of the fact that there’s no data in it yet. Why will that be ? Cos i know also created it and i can see it in indexes.conf on the CM. Thanxx bro for ur time. I appreciate .

[PickleRick]

As I said before - you must send some events to an index if you want splunk to find something. Otherwise the "0 results" result will not indicate lack of index but simply will show you that there are no events in that index.

It will most probably even show as a result of dbinspect command since you haven't written anything to it.

It should however show in a response to

| rest splunk_server=* /services/data/indexes 
| table splunk_server title
| where title="my_index

Along with the names of the servers on which it has been defined. Run it from the monitoring console.

[Lorenzo1]

ok am gonna try this and revert. Thanks 

[Lorenzo1]

i forgot to add this:

[monitor:///var/log]

whitelist=(\.log|log$|messages|secure|auth|mesg$|cron$|acpid$|\.out)

blacklist=(lastlog|anaconda\.syslog)

disabled = 0

[gcusello]

Hi @Lorenzo1,

let me understand: you want to create an index  (e.g. "os") on an Indexer Cluster and you want to use it to send logs from some other servers using a TA like Splunk_TA_nix.

The correct approach should be:

• create indexes.conf on the Master Node in _cluster or (better) in a dedicated TA (called e.g. TA_indexers),
• Configure the Deployment Server (and the other clients) to send logs to indexers by GUI or using outputs.conf
• install Spluk_TA_nix on the DS to monitor the DS itself, you can do it by GUI o by SSH,
• put the TA in the $SPLUNK_HOME/etc/deployment-apps folder to deploy this app to other servers using ServerClasses.conf.

To deploy a TA using  the DS you don't need to install it on DS but only to put it the $SPLUNK_HOME/etc/deployment-apps folder and create servercalsses.conf.

Anyway, you don't need to create index also on the clients; sometimes it could be useful doing it only to be able to see in in some GUI feature (e.g. the Add Data feature).

Ciao.

Giuseppe

[Lorenzo1]

attached are the indexes i created in _cluster on the CM. but when i try to search them on the SH i do not see them. The only one i see is the linux index because i also created it on the SH GUI.

[gcusello]

Hi @Lorenzo1,

on Cluster Master it's the same thing: it's always better to put conf files in a dedicated Add-On in $SPLUNK_HOME/etc/manager-apps than in _cluster.

Ciao.

Giuseppe

[Lorenzo1]

hi @gcusello,

thanks alot for your reply. so i did everything like you instructed but i also installed the TA on the cluster master. does it affect anything? and if it does, how do i uninstall it? and also how do i know what files to monitor. am new to splunk that's why am trying to get my footing. thank you.

[gcusello]

Hi @Lorenzo1,

if you installed te Splunk_TA_nix on the Master Node is relevant only to index Master Node itself logs, but it isn't relevant for Indexers deployment.

To deploy an app or a configuration to Indexers, you have to follow the instructions at https://docs.splunk.com/Documentation/Splunk/9.0.2/Indexer/Manageappdeployment 

in few words, you have to copy the untared apps to deploy in $SPLUNK_HOME/etc/manager-apps and push them by GUI.

As i said, it is prefereable to put indexes.conf in a dedicated Add-On han in _cluster, there's no difference in performances or features it's only better to have an organized installation.

Ciao.

Giuseppe

[Lorenzo1]

hi @gcusello

please can you explain the 6th line on the attached screenshot with some examples.

"Install the app or add-on that contains the inputs you want."

Thanks a lot.

[gcusello]

Hi @Lorenzo1,

inputs are usually contained in Technical Add-Ons (TA), in your case the Splunk_TA_nix.

Ciao.

Giuseppe

[Lorenzo1]

ok i think i understand it now. I must always use add-on for configuring inputs cos i was wandering where they keep getting the input stanzas that they copy all the time. So for every type of input i must have to look for a TA for it? i appreciate your time . Thanks a lot .

[gcusello]

Hi @Lorenzo1,

yes, it's always a best practive to have inputs in an Add-On and I always prefer to use Add-on from Splunkbase than custom, because events should be normalized for using in following apps (as ES).

In other words, never put conf files in system/local always in an app, possibly standard eventually with custom inputs.conf.

Ciao.

Giuseppe

[Lorenzo1]

Ok Guiseppe, i appreciate. Can you send me a link or tell me how to install a dedicated TA. Thanxx

[gcusello]

Hi @Lorenzo1,

as described in the above link to documentation, to install an App in a Search peer per an Indexer Cluster, you have to:

• untar and put it into the $SPLUNK_HOME/etc/manager-apps folder,
• go in the GUI at [Settings > Indexer Clustering] and Push the configuration.

If instead you want to install the Add-On on the Master Node itself, you can use the usual procedure via GUI.

Ciao.

Giuseppe

[blbr123]

Check if that index is associated to any splunk role if yes check if you have access to that role then you can see that index while you search may be 

One more to check is there any data sent to that index

[Lorenzo1]

thanks @blbr123 

yes i ve not been able to send any data into the indexes but they're not even showing up on the search assistant when am typing "index=...". The only one that shows up is the linux index which bcos ive created that one on the search head gui but then i tried to point to it using the TA_nix that i installed on the DS but still wasnt able to fetch any data.