Getting Data In

Why can't I see the indexes I created in search results?

Lorenzo1
Path Finder

hi pls am having problem viewing the indexes i created in my clustered environment. They were all created on the cluster manager ..._cluster and also same on the deployer but when i try to search them i don't get to see any of them. When i tried to check the indexer GUI i see them under indexes but not on the seacrhhead GUI. What am i doing wrong?

Also i installed a TA (add-on for unix and linux) and tried to use one of the monitor stanza as a input on the DS; yet still not working. My serverclasses are fine. Below is the stanza i copied from the TA which i used in my inputs.conf in the local folder of the TA under deployment apps. Kindly assist. Thanxx 

Labels (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

As a general rule - you don't see indexes on the search-heads. And you shouldn't.

If you have a multi-layer architecture (separate SHs, separate indexers), search-heads don't typically hold any indexes locally (they might have _internal index but in a well-architected environment the events destined for _internal should be pushed to the indexer layer). So you don't see any indexes in the webUI.

It makes even more sense if you take into account that a single SH (or SH cluster) can search multiple indexers (or indexer clusters) holding different sets of indexes.

So managing indexes from the webUI makes sense only in case of an all-in-one installation.

With more advanced architecture you need to use rest, tstats or dbinspect, depending on the information you want to get.

Lorenzo1
Path Finder

@PickleRickso what i meant was that i don't see those indexes i created on the cluster master when i try to search for them using any of the SH. Also can you explain to me (am confused at this concept) why when configuring input on the deployment server, how and why they copy input monitor stanza from somewhere. I simply fail to understand that from the videos i've watched on youtube. Will appreciate pls. Thanxx

0 Karma

PickleRick
SplunkTrust
SplunkTrust

OK. There are several separate things here 🙂

1. You don't see the indexes from the indexers on the search-heads in configuration part. You simply don't. Since you can't manage them from indexers, it doesn't make sense to show them there. And since you can have single search head (or search head cluster) search across many separate indexers and/or indexer clusters, those indexers can have different indexes defined on them.

2. Just because an index is defined, it doesn't mean there's anything searchable in that index. So if you create and index called - for example - indexXYZ and do a search with

index=indexXYZ earliest=0

will still show 0 results unless you sent something to that index.

3. There are three separate deployment mechanisms within splunk environment.

- deployer - pushes configuration to search-head clusters

- cluster manager (formerly cluster master) - pushes configuration bundle to indexer cluster members

- deployment server - serves apps to deployment clients which connect to it and pull their associated apps - typically it's used for distributing configuration to forwarders but can also be used for stand-alone indexers and search-head as well as cluster managers and deployers in more advanced configurations.

Lorenzo1
Path Finder

thanks alot @PickleRick . i fully understand all you wrote. Now for no. 2, what of a situation whereby am not able to see "indexXYZ" at all when trying to search for it even though am aware of the fact that there’s no data in it yet. Why will that be ? Cos i know also created it and i can see it in indexes.conf on the CM. Thanxx bro for ur time. I appreciate .

0 Karma

PickleRick
SplunkTrust
SplunkTrust

As I said before - you must send some events to an index if you want splunk to find something. Otherwise the "0 results" result will not indicate lack of index but simply will show you that there are no events in that index.

It will most probably even show as a result of dbinspect command since you haven't written anything to it.

It should however show in a response to

| rest splunk_server=* /services/data/indexes 
| table splunk_server title
| where title="my_index

Along with the names of the servers on which it has been defined. Run it from the monitoring console.

Lorenzo1
Path Finder

ok am gonna try this and revert. Thanks 

0 Karma

Lorenzo1
Path Finder

i forgot to add this:

[monitor:///var/log]

whitelist=(\.log|log$|messages|secure|auth|mesg$|cron$|acpid$|\.out)

blacklist=(lastlog|anaconda\.syslog)

disabled = 0

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Lorenzo1,

let me understand: you want to create an index  (e.g. "os") on an Indexer Cluster and you want to use it to send logs from some other servers using a TA like Splunk_TA_nix.

The correct approach should be:

  • create indexes.conf on the Master Node in _cluster or (better) in a dedicated TA (called e.g. TA_indexers),
  • Configure the Deployment Server (and the other clients) to send logs to indexers by GUI or using outputs.conf
  • install Spluk_TA_nix on the DS to monitor the DS itself, you can do it by GUI o by SSH,
  • put the TA in the $SPLUNK_HOME/etc/deployment-apps folder to deploy this app to other servers using ServerClasses.conf.

To deploy a TA using  the DS you don't need to install it on DS but only to put it the $SPLUNK_HOME/etc/deployment-apps folder and create servercalsses.conf.

Anyway, you don't need to create index also on the clients; sometimes it could be useful doing it only to be able to see in in some GUI feature (e.g. the Add Data feature).

Ciao.

Giuseppe

Lorenzo1
Path Finder

 

attached are the indexes i created in _cluster on the CM. but when i try to search them on the SH i do not see them. The only one i see is the linux index because i also created it on the SH GUI.

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Lorenzo1,

on Cluster Master it's the same thing: it's always better to put conf files in a dedicated Add-On in $SPLUNK_HOME/etc/manager-apps than in _cluster.

Ciao.

Giuseppe

0 Karma

Lorenzo1
Path Finder

hi @gcusello,

thanks alot for your reply. so i did everything like you instructed but i also installed the TA on the cluster master. does it affect anything? and if it does, how do i uninstall it? and also how do i know what files to monitor. am new to splunk that's why am trying to get my footing. thank you.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Lorenzo1,

if you installed te Splunk_TA_nix on the Master Node is relevant only to index Master Node itself logs, but it isn't relevant for Indexers deployment.

To deploy an app or a configuration to Indexers, you have to follow the instructions at https://docs.splunk.com/Documentation/Splunk/9.0.2/Indexer/Manageappdeployment 

in few words, you have to copy the untared apps to deploy in $SPLUNK_HOME/etc/manager-apps and push them by GUI.

As i said, it is prefereable to put indexes.conf in a dedicated Add-On han in _cluster, there's no difference in performances or features it's only better to have an organized installation.

Ciao.

Giuseppe

0 Karma

Lorenzo1
Path Finder

hi @gcusello

please can you explain the 6th line on the attached screenshot with some examples.

"Install the app or add-on that contains the inputs you want."

Thanks a lot.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Lorenzo1,

inputs are usually contained in Technical Add-Ons (TA), in your case the Splunk_TA_nix.

Ciao.

Giuseppe

Lorenzo1
Path Finder

ok i think i understand it now. I must always use add-on for configuring inputs cos i was wandering where they keep getting the input stanzas that they copy all the time. So for every type of input i must have to look for a TA for it? i appreciate your time . Thanks a lot .

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Lorenzo1,

yes, it's always a best practive to have inputs in an Add-On and I always prefer to use Add-on from Splunkbase than custom, because events should be normalized for using in following apps (as ES).

In other words, never put conf files in system/local always in an app, possibly standard eventually with custom inputs.conf.

Ciao.

Giuseppe

0 Karma

Lorenzo1
Path Finder

Ok Guiseppe, i appreciate. Can you send me a link or tell me how to install a dedicated TA. Thanxx

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Lorenzo1,

as described in the above link to documentation, to install an App in a Search peer per an Indexer Cluster, you have to:

  • untar and put it into the $SPLUNK_HOME/etc/manager-apps folder,
  • go in the GUI at [Settings > Indexer Clustering] and Push the configuration.

If instead you want to install the Add-On on the Master Node itself, you can use the usual procedure via GUI.

Ciao.

Giuseppe

0 Karma

blbr123
Path Finder

Check if that index is associated to any splunk role if yes check if you have access to that role then you can see that index while you search may be 

One more to check is there any data sent to that index

 

 

 

 

Lorenzo1
Path Finder

thanks @blbr123 

yes i ve not been able to send any data into the indexes but they're not even showing up on the search assistant when am typing "index=...". The only one that shows up is the linux index which bcos ive created that one on the search head gui but then i tried to point to it using the TA_nix that i installed on the DS but still wasnt able to fetch any data.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...