After deploying the Splunk App for Unix and Linux in our search head clustering environment, why are no dashboards populated with data?

[msudhindra]

Hi,

We have a search head cluster, consisting of 3 nodes.

I have followed the instructions to deploy the Splunk App for Unix and Linux correctly, and deployed the same across our infrastructure.
The problem we have is that none of the dashboards populate.

All the dashboards come up blank.

Any idea on what the possible issue could be?

Thanks in advance,

Madan Sudhindra

[masonmorales]

A few things to check:

1. Are the scripted inputs enabled?
2. Are they set to executable (i.e. chmod +x)?
3. Does the Linux account running your Splunk instance have the permissions necessary to execute those scripts?

[msudhindra]

Hi @masonmorales

I have checked all these items and they are indeed the way they are supposed to be.

Splunk services are running as root, so privileges should not be an issue.

The weird thing is that if I try accessing app on a standalone search-head (which is part of the cluster, by accessing the search head directly using the hosts's URL), the dashboards populate fine. Just the cluster URL (which goes through a load balancer), does not seem to want to populate the dashboards.

Also, in a SH cluster scenario, how are the Unix groups populated ? I created a group "All Unix Hosts" and added all my *nix hosts to that group, but as I connect to different nodes in the search-head, the group definition seems to disappear. It appears that this information is not replicated amongst the different search-head nodes in the cluster.

Thanks,
Madan Sudhindra

[msudhindra]

On the part where the groups created on one search-head node not propagating to the other nodes in the cluster, the issue is logged as a BUG. Bug ID: SPL-98866

[abrarfakhri]

So this means we should make all the config changes on one search-head. Then manually tar gzip / gunzip on the other search-heads in the cluster - correct?

I think putting the newly configured app dir can be placed back on cluster master and we can deploy the bundle.

Can someone with more expertise at this confirm which of the above scenarios is preferred?

Also, confirm which of the following will need to be redeployed:
SA-nix
splunk_app_for_nix
Splunk_TA_nix

Regards,
Abrar