Setting up a Splunk Dev Environment

[Abass42]

So i have three servers in this Splunk infrastructure, a SH, an Indexer, and a forwarder. I have installed the free 10gb dev license as well as the 50Gb one, and am not using clustering anywhere. 

I have installed and followed this guide to send test data to our boxes. https://splunkbase.splunk.com/app/1924

I set the app up on the forwarder, and i can see the data in the index i created, testindex on the indexer. I can view the sample data. I cannot however, view the data from the SH. 

My problem rn is I can not find what I am missing. I have looked everywhere and cant figure it out. I have confirmed my server.conf, distsearch.conf, the outputs.conf on the forwarder, I have made the pass4symm keys on all machines similar, I can ping each server from one another, so connection is good.

What else can i check? Most of the splunk docs i see are for clustered env, and am struggling to find relevant docs. 

I have set the SH to be the License Master, and both machines point to the SH as License Manager, yet on the SH, I do not see any instance other than itself to be the indexer. 

When i go to add a new pool, like i see it on our DMC, i can only add itself as available indexers

On our production DMC, we have all of the indexers listed. I should be seeing the indexer or something showing up somewhere within the SH, but i dont see any mention anywhere. Checking _internal Logs, I just see its own Hostname Mentioned. Im having issues figuring out where im going wrong. The SH should see the indexer based on my findings and set up. 

Any help or guidance would be appreciated. Thank you. 

[deepakc]

It sounds like you have:

1. You have a SH (Can't Search Data)
2. You have an Indexer
3. A UF which is sending eventgen data to the indexer to your index and you have verified this is working and can see data via CLI I suspect.
4. The SH is also acting a License Manager (Therefore the indexer must point to the License manager)

Try the below steps and see if that fixes it.

#Add the Indexer to your SH
On the SH via the GUI
Go to Settings- Distributed search » Search peers » Add new
Normally its something like https://MY_INDEXER:8089
Add your admin and password
Restart Splunk

#Add the Indexer to the Licence Manager as a Licence Peer
From the Indexer GUI > Settings > Licensing > Change to Peer Point to the Licence Manager
https://MY_LICENCE_MANAGER:8089 (This is also your SH)
Restart Splunk

[Abass42]

That's what I was thinking as well. But when I go to the Distributed Search, I get this message, like i cant add anything

And for ONE SH, that is def allowed with the free license. Ive reached out to our splunk rep to ask about the license. 

Thanks for any help

[Abass42]

After a bit of work, I made the indexer the License Master. I already wanted the SH to also server as the DMC, and im not sure what was happening, but i made the indexer the License master, confirmed a few settings, and I was able to add a new search peer. That window now allows me to see search peers under Distributed peers:

Not entirely sure what the problem was. 

But in this instance, I am trying to get one indexer, one SH, and one forwarder working. i made the indexer the License master, the forwarder just a forwarder, and hopefully the SH as a SH and DMC.

Thanks for the help. 

[deepakc]

It's most likely some config setting under the hood as you should be able to use the SH as licence manager. 

Not saying you did this, but did you set the distsearch.conf manually ? 

If you look here it states 

You must specify the non-clustered search peers through either Splunk Web or the CLI. Due to authentication issues, you cannot specify the search peers by directly editing distsearch.conf. When you add a search peer with Splunk Web or the CLI, the search head prompts you for public key credentials. It has no way of obtaining those credentials when you add a search peer by directly editing distsearch.conf

https://docs.splunk.com/Documentation/Splunk/9.2.1/Indexer/Configureclusteredandnonclusteredsearch 

[Abass42]

I was testing out a lot of different things. I know I def did edit the distsearch manually. I did most everything from the CLI. Redoing and moving the License Manager through the GUI fixed some of the issues, as i can now search the data. 

Thanks

[deepakc]

Good its working and yes lots of moving parts / configs and scenarios with Splunk. 

[Abass42]

Just to follow up with what my problem was, I had a license set for an individual instance. I thought distributed meant multiple instances of each type of Splunk Server, ie, multiple indexers, SH, forwarders, etc. I didnt realize one SH, one Indexer, and one Forwarder counted as a distributed. Either way, putting the 10 GB/day distributed license did the trick.

Now dev works 🙂 

[PickleRick]

Yes. Dev and trial licenses are for single instance installations only. If you try to set up multiple servers in your setup with the same license you'll get errors and/or warnings about not-working functionalities or duplicate license keys, depending on your architecture.