Questions on best practices for a new Splunk environment

[hrithiktej]

Sorry for too many questions

This is our environment

6 Splunk servers

1) splunk01 – Ad HOC Search head used for standalone searches

47.14 GB Physical Memory, 10 CPU Cores

2) splunk02 – Enterprise Security Search Head has Enterprise Security app installed on it.

125.75 GB Physical Memory, 24 CPU Cores

3) splunk03 – Indexer – Syslog plus Indexer server

62.75 GB Physical Memory, 24 CPU Cores

4) splunk04 – Indexer – Syslog plus Indexer server

62.75 GB Physical Memory, 24 CPU Cores

Below two Splunk servers are on a host that has several other VMs hosted on it.

5) splunk05 – License Master plus Indexer cluster master

7.64 GB Physical Memory, 4 CPU Cores

6) splunk06 – Deployment Server

3.7 GB Physical Memory, 2 CPU Cores

Question 1) Our indexers 3&4 are also Syslog servers with HD of 5tb each is it a best practice to have Indexers and Syslog servers on the same box?

Question 2) Our License master with its current RAM and CPU config as stated above is it enough to be a License master?

Question 3) Since our Syslog and indexer reside on the same box does that mean our HFs don't play any role in forwarding data?

Question 4) Can we install DMC on our license master?

[jkat54 · Accepted solution]

Question 1) Our indexers 3&4 are also Syslog servers with HD of 5tb each is it a best practice to have Indexers and Syslog servers on the same box?

Answer 1) No, not really. You can do it but then you're going to decrease the available incoming network ports, add extra load, create a maintenance / patching/ upgrade nightmare, etc.

Question 2) Our License master with its current RAM and CPU config as stated above is it enough to be a License master?
Answer 2) Yes its enough

Question 3) Since our Syslog and indexer reside on the same box does that mean our HFs don't play any role in forwarding data?
Answer 3) there are very few use cases where HFs are needed. You can usually use a UF instead of HF for just about everything. Syslog will not address getting windows event logs into splunk for example... however a UF will.

Question 4) Can we install DMC on our license master?
Answer 4) http://docs.splunk.com/Documentation/Splunk/6.6.2/DMC/WheretohostDMC
It should go on your master node according to the documentation, which in your case, is the same as the license master.

[hrithiktej]

Hi Many thanks for your help I have separated the syslog server from the indexers now and have installed UFs on them to forward the data, I did ran into some problems but everything is cool now and the performance in terms of searches is a lot better.

[hrithiktej]

Thank you very much for your quick help. Sorry I am new to splunk

Can you elaborate on Question no. 1 I ask again because we have a lot of performance issues our searches are slow.

& Question 3 I read the doc thanks and I think below is our scenario

Distributed mode Yes
Indexer clustering yes
Search head clustering Not relevant
Monitoring Console options
The master node. If preferred, you can instead run the Monitoring Console on a dedicated search head not used for other purposes. So does this mean I should install DMC on our master server?

one more question
Question 5)
All our Router, Switches, FWs, forward data directly to our Syslog servers which are nothing but our indexers 3 & 4 but our estreamer i.e. Cisco IPS/Firepower manager forwards data to Splunk 02 server i.e. our Enterprise Security app search head now I want to know whether Splunk 02 does the indexing of data on its own or does it forward it to indexers 3 &4 for indexing and then request it back during searches.

[jkat54]

Put syslog on dedicated servers = best practice.

So don't do what you're doing right now. Build new servers for syslog and put universal forwarders on them to send the data to Splunk indexers.

Question 3: you said your cluster master is your license master... so there's no difference but yes it is supposed to be on the cluster master in your case.

Question 5) you tell me the answer. Do you have forwarding enabled on server 2? If so, then the data is forwarding to whatever you've configured unless the inputs have indexAndForward enabled.

[hrithiktej]

Thanks a lot once again. for Question 5: Under settings > Data > Forwarding $ Receiving > Forward data > Configure forwarding > I do see names of my indexers:9997 (splunk03:9997 & splunk04:9997) status as enabled which means it is forwarding I guess. Sorry but I did not get your statement "unless the inputs have indexAndForward enabled" how do I check this index & forward?

[jkat54]

Index and forward is an outputs.conf setting. I mis-spoke when I say inputs.

https://docs.splunk.com/Documentation/Splunk/6.6.3/Admin/Outputsconf

To check if it's enabled you can use btool

./splunk btool outputs list

[hrithiktej]

Thank you, from the below output I guess it is just forwarding to 3 & 4 and not doing indexing as I see index = false.

[root@splunk02 bin]# ./splunk btool outputs list
[indexAndForward]
index = false
[syslog]
dropEventsOnQueueFull = -1
maxEventSize = 1024
priority = <13>
type = udp
[tcpout]
ackTimeoutOnShutdown = 30
autoLBFrequency = 30
blockOnCloning = true
blockWarnThreshold = 100
compressed = false
connectionTimeout = 20
defaultGroup = primary_indexers
disabled = false
dropClonedEventsOnQueueFull = 5
dropEventsOnQueueFull = -1
forceTimebasedAutoLB = false
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_internal|_introspection)
forwardedindex.filter.disable = false
heartbeatFrequency = 30
indexAndForward = false
maxConnectionsPerIndexer = 2
maxFailuresPerInterval = 2
maxQueueSize = 7MB
readTimeout = 300
secsInFailureInterval = 1
sendCookedData = true
sslQuietShutdown = false
tcpSendBufSz = 0
useACK = true
writeTimeout = 300
[tcpout:primary_indexers]
server = splunk03:9997, splunk04. :9997

I am wondering why my prev admin chose to forward data from estreamer to splunk02 i.e. our enterprise security server rather than directly to the indexers.
Is there any added benefit for forwarding estreamer to enterprise security Splunk rather than indexers first?

[jkat54]

The cisco apps can be a bit special at times... you should probably open a new question on that.

[hrithiktej]

ok, sure I will thank you very much for all your help very grateful.