How to store or index data from multiple clients that have multiple servers?

[makincerdas]

Hello.

First time I'm posting a question, and a relative new to Splunk so I apologize up front if this has already been asked and answered, or if this is a silly question. We are planning to use Splunk for log monitoring.

Scenario:
Let's say we have 3 clients (A,B,C) to simplify the situation (in real situation we have more than 3 clients).
Each client has 20 servers to monitor.
Currently, when problem happens on Client A, we manually login to every server that belong to Client A and start checking logs.
We want to use Splunk to help us to speed up investigation in finding issue.

Question:
How should I setup Splunk to segment or partition logs?
logs from Client A are stored and indexed as Client-A logs
logs from Client B are stored and indexed as Client-B logs.
logs from Client C are stored and indexed as Client-C logs.

Reason: When issue happens on Client-A, I want to view and analyze logs from Client-A only.
I don't want to see logs from Client-B and Client-C.

Thank you in advance.

[lukejadamec · Accepted solution]

With Splunk, each server would be identified by the server name, so if you wanted to look at just one server, then you simply enter a search that only looks at that one server.
With Splunk, you can create groups of servers a number of different ways to isolate your Client groups, and search on all groups, or just one group.
Once set up, all logs from all servers are collected in one location in pretty much real time, so there is no need to log into each server.

[makincerdas]

Which high level diagram is suitable for this kind implementation? (give each cluster it's own index, or set of indexes)

UF : Universal Forwarder
IDX : Indexer
LB : Load Balancer
SH : Search Head

Diagram 1 (2 LB)

A-UF---\     /---A-IDX----\
A-UF----\   /----A-IDX-----\

B-UF------L----- B-IDX-------L------S
B-UF------B----- B-IDX-------B------H

C-UF----/   \----C-IDX-----/
C-UF---/     \---C-IDX----/

Diagram 2 (4 LB)

A-UF---\     /---A-IDX-----L---\
A-UF----\   /----A-IDX-----B----\

B-UF------L------B-IDX-----L------S
B-UF------B------B-IDX-----B------H

C-UF----/   \----C-IDX-----L----/
C-UF---/     \---C-IDX-----B---/

Diagram 3 (4 LB)

A-UF------L------A-IDX---\
A-UF------B------A-IDX----\
                           \
B-UF------L------B-IDX-----L------S
B-UF------B------B-IDX-----B------H
                           /
C-UF------L------C-IDX----/
C-UF------B------C-IDX---/

Diagram 4 (6 LB)

A-UF-----L-----A-IDX-----L---\
A-UF-----B-----A-IDX-----B----\
                               \
B-UF-----L-----B-IDX-----L------S
B-UF-----B-----B-IDX-----B------H
                               /
C-UF-----L-----C-IDX-----L----/
C-UF-----B-----C-IDX-----B---/

[lukejadamec]

Well, it really depends on the scale. I can do this with one indexer and no forwarders monitoring a very complex system with different groups of logs going to different indexes on one piece of powerful hardware, or I can add load balancers, multiple indexers, and separate search heads for large enterprise deployments.
What you will need will depend on network speed, log volume, server capability, number of users accessing search, and the like.
From what you have described, I could have all servers report to one indexer that also serves as a search head, provided the network speed was good, the volume was within tolerance, and the indexer hardware was good enough to provide both indexing and searching capability.
You should really take this up with "Splunk the Company" to plan a large deployment.

[lukejadamec]

Also, this is Splunk Answers, not a chat line. You ask a question, get an answer, accept the answer, ask a new question, etc....
Let's try to keep Splunk Answers in focus and on point. It makes it more valuable in the long run for everyone.

[makincerdas]

Thank you @lukejadamec for your help.

[makincerdas]

Is this below what you mean by...Give each cluster it's own index, or set of indexes?

Client A : Servers for Client A has forwarder set to Cluster-Indexer-A (let's say 2 Indexer machines)
Client B : Servers for Client B has forwarder set to Cluster-Indexer-B (let's say 2 Indexer machines)
Client C : Servers for Client C has forwarder set to Cluster-Indexer-C (let's say 2 Indexer machines)

So when issue happens to Client, I can do the search, the search head will go directly to Cluster-Indexer-A.
I see similiar approach on this link:
https://answers.splunk.com/answers/8226/selective-indexing-and-forwarding-based-on-source.html

Am I understanding correctly here?

Can I still have ONE big cluster of Index consists of Indexer-A, Indexer-B, Indexer-C (total 6 machines)?

[lukejadamec]

Basically yes. I do this exact thing. Here are some examples of how you would construct the searches you mentioned if each cluster was given it's own index. Note: the index names cannot include dashes, so I changed them to underscores. FYI, these are the exact search string that you would enter into Splunk:
To search all logs on all servers from Cluster-Indexer-A:

index=Cluster_Indexer_A

To search all logs on all servers from all Clusters:

index=Cluster_Indexer_*

[makincerdas]

lukejadamec,

I appreciate your quick response.
Can you provide one example on how to create groups of servers on Splunk?
Is this done on Indexer or on Forwarder?
Where can I find document about creating groups on Splunk to isolate Client groups?

Thank you.

[lukejadamec]

Actually, @ddrillic posted an example for doing this with the Calculated Fields UI yesterday in this answer:
https://answers.splunk.com/answers/453481/splunk-calculated-fields.html#answer-453486

Another way to do it would be give each cluster it's own index, or set of indexes.

The Calculated Field solution would be a search time solution configured on the search head. The index solution would separate things at index time and is configured on the forwarder that is monitoring the logs.