Splunk Enterprise

What data is sent back to Search Head in a Distributed Environment?

b_chris21
Communicator

Hello,

I am administrating a distributed environment with 1 Search Head and 10 peers. Something special is that communication is established via a satellite therefore the bandwidth is limited.

Search Head has Splunk Enterprise Security installed and is a deployment server.

Peers have the indexer role and all ingest Suricata IDS logs, while only one of them also ingests Windows Logs.

I have measured that 3GB per day is the size of data exchanged between Search Head and Indexers, which seems quite a lot to me.

Can someone please explain me what kind of data is transferred by default in a distributed environment?

Some things to note:

1. Notable index and internal logs are stored locally in Search Head and not forwarded to peers.

2. Replication bundle is 16M

Thank you in advance.

With kind regards,

Chris

Labels (2)
Tags (1)
0 Karma
1 Solution

isoutamo
SplunkTrust
SplunkTrust

Hi

if you have splunk 8.2.x you can try to look "Job Details Dashboard" via Job inspector. There are some statistics which you could use when you are doing estimations how much data has transferred between different instances and layers. With older versions you can try to found that information from search.log. Unfortunately I cannot found any exact fields which told this.

r. Ismo

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

All search queries and search results are sent between search heads and indexers.  The more you search, the more data is exchanged.  The less efficient the searches, the more data is returned from the peers.

Windows logs tend to be verbose so they can run up the size of the results.

If the peers are clients of the DS then additional data is transferred when the peers phone home every few minutes, plus the size of the apps they download and install.

I should take this opportunity to point out some architectural "quirks" in the described environment.

  • ES is supposed to run on a dedicated search head.
  • ES and DS should not be on the same instance.
  • For better search performance, the Windows logs should be ingested on all indexers.
  • Search heads should forward their logs to the indexers.
---
If this reply helps you, Karma would be appreciated.

b_chris21
Communicator

Thanks for your reply @richgalloway .

How can I see every single sourcetype that is transferred between my indexers (deployment clients) and my search head (deployment server) split by host and total size in GBs?

I would like preferably to see the data transferred from both sides. I mean sourcetypes and size of data transferred from indexers to search head and vice versa.

Does the data travel compressed?

Many thanks in advance.

Christos

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

if you have splunk 8.2.x you can try to look "Job Details Dashboard" via Job inspector. There are some statistics which you could use when you are doing estimations how much data has transferred between different instances and layers. With older versions you can try to found that information from search.log. Unfortunately I cannot found any exact fields which told this.

r. Ismo

b_chris21
Communicator

@isoutamo 

In Search Job Properties under diskUsage I found exactly what I needed. Great tip!

Thank you both @richgalloway and @isoutamo  🙂

Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...