Splunk Enterprise

What data is sent back to Search Head in a Distributed Environment?

b_chris21
Communicator

Hello,

I am administrating a distributed environment with 1 Search Head and 10 peers. Something special is that communication is established via a satellite therefore the bandwidth is limited.

Search Head has Splunk Enterprise Security installed and is a deployment server.

Peers have the indexer role and all ingest Suricata IDS logs, while only one of them also ingests Windows Logs.

I have measured that 3GB per day is the size of data exchanged between Search Head and Indexers, which seems quite a lot to me.

Can someone please explain me what kind of data is transferred by default in a distributed environment?

Some things to note:

1. Notable index and internal logs are stored locally in Search Head and not forwarded to peers.

2. Replication bundle is 16M

Thank you in advance.

With kind regards,

Chris

Labels (2)
Tags (1)
0 Karma
1 Solution

isoutamo
SplunkTrust
SplunkTrust

Hi

if you have splunk 8.2.x you can try to look "Job Details Dashboard" via Job inspector. There are some statistics which you could use when you are doing estimations how much data has transferred between different instances and layers. With older versions you can try to found that information from search.log. Unfortunately I cannot found any exact fields which told this.

r. Ismo

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

All search queries and search results are sent between search heads and indexers.  The more you search, the more data is exchanged.  The less efficient the searches, the more data is returned from the peers.

Windows logs tend to be verbose so they can run up the size of the results.

If the peers are clients of the DS then additional data is transferred when the peers phone home every few minutes, plus the size of the apps they download and install.

I should take this opportunity to point out some architectural "quirks" in the described environment.

  • ES is supposed to run on a dedicated search head.
  • ES and DS should not be on the same instance.
  • For better search performance, the Windows logs should be ingested on all indexers.
  • Search heads should forward their logs to the indexers.
---
If this reply helps you, Karma would be appreciated.

b_chris21
Communicator

Thanks for your reply @richgalloway .

How can I see every single sourcetype that is transferred between my indexers (deployment clients) and my search head (deployment server) split by host and total size in GBs?

I would like preferably to see the data transferred from both sides. I mean sourcetypes and size of data transferred from indexers to search head and vice versa.

Does the data travel compressed?

Many thanks in advance.

Christos

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

if you have splunk 8.2.x you can try to look "Job Details Dashboard" via Job inspector. There are some statistics which you could use when you are doing estimations how much data has transferred between different instances and layers. With older versions you can try to found that information from search.log. Unfortunately I cannot found any exact fields which told this.

r. Ismo

b_chris21
Communicator

@isoutamo 

In Search Job Properties under diskUsage I found exactly what I needed. Great tip!

Thank you both @richgalloway and @isoutamo  🙂

Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...