What data is sent back to Search Head in a Distributed Environment?

[b_chris21]

Hello,

I am administrating a distributed environment with 1 Search Head and 10 peers. Something special is that communication is established via a satellite therefore the bandwidth is limited.

Search Head has Splunk Enterprise Security installed and is a deployment server.

Peers have the indexer role and all ingest Suricata IDS logs, while only one of them also ingests Windows Logs.

I have measured that 3GB per day is the size of data exchanged between Search Head and Indexers, which seems quite a lot to me.

Can someone please explain me what kind of data is transferred by default in a distributed environment?

Some things to note:

1. Notable index and internal logs are stored locally in Search Head and not forwarded to peers.

2. Replication bundle is 16M

Thank you in advance.

With kind regards,

Chris

[richgalloway]

All search queries and search results are sent between search heads and indexers.  The more you search, the more data is exchanged.  The less efficient the searches, the more data is returned from the peers.

Windows logs tend to be verbose so they can run up the size of the results.

If the peers are clients of the DS then additional data is transferred when the peers phone home every few minutes, plus the size of the apps they download and install.

I should take this opportunity to point out some architectural "quirks" in the described environment.

• ES is supposed to run on a dedicated search head.
• ES and DS should not be on the same instance.
• For better search performance, the Windows logs should be ingested on all indexers.
• Search heads should forward their logs to the indexers.

[b_chris21]

Thanks for your reply @richgalloway .

How can I see every single sourcetype that is transferred between my indexers (deployment clients) and my search head (deployment server) split by host and total size in GBs?

I would like preferably to see the data transferred from both sides. I mean sourcetypes and size of data transferred from indexers to search head and vice versa.

Does the data travel compressed?

Many thanks in advance.

Christos

[isoutamo · Accepted solution]

Hi

if you have splunk 8.2.x you can try to look "Job Details Dashboard" via Job inspector. There are some statistics which you could use when you are doing estimations how much data has transferred between different instances and layers. With older versions you can try to found that information from search.log. Unfortunately I cannot found any exact fields which told this.

r. Ismo

[b_chris21]

@isoutamo 

In Search Job Properties under diskUsage I found exactly what I needed. Great tip!

Thank you both @richgalloway and @isoutamo  🙂