Universal Forwarder to Intermediate Forwarding to Splunk Enterprise Instance,Indexer Cluster & Heavy Forwarder

[hectorvp]

I've a scenario where I've got around 250 servers where UF has to be installed. These data would be forwarded to Indexer cluster or heavy forwarder via Intermediate Forwarder. 

I need to use Intermediate forwarder(Universal forwarder itself).

Now I need to route data from Intermediate Forwarder in this way:

if hostname=x

( Indexer Cluster  AND Other Splunk Enterprise Instance) 

else if hostname=y

(Heay Forwarder AND Other Splunk Enterprise Instance) 

Note: Splunk Enterprise Instance is other independent instance which has no relation with Indexer cluster and heavy forwarder)

What should by inputs.conf & outputs.conf in UF and Intermediate Forwarder?

How can I achieve this?

[gcusello · Accepted solution]

Hi @hectorvp,

if you have only to forward logs, you can use a Universal Forwarder as intermediate Forwarder, but, if you have to make some eleboration (e.g. choose the destination indexer) you have to use an Heavy Forwarder as Intermediate Forwarder (remeber that in this way you have to do parsing and transformation in this HF).

But i think that the correct approach should be another:

I think that you manage your Universal Forwarders using a Deployment Server.

So you could create two apps (called e.g. TA_ForwardersX and TA_ForwardersY) in which you put only two files:

• deploymentclient.conf, addressing your Deployment Server: it will be the same in both the TAs;
• outputs.conf addressing, the indexers to send data: they will be specific for each TA.

In the first outputs.conf you use the destinations for hostnames=x and put it in TA_ForwardersX.

In the second you put the destinations for hostnames=y and put it in TA_ForwardersY.

Then you create two ServerClasses so you'll have your correct distribution.

Ciao.

Giuseppe

[hectorvp]

@gcusello Thanks afa lot for replying

this provided solution sends logs directly to indexer right? And no IF right?

Actually I've to use IF which is UF due to the business requirement, what I've thought is to make IF listen on two different ports where port 1 will receive host x logs and port 2 will receive host y logs then I can forward them...I knw this isn't a best way but I guess this will work.... 

Since I'm new with splunk and I've other questions as well,

I've around 250 servers where only OS logs has to be collected and no application logs which would be send to IF,

Use of 2 IF will cause bottleneck or it won't??? Or do I've to add more??

And how much data would be ingested to indexer so I can estimate costing for this??

These are very practical scenarios where it is difficult to find answers in documentation or anywhere, your reply would really help.....

[gcusello]

Hi @hectorvp,

I use IFs only when I have UFs in separate networks so I don't want to open too many routes between targets and Indexers (especially if I have to use Splunk Cloud!).

If there isn't this requirement I usually send logs from targets to Indexers.

IFs could be a bottleneck (it depends on the log volume and configurations) and anyway they are an additional complication to architecture, so, if there isn't an explicit requirement I don't use them!

About indexing costing, they are indipendent from the presence of IFs, it depends only on the volume of indexed logs.

To better understand and create your architecture, the best approach is to have in your team a Splunk Architect from your System Integrator or a Splunk PS.

In addition I suggest the Splunk Architect Certification Path that's very useful for this.

Ciao.

Giuseppe

[hectorvp]

@gcusello 

Thanks a lot for your guidance, surely I've certification in my roadmap, however things are changing rapidly here in my organisation so not able to reach out splunk professional consultant.

I was able to explain stakeholders not to use IF  as they were adding unnecessary overhead in a design.

I'm right now calculating  splunk license sizing and these are my calculations, please suggest incase I'm going wrong:

Total servers : 250

Expected license : 10GB

Each server gets 40MB of logs to get ingested in indexer daily,

Most of the servers are windows server and requirement is to only pull OS logs(system, security, application,setup) and no application logs.

Considering each event size as 700bytes.

So total events one server can have at every day= 40* 10^6 / 700 =  57,142.

I know event size and no. of events changes from server to server but this is how I'm planning to estimate it.

Avg event size for windows is 500bytes as per I saw over internet and I've added 200 bytes as a buffer or for normalisation.

So is this calculation in a right way?

Am i missing something?

Practically 

I'm in no position to make a POC to find right estimates perhaps I can check no. of events generated per day on one of the 250 servers.

And can revise  my license  capacity prior to procuring it.

[gcusello]

Hi @hectorvp,

I understand your needs, we're all in the same situation, but anyway don't  forget training and certifications (it's very useful also for you!).

Anyway, if you have 40 MB/day for each server, the total license consuption (in GB/day) is 40*250/1024, then you have to consider a 25/30% of additional license because you could have some extraordinary days.

Sincerely I think that 40 MB/day for a windows server are really few because a windows server has normally more than 20,000 events by day and Domain Controllers much more than.

But anyway you can identify them in a PoC, using at least one server and one DC.

If you cannot do a PoC, you could consider 40-50 MB/day per server (eventually filtering non interesting events) and 200 MB/day for the domain controllers.

Then As I said, consider 25% of margin.

Then, remeber that if you want to monitor file servers, they are very verbose!

Ciao.

Giuseppe

[hectorvp]

@gcusello : 

Thanks a lot again for these responses, it is really helping me a lot.