topic Re: Universal forwarder and Deployment Server in Monitoring Splunk

Universal forwarder and Deployment Server

hectorvp — Wed, 23 Sep 2020 15:45:38 GMT

Hi,

I've a scenario where our organisation is supposed to only send logs from servers to clients indexers.

We have decided to use UF and deployment server.

We need to know what are known downtimes, performance issues for for UFs and deployment servers.

For example incase there may be any downtime while upgrade of UFs or any maintenance aspects.

Are there any exceptions with capabilities of UF to forward logs like for certain application (commonly used) logs cannot be forwarded since they are in xyz format.....

For example incase there may be any downtime while upgrade of UF.

We need this information for certain agreements with the customer.

Can anyone enlist few points here.

Re: Universal forwarder and Deployment Server

gcusello — Wed, 23 Sep 2020 15:59:44 GMT

Hi @hectorvp,

forwarders is the best approach to take logs from servers because UF guarantee to you some feature improvement than other methods (e.g. WMI or syslogs), these are the main:

packets are compressed so you consume less bandwidth,
UFs has a local cache for logs in case of unavailability of Indexers;
it's possible to configure packets dimension to limit the bandwidth.

UFs cosumes just a little part of server resources (e.g.: on it uses around Windows 70-80 MB RAM and 2-3 % of CPU usage).

Deployment Server is the best approach to manage UFs.

UFs continue to work also with the DS down, so it isn't a Single Point of Failure.

Downtime isn't relevant because installation, upgrade of UF or configurations don't require a server restart.

DS must be a dedicated machine if it has to manage more than 50 clients.

DS can also be a virtual server, but it needs of the same resources of a stand-alone Splunk (12 CPUs and 12 GB of RAM).

Here you can find all the documentation about DS https://docs.splunk.com/Documentation/Splunk/8.0.6/Updating/Aboutdeploymentserver

Ciao.

Giuseppe

Re: Universal forwarder and Deployment Server

hectorvp — Wed, 23 Sep 2020 16:54:48 GMT

Hi @gcusello ,

Thanks again for the response.

Can I expect uptime of 99.99% ? (Considering UFs and DS are properly configured)

Is there any situation where agent may crash and need to take a look??

For example if clients indexers aren't receiving any logs.

From ur above response I consider there won't be any downtime with UF maintenance.

But still would there be any data loss while upgrading UF?

And the last one

Are there any exceptions where UFs cannot pick logs from server (ex: not supported any file extensions like etl ). I'm afraid of with application logs mostly since they may not have been logging data as windows event logs.

Re: Universal forwarder and Deployment Server

gcusello — Thu, 24 Sep 2020 07:30:56 GMT

Hi @hectorvp,

answering to your questions:

uptime depends on the maintenance you schedule for your systems, as I said, Splunk doesn't require server restart;

if you're speking of monitoring uptime, Splunk doesn't lose any log because it cashes logs when cannot send them to Indexers.

In my experience I saw agent crashes only on some Windows server (especially if they didn't have sufficient resources), when it happened I opened a case to Splunk Support.

if Indexers don't receive logs, you have to configure an alert to notice this event and immediately intervene (I usually configure an alert triggering every 5 minutes).

as I said you don't lose logs during maintenance.

The only logs you risk to lose are syslogs because you have to ingest them when they arrive, for this reason I hint to use two Heavy Forwarders with a Load Balancer, in this way you put in maintenance only one at a time of them.

when you upgrade UFs, they obviously don't send logs but they send them as soon as they are connected.

Splunk takes avery kind of text logs and some special logs as wineventlogs, to know which logs Splunk can index see at https://docs.splunk.com/Documentation/SplunkCloud/8.1.2008/Data/WhatSplunkcanmonitor#What_data_can_I_index?

for other kind of data, see in splunkbase (apps.splunk.com) if there's a special Technical Add-Ons (TA), otherwise, you have to preparse them before indexing by script (e.g. encrypted data).

Ciao.

Giuseppe

Re: Universal forwarder and Deployment Server

hectorvp — Fri, 25 Sep 2020 11:22:12 GMT

Thanks @gcusello ,

Just one follow up question

Since we have a task only to forward OS and application logs from servers to the customers indexer, we only meed Splunk Core license, right?

Or is there any possibility that any other license for example ITSI would be needed?

Re: Universal forwarder and Deployment Server

gcusello — Fri, 25 Sep 2020 13:17:04 GMT

Hi @hectorvp,

this is another question and, for the future, it should be better to open a new question!

Anyway, Splunk licensing is related only to the daily indexed logs, not other thing as number of forwarders, Splunk servers, installed apps, etc...

The only exception are premium apps (like ITSI or ES) that you have to pay in addition to the Splunk Enterprise license.

Also ITSI and ES licenses are measured using the daily log volume .

Ciao.

Giuseppe

P.S.: Karma Points are appreciated 😉

Re: Universal forwarder and Deployment Server

hectorvp — Fri, 25 Sep 2020 12:50:10 GMT

Thanks @gcusello , sure, new question from next time on wards 😊