How to block few logs from some specific hosts?

[blbr123]

Hi All,

I have around 30 Hosts forwarding logs to splunk.

I have the below same paths in all the servers

/data/abc/vault.logs

/data/abc/vault_audit.logs

/data/xyz/proxy.logs

So I have created an app included inputs with all those above stanzas and pushed the app to all hosts.

So by default all those hosts are sending the above mentioned logs to splunk.

But I want 5 servers to send just the below log but not other logs

/data/xyz/proxy.logs

How to achieve this?

[gcusello]

Hi @blbr123,

you have two ways to reach your target:

• the easiest (that I hint) is creating two Add-Ons: one for the 5 hosts with only one input and another one with all the inputs, then you have to deploy the two Add-Ons using two different ServerClasses,
• If you don't want to have two Add-Ons, you can have only one Add-On and put a filter on your Indexers to delete the other logs coming from the 5 hosts.

About the first solution, I think that you don't need any help to create the two Add-Ons and the two ServerClasses, if you need it, please, tell me.

About the second solution, you have to put in your Indexers or (if present) on your Heavy Forwarders the following props.conf

[host::host1]
TRANSFORMS-null= setnull

[host::host2]
TRANSFORMS-null= setnull

[host::host3]
TRANSFORMS-null= setnull

[host::host4]
TRANSFORMS-null= setnull

[host::host5]
TRANSFORMS-null= setnull

and in your transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

as you can read at https://docs.splunk.com/Documentation/Splunk/8.2.6/Forwarding/Routeandfilterdatad#Filter_event_data_...

Ciao.

Giuseppe

[blbr123]

Yes I need help on creating the add-on if I have to apply the first solution

[gcusello]

Hi @blbr123,

I suppose that you're using a Deployment Server to deploy configurations to your Forwarders, tell me if not and anyway, put in mind to use it as soon as possible!

you can find information about how to get data in at https://docs.splunk.com/Documentation/Splunk/latest/Data/Getstartedwithgettingdatain 

and https://docs.splunk.com/Documentation/Splunk/8.2.6/Updating/Createdeploymentapps https://docs.splunk.com/Documentation/Splunk/8.2.6/Updating/Updateconfigurations

Anyway, you have to create two addons both containing the following folders structure:

• bin
• default
• local
• metadata
• static

in each Add-On put in the default folder app.conf file containing something like this:

[default]

[launcher]
author = you
description = Add-On for all hosts
version = 1.0.0

[package]
check_for_updates = 0

[ui]
is_visible = 0
label = TA-All_Servers

obviously changing label and description for each one.

Then put in the local folder of the first (the one for all servers) the following inputs.conf:

[monitor:///data/abc/vault.logs]
disabled = 0
index = your_index
sourcetype = your_sourcetype1

[monitor:///data/abc/vault_audit.logs]
disabled = 0
index = your_index
sourcetype = your_sourcetype2

[monitor:///data/xyz/proxy.logs]
disabled = 0
index = your_index
sourcetype = your_sourcetype3

and in the local folder of the second Add-On:

[monitor:///data/xyz/proxy.logs]
disabled = 0
index = your_index
sourcetype = your_sourcetype3

Then you have to deploy these two Add-Ons using the Deployment Server, following the instructions at https://docs.splunk.com/Documentation/Splunk/8.2.6/Updating/Updateconfigurations

in few words you have to:

• copy both apps is %SPLUNK_HOME/etc/deployment-apps folder of your Deployment Server,
• using GUI create a serverclass for the first group of Forwarders adding:
  • the IP addresses or the hostnames of the first group servers,
  • the related Add-On to deploy,
• then you have to repeat this operation for the second group of servers and Add-On.

Obviously you had to configure your clients as client of the Deployment Server, if you didn't do it follow the instructiona at the above link.

If you don't want to configure a Deployment Server in your infrastructure (I don't hint this!) you could manually copy the Add-Ons into the related servers in the %SPLUNK_HOME/etc/apps folder, remembering to restart Splunk on each one.

My final hint is to follow a training for Splunk Admin to better understand how to do all these things.

Ciao.

Giuseppe

[blbr123]

Can't we achieve this mentioning the host details in inputs.conf

Let's say

[monitor://data/abc/vault.log]

index=applog

Host=dx096865

By doing this don't I  get just the vault.log from just that host?

[blbr123]

About second solution,

We actually don't use transforms much, but work on props based on sourcetypes

So not sure if this can be achieved just in props

[gcusello]

Hi @blbr123,

what's the problem to use also transforms.conf? it's a part of the solution.

This is the usual method to filter unwanted logs. 

Ciao.

Giuseppe

[gcusello]

Hi @blbr123,

as I said: you have two choices:

1. two Add-Ons with different inputs.conf to intervene on Forwarders,
2. intervene on Indexers if you want to have only one Add-On.

You cannot put a condition in inputs.conf.

My hint is to have two Add_Ons (solution 1), but also the second solution, as I said, it's an easy to implement solution.

Ciao.

Giuseppe

[blbr123]

Ok then for what purpose the hosts is mentioned in inputs which I saw in some configurations

[gcusello]

Hi @blbr123,

using the filtering solution, you have only one inputs. conf and the filter (mentioning hosts) is on props.conf on Indexers.

The option "host=your_host" in inputs.conf is used to force the value of host for that data source.

If you don't use it, by default, the host value of that data source is setted to the value of the forwarder you're using (you can find it in $SPLUNK_HOME/etc/system/local/server.conf of the Forwarder).

Ciao.

Giuseppe