Is there a way to have splunk filter read values externally?

[salavi]

We want to limit the ingestion of data that is coming from some sources (in this case the value would be in Properties.HostName) because they basically are not working correctly (customer machines) and continue to spam the system. (Turning them off is not an option. ). I know that we can add hardcoded filters such as below:

Name: Serilog:Filter:nn:Args:expression

Value: @p['AssemlyName'] = 'SomeAssembly.xxx.yyy' and @p['HostName'] in ['Spammer1', 'Spammer2', ...]

But the spammers change from time to time and we can generate their list. 

The question is, if I have a list of these spammers (in any form needed) can I somehow use some sort of a value above of some other method to read from that list (in place of the "in [... ]" expression above)? 

[gcusello]

Hi @salavi ,

are you speaking of a filter at index or search time?

if at search time, you can put the list in a lookup and use it for the search:

<your_search> [ | inputlookup your_lookup.csv | fields HostName]

you can eventually refresh your lookup  taking values from a scheduled search:

<your_lookup_search>
| dedup HostName
| table HostName
| outputlookup your_lookup.csv

that you can schedule e.g. every hour.

Ciao.

Giuseppe

[salavi]

Thanks for your reply. 

No, not in the search. I want to prevent these events (from certain hostnames) to even get ingested into Splunk to begin with. As I mentioned before, we can do this by adding a logging app service setting (on Azure), but it requires listing the hostnames individually. I was hoping there was a way to read from a list instead.  

Currently we are doing something like this. 

[PickleRick]

Are you asking if you can do this on egress in Azure or are you trying to do equivalent thing on ingress in Splunk?

You can do filtering on input, if you use ingest-evals even using lookups (but not in the Cloud).

[salavi]

Basically I am trying to find a way to prevent data from certain hostnames to even get ingested into Splunk (cost cutting measure for one thing). 

[PickleRick]

The general answer is yes - you can filter out events. The way to do it specific to your need will depend on your precise use case.

Within Splunk you can do it like this

https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad

If you can filter out in Azure so you simply don't send data to Splunk - even better. But this is out of scope of this forum and you have to ask some experienced Azure admins how to do so.

[gcusello]

Hi @salavi ,

at index time, I'm not sure that there isn't any other solution, unless the Splunk Edge Processor will be available.

Ciao.

Giuseppe