topic Re: Index redirection in Getting Data In

Index redirection

francois — Fri, 14 Jan 2022 11:13:29 GMT

Hi,

We are setting up a Splunk infrastructure where we would like to redirect event coming in particular indexes to an external SOC.

For example, logs from multiple firewall technologies would be put into the index "clientX_firewall" by an SC4S and this whole index would have to be forwarded to both my indexing tier and the external SOC, whatever the sourcetype / host / source.

Is there a way to properly redirect this whole index ? Without having to specify the source / host / sourcetype involved for each type involved ?

Thanks for your help.

Re: Index redirection

SinghK — Fri, 14 Jan 2022 11:17:15 GMT

I think you need to do a syslogrouting on hf and send everything to indexer and that external location from there with outputs.conf having config for both

Re: Index redirection

gcusello — Fri, 14 Jan 2022 11:21:04 GMT

Hi @francois,

do you want to redirect the logs by Indexers or by Heavy Forwarders?

If you want to use HFs, you can find all instructions at https://docs.splunk.com/Documentation/Splunk/8.2.4/Forwarding/Routeandfilterdatad#Replicate_a_subset_of_data_to_a_third-party_system

in this case, put attention to the volume of syslogs: I had problems with large volume of syslogs.

If you want to send from Indexers, you could use the Splunk Connect for Syslogs Apps (https://splunkbase.splunk.com/app/4740/#/details) that can help you.

Ciao.

Giuseppe

Re: Index redirection

francois — Fri, 14 Jan 2022 12:40:54 GMT

Hi @gcusello,

Thank you for your response,

Once the event has been processed by the SC4S (Splunk-connect-for-syslog), it is sent as HTTP so I don't think I'll have a problem with volume. From the documentation, a single SC4S instance with proper hardware requirements can handle up to 6TB/day.

The events will be replicated on my heavy forwarder, I did try the method described on https://docs.splunk.com/Documentation/Splunk/8.2.4/Forwarding/Routeandfilterdatad#Replicate_a_subset_of_data_to_a_third-party_system but using this method, I'd have to create an entry in my props.conf for each source / host /sourcetype. This would mean a lot of repetitive / unnecessary work to maintain.

It would be much simpler to be able to replicate an entire index to a third party system.

Do you know if it is possible ?

Best regards,

François.

Re: Index redirection

gcusello — Fri, 14 Jan 2022 13:09:46 GMT

Hi @francois,

my hint is to use the Syslog Connect App, but you need a search as input for it, this means that you have to use it on Indexers or configure your Heavy Forwarder as a Search Head or to duplicate data to forward on the HF.

For these reasons I hint to use it on Indexers or on Search Heads.

Ciao.

Giuseppe

Re: Index redirection

francois — Fri, 14 Jan 2022 13:45:05 GMT

@gcusello

Sorry I don't understand your answer. Syslog-connect ( https://splunkbase.splunk.com/app/4740/#/details ) is, per my understanding, not an app but an appliance (containerized syslog-ng with pre-defined filters) that allows for syslog traffic to be properly categorized into sourcetype, host, index, etc. and then sent to an HEC.

Therefore :

1. How can I install this on a search head / Indexer ?

2. How can I pass seaches as inputs ?

Re: Index redirection

PickleRick — Fri, 14 Jan 2022 13:58:26 GMT

What I would check (but I'm not sure if it will work that's why I say I'd check it first) is matching to some wildcard in props.conf to apply a transform and then in that transform matching to a particular index metadata field.

But as I said - haven't tried it, that's just a quick idea from the top of my head.

Re: Index redirection

gcusello — Fri, 14 Jan 2022 13:58:29 GMT

Hi @francois,

Splunk is a software solution hardware indipendent.

The Syslog Connect for Splunk is an App to install on a Full Splunk instance (all except Universal Forwarders).

You can install it on an Indexer or a Search Head because you have to execute a search and the results are the input for the App for sending to a third party system.

You can also install it on an Heavy Forwarder, but HFs usually don't access data so you have to configure it as a Search Head or locally index a copy of the data to send to third party, but this solution is expensive because you duplicate the license consuption.

This app is usually used for syslog ingesting but it also offers features for syslog sending to third party.

I used it in a project to send a part of logs to an external SIEM.

Ciao.

Giuseppe

Re: Index redirection

francois — Tue, 18 Jan 2022 12:30:18 GMT

Thanks for your help 🙂

We ended up using the stanza "default" in the props.conf and sending it to various transforms.conf group so that it can match multiple regex.