Index redirection

[francois]

Hi,

We are setting up a Splunk infrastructure where we would like to redirect event coming in particular indexes to an external SOC.

For example, logs from multiple firewall technologies would be put into the index "clientX_firewall" by an SC4S and this whole index would have to be forwarded to both my indexing tier and the external SOC, whatever the sourcetype / host / source.

Is there a way to properly redirect this whole index ? Without having to specify the source / host / sourcetype involved for each type involved ? 

Thanks for your help.

[PickleRick · Accepted solution]

What I would check (but I'm not sure if it will work that's why I say I'd check it first) is matching to some wildcard in props.conf to apply a transform and then in that transform matching to a particular index metadata field.

But as I said - haven't tried it, that's just a quick idea from the top of my head.

[francois]

Thanks for your help 🙂

We ended up using the stanza "default" in the props.conf and sending it to various transforms.conf group so that it can match multiple regex.

[gcusello]

Hi @francois,

do you want to redirect the logs by Indexers or by Heavy Forwarders?

If you want to use HFs, you can find all instructions at https://docs.splunk.com/Documentation/Splunk/8.2.4/Forwarding/Routeandfilterdatad#Replicate_a_subset...

in this case, put attention to the volume of syslogs: I had problems with large volume of syslogs.

If you want to send from Indexers, you could use the Splunk Connect for Syslogs Apps (https://splunkbase.splunk.com/app/4740/#/details) that can help you.

Ciao.

Giuseppe

[francois]

Hi @gcusello,

Thank you for your response,

Once the event has been processed by the SC4S (Splunk-connect-for-syslog), it is sent as HTTP so I don't think I'll have a problem with volume. From the documentation, a single SC4S instance with proper hardware requirements can handle up to 6TB/day.

The events will be replicated on my heavy forwarder, I did try the method described on https://docs.splunk.com/Documentation/Splunk/8.2.4/Forwarding/Routeandfilterdatad#Replicate_a_subset... but using this method, I'd have to create an entry in my props.conf for each source / host /sourcetype. This would mean a lot of repetitive / unnecessary work to maintain. 

It would be much simpler to be able to replicate an entire index to a third party system.

Do you know if it is possible ? 

Best regards, 

François.

[gcusello]

Hi @francois,

my hint is to use the Syslog Connect App, but you need a search as input for it, this means that you have to  use it on Indexers or configure your Heavy Forwarder as a Search Head or to duplicate data to forward on the HF.

For these reasons I hint to use it on Indexers or on Search Heads.

Ciao.

Giuseppe

[francois]

@gcusello 

Sorry I don't understand your answer. Syslog-connect ( https://splunkbase.splunk.com/app/4740/#/details ) is, per my understanding, not an app but an appliance (containerized syslog-ng with pre-defined filters) that allows for syslog traffic to be properly categorized into sourcetype, host, index, etc. and then sent to an HEC.

Therefore :

1. How can I install this on a search head / Indexer ?

2. How can I pass seaches as inputs ?

[gcusello]

Hi @francois,

Splunk is a software solution hardware indipendent.

The Syslog Connect for Splunk is an App to install on a Full Splunk instance (all except Universal Forwarders).

You can install it on an Indexer or a Search Head because you have to execute a search and the results are the input for the App for sending to a third party system.

You can also install it on an Heavy Forwarder, but HFs usually don't access data so you have to configure it as a Search Head or locally index a copy of the data to send to third party, but this solution is expensive because you duplicate the license consuption.

This app is usually used for syslog ingesting but it also offers features for syslog sending to third party.

I used it in a project to send a part of logs to an external SIEM.

Ciao.

Giuseppe

[SinghK]

I think you need to do a syslogrouting on hf and send everything to indexer and that external location from there with outputs.conf having config for both