Redirecting specific logs using a regex

[Nicolas2203]

Hi splunk community, I have a question on logs cloning/redirection

Purpose :

Extract logs containing "network-guest", and don't redirect this logs to a distant HF, but only to local indexers

LOGS ENTRY CONFIG
Into an app Splunk_TA_FIREWALL

inputs.conf
[tcp://22000]
sourcetype = fw:firewall
index = fw_index
_TCP_ROUTING = local_indexers

This logs are perfectly working and are stored on my local indexers

Now this logs must be cloned and redirected to a distant HF but not the logs containing "network-guest"
THat my props and transforms config

props.conf
[fw:firewall]
TRANSFORMS-clone = fwfirewall-route-network-guest-, fwfirewall-clone

transforms.conf
[fwfirewall-route-network-guest]
REGEX = \bNETWORK-GUEST\b
DEST_KEY = _SYSLOG_ROUTING
FORMAT = local_indexers

[fwfirewalll-clone]
DEST_KEY = _SYSLOG_ROUTING
FORMAT = distant_HF
REGEX = .

When I check into the logs, on the distant splunk, I don't see NETWORK-GUEST logs anymore, and I can see those logs on the local splunk
Question is, I'm not sure I'm doing that the right way, and not sure if it works 100%
Has someone a good knowledge on this kind of configuration ?

Thanks a lot for the help

Nico

[livehybrid]

Hi @Nicolas2203 

You are referencing _SYSLOG_ROUTING which is for syslog routing, whereas your input is using _TCP_ROUTING.

Did you mean to use _TCP_ROUTING in your transforms?

Another thing is that this will not clone your data, it will only *change* the routing. 

When you specify multiple items in a TRANSFORMS they are processed in order, meaning that your network guest route is applied, then the second one. In your scenario the second transform applies to ALL events because of the "." in the REGEX which means that will be the routing which is applied.

I think what you are looking for is the following:

 == props.conf ==
[fw:firewall]
TRANSFORMS-clone = fwfirewall-route-network-guest, fwfirewall-clone

== transforms.conf ==
[fwfirewall-clone]
DEST_KEY = _TCP_ROUTING
FORMAT = distant_HF,local_indexers
REGEX = .

[fwfirewall-route-network-guest]
REGEX = \bNETWORK-GUEST\b
DEST_KEY = _TCP_ROUTING
FORMAT = local_indexers

How this works is by specifying both outputs in _TCP_ROUTING when the REGEX matches "." (always) and then changes it to local_indexers IF the event contains NETWORK-GUEST.

This could actually be simplified by setting the duplicate output in the input, then just overriding for the local_indexers if it contains NETWORK-GUEST:

== inputs.conf ==
[tcp://22000]
sourcetype = fw:firewall
index = fw_index
_TCP_ROUTING = distant_HF,local_indexers

== props.conf ==
[fw:firewall]
TRANSFORMS-redirectLocal = fwfirewall-route-network-guest

== transforms.conf ==
[fwfirewall-route-network-guest]
REGEX = \bNETWORK-GUEST\b
DEST_KEY = _TCP_ROUTING
FORMAT = local_indexers

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[Nicolas2203]

Hello @livehybrid 

Thanks for your time

OK, I understand now. I see what I was missing.

Strangely, what I had done was working, and I was perplexed about that. 😕

I will test with the configuration you provided; it makes more sense.

But I have a quick question: if the logs need to be anonymized before they are sent to the distant_HF, will putting the two outputs in the _TCP_ROUTING in the inputs.conf work?

Many thanks for you clear answer !!!

[livehybrid]

Hi @Nicolas2203 

Soo...if you want to redact your logs sent to one place but not redact them sent to the other then I think you would have to use CLONE_SOURCETYPE and then apply some redaction and routing of this new sourcetype as required.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[isoutamo · Accepted solution]

It’s exactly this way what you need to do. In Yourcase this must do everything with props and transforms instead of defining it on inputs.conf.
Clone sourcetype to sent it to HF and filter it like you need. And send original into local indexers.

You can check the next links:
- https://community.splunk.com/t5/Getting-Data-In/How-can-I-use-CLONE-SOURCETYPE-to-send-a-cloned-modi...
and
- https://www.tekstream.com/blog/routing-pii-data-to-multiple-indexes/
Those explains this with samples.

[Nicolas2203]

Hi @livehybrid , @isoutamo 

Thanks a lot for your help! I’ll try the configuration with CLONE_SOURCETYPE and will come back here to let you know if it works for me. 🙂
Thanks again for your support!

[Nicolas2203]

Hello

Had an additional questions, if I have to extract datas from two fields
For example, I will NOT send to distant splunk, all the datas that in the dest_zone and src_zone contains DMZ-NETWORK and GUEST-NETWORK

My config is something like that

TRANSFORMS.CONF

[clone_only_dmz-network]
REGEX = ^((?!dmz-network).)*$
CLONE_SOURCETYPE = cloned:firewall_clone
DEST_KEY = _SYSLOG_ROUTING #it's syslog logs
FORMAT = distant_splunk

PROPS.CONF
[firewall_sourcetype]
TRANSFORMS-clone = clone_only_dmz-network

OUTPUTS.CONF
[syslog:distant_splunk]
server = ip of the distant HF

This is actually working, I tested it

But, if I want to not send mutliple fileds, how can I achieve that ?

[PickleRick]

In the ingestion pipeline Splunk has no notion of most of the fields since they're usually mostly search-time extracted so you can either bend over backwards trying to create a regex which will acount for all possible situations (which might get tricky if the order of fields isn't fixed or if their presence isn't mandatory) or you can do anotner dirty trick - extract fields in index time, do a INGEST_EVAL based on their value and then remove them (assign null to them) so they're not getting indexed.

[isoutamo]

You should try to create suitable REGEX to match only those events.
One other comment, never put comment # mark in middle of conf line! At least in some conf file splunk cannot recognize it as a comment and try to do something with it with unknown results.

[Nicolas2203]

Thanks all for the help, I will try a regex that match both. I learn a lot with you guys thanks !!!!

[Nicolas2203]

Hi Splunk Community,

I have a new issue concerning this config, some particular behaviour that I don't understand

Here is my configuration

#classic input on TCP, this is syslog logs
[tcp://22000]
sourcetype = mydevice:sourcetype
index = local_index
_TCP_ROUTING = local_indexers:9997

# Idea is to clone the sourcetype, but not logs containing LAN1 and LAN2 logs, it's not necessary for the second splunk
[mydevice-clone]
CLONE_SOURCETYPE = mydevice:clone
REGEX = ^((?!LAN1|LAN2).)*$
DEST_KEY = _SYSLOG_ROUTING
FORMAT = sending_to_second_splunk

# on the props I apply the configuration made on the transforms
[mydevice:sourcetype]
TRANSFORMS-clone = mydevice-clone

#IP of the HF that will send data to second splunk
[syslog:sending_to_second_splunk]
server = 10.10.10.10:port
type = tcp

Issue Encountered

This configuration works partially:

• Data is properly indexed to the second Splunk, without LAN1 and LAN2 data.
• Data containing LAN1 and LAN2 is indexed on the local indexer.
• However, the sourcetype mydevice:clone is also indexed on my local indexer, resulting in some data being indexed twice with two different sourcetypes.

I don't understand why this is happening and I am seeking help to resolve this issue, I have the feeling that I miss something

Thanks,

Nicolas

[livehybrid]

Hi @Nicolas2203 

It looks like you are setting _TCP_ROUTING to your local indexers in the input but then do not change it on the cloned data, you are setting _SYSLOG_ROUTING but the TCP_ROUTING is still also sending to the local indexers.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[Nicolas2203]

Hi @livehybrid 

Yes in my meening, setting a _TCP_ROUTING on the input will forward the logs on the local indexers.

And, at the sametime, sourcetype is cloned to be forwarded to the second splunk.

Maybe I didn't understand something 😕

[livehybrid]

So in terms of "the sourcetype mydevice:clone is also indexed on my local indexer" - you have cloned it but because it still has _TCP_ROUTING=local_indexers it will also index on the local indexers. 

How come are you are a secondary Splunk server via syslog instead of Splunk2Splunk? If you dont want to send the cloned sourcetype to local indexers then you need to use another transform to set "_TCP_ROUTING=" (No value) as well as setting your syslog routing in the other transforms.c

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[Nicolas2203]

Ok so if I understand, when I clone a sourcetype, he will clone it's destination too ?

Not sure to understand, I have some other log sources that I clone, and forward to a secondary splunk with the same clone methods.

Cloning the sourcetype to sourcetype:anonymized

In the transforms I applied on the cloned sourcetype some regex for anonymization

And this sourcetype is routed via _TCP_ROUTING to an output that is a heavy forwarder that route to the secondary splunk.

For example 

Transforms.conf
[firewall_log-clone]
CLONE_SOURCETYPE = firewall_log:clone
REGEX = .*
DEST_KEY = _TCP_ROUTING
FORMAT = output_externalhf

Props.conf
[firewall_log]
TRANSFORMS-clone = firewall_log-clone

This is working, logs are properly sent to an HF that will forward those logs to a secondary splunk

But it's not syslog log source so maybe this is the difference ?

Thanks for the help

[livehybrid]

Hi @Nicolas2203 

With the anonymized sourcetype you are overwriting the original _TCP_ROUTING with output_externalhf:

DEST_KEY = _TCP_ROUTING
FORMAT = output_externalhf

 However with mydevice:clone you are *not* overwriting the existing _TCP_ROUTING, instead you are also adding _SYSLOG_ROUTING but this does not overwrite the _TCP_ROUTING.

You will need to apply a transform to mydevice:clone to set _TCP_ROUTING to a blank value to prevent it using the original local_indexers output. 

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[Nicolas2203]

OK now I get it, so I need to define one more stanza in my transforms to overwrite the _TCP_ROUTING

For example

[mydevice_overwrite_tcprouting]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = an_empty_output

Props.conf
[my_device_clone]
TRANSFORMS-deletetcprouting = mydevice_overwrite_tcprouting

Something like this ? is it possible to define an empty values in an outputs.conf ?

Thanks

Nicolas

[PickleRick]

The CLONE_SOURCETYPE option in a transform causes Splunk to create a copy (at this moment of the ingestion pipeline, so all the state of the event at this point is retained) of the processed event, changes its sourcetype to the one specified in the CLONE_SOURCETYPE option and reingests the event back at the (almost) beginning of the pipeline (skipping the initial phases of line breaking and time recognition).

See the usual https://community.splunk.com/t5/Getting-Data-In/Diagrams-of-how-indexing-works-in-the-Splunk-platfor...

CLONE_SOURCETYPE is called from a transform in the typing phase. The original event is processed without changes as if the transform containing the CLONE_SOURCETYPE option wasn't there. But the copy is moved back to the typing queue and starts the whole typing phase with a new sourcetype and triggers completely new set of transforms according to the new sourcetype (and possibly new source and host if they were overwritten during the initial transforms run).

[kiran_panchavat]

@Nicolas2203 

Could you please check this? In your props.conf, you've referenced "fwfirewall-route-network-guest-", but in transforms.conf, the stanza is named "fwfirewall-route-network-guest". Is this a typo?

[Nicolas2203]

Hello kiran, yes this is a typo .