traffic events not getting routed to nw_fortigate and non-traffic events not getting routed to os_linux

[sureshkumaar]

traffic events not getting routed to nw_fortigate and non-traffic events not getting routed to os_linux

Can someone help?

props.conf

[source::.../TUC-*/OOB/TUC-*(50M)*.log]
TRANSFORMS-routing = route_fortigate_traffic, route_nix_messages

transforms.conf

[route_fortigate_traffic]
REGEX = (?i)traffic|session|firewall|deny|accept
DEST_KEY = _MetaData:Index
FORMAT = nw_fortigate

[route_nix_messages]
REGEX = .*
DEST_KEY = _MetaData:Index
FORMAT = os_linux

[sureshkumaar]

Hi @livehybrid and @isoutamo 

Below is the complete conf data.

requirement is  from same log file "traffic|session|firewall|deny|accept" related events should get indexed in "index=nw_fortigate" sourcetype=fortigate

and other events should get indexed in "index=os_linux" sourcetype=os

Kindly let me know do i need to any corrections in the conf and change the order in the transforms as well.

inputs.conf

[monitor:///TUC-RST50/OOB/TUC-RST50M*.log]
disabled = false

props.conf

[source::.../TUC-*/OOB/TUC-*(50M)*.log]
TRANSFORMS-routing = route_fortigate_traffic, route_nix_messages

transforms.conf

[route_fortigate_traffic]
DEST_KEY = _MetaData:Index
REGEX = (?i)\b(traffic|session|firewall|deny|accept)\b
FORMAT = nw_fortigate

[route_nix_messages]
DEST_KEY = _MetaData:Index
REGEX = .*
FORMAT = os_linux

[PickleRick]

You've already been told - transforms are not ACL-s. It's not that first matching transforms runs and execution stops. No, it's the other way around. Every configured transform is executed (as long as its regex matches; with some exceptions like CLONE_SOURCETYPE but let's not dig into this here, I'm just listing it for completness). So if you _first_ redirect some events to nw_fortigate index and then have a transform redirecting all events to os_linux index... all events will end up in os_linux.

[sureshkumaar]

Hi @PickleRick - Yes i did the changes accordingly.

Now i am facing below

1. Able to get the expected results running without sourcetype, but while running the search with sourcetype=nix:messages OR sourcetype=fortigate_traffic 0 results returning.

2. the host extraction from the source which was there earlier now it's not working

props.conf

### to send traffic and non-traffic events ###
[source::.../TUC-*/OOB/TUC-*(50M)*.log]
TRANSFORMS-routing = route_nix_messages, route_fortigate_traffic
TRANSFORMS-sourcetype = set_nix_sourcetype_if_not_traffic, set_fortigate_sourcetype_if_routed

### to extract host from source ###
[nix:messages]
TRANSFORMS-set_host = set_custom_host

[fortigate_traffic]
TRANSFORMS-set_host = set_custom_host

transforms.conf

### to send traffic and non-traffic events ###

[route_nix_messages]
DEST_KEY = _MetaData:Index
REGEX = .*
FORMAT = os_linux

[set_nix_sourcetype_if_not_traffic]
DEST_KEY = MetaData:Sourcetype
REGEX = .*
FORMAT = nix:messages

[route_fortigate_traffic]
DEST_KEY = _MetaData:Index
REGEX = (?i)\b(traffic|session|firewall|deny|accept)\b
FORMAT = nw_fortigate

[set_fortigate_sourcetype_if_routed]
DEST_KEY = MetaData:Sourcetype
REGEX = (?i)\b(traffic|session|firewall|deny|accept)\b
FORMAT = fortigate_traffic

### to extract host from source ###

[set_custom_host]
REGEX = /TUC-[^/]+/[^/\n]+/([^-\n]+(?:-[^-\n]+){0,5})-(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})-\d{2}-\d{2}-\d{4}\.log
FORMAT = host::$1
DEST_KEY = MetaData:Host
SOURCE_KEY = MetaData:Source

[PickleRick]

You're expecting Splunk to "pick up" the rewritten sourcetype and apply transforms defined for it, right?

It doesn't work that way. An event's path through the ingestion pipeline is determined at the start by its sourcetype/source/host triplet. Anything you do to those field during ingestio  doesn't change the processing path - it can only affect search-time operations later.

You can use CLONE_SOURCETYPE to make event go through (almost) whole ingestion pipeline again with a new sourcetype but the caveat is that CLONE_SOURCETYPE doesn't work selectively - you can't limit its scope with regex. So its usage is fairly complicated and I wouldn't advise it.

[isoutamo]

Hi

when you are matching without SOURCE_KEY you are using _raw. Are you sure that this information is in event itself or is it in metadata field? If it's somewhere else than _raw you must add that information into SOURCE_KEY.

It's also good to use capture group especially when you have this kind of if then if then else selection.

Also as @livehybrid said if you have REGEX = .* then it must be in 1st transformation in list as it catch everything.

One excellent place to test your regex with data is regex101.com there you can ensure that those are correct and match how you expecting.

r. Ismo

[livehybrid]

Hi @sureshkumaar 

I think one of the issues here is that your transforms are in the wrong order, a list of TRANSFORMS are applied in order and do not stop once the criteria of one is met, this means in this situation it would apply the route_fortigate_traffic index change, then route_nix_messages which would just set it to os_linux regardless. Change the order of these to first set os_linux and then override to the nw_fortigate index if appropriate.

That being said, it sounds like neither of your transforms are being applied? Please could you confirm the following:

• Does [source::.../TUC-*/OOB/TUC-*(50M)*.log] definately match your file name? Please could you provide a sample filename/path?
• Are you sending the data from a Universal Forwarder (UF) or Heavy Forwarder (HF)? Data from a HF wont be reparsed using this approach.

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[kiran_panchavat]

@sureshkumaar 

Ensure that the REGEX in route_fortigate_traffic is correctly matching the events. 

Verify that the source path in props.conf is correct and matches the actual log file paths.

Where do you have props.conf and trasnforms.conf?

Route and filter data - Splunk Documentation

Refer this community link https://community.splunk.com/t5/Splunk-Cloud-Platform/Routing-log-data-to-different-indexes-based-on... 

https://community.splunk.com/t5/Getting-Data-In/Route-index-data-based-on-source/m-p/688917