topic Re: Routing data to separate index using (?!...) in All Apps and Add-ons

Routing data to separate index using (?!...)

vpsmax — Wed, 12 Aug 2020 23:59:53 GMT

Hello.

Trying to resolve an issue with routing log events. The goal is to route log events with an "Api" keyword to a separate index. Here is a log sample of three events (the first has no "Api" embedded, the next two have "Api"):

2020-08-12 23:04:24 W3SVC5 SERVER_1 XX.XX.XX.XXX GET / - 443 - XX.XX.XX.XX HTTP/0.9 - - - - 302 0 0 389 7 10

2020-08-12 23:04:24 W3SVC5 SERVER_1 XX.XX.XX.XXX GET /Api/TopicsUpdate/GetRecalculationServiceTopicsThatMustBeUpdated pageSize=1 443 system.service XX.XX.XX.XX HTTP/1.1 - - - XX.XX.XX.XX 200 0 0 597 175 44

2020-08-12 23:04:22 W3SVC5 SERVER_1 XX.XX.XX.XXX GET /Api/TopicsUpdate/GetRecalculationServiceTopicsThatMustBeUpdated pageSize=1 443 system.service XX.XX.XX.XX HTTP/1.1 - - - XX.XX.XX.XX 200 0 0 597 175 54

Here is the props.conf files:

[sourcetype1]
TRANSFORMS-set = sourcetype_web_rename_iis,sourcetype_api_rename_iis,web_index_rename_iis,api_index_rename_iis

Here is the transforms.conf:

[sourcetype_web_rename_iis]
REGEX = \d+\-\d+\-\d+\s\d+\:\d+\:\d+\s\w+\s\w+\s\d+.\d+\.\d+\.\d+\s\w+\s\/(?!Api)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::ms:iis

[sourcetype_api_rename_iis]
REGEX = \d+\-\d+\-\d+\s\d+\:\d+\:\d+\s\w+\s\w+\s\d+.\d+\.\d+\.\d+\s\w+\s\/Api\/
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::ms:iis

[web_index_rename_iis]
REGEX = \d+\-\d+\-\d+\s\d+\:\d+\:\d+\s\w+\s\w+\s\d+.\d+\.\d+\.\d+\s\w+\s\/(?!Api)
DEST_KEY = _MetaData:Index
FORMAT = index1

[api_index_rename_iis]
REGEX = \d+\-\d+\-\d+\s\d+\:\d+\:\d+\s\w+\s\w+\s\d+.\d+\.\d+\.\d+\s\w+\s\/Api\/
DEST_KEY = _MetaData:Index
FORMAT = index2

Are there any special considerations when using (?!...)?

Regards,
Max

Re: Routing data to separate index using (?!...)

gcusello — Thu, 13 Aug 2020 08:26:52 GMT

Hi @vpsmax,

what's the problem in your configuration?

is the sourcetype override running or not?
is the index override running or not?

I don't like to have all the transformations in the same command, try to divide transformation in two commands in props.conf:

[sourcetype1] TRANSFORMS-sourcetype_override = sourcetype_web_rename_iis, sourcetype_api_rename_iis TRANSFORMS-index_override = web_index_rename_iis, api_index_rename_iis

Ciao.

Giuseppe

Re: Routing data to separate index using (?!...)

vpsmax — Thu, 13 Aug 2020 14:13:27 GMT

Hi Giuseppe,

Good morning. As suggested, I have tried separating the sourcetype and the index TRANSFORMS. As I am now seeing API data in the API index, I am good with ...

sourcetype_api_rename_iis
api_index_rename_iis

But for the ...

sourcetype_web_rename_iis
web_index_rename_iis

As all the IIS data is being stored in the IIS index, it appears the parsing is not happening. The goal is to simply store IIS data (and not API data) in the IIS index. Do you have any other suggestions?

Regards,
Max

Re: Routing data to separate index using (?!...)

gcusello — Thu, 13 Aug 2020 17:25:41 GMT

Hi @vpsmax,

regexes aren't correct, please try these:

props.conf

[sourcetype1] TRANSFORMS-sourcetype_override = sourcetype_web_rename_iis, sourcetype_api_rename_iis TRANSFORMS-index_override = web_index_rename_iis, api_index_rename_iis

transforms.conf

[sourcetype_web_rename_iis] REGEX = . DEST_KEY = MetaData:Sourcetype FORMAT = sourcetype::ms:iis [sourcetype_api_rename_iis] REGEX = \/Api\/ DEST_KEY = MetaData:Sourcetype FORMAT = sourcetype::ms:iis [web_index_rename_iis] REGEX = . DEST_KEY = _MetaData:Index FORMAT = index1 [api_index_rename_iis] REGEX = \/Api\/ DEST_KEY = _MetaData:Index FORMAT = index2

Or still easier:

in inputs.conf:

index=index1 sourcetype=sourcetype::ms:iis

in props.conf:

[sourcetype::ms::iis] TRANSFORMS-index_override = api_index_rename_iis

in transforms.conf:

[api_index_rename_iis] REGEX = \/Api\/ DEST_KEY = _MetaData:Index FORMAT = index2

In this way you have to override only index for api events.

Ciao.

Giuseppe