Routing data to separate index using (?!...)

[vpsmax]

Hello.

Trying to resolve an issue with routing log events.  The goal is to route log events with an "Api" keyword to a separate index.  Here is a log sample of three events (the first has no "Api" embedded, the next two have "Api"):

2020-08-12 23:04:24 W3SVC5 SERVER_1 XX.XX.XX.XXX GET / - 443 - XX.XX.XX.XX HTTP/0.9 - - - - 302 0 0 389 7 10

2020-08-12 23:04:24 W3SVC5 SERVER_1 XX.XX.XX.XXX GET /Api/TopicsUpdate/GetRecalculationServiceTopicsThatMustBeUpdated pageSize=1 443 system.service XX.XX.XX.XX HTTP/1.1 - - - XX.XX.XX.XX 200 0 0 597 175 44

2020-08-12 23:04:22 W3SVC5 SERVER_1 XX.XX.XX.XXX GET /Api/TopicsUpdate/GetRecalculationServiceTopicsThatMustBeUpdated pageSize=1 443 system.service XX.XX.XX.XX HTTP/1.1 - - - XX.XX.XX.XX 200 0 0 597 175 54

Here is the props.conf files:

[sourcetype1]
TRANSFORMS-set = sourcetype_web_rename_iis,sourcetype_api_rename_iis,web_index_rename_iis,api_index_rename_iis

Here is the transforms.conf:

[sourcetype_web_rename_iis]
REGEX = \d+\-\d+\-\d+\s\d+\:\d+\:\d+\s\w+\s\w+\s\d+.\d+\.\d+\.\d+\s\w+\s\/(?!Api)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::ms:iis

[sourcetype_api_rename_iis]
REGEX = \d+\-\d+\-\d+\s\d+\:\d+\:\d+\s\w+\s\w+\s\d+.\d+\.\d+\.\d+\s\w+\s\/Api\/
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::ms:iis

[web_index_rename_iis]
REGEX = \d+\-\d+\-\d+\s\d+\:\d+\:\d+\s\w+\s\w+\s\d+.\d+\.\d+\.\d+\s\w+\s\/(?!Api)
DEST_KEY = _MetaData:Index
FORMAT = index1

[api_index_rename_iis]
REGEX = \d+\-\d+\-\d+\s\d+\:\d+\:\d+\s\w+\s\w+\s\d+.\d+\.\d+\.\d+\s\w+\s\/Api\/
DEST_KEY = _MetaData:Index
FORMAT = index2

Are there any special considerations when using (?!...)?

Regards,
Max

[gcusello]

Hi @vpsmax,

what's the problem in your configuration?

• is the sourcetype override running or not?
• is the index override running or not?

I don't like to have all the transformations in the same command, try to divide transformation in two commands in props.conf:

[sourcetype1]
TRANSFORMS-sourcetype_override = sourcetype_web_rename_iis, sourcetype_api_rename_iis
TRANSFORMS-index_override = web_index_rename_iis, api_index_rename_iis

 Ciao.

Giuseppe

[vpsmax]

Hi Giuseppe,

Good morning.  As suggested, I have tried separating the sourcetype and the index TRANSFORMS.  As I am now seeing API data in the API index, I am good with ...

• sourcetype_api_rename_iis
• api_index_rename_iis

But for the ...

• sourcetype_web_rename_iis
• web_index_rename_iis

As all the IIS data is being stored in the IIS index, it appears the parsing is not happening.  The goal is to simply store IIS data (and not API data) in the IIS index.  Do you have any other suggestions?

Regards,
Max

[vpsmax]

Regards,
Max

[gcusello]

Hi @vpsmax,

regexes aren't correct, please try these:

props.conf

[sourcetype1]
TRANSFORMS-sourcetype_override = sourcetype_web_rename_iis, sourcetype_api_rename_iis
TRANSFORMS-index_override = web_index_rename_iis, api_index_rename_iis

transforms.conf

[sourcetype_web_rename_iis]
REGEX = .
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::ms:iis

[sourcetype_api_rename_iis]
REGEX = \/Api\/
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::ms:iis

[web_index_rename_iis]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = index1

[api_index_rename_iis]
REGEX = \/Api\/
DEST_KEY = _MetaData:Index
FORMAT = index2

Or still easier:

in inputs.conf:

index=index1
sourcetype=sourcetype::ms:iis

in props.conf:

[sourcetype::ms::iis]
TRANSFORMS-index_override = api_index_rename_iis

in transforms.conf:

[api_index_rename_iis]
REGEX = \/Api\/
DEST_KEY = _MetaData:Index
FORMAT = index2

In this way you have to override only index for api events.

Ciao.

Giuseppe