Hello.
Trying to resolve an issue with routing log events. The goal is to route log events with an "Api" keyword to a separate index. Here is a log sample of three events (the first has no "Api" embedded, the next two have "Api"):
2020-08-12 23:04:24 W3SVC5 SERVER_1 XX.XX.XX.XXX GET / - 443 - XX.XX.XX.XX HTTP/0.9 - - - - 302 0 0 389 7 10
2020-08-12 23:04:24 W3SVC5 SERVER_1 XX.XX.XX.XXX GET /Api/TopicsUpdate/GetRecalculationServiceTopicsThatMustBeUpdated pageSize=1 443 system.service XX.XX.XX.XX HTTP/1.1 - - - XX.XX.XX.XX 200 0 0 597 175 44
2020-08-12 23:04:22 W3SVC5 SERVER_1 XX.XX.XX.XXX GET /Api/TopicsUpdate/GetRecalculationServiceTopicsThatMustBeUpdated pageSize=1 443 system.service XX.XX.XX.XX HTTP/1.1 - - - XX.XX.XX.XX 200 0 0 597 175 54
Here is the props.conf files:
[sourcetype1]
TRANSFORMS-set = sourcetype_web_rename_iis,sourcetype_api_rename_iis,web_index_rename_iis,api_index_rename_iis
Here is the transforms.conf:
[sourcetype_web_rename_iis]
REGEX = \d+\-\d+\-\d+\s\d+\:\d+\:\d+\s\w+\s\w+\s\d+.\d+\.\d+\.\d+\s\w+\s\/(?!Api)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::ms:iis
[sourcetype_api_rename_iis]
REGEX = \d+\-\d+\-\d+\s\d+\:\d+\:\d+\s\w+\s\w+\s\d+.\d+\.\d+\.\d+\s\w+\s\/Api\/
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::ms:iis
[web_index_rename_iis]
REGEX = \d+\-\d+\-\d+\s\d+\:\d+\:\d+\s\w+\s\w+\s\d+.\d+\.\d+\.\d+\s\w+\s\/(?!Api)
DEST_KEY = _MetaData:Index
FORMAT = index1
[api_index_rename_iis]
REGEX = \d+\-\d+\-\d+\s\d+\:\d+\:\d+\s\w+\s\w+\s\d+.\d+\.\d+\.\d+\s\w+\s\/Api\/
DEST_KEY = _MetaData:Index
FORMAT = index2
Are there any special considerations when using (?!...)?
Regards,
Max
Hi @vpsmax,
what's the problem in your configuration?
I don't like to have all the transformations in the same command, try to divide transformation in two commands in props.conf:
[sourcetype1]
TRANSFORMS-sourcetype_override = sourcetype_web_rename_iis, sourcetype_api_rename_iis
TRANSFORMS-index_override = web_index_rename_iis, api_index_rename_iis
Ciao.
Giuseppe
Hi Giuseppe,
Good morning. As suggested, I have tried separating the sourcetype and the index TRANSFORMS. As I am now seeing API data in the API index, I am good with ...
But for the ...
As all the IIS data is being stored in the IIS index, it appears the parsing is not happening. The goal is to simply store IIS data (and not API data) in the IIS index. Do you have any other suggestions?
Regards,
Max
Hi @vpsmax,
regexes aren't correct, please try these:
props.conf
[sourcetype1]
TRANSFORMS-sourcetype_override = sourcetype_web_rename_iis, sourcetype_api_rename_iis
TRANSFORMS-index_override = web_index_rename_iis, api_index_rename_iis
transforms.conf
[sourcetype_web_rename_iis]
REGEX = .
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::ms:iis
[sourcetype_api_rename_iis]
REGEX = \/Api\/
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::ms:iis
[web_index_rename_iis]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = index1
[api_index_rename_iis]
REGEX = \/Api\/
DEST_KEY = _MetaData:Index
FORMAT = index2
Or still easier:
in inputs.conf:
index=index1
sourcetype=sourcetype::ms:iis
in props.conf:
[sourcetype::ms::iis]
TRANSFORMS-index_override = api_index_rename_iis
in transforms.conf:
[api_index_rename_iis]
REGEX = \/Api\/
DEST_KEY = _MetaData:Index
FORMAT = index2
In this way you have to override only index for api events.
Ciao.
Giuseppe