All Apps and Add-ons

Routing data to separate index using (?!...)

vpsmax
Path Finder

Hello.

Trying to resolve an issue with routing log events.  The goal is to route log events with an "Api" keyword to a separate index.  Here is a log sample of three events (the first has no "Api" embedded, the next two have "Api"):

2020-08-12 23:04:24 W3SVC5 SERVER_1 XX.XX.XX.XXX GET / - 443 - XX.XX.XX.XX HTTP/0.9 - - - - 302 0 0 389 7 10

2020-08-12 23:04:24 W3SVC5 SERVER_1 XX.XX.XX.XXX GET /Api/TopicsUpdate/GetRecalculationServiceTopicsThatMustBeUpdated pageSize=1 443 system.service XX.XX.XX.XX HTTP/1.1 - - - XX.XX.XX.XX 200 0 0 597 175 44

2020-08-12 23:04:22 W3SVC5 SERVER_1 XX.XX.XX.XXX GET /Api/TopicsUpdate/GetRecalculationServiceTopicsThatMustBeUpdated pageSize=1 443 system.service XX.XX.XX.XX HTTP/1.1 - - - XX.XX.XX.XX 200 0 0 597 175 54

Here is the props.conf files:

[sourcetype1]
TRANSFORMS-set = sourcetype_web_rename_iis,sourcetype_api_rename_iis,web_index_rename_iis,api_index_rename_iis

Here is the transforms.conf:

[sourcetype_web_rename_iis]
REGEX = \d+\-\d+\-\d+\s\d+\:\d+\:\d+\s\w+\s\w+\s\d+.\d+\.\d+\.\d+\s\w+\s\/(?!Api)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::ms:iis

[sourcetype_api_rename_iis]
REGEX = \d+\-\d+\-\d+\s\d+\:\d+\:\d+\s\w+\s\w+\s\d+.\d+\.\d+\.\d+\s\w+\s\/Api\/
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::ms:iis

[web_index_rename_iis]
REGEX = \d+\-\d+\-\d+\s\d+\:\d+\:\d+\s\w+\s\w+\s\d+.\d+\.\d+\.\d+\s\w+\s\/(?!Api)
DEST_KEY = _MetaData:Index
FORMAT = index1

[api_index_rename_iis]
REGEX = \d+\-\d+\-\d+\s\d+\:\d+\:\d+\s\w+\s\w+\s\d+.\d+\.\d+\.\d+\s\w+\s\/Api\/
DEST_KEY = _MetaData:Index
FORMAT = index2

Are there any special considerations when using (?!...)?

Regards,
Max

Labels (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @vpsmax,

what's the problem in your configuration?

  • is the sourcetype override running or not?
  • is the index override running or not?

I don't like to have all the transformations in the same command, try to divide transformation in two commands in props.conf:

[sourcetype1]
TRANSFORMS-sourcetype_override = sourcetype_web_rename_iis, sourcetype_api_rename_iis
TRANSFORMS-index_override = web_index_rename_iis, api_index_rename_iis

 Ciao.

Giuseppe

0 Karma

vpsmax
Path Finder

Hi Giuseppe,

Good morning.  As suggested, I have tried separating the sourcetype and the index TRANSFORMS.  As I am now seeing API data in the API index, I am good with ...

  • sourcetype_api_rename_iis
  • api_index_rename_iis

But for the ...

  • sourcetype_web_rename_iis
  • web_index_rename_iis

As all the IIS data is being stored in the IIS index, it appears the parsing is not happening.  The goal is to simply store IIS data (and not API data) in the IIS index.  Do you have any other suggestions?

Regards,
Max

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @vpsmax,

regexes aren't correct, please try these:

props.conf

 

[sourcetype1]
TRANSFORMS-sourcetype_override = sourcetype_web_rename_iis, sourcetype_api_rename_iis
TRANSFORMS-index_override = web_index_rename_iis, api_index_rename_iis

 

transforms.conf

 

[sourcetype_web_rename_iis]
REGEX = .
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::ms:iis

[sourcetype_api_rename_iis]
REGEX = \/Api\/
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::ms:iis

[web_index_rename_iis]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = index1

[api_index_rename_iis]
REGEX = \/Api\/
DEST_KEY = _MetaData:Index
FORMAT = index2

 

Or still easier:

in inputs.conf:

index=index1
sourcetype=sourcetype::ms:iis

in props.conf:

[sourcetype::ms::iis]
TRANSFORMS-index_override = api_index_rename_iis

in transforms.conf:

[api_index_rename_iis]
REGEX = \/Api\/
DEST_KEY = _MetaData:Index
FORMAT = index2

In this way you have to override only index for api events.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...