All Apps and Add-ons

Routing data to separate index using (?!...)

vpsmax
Path Finder

Hello.

Trying to resolve an issue with routing log events.  The goal is to route log events with an "Api" keyword to a separate index.  Here is a log sample of three events (the first has no "Api" embedded, the next two have "Api"):

2020-08-12 23:04:24 W3SVC5 SERVER_1 XX.XX.XX.XXX GET / - 443 - XX.XX.XX.XX HTTP/0.9 - - - - 302 0 0 389 7 10

2020-08-12 23:04:24 W3SVC5 SERVER_1 XX.XX.XX.XXX GET /Api/TopicsUpdate/GetRecalculationServiceTopicsThatMustBeUpdated pageSize=1 443 system.service XX.XX.XX.XX HTTP/1.1 - - - XX.XX.XX.XX 200 0 0 597 175 44

2020-08-12 23:04:22 W3SVC5 SERVER_1 XX.XX.XX.XXX GET /Api/TopicsUpdate/GetRecalculationServiceTopicsThatMustBeUpdated pageSize=1 443 system.service XX.XX.XX.XX HTTP/1.1 - - - XX.XX.XX.XX 200 0 0 597 175 54

Here is the props.conf files:

[sourcetype1]
TRANSFORMS-set = sourcetype_web_rename_iis,sourcetype_api_rename_iis,web_index_rename_iis,api_index_rename_iis

Here is the transforms.conf:

[sourcetype_web_rename_iis]
REGEX = \d+\-\d+\-\d+\s\d+\:\d+\:\d+\s\w+\s\w+\s\d+.\d+\.\d+\.\d+\s\w+\s\/(?!Api)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::ms:iis

[sourcetype_api_rename_iis]
REGEX = \d+\-\d+\-\d+\s\d+\:\d+\:\d+\s\w+\s\w+\s\d+.\d+\.\d+\.\d+\s\w+\s\/Api\/
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::ms:iis

[web_index_rename_iis]
REGEX = \d+\-\d+\-\d+\s\d+\:\d+\:\d+\s\w+\s\w+\s\d+.\d+\.\d+\.\d+\s\w+\s\/(?!Api)
DEST_KEY = _MetaData:Index
FORMAT = index1

[api_index_rename_iis]
REGEX = \d+\-\d+\-\d+\s\d+\:\d+\:\d+\s\w+\s\w+\s\d+.\d+\.\d+\.\d+\s\w+\s\/Api\/
DEST_KEY = _MetaData:Index
FORMAT = index2

Are there any special considerations when using (?!...)?

Regards,
Max

Labels (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @vpsmax,

what's the problem in your configuration?

  • is the sourcetype override running or not?
  • is the index override running or not?

I don't like to have all the transformations in the same command, try to divide transformation in two commands in props.conf:

[sourcetype1]
TRANSFORMS-sourcetype_override = sourcetype_web_rename_iis, sourcetype_api_rename_iis
TRANSFORMS-index_override = web_index_rename_iis, api_index_rename_iis

 Ciao.

Giuseppe

0 Karma

vpsmax
Path Finder

Hi Giuseppe,

Good morning.  As suggested, I have tried separating the sourcetype and the index TRANSFORMS.  As I am now seeing API data in the API index, I am good with ...

  • sourcetype_api_rename_iis
  • api_index_rename_iis

But for the ...

  • sourcetype_web_rename_iis
  • web_index_rename_iis

As all the IIS data is being stored in the IIS index, it appears the parsing is not happening.  The goal is to simply store IIS data (and not API data) in the IIS index.  Do you have any other suggestions?

Regards,
Max

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @vpsmax,

regexes aren't correct, please try these:

props.conf

 

[sourcetype1]
TRANSFORMS-sourcetype_override = sourcetype_web_rename_iis, sourcetype_api_rename_iis
TRANSFORMS-index_override = web_index_rename_iis, api_index_rename_iis

 

transforms.conf

 

[sourcetype_web_rename_iis]
REGEX = .
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::ms:iis

[sourcetype_api_rename_iis]
REGEX = \/Api\/
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::ms:iis

[web_index_rename_iis]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = index1

[api_index_rename_iis]
REGEX = \/Api\/
DEST_KEY = _MetaData:Index
FORMAT = index2

 

Or still easier:

in inputs.conf:

index=index1
sourcetype=sourcetype::ms:iis

in props.conf:

[sourcetype::ms::iis]
TRANSFORMS-index_override = api_index_rename_iis

in transforms.conf:

[api_index_rename_iis]
REGEX = \/Api\/
DEST_KEY = _MetaData:Index
FORMAT = index2

In this way you have to override only index for api events.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...