topic Re: How to extract fields from a multi line messages in Splunk Search

How to extract fields from a multi line messages

jdagenais — Wed, 01 Dec 2010 06:20:17 GMT

We have a multi line message that looks like this:

11/30/10 16:28:34 Verifying pricing env CLOSE,FX_CLOSE,XLA_ENV,INTRADAY,CPTY_CREDIT
   No exceptions for CLOSE, loaded in 0.05 secs
   Messages for FX_CLOSE
      PricerConfigRefresh: No item found for 1246892/CurveZero (CLOSE)  before Tue Nov 30 16:28:34 EST 2010
   No exceptions for FX_CLOSE, loaded in 0.17 secs
   No exceptions for XLA_ENV, loaded in 0.05 secs
   No exceptions for INTRADAY, loaded in 0.10 secs
   No exceptions for CPTY_CREDIT, loaded in 0.40 sec

I would like to create the following fields and assign the numeric values as response time, so I can search and chart on the RT values.

  close_pe_rt=0.05
  fx_close_pe_rt=0.17
  xla_env_pe_rt=0.05
  intraday_pe_rt=0.10
  cpty_credit_pe_rt=0.40

What is the best way to handle such extraction and assignments?

Thanks very much for your help, Jean

Re: How to extract fields from a multi line messages

ziegfried — Wed, 01 Dec 2010 07:53:37 GMT

You can either use multiple regular expressions (multiple rex command calls or multiple EXTRACT-* options in props.conf) or a single regex with the (?m) multiline flag (maybe as well the (?s) DOTALL flag).

Someting like this might work:

... | rex "(?ms)No exceptions for CLOSE, loaded in (?<close_pe_rt>[\d\.]+).+FX_CLOSE, loaded in (?<fx_close_pe_rt>[\d\.]+).+XLA_ENV, loaded in (?<xla_env_pe_rt>[\d\.]+).+INTRADAY, loaded in (?<intraday_pe_rt>[\d\.]+).+CPTY_CREDIT, loaded in (?<cpty_credit_pe_rt>[\d\.]+)"

Re: How to extract fields from a multi line messages

hulahoop — Wed, 01 Dec 2010 07:54:14 GMT

These are the 2 options I would try

configuration files
rex command in the search bar

The easiest, but also most transient, option is to use rex command inline in your search. For example:

sourcetype="multiline" | rex "CLOSE, loaded in (?<close_pe_rt>\S+)" | rex "FX_CLOSE, loaded in (?<fx_close_pe_rt>\S+)" | rex "XLA_ENV, loaded in (?<xla_env_pe_rt>\S+)" | rex "INTRADAY, loaded in (?<intraday_pe_rt>\S+)" | rex "CPTY_CREDIT, loaded in (?<cpty_credit_pe_rt>\S+)"

Maybe there's a way to do this in one rex invocation, but I tried several things which didn't work.

The other option is to add a few stanzas to props.conf and transforms.conf. For example,

in props.conf:

[multiline] REPORT-foo = mlFields

in transforms.conf:

[mlFields] REGEX = CLOSE, loaded in (\S+).* FX_CLOSE, loaded in (\S+).* XLA_ENV, loaded in (\S+).* INTRADAY, loaded in (\S+).* CPTY_CREDIT, loaded in (\S+) FORMAT = close_pe_rt::$1 fx_close_pe_rt::$2 xla_env_pe_rt::$3 intraday_pe_rt::$4 cpty_credit_pe_rt::$5

You could also try using the Interactive Field Extractor (IFX).

Re: How to extract fields from a multi line messages

hulahoop — Wed, 01 Dec 2010 07:57:17 GMT

Note: I did not have to use the (?m) regex modifier in the REGEX field for transforms.conf. Somewhere along the way, Splunk automatically knows how to deal with multiline events.

Re: How to extract fields from a multi line messages

jdagenais — Thu, 02 Dec 2010 23:22:44 GMT

Thank you very much.

This solution worked quite well and I implemented it with the props.conf and transforms.conf.