Splunk Search

How to extract fields from a multi line messages

jdagenais
Explorer

We have a multi line message that looks like this:

11/30/10 16:28:34 Verifying pricing env CLOSE,FX_CLOSE,XLA_ENV,INTRADAY,CPTY_CREDIT
   No exceptions for CLOSE, loaded in 0.05 secs
   Messages for FX_CLOSE
      PricerConfigRefresh: No item found for 1246892/CurveZero (CLOSE)  before Tue Nov 30 16:28:34 EST 2010
   No exceptions for FX_CLOSE, loaded in 0.17 secs
   No exceptions for XLA_ENV, loaded in 0.05 secs
   No exceptions for INTRADAY, loaded in 0.10 secs
   No exceptions for CPTY_CREDIT, loaded in 0.40 sec

I would like to create the following fields and assign the numeric values as response time, so I can search and chart on the RT values.

  close_pe_rt=0.05
  fx_close_pe_rt=0.17
  xla_env_pe_rt=0.05
  intraday_pe_rt=0.10
  cpty_credit_pe_rt=0.40

What is the best way to handle such extraction and assignments?

Thanks very much for your help, Jean

Tags (1)
1 Solution

hulahoop
Splunk Employee
Splunk Employee

These are the 2 options I would try

  • configuration files
  • rex command in the search bar

The easiest, but also most transient, option is to use rex command inline in your search. For example:

sourcetype="multiline" | rex "CLOSE, loaded in (?<close_pe_rt>\S+)" | rex "FX_CLOSE, loaded in (?<fx_close_pe_rt>\S+)" | rex "XLA_ENV, loaded in (?<xla_env_pe_rt>\S+)" | rex "INTRADAY, loaded in (?<intraday_pe_rt>\S+)" | rex "CPTY_CREDIT, loaded in (?<cpty_credit_pe_rt>\S+)"

Maybe there's a way to do this in one rex invocation, but I tried several things which didn't work.

The other option is to add a few stanzas to props.conf and transforms.conf. For example,

in props.conf:

[multiline] REPORT-foo = mlFields

in transforms.conf:

[mlFields] REGEX = CLOSE, loaded in (\S+).* FX_CLOSE, loaded in (\S+).* XLA_ENV, loaded in (\S+).* INTRADAY, loaded in (\S+).* CPTY_CREDIT, loaded in (\S+) FORMAT = close_pe_rt::$1 fx_close_pe_rt::$2 xla_env_pe_rt::$3 intraday_pe_rt::$4 cpty_credit_pe_rt::$5

You could also try using the Interactive Field Extractor (IFX).

View solution in original post

hulahoop
Splunk Employee
Splunk Employee

These are the 2 options I would try

  • configuration files
  • rex command in the search bar

The easiest, but also most transient, option is to use rex command inline in your search. For example:

sourcetype="multiline" | rex "CLOSE, loaded in (?<close_pe_rt>\S+)" | rex "FX_CLOSE, loaded in (?<fx_close_pe_rt>\S+)" | rex "XLA_ENV, loaded in (?<xla_env_pe_rt>\S+)" | rex "INTRADAY, loaded in (?<intraday_pe_rt>\S+)" | rex "CPTY_CREDIT, loaded in (?<cpty_credit_pe_rt>\S+)"

Maybe there's a way to do this in one rex invocation, but I tried several things which didn't work.

The other option is to add a few stanzas to props.conf and transforms.conf. For example,

in props.conf:

[multiline] REPORT-foo = mlFields

in transforms.conf:

[mlFields] REGEX = CLOSE, loaded in (\S+).* FX_CLOSE, loaded in (\S+).* XLA_ENV, loaded in (\S+).* INTRADAY, loaded in (\S+).* CPTY_CREDIT, loaded in (\S+) FORMAT = close_pe_rt::$1 fx_close_pe_rt::$2 xla_env_pe_rt::$3 intraday_pe_rt::$4 cpty_credit_pe_rt::$5

You could also try using the Interactive Field Extractor (IFX).

jdagenais
Explorer

Thank you very much.

This solution worked quite well and I implemented it with the props.conf and transforms.conf.

0 Karma

hulahoop
Splunk Employee
Splunk Employee

Note: I did not have to use the (?m) regex modifier in the REGEX field for transforms.conf. Somewhere along the way, Splunk automatically knows how to deal with multiline events.

0 Karma

ziegfried
Influencer

You can either use multiple regular expressions (multiple rex command calls or multiple EXTRACT-* options in props.conf) or a single regex with the (?m) multiline flag (maybe as well the (?s) DOTALL flag).

Someting like this might work:

... | rex "(?ms)No exceptions for CLOSE, loaded in (?<close_pe_rt>[\d\.]+).+FX_CLOSE, loaded in (?<fx_close_pe_rt>[\d\.]+).+XLA_ENV, loaded in (?<xla_env_pe_rt>[\d\.]+).+INTRADAY, loaded in (?<intraday_pe_rt>[\d\.]+).+CPTY_CREDIT, loaded in (?<cpty_credit_pe_rt>[\d\.]+)"
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...