Extracting fields in payload xml multiline log

[changux]

Hi all.

I need help setting an input and extracting multiline fields with one entry like this:

####<May 2, 2015 23:37:26 PM PCT> <Warning> <TGFG Logging> <host> <source> <[ACTIVE] ExecuteThread: '26' for queue: 'jboss.kernel.Default (self-tuning)'> <<user>> <> <y7433edf553abf7c3:-12a453ef2:148c05609dd:-5000-000000000000342> <54564444> <JB-000000> < [FILE34, null, null, REQUEST] <!--- Input BD Profiles -->: <soapenv:Body xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <ns:RunService xmlns:ns="http://mysite.com/s3" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <ns:data>
      <con:java-content ref="jcid:-12ac44f2:35t54ed33:-4ea8" xmlns:con="http://www.jobss.com/wli/sb/context"/>
    </ns:data>
  </ns:RunService>
</soapenv:Body>>

Useful info:

• All the events begin with the same:

• I need to extract the field "Status", in the example is the single word between the < > after the timestamp (<Warning>).

• Also, i need to extract the <host>

• I need to extract the <source>

• Everything left must be called "Payload".

Anyone can help me?

Thanks!

[woodcock · Accepted solution]

Like I said, fix your linebreaking and timestampinginprops.conf` first or you'll never get anything off the ground but once you do, this will work:

| rex "^####<[^>]+>\s*<(?<Status>[^>]+)>\s*<[^>]+>\s*<(?<Host>[^>]+)>\s*<(?<Source>[^>]+)>\s*(?<Payload>.*)$"

[changux]

You rock! thanks!

[changux]

Works very nice. How i can extract only the "payload"?

Thank you so much!

[woodcock]

| rex "^####<[^>]+>\s*<[^>]+>\s*<[^>]+>\s*<[^>]+>\s*<[^>]+>\s*(?<Payload>.*)$"

Don't forget to click "Accept".

[aljohnson_splun]

If they're already editing props.conf, why would you suggest they use rex instead of adding an EXTRACT ?

[lcrielaa]

| rex field=_raw "####<[^>]+>\s<(?[^>]+)>\s<[^>]+>\s<(?[^>]+)>\s<(?[^>]+)>\s(?.*)"

That'll do the extracting you want but like the other said, you'll need to fix your linebreaking and timestamping via your props.conf.

[woodcock]

All that you are asking is possible and not too difficult but you are WAY ahead of yourself. You need to get timestamping and linebreaking working first. Are you working on that?

[changux]

Yes, but doesn't work properly.

[woodcock]

Do you have anything at all coming into Splunk yet? In other words, do you have inputs.conf working and timestamp recognition working?

[changux]

Not for now. My test extracts fields with the symbols ( <> ), not only the content 😞