topic Re: Extracting fields in payload xml multiline log in Splunk Search

Extracting fields in payload xml multiline log

changux — Tue, 26 May 2015 02:41:20 GMT

Hi all.

I need help setting an input and extracting multiline fields with one entry like this:

####<May 2, 2015 23:37:26 PM PCT> <Warning> <TGFG Logging> <host> <source> <[ACTIVE] ExecuteThread: '26' for queue: 'jboss.kernel.Default (self-tuning)'> <<user>> <> <y7433edf553abf7c3:-12a453ef2:148c05609dd:-5000-000000000000342> <54564444> <JB-000000> < [FILE34, null, null, REQUEST] <!--- Input BD Profiles -->: <soapenv:Body xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <ns:RunService xmlns:ns="http://mysite.com/s3" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <ns:data>
      <con:java-content ref="jcid:-12ac44f2:35t54ed33:-4ea8" xmlns:con="http://www.jobss.com/wli/sb/context"/>
    </ns:data>
  </ns:RunService>
</soapenv:Body>>

Useful info:

All the events begin with the same:
I need to extract the field "Status", in the example is the single word between the < > after the timestamp (<Warning>).
Also, i need to extract the <host>
I need to extract the <source>
Everything left must be called "Payload".

Anyone can help me?

Thanks!

Re: Extracting fields in payload xml multiline log

woodcock — Tue, 26 May 2015 02:56:27 GMT

Do you have anything at all coming into Splunk yet? In other words, do you have inputs.conf working and timestamp recognition working?

Re: Extracting fields in payload xml multiline log

changux — Tue, 26 May 2015 03:03:15 GMT

Not for now. My test extracts fields with the symbols ( <> ), not only the content 😞

Re: Extracting fields in payload xml multiline log

woodcock — Tue, 26 May 2015 12:31:28 GMT

All that you are asking is possible and not too difficult but you are WAY ahead of yourself. You need to get timestamping and linebreaking working first. Are you working on that?

Re: Extracting fields in payload xml multiline log

changux — Tue, 02 Jun 2015 19:49:11 GMT

Yes, but doesn't work properly.

Re: Extracting fields in payload xml multiline log

lcrielaa — Wed, 03 Jun 2015 06:38:32 GMT

| rex field=_raw "####<[^>]+>\s<(?[^>]+)>\s<[^>]+>\s<(?[^>]+)>\s<(?[^>]+)>\s(?.*)"

That'll do the extracting you want but like the other said, you'll need to fix your linebreaking and timestamping via your props.conf.

Re: Extracting fields in payload xml multiline log

woodcock — Wed, 03 Jun 2015 13:22:26 GMT

Like I said, fix your linebreaking and timestampinginprops.conf` first or you'll never get anything off the ground but once you do, this will work:

| rex "^####<[^>]+>\s*<(?<Status>[^>]+)>\s*<[^>]+>\s*<(?<Host>[^>]+)>\s*<(?<Source>[^>]+)>\s*(?<Payload>.*)$"

Re: Extracting fields in payload xml multiline log

changux — Wed, 03 Jun 2015 20:35:13 GMT

Works very nice. How i can extract only the "payload"?

Thank you so much!

Re: Extracting fields in payload xml multiline log

woodcock — Wed, 03 Jun 2015 20:55:02 GMT

| rex "^####<[^>]+>\s*<[^>]+>\s*<[^>]+>\s*<[^>]+>\s*<[^>]+>\s*(?<Payload>.*)$"

Don't forget to click "Accept".

Re: Extracting fields in payload xml multiline log

changux — Wed, 03 Jun 2015 21:11:36 GMT

You rock! thanks!

Re: Extracting fields in payload xml multiline log

aljohnson_splun — Wed, 03 Jun 2015 21:48:22 GMT

If they're already editing props.conf, why would you suggest they use rex instead of adding an EXTRACT ?