How to extract multiple records from 1 log message

[markrussell]

I am having trouble extract the data from an apache log. Below is one message from the log, there is some header info, and then a string with a variable number of records which is the data that I am trying to extract.

99.59.54.101 - - [24/Aug/2010:00:01:04 -0700] "POST /PingCompanyRq HTTP/1.1" 200 474 "-" "QBUpdate/19.0/R10/pro" tcontent.quickbooks.com - - - 0 "(date-2010_08_23_04_02_19;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-NewFeatures;dur-0;bytes-0;a-0)(date-2010_08_23_04_02_19;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-Patch;dur-2;bytes-0;a-0)(date-2010_08_23_04_02_21;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-Help0;dur-0;bytes-0;a-0)(date-2010_08_23_04_02_21;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-Pro00;dur-0;bytes-0;a-0)(date-2010_08_23_04_02_21;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-SyncMgr;dur-0;bytes-0;a-0)(date-2010_08_24_03_00_54;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-Guide;dur-7;bytes-0;a-0)" "-"

That is one log entry, but you can see the sub records in (), that I want to pull out and run reports on each of those. There can be 1 to many of those records

I’d like to pull each field out of there, which you can see the format field name “ –“ Value. (date-2010_08_24_03_00_54;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-Guide;dur-7;bytes-0;a-0)

If anyone has any idea on a good way to extract these records, I would appreciate it.

[southeringtonp]

Or if you need a fully search-time approach:

...
| rex max_match=100 field=_raw "\((?<subrecord>.*?)\)"
| mvexpand subrecord
| eval _raw = record
| extract pairdelim=";", kvdelim="=-", auto=f
| table _time, src_ip, ver, n, err, s, conn, fg, bg, sd, chan, dur, bytes, a, subrecord

Advantage: Your web server logs are kept intact.

Disadvantage: Messy search strings

[jbsplunk]

You could do this in your props.conf:

   [sourcetype]
    LINE_BREAKER=()\(date-\d+_
    SHOULD_LINEMERGE=False 

That should pull out each 'sub-record' as a separate event.