I am having trouble extract the data from an apache log. Below is one message from the log, there is some header info, and then a string with a variable number of records which is the data that I am trying to extract.
99.59.54.101 - - [24/Aug/2010:00:01:04 -0700] "POST /PingCompanyRq HTTP/1.1" 200 474 "-" "QBUpdate/19.0/R10/pro" tcontent.quickbooks.com - - - 0 "(date-2010_08_23_04_02_19;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-NewFeatures;dur-0;bytes-0;a-0)(date-2010_08_23_04_02_19;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-Patch;dur-2;bytes-0;a-0)(date-2010_08_23_04_02_21;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-Help0;dur-0;bytes-0;a-0)(date-2010_08_23_04_02_21;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-Pro00;dur-0;bytes-0;a-0)(date-2010_08_23_04_02_21;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-SyncMgr;dur-0;bytes-0;a-0)(date-2010_08_24_03_00_54;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-Guide;dur-7;bytes-0;a-0)" "-"
That is one log entry, but you can see the sub records in (), that I want to pull out and run reports on each of those. There can be 1 to many of those records
I’d like to pull each field out of there, which you can see the format field name “ –“ Value. (date-2010_08_24_03_00_54;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-Guide;dur-7;bytes-0;a-0)
If anyone has any idea on a good way to extract these records, I would appreciate it.
Or if you need a fully search-time approach:
...
| rex max_match=100 field=_raw "\((?<subrecord>.*?)\)"
| mvexpand subrecord
| eval _raw = record
| extract pairdelim=";", kvdelim="=-", auto=f
| table _time, src_ip, ver, n, err, s, conn, fg, bg, sd, chan, dur, bytes, a, subrecord
Advantage: Your web server logs are kept intact.
Disadvantage: Messy search strings
You could do this in your props.conf:
[sourcetype]
LINE_BREAKER=()\(date-\d+_
SHOULD_LINEMERGE=False
That should pull out each 'sub-record' as a separate event.