Splunk Search

How to extract multiple records from 1 log message

markrussell
New Member

I am having trouble extract the data from an apache log. Below is one message from the log, there is some header info, and then a string with a variable number of records which is the data that I am trying to extract.

99.59.54.101 - - [24/Aug/2010:00:01:04 -0700] "POST /PingCompanyRq HTTP/1.1" 200 474 "-" "QBUpdate/19.0/R10/pro" tcontent.quickbooks.com - - - 0 "(date-2010_08_23_04_02_19;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-NewFeatures;dur-0;bytes-0;a-0)(date-2010_08_23_04_02_19;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-Patch;dur-2;bytes-0;a-0)(date-2010_08_23_04_02_21;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-Help0;dur-0;bytes-0;a-0)(date-2010_08_23_04_02_21;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-Pro00;dur-0;bytes-0;a-0)(date-2010_08_23_04_02_21;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-SyncMgr;dur-0;bytes-0;a-0)(date-2010_08_24_03_00_54;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-Guide;dur-7;bytes-0;a-0)" "-"

That is one log entry, but you can see the sub records in (), that I want to pull out and run reports on each of those. There can be 1 to many of those records

I’d like to pull each field out of there, which you can see the format field name “ –“ Value. (date-2010_08_24_03_00_54;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-Guide;dur-7;bytes-0;a-0)

If anyone has any idea on a good way to extract these records, I would appreciate it.

Tags (1)
0 Karma

southeringtonp
Motivator

Or if you need a fully search-time approach:

...
| rex max_match=100 field=_raw "\((?<subrecord>.*?)\)"
| mvexpand subrecord
| eval _raw = record
| extract pairdelim=";", kvdelim="=-", auto=f
| table _time, src_ip, ver, n, err, s, conn, fg, bg, sd, chan, dur, bytes, a, subrecord

Advantage: Your web server logs are kept intact.

Disadvantage: Messy search strings

jbsplunk
Splunk Employee
Splunk Employee

You could do this in your props.conf:

   [sourcetype]
    LINE_BREAKER=()\(date-\d+_
    SHOULD_LINEMERGE=False 

That should pull out each 'sub-record' as a separate event.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...