Splunk Search

How to extract multiple records from 1 log message

markrussell
New Member

I am having trouble extract the data from an apache log. Below is one message from the log, there is some header info, and then a string with a variable number of records which is the data that I am trying to extract.

99.59.54.101 - - [24/Aug/2010:00:01:04 -0700] "POST /PingCompanyRq HTTP/1.1" 200 474 "-" "QBUpdate/19.0/R10/pro" tcontent.quickbooks.com - - - 0 "(date-2010_08_23_04_02_19;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-NewFeatures;dur-0;bytes-0;a-0)(date-2010_08_23_04_02_19;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-Patch;dur-2;bytes-0;a-0)(date-2010_08_23_04_02_21;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-Help0;dur-0;bytes-0;a-0)(date-2010_08_23_04_02_21;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-Pro00;dur-0;bytes-0;a-0)(date-2010_08_23_04_02_21;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-SyncMgr;dur-0;bytes-0;a-0)(date-2010_08_24_03_00_54;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-Guide;dur-7;bytes-0;a-0)" "-"

That is one log entry, but you can see the sub records in (), that I want to pull out and run reports on each of those. There can be 1 to many of those records

I’d like to pull each field out of there, which you can see the format field name “ –“ Value. (date-2010_08_24_03_00_54;ver-QBUpdate/19.0/R10/pro;n-1;nfg-0;err-0;s-1;conn-L;fg=0;bg-1;sd=0;chan-Guide;dur-7;bytes-0;a-0)

If anyone has any idea on a good way to extract these records, I would appreciate it.

Tags (1)
0 Karma

southeringtonp
Motivator

Or if you need a fully search-time approach:

...
| rex max_match=100 field=_raw "\((?<subrecord>.*?)\)"
| mvexpand subrecord
| eval _raw = record
| extract pairdelim=";", kvdelim="=-", auto=f
| table _time, src_ip, ver, n, err, s, conn, fg, bg, sd, chan, dur, bytes, a, subrecord

Advantage: Your web server logs are kept intact.

Disadvantage: Messy search strings

jbsplunk
Splunk Employee
Splunk Employee

You could do this in your props.conf:

   [sourcetype]
    LINE_BREAKER=()\(date-\d+_
    SHOULD_LINEMERGE=False 

That should pull out each 'sub-record' as a separate event.

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...