multiline regex

[briang67]

I have an unstructured log file that looks like the following. How would I go about creating key/value pairs for metrics like "Queue Additions Max Time" or "Data Insertions Avg Time" when part of the qualifier for the field name spans a different line than the metric name and value?
thanks

17-09-2013 17:36:58,489 CDT INFO  [scheduler-2] [org.hyperic.hq.common.DiagnosticsLogger@105] [com.hyperic.hq.measurement.server.session.BatchAggregateDiagnostic@75a20d1c] Batch Aggregate DataInserter Diagnostics
Configuration:
    Workers:    10
    QueueSize:  500000
    BatchSize:  1000
Queue Additions:
    # calls:    1585
    Max time:   12 ms
    Avg time:   0 ms
Data Insertions:
    # calls:    966
    Max time:   640 ms
    Avg time:   27 ms
Queue size:
    # entries:  0
    Max size:   1403

[briang67]

I eventually got this to work using a complex regex that included newline chars. This is probably not the most efficient/elegant way to handle this, but I was able to make it work. Thanks to everyone who answered.

[_d_]

Unfortunately there is no automatic way to do this but you can use multiple extractions ordered appropriately. In props.conf you can do inline EXTRACTS-xxx that extract configuration, queue_additions, data_insertions and queue_size fields, then use REPORT-yyy scoped on each one with FIELDS names of your liking.

[lukejadamec]

Assuming “Max time” and “Ave time” are recognized as fields:
You can use mvindex to identify which value you want. For example, the following search will pull out "Queue Additions Max Time" and "Data Insertions Avg Time":

search | eval Queue_Additions_Max_Time =mvindex(Max_time,0) | eval Queue_Additions_Avg_Time =mvindex(Ave_time,0) |  eval Data_Insertions_Max_Time =mvindex(Max_time,1) eval Data_Insertions_Avg_Time =mvindex(Ave_time,1) |table Queue_Additions_Max_Time, Queue_Additions_Avg_Time ,Data_Insertions_Max_Time, Data_Insertions_Avg_Time

As you can see, the 0 pulls the first occurrence in a multivalue field, and 1 pulls the second occurrence.

Let us know if “Max time” and “Ave time” are not automatically recognized as fields because a rex function can be used to create the fields.

[lukejadamec]

Can you post your regex?

A rex might be better.

[briang67]

This search is failing for me with an "Error in 'eval' command: The operator at 'eval Data_Insertions_Avg_Time =mvindex(avg_time,1)' is invalid."

I believe this is because my regex that creates the max_time field is only matching against the first instance of max_time in the event. I think if I can fix that this query will work.

[lguinn2]

Did you look at other answers? This one seems to be relevant

http://answers.splunk.com/answers/38753/regex-for-multiline-events

[lukejadamec]

Have you tried a search | regex yet?