topic Break Log file with header and details into events in Getting Data In

Break Log file with header and details into events

warrenpage — Wed, 27 Apr 2011 01:11:06 GMT

I have a log file that looks generally like this

Header data  time=xxxxxx  databasename=yyyyyyy  numberortables=xx
Detail tablename=table1  rowsread=1111
Detail tablename=table2  rowsread=2222

I know I could break events on each detail record. However is there a way to configure my props.conf file so it pushes/collect the Data from the header into the detail events somehow?

This is so I could do a search on say databasename=xxx and tablename=yyy?

Re: Break Log file with header and details into events

hazekamp — Wed, 27 Apr 2011 06:32:55 GMT

Generally speaking there is no way to force that Header into each Detail record such that you have:

Event 1:
Header 1
Detail A

Event 2:
Header 1
Detail B

Depending on the number of "Detail" records you could create a multi-line event. I wouldn't recommend creating events w/ linecount>500. Given that the row count for an event = (# of tables) + 1 (header) you should be under that mark. You would essentially be creating a single event for each database.

Event 1:
Header 1
Detail A
Detail B
Detail C

Event 2:
Header 2
Detail X
Detail Y
Detail Z

Re: Break Log file with header and details into events

warrenpage — Fri, 29 Apr 2011 01:38:27 GMT

Thanks I suspected as much.

I think I will just need to run the file through awk first to create full detail records like

time=xxxxxx databasename=yyyyyyy tablename=table1 rowsread=1111
time=xxxxxx databasename=yyyyyyy tablename=table2 rowsread=2222

In the meantime just doing without the header fields and marking the details as events separately.