I have a bunch of log files which as part of the first 4 lines or so sends a handshake in the form of
20140930093723895211 R 0019 ,prepare ENSAll)
Where ENSAll is a profile name. There is only ever one profile per source (each source file is actually per socket). What I need is to be able to attache this profile name to every event for a given source.
I started with a basic join but as we get up to 100k entries per file and the prepare is the first one, this search can take forever. Also, if the prepare is out of the time window it may not be registered.
I am at a complete loss how to speed up this query, it feels like it should be able to be indexed somehow since it is once per file, though not the first line necessarily.
Edit
I should clarify, the files look more like this:
...snip...
20140930084819093639 R 0010 (CM,000026
20140930084819561640 R 0016 ,prepare ENSall)
20140930084819561640 R 0010 (CM,000026
20140930084819561640 W 0019 (AK,000019,welcome)
20140930084819811240 W 0027 (AK,000027,starting:resume)
20140930084819811240 R 0014 ,start resume)
...snip...
And I need the profile name for each event following the first prepare.
... View more