How to split an event into parts and recombine with a common header?

[curtisjester]

Hey all - thanks in advance!

I have _raw log data that contains a header section and then what appears to be two entries within itself. I want to split these entries (they are formatted the same, except the latter appends a '1' onto each fieldname) and then create two events from this one event, like so:

Before
_raw = HEADER|PART1|PART2
After
event1 = HEADER|PART1
event2 = HEADER|PART2

An event will come from the same IP and device name; the parts are paths and simple fields. Here is a sample log (bracketed to show how I want it split, but these brackets are not in the raw data):

[Jun 12 23:00:09 server.i.j WRD: 0|AName|Named Application Server|1|0|Rule|0|ClientTime=7:02:28-PM CompName=sparse.info.given.here IPv4=x.x.x.x] [Path=No-Results-Found MD5= Size= Modified= RuleID= ValidHits= InvalidHits= NoValidationHits=] [Path1=No-Results-Found MD51= Size1= Modified1= RuleID1= ValidHits1= InvalidHits1= NoValidationHits1= Count=1]

I would like the final results to be:

Jun 12 23:00:09 server.i.j WRD: 0|AName|Named Application Server|1|0|Rule|0|ClientTime=7:02:28-PM CompName=sparse.info.given.here IPv4=x.x.x.x Path=No-Results-Found MD5= Size= Modified= RuleID= ValidHits= InvalidHits= NoValidationHits=

Jun 12 23:00:09 server.i.j WRD: 0|AName|Named Application Server|1|0|Rule|0|ClientTime=7:02:28-PM CompName=sparse.info.given.here IPv4=x.x.x.x Path1=No-Results-Found MD51= Size1= Modified1= RuleID1= ValidHits1= InvalidHits1= NoValidationHits1= Count=1

Count is not really a big deal here, it can be on either log (the latter by default as it is the final field in the log)

I have the regex to perform the part-splitting if rex is the move here:

| rex field=_raw "(?<header>.*IPv4Address=\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}) (?<part1>Path.*) (?<part2>Path.*)"

Once recombined, I will still perform manipulation on the resulting logs, and I do not need to write to file or CSV. The issue this is causing relates to finding accurate hits on files (the ValidHits1 field is annoying; same with Path1). I can happily rename fields after rejoining my Parts to the header so I can then correlate on top of all data with common field names.

Please feel free to ask for more information to help me out with this, and I appreciate any help you can give for this project!

[yuanliu]

How about this:

| eval data = split(_raw, "] [")
| eval header = ltrim(mvindex(data, 0), "["), path = mvappend(mvindex(data, 1), rtrim(mvindex(data, 2), "]"))
| mvexpand path
| eval _raw = mvjoin(mvappend(header, path), " ")

Your sample data gives

_raw

Jun 12 23:00:09 server.i.j WRD: 0|AName|Named Application Server|1|0|Rule|0|ClientTime=7:02:28-PM CompName=sparse.info.given.here IPv4=x.x.x.x Path=No-Results-Found MD5= Size= Modified= RuleID= ValidHits= InvalidHits= NoValidationHits=

Jun 12 23:00:09 server.i.j WRD: 0|AName|Named Application Server|1|0|Rule|0|ClientTime=7:02:28-PM CompName=sparse.info.given.here IPv4=x.x.x.x Path1=No-Results-Found MD51= Size1= Modified1= RuleID1= ValidHits1= InvalidHits1= NoValidationHits1= Count=1

[ITWhisperer]

@yuanliu  the brackets aren't in the data, but you are on the right lines

| eval data = split(_raw, " Path")
| eval header = mvindex(data, 0), path = mvappend("Path".mvindex(data, 1), "Path".mvindex(data, 2))
| mvexpand path
| eval _raw = mvjoin(mvappend(header, path), " ")

[curtisjester]

This works and I am now focused on renaming the fields within the path mv-object -> the last part of the index(2) is what has fields appended with '1' that need to be removed.

After running the mvexpand on path, I tried to do "rename var1 as var vary1 as vary" etc... but to no avail. I may be out of my scope of understanding on how Multi-Value commands manipulate or stream data. I am going to be searching for that answer on how to modify those field names with this mvindex and mvexpand then mvjoin answer... If I can't seem to find what I want I will accept this as a solution in the coming days.

Thanks to you both, @ITWhisperer @yuanliu 

____________________________
Edit:
I think when doing my renames I recognized my mistake; they are not extracted fields. So I either have to get it to recognize there are Path and Path1, ValidHits and ValidHits1, etc. for every field or change the data when doing the MV stuff.

@ITWhispererDoes mvindex just "look at" or "copy" data into new, mutable sections? If It just looks at, then I can understand how renaming (read: editing raw data) isn't possible. But If it's copying into a new field and then we're rejoining, shouldn't I be able to manipulate those internal values -- not using rename (as they are not extracted fields)? By using a combo of rex and something else?

[ITWhisperer · Accepted solution]

You could do this (although it only works for 1, 2 and 3, because IPv4 and MD5 complicate matters)!

| eval data = split(_raw, " Path")
| eval header = mvindex(data, 0), path = mvappend("Path".mvindex(data, 1), "Path".mvindex(data, 2))
| mvexpand path
| eval _raw = mvjoin(mvappend(header, path), " ")
| rex mode=sed "s/(?<name>\w+)(?<digit>[1-3])=/\1=/g"

[curtisjester]

Thank you very much! That has got me what I needed!