Splunk Search

How to split an event into parts and recombine with a common header?

curtisjester
Explorer

Hey all - thanks in advance!

I have _raw log data that contains a header section and then what appears to be two entries within itself. I want to split these entries (they are formatted the same, except the latter appends a '1' onto each fieldname) and then create two events from this one event, like so:

Before
_raw = HEADER|PART1|PART2
After
event1
= HEADER|PART1
event2 = HEADER|PART2

 

An event will come from the same IP and device name; the parts are paths and simple fields. Here is a sample log (bracketed to show how I want it split, but these brackets are not in the raw data):

[Jun 12 23:00:09 server.i.j WRD: 0|AName|Named Application Server|1|0|Rule|0|ClientTime=7:02:28-PM CompName=sparse.info.given.here IPv4=x.x.x.x] [Path=No-Results-Found MD5= Size= Modified= RuleID= ValidHits= InvalidHits= NoValidationHits=] [Path1=No-Results-Found MD51= Size1= Modified1= RuleID1= ValidHits1= InvalidHits1= NoValidationHits1= Count=1]

I would like the final results to be:

Jun 12 23:00:09 server.i.j WRD: 0|AName|Named Application Server|1|0|Rule|0|ClientTime=7:02:28-PM CompName=sparse.info.given.here IPv4=x.x.x.x Path=No-Results-Found MD5= Size= Modified= RuleID= ValidHits= InvalidHits= NoValidationHits=

Jun 12 23:00:09 server.i.j WRD: 0|AName|Named Application Server|1|0|Rule|0|ClientTime=7:02:28-PM CompName=sparse.info.given.here IPv4=x.x.x.x Path1=No-Results-Found MD51= Size1= Modified1= RuleID1= ValidHits1= InvalidHits1= NoValidationHits1= Count=1

Count is not really a big deal here, it can be on either log (the latter by default as it is the final field in the log)

I have the regex to perform the part-splitting if rex is the move here:

| rex field=_raw "(?<header>.*IPv4Address=\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}) (?<part1>Path.*) (?<part2>Path.*)"

Once recombined, I will still perform manipulation on the resulting logs, and I do not need to write to file or CSV. The issue this is causing relates to finding accurate hits on files (the ValidHits1 field is annoying; same with Path1). I can happily rename fields after rejoining my Parts to the header so I can then correlate on top of all data with common field names.

Please feel free to ask for more information to help me out with this, and I appreciate any help you can give for this project!

Labels (2)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust

You could do this (although it only works for 1, 2 and 3, because IPv4 and MD5 complicate matters)!

| eval data = split(_raw, " Path")
| eval header = mvindex(data, 0), path = mvappend("Path".mvindex(data, 1), "Path".mvindex(data, 2))
| mvexpand path
| eval _raw = mvjoin(mvappend(header, path), " ")
| rex mode=sed "s/(?<name>\w+)(?<digit>[1-3])=/\1=/g"

View solution in original post

0 Karma

yuanliu
SplunkTrust
SplunkTrust

How about this:

 

| eval data = split(_raw, "] [")
| eval header = ltrim(mvindex(data, 0), "["), path = mvappend(mvindex(data, 1), rtrim(mvindex(data, 2), "]"))
| mvexpand path
| eval _raw = mvjoin(mvappend(header, path), " ")

 

Your sample data gives

_raw
Jun 12 23:00:09 server.i.j WRD: 0|AName|Named Application Server|1|0|Rule|0|ClientTime=7:02:28-PM CompName=sparse.info.given.here IPv4=x.x.x.x Path=No-Results-Found MD5= Size= Modified= RuleID= ValidHits= InvalidHits= NoValidationHits=
Jun 12 23:00:09 server.i.j WRD: 0|AName|Named Application Server|1|0|Rule|0|ClientTime=7:02:28-PM CompName=sparse.info.given.here IPv4=x.x.x.x Path1=No-Results-Found MD51= Size1= Modified1= RuleID1= ValidHits1= InvalidHits1= NoValidationHits1= Count=1
Tags (3)
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

@yuanliu  the brackets aren't in the data, but you are on the right lines

| eval data = split(_raw, " Path")
| eval header = mvindex(data, 0), path = mvappend("Path".mvindex(data, 1), "Path".mvindex(data, 2))
| mvexpand path
| eval _raw = mvjoin(mvappend(header, path), " ")
0 Karma

curtisjester
Explorer

This works and I am now focused on renaming the fields within the path mv-object -> the last part of the index(2) is what has fields appended with '1' that need to be removed.

After running the mvexpand on path, I tried to do "rename var1 as var vary1 as vary" etc... but to no avail. I may be out of my scope of understanding on how Multi-Value commands manipulate or stream data. I am going to be searching for that answer on how to modify those field names with this mvindex and mvexpand then mvjoin answer... If I can't seem to find what I want I will accept this as a solution in the coming days.

Thanks to you both, @ITWhisperer @yuanliu 

 

____________________________
Edit:
I think when doing my renames I recognized my mistake; they are not extracted fields. So I either have to get it to recognize there are Path and Path1, ValidHits and ValidHits1, etc. for every field or change the data when doing the MV stuff.

@ITWhispererDoes mvindex just "look at" or "copy" data into new, mutable sections? If It just looks at, then I can understand how renaming (read: editing raw data) isn't possible. But If it's copying into a new field and then we're rejoining, shouldn't I be able to manipulate those internal values -- not using rename (as they are not extracted fields)? By using a combo of rex and something else?

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

You could do this (although it only works for 1, 2 and 3, because IPv4 and MD5 complicate matters)!

| eval data = split(_raw, " Path")
| eval header = mvindex(data, 0), path = mvappend("Path".mvindex(data, 1), "Path".mvindex(data, 2))
| mvexpand path
| eval _raw = mvjoin(mvappend(header, path), " ")
| rex mode=sed "s/(?<name>\w+)(?<digit>[1-3])=/\1=/g"
0 Karma

curtisjester
Explorer

Thank you very much! That has got me what I needed!

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...