How to split access.log data

slipinski — Wed, 03 Mar 2021 13:31:15 GMT

Hello Splunkers,

I've got a problem with data splitting. I would like to split data into separate lines.

Please take a look at my data:

10.62.19.11 - - [03/Mar/2021:12:49:02 +0100] "POST /api/setModernServices HTTP/1.1" 200 1315 0.154 0.148 "http://10.69.10.170/radio/web/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36" "{header:{id:0},data:{entries:[{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.33,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.3,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.7,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.18,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.8,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.21,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.30,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.57,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.20,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.29,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.41,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.23,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.42,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.43,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.24,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.44,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.39,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition\x

I want to have data split like below:

10.62.19.11 - - [03/Mar/2021:12:49:02 +0100] type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.33,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true 10.62.19.11 - - [03/Mar/2021:12:49:02 +0100] type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.3,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true

I suppose I need to use mvexpand in combination with static regex, but I cannot find correct formula. Could you help please.

Re: How to split access.log data

manjunathmeti — Wed, 03 Mar 2021 19:02:43 GMT

hi @slipinski,

Try this:

| rex "^(?<field1>.*\d{4}\])\s" | rex "{scheduleInfo:{(?<field2>.+)}}\]}}" | eval field2=split(field2, "}},{scheduleInfo:{") | mvexpand field2 | strcat field1 " " field2 newfield

If this reply helps you, an upvote/like would be appreciated.

Re: How to split access.log data

richgalloway — Wed, 03 Mar 2021 19:28:29 GMT

Try adding this to the relevant stanza of props.conf:

LINE_BREAKER = ()\d\d\.\d\d\.\d\d\.\d\d\s

This should properly split events on newly-indexed data. It won't change what's already indexed.

Re: How to split access.log data

slipinski — Tue, 09 Mar 2021 14:29:37 GMT

Hi manjunathmeti,

Thanks for your effort.

Your query extracts field1 correctly, but field2 is empty. No idea why.

topic Re: How to split access.log data in Getting Data In

How to split access.log data

Re: How to split access.log data

Re: How to split access.log data

Re: How to split access.log data