Getting Data In
×

Join the Conversation

Without signing in, you're just watching from the sidelines. Sign in or Register to connect, share, and be part of the Splunk Community.
Ask a Question

How to split access.log data

slipinski
Path Finder
‎03-03-2021 05:31 AM

Hello Splunkers,

I've got a problem with data splitting. I would like to split data into separate lines.

Please take a look at my data: 

 

 

10.62.19.11 - - [03/Mar/2021:12:49:02 +0100] "POST /api/setModernServices HTTP/1.1" 200 1315 0.154 0.148 "http://10.69.10.170/radio/web/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36" "{header:{id:0},data:{entries:[{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.33,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.3,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.7,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.18,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.8,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.21,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.30,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.57,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.20,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.29,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.41,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.23,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.42,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.43,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.24,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.44,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.39,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition\x

 

 

I want to have data split like below:

 

 

10.62.19.11 - - [03/Mar/2021:12:49:02 +0100] type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.33,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true

10.62.19.11 - - [03/Mar/2021:12:49:02 +0100] 
type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.3,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true

 

 

 

I suppose I need to use mvexpand in combination with static regex,  but I cannot find correct formula. Could you help please. 

Labels (1)
Labels
Tags (2)
0 Karma
Reply

richgalloway
SplunkTrust
SplunkTrust
‎03-03-2021 11:28 AM

Try adding this to the relevant stanza of props.conf:

LINE_BREAKER = ()\d\d\.\d\d\.\d\d\.\d\d\s

This should properly split events on newly-indexed data.  It won't change what's already indexed.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Reply

manjunathmeti
Champion
‎03-03-2021 11:02 AM

hi @slipinski,

Try this:

| rex "^(?<field1>.*\d{4}\])\s" 
| rex "{scheduleInfo:{(?<field2>.+)}}\]}}" 
| eval field2=split(field2, "}},{scheduleInfo:{") 
| mvexpand field2 
| strcat field1 " " field2 newfield

 

If this reply helps you, an upvote/like would be appreciated.

0 Karma
Reply

slipinski
Path Finder
‎03-09-2021 06:29 AM

Hi manjunathmeti,

 

Thanks for your effort. 

 

Your query extracts field1 correctly, but field2 is empty.  No idea why.

0 Karma
Reply
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Agentic Operations Start with Context: Build the Right Data Foundation

Agentic Operations Start with Context: Build the Right Data Foundation   By Courtney Wright, Product Marketing ...

Assisted, Augmented or Agentic? Choose Your Splunk Starting Point

Assisted, Augmented or Agentic? Choose Your Splunk Starting Point   By Courtney Wright, Product Marketing ...

Session 2 | Beyond the Thread: Operationalizing AI with Confidence

Session 2   Beyond the Thread: Operationalizing AI with Confidence    The true power of the Cisco Data Fabric ...