Getting Data In
×

Join the Conversation

Without signing in, you're just watching from the sidelines. Sign in or Register to connect, share, and be part of the Splunk Community.
Ask a Question

How to split access.log data

slipinski
Path Finder
‎03-03-2021 05:31 AM

Hello Splunkers,

I've got a problem with data splitting. I would like to split data into separate lines.

Please take a look at my data: 

 

 

10.62.19.11 - - [03/Mar/2021:12:49:02 +0100] "POST /api/setModernServices HTTP/1.1" 200 1315 0.154 0.148 "http://10.69.10.170/radio/web/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36" "{header:{id:0},data:{entries:[{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.33,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.3,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.7,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.18,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.8,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.21,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.30,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.57,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.20,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.29,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.41,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.23,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.42,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.43,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.24,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.44,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.39,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition\x

 

 

I want to have data split like below:

 

 

10.62.19.11 - - [03/Mar/2021:12:49:02 +0100] type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.33,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true

10.62.19.11 - - [03/Mar/2021:12:49:02 +0100] 
type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.3,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true

 

 

 

I suppose I need to use mvexpand in combination with static regex,  but I cannot find correct formula. Could you help please. 

Labels (1)
Labels
Tags (2)
0 Karma
Reply

richgalloway
SplunkTrust
SplunkTrust
‎03-03-2021 11:28 AM

Try adding this to the relevant stanza of props.conf:

LINE_BREAKER = ()\d\d\.\d\d\.\d\d\.\d\d\s

This should properly split events on newly-indexed data.  It won't change what's already indexed.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Reply

manjunathmeti
Champion
‎03-03-2021 11:02 AM

hi @slipinski,

Try this:

| rex "^(?<field1>.*\d{4}\])\s" 
| rex "{scheduleInfo:{(?<field2>.+)}}\]}}" 
| eval field2=split(field2, "}},{scheduleInfo:{") 
| mvexpand field2 
| strcat field1 " " field2 newfield

 

If this reply helps you, an upvote/like would be appreciated.

0 Karma
Reply

slipinski
Path Finder
‎03-09-2021 06:29 AM

Hi manjunathmeti,

 

Thanks for your effort. 

 

Your query extracts field1 correctly, but field2 is empty.  No idea why.

0 Karma
Reply
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Federated Search for Snowflake Is Now Generally Available on Splunk Cloud Platform

Splunk is excited to announce the General Availability (GA) of Federated Search for ...

Help Us Build Better Splunk Regex Puzzles (And Win Prizes!)

If you’ve spent any time in the Splunk Community Slack, you’ve likely seen our resident Splunk Trust ...

Fuel Your Journey: What’s Waiting for You at the .conf26 Acceleration Station

Navigating the show floor at .conf26 isn't just about keynotes and technical breakout sessions; it's also ...