Getting Data In
×

Are you a member of the Splunk Community?

Sign in or Register with your Splunk account to get your questions answered, access valuable resources and connect with experts!
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Ask a Question

How to split access.log data

slipinski
Path Finder
‎03-03-2021 05:31 AM

Hello Splunkers,

I've got a problem with data splitting. I would like to split data into separate lines.

Please take a look at my data: 

 

 

10.62.19.11 - - [03/Mar/2021:12:49:02 +0100] "POST /api/setModernServices HTTP/1.1" 200 1315 0.154 0.148 "http://10.69.10.170/radio/web/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36" "{header:{id:0},data:{entries:[{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.33,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.3,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.7,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.18,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.8,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.21,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.30,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.57,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.20,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.29,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.41,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.23,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.42,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.43,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.24,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.44,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.39,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true}},{scheduleInfo:{type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition\x

 

 

I want to have data split like below:

 

 

10.62.19.11 - - [03/Mar/2021:12:49:02 +0100] type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.33,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.58,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true

10.62.19.11 - - [03/Mar/2021:12:49:02 +0100] 
type:once,startTimestamp:1614772142364,endTimestamp:8999999999000},serviceDefinition:{descriptor:{label:,desc:},dests:[{destId:virtual.46.0.3,destName:}],groupOrder:1,profileId:7637a321-8628-47d3-92e8-5fd85b54aa6b,skipReuseJoinId:true,routing:p2mp,srcId:virtual.96.0.39,tags:[LastOnAirId:null,EmerOnAirId:null],type:connection,force:true

 

 

 

I suppose I need to use mvexpand in combination with static regex,  but I cannot find correct formula. Could you help please. 

Labels (1)
Labels
Tags (2)
0 Karma
Reply

richgalloway
SplunkTrust
SplunkTrust
‎03-03-2021 11:28 AM

Try adding this to the relevant stanza of props.conf:

LINE_BREAKER = ()\d\d\.\d\d\.\d\d\.\d\d\s

This should properly split events on newly-indexed data.  It won't change what's already indexed.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Reply

manjunathmeti
Champion
‎03-03-2021 11:02 AM

hi @slipinski,

Try this:

| rex "^(?<field1>.*\d{4}\])\s" 
| rex "{scheduleInfo:{(?<field2>.+)}}\]}}" 
| eval field2=split(field2, "}},{scheduleInfo:{") 
| mvexpand field2 
| strcat field1 " " field2 newfield

 

If this reply helps you, an upvote/like would be appreciated.

0 Karma
Reply

slipinski
Path Finder
‎03-09-2021 06:29 AM

Hi manjunathmeti,

 

Thanks for your effort. 

 

Your query extracts field1 correctly, but field2 is empty.  No idea why.

0 Karma
Reply
Get Updates on the Splunk Community!

Fun with Regular Expression - multiples of nine

Fun with Regular Expression - multiples of nineThis challenge was first posted on Slack #regex channel ...

[Live Demo] Watch SOC transformation in action with the reimagined Splunk Enterprise ...

Overwhelmed SOC? Splunk ES Has Your Back Tool sprawl, alert fatigue, and endless context switching are making ...

What’s New & Next in Splunk SOAR

Security teams today are dealing with more alerts, more tools, and more pressure than ever.  Join us on ...