I just recently started running into issues with my activeMQ server. I convinced the business to allow me to push these log files into Splunk in order to trace the issue. My problem is, I don't know regex and I'm no Splunk guru, so I need help on parsing my log file so it is traceable and or readable in some sort of table. This way, I will be able to see trends and further work on the fix.
My configuration:
index = activemq
sourcetype = kahadb_log
source = kahadb
Below is a sample log file that i have in ActiveMQ as well as what splunk outputs.
ActiveMQ LOG
TRACE | Last update: 164:41712, full gc candidates set: [86, 87, 163, 164] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
TRACE | gc candidates after first tx:164:41712, [86, 87, 163] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
TRACE | gc candidates after dest:0:A, [86, 87, 163] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
TRACE | gc candidates after dest:1:B, [86, 87, 163] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
TRACE | gc candidates after dest:0:D, [86, 87, 163] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
TRACE | gc candidates after dest:0:E, [86, 87] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
TRACE | gc candidates after dest:0:H, [86, 87] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
TRACE | gc candidates after dest:0:I, [86, 87] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
TRACE | gc candidates after dest:0:J, [87] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
TRACE | gc candidates: [87] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
DEBUG | Cleanup removing the data files: [87] | org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint Worker
TRACE MessageDatabase - Last update: 1502:31820100, full gc candidates set: [1502]
TRACE MessageDatabase - gc candidates after first tx:1502:31820100, []
TRACE MessageDatabase - gc candidates: []
DEBUG MessageDatabase - Checkpoint done.
output from splunk in csv
"DEBUG | MessageDatabase | Checkpoint started. | ActiveMQ Journal Checkpoint Worker
DEBUG | MessageDatabase | Checkpoint done. | ActiveMQ Journal Checkpoint Worker","2016-02-17T15:54:08.000-0500",GDPCCG01,activemq,2,kahadb,"kahadb_log",vA006
"INFO | MessageDatabase | KahaDB is version 4 | WrapperSimpleAppMain
INFO | MessageDatabase | Recovering from the journal ... | WrapperSimpleAppMain
INFO | MessageDatabase | Recovery replayed 1 operations from the journal in 0.0 seconds. | WrapperSimpleAppMain","2016-02-17T15:54:08.000-0500",GDPCCG01,activemq,3,kahadb,"kahadb_log",vA006
"DEBUG | Checkpoint started. | ActiveMQ ShutdownHook
TRACE | Last update: 1502:31909638, full gc candidates set: [1502] | ActiveMQ ShutdownHook
TRACE | gc candidates after first tx:1502:31909638, [] | ActiveMQ ShutdownHook
TRACE | gc candidates: [] | ActiveMQ ShutdownHook
DEBUG | Checkpoint done. | ActiveMQ ShutdownHook","2016-02-17T15:54:08.000-0500",GDPCCG01,activemq,5,kahadb,"kahadb_log",vA006
"DEBUG | Checkpoint started. | ActiveMQ Journal Checkpoint Worker
DEBUG | Checkpoint done. | ActiveMQ Journal Checkpoint Worker","2016-02-17T15:54:00.000-0500",GDPCCG01,activemq,2,kahadb,"kahadb_log",vA006
"DEBUG | Checkpoint started. | ActiveMQ Journal Checkpoint Worker
DEBUG | Checkpoint done. | ActiveMQ Journal Checkpoint Worker","2016-02-17T15:54:00.000-0500",GDPCCG01,activemq,2,kahadb,"kahadb_log",vA006
"DEBUG | Checkpoint started. | ActiveMQ Journal Checkpoint Worker
TRACE | Last update: 1502:31908429, full gc candidates set: [1502] | ActiveMQ Journal Checkpoint Worker
TRACE | gc candidates after first tx:1502:31908429, [] | ActiveMQ Journal Checkpoint Worker
TRACE | gc candidates: [] | ActiveMQ Journal Checkpoint Worker
DEBUG | Checkpoint done. | ActiveMQ Journal Checkpoint Worker","2016-02-17T15:53:50.000-0500",GDPCCG01,activemq,5,kahadb,"kahadb_log",vA006
"DEBUG | Checkpoint started. | ActiveMQ Journal Checkpoint Worker
DEBUG | Checkpoint done. | ActiveMQ Journal Checkpoint Worker","2016-02-17T15:53:50.000-0500",GDPCCG01,activemq,2,kahadb,"kahadb_log",vA006
"DEBUG | Checkpoint started. | ActiveMQ Journal Checkpoint Worker
DEBUG | Checkpoint done. | ActiveMQ Journal Checkpoint Worker","2016-02-17T15:53:40.000-0500",GDPCCG01,activemq,2,kahadb,"kahadb_log",vA006
... View more