Getting Data In
Highlighted

Events not breaking correctly - using mv-add

Communicator

Hello Splunkers.

I'm helping a client to find out why some of his events are not being broken correctly.
They are currently running a Search Head Cluster with 3 SHs, 2 Indexers, 1 Master Cluster and 1 License/Deployer.
Here is a example of log:

--
tstamp="20160105 23:59:39.893"
IdCmd=01
Port=01
tstampResp="20160105 23:59:40.390"
Cmd="XXXXXX"
tipoAcao=ABC
Pri=I
Rsgmt=H
mainkey=12345678
acao="A";vAcao="000100000000A";resp="O"
acao="A";vAcao="000200000000A";resp="O"
acao="A";vAcao="000300000000A";resp="O"
acao="A";vAcao="000400000000A";resp="O"
acao="A";vAcao="000500000000A";resp="O"
acao="A";vAcao="000600000000A";resp="O"
acao="A";vAcao="000700000000A";resp="O"
acao="A";vAcao="000800000000A";resp="O"
acao="A";vAcao="000900000000A";resp="O"
acao="A";vAcao="001000000000A";resp="O"
acao="A";vAcao="001100000000A";resp="O"
acao="A";vAcao="001200000000A";resp="O"
acao="A";vAcao="001300000000A";resp="O"
acao="A";vAcao="001400000000A";resp="O"
acao="A";vAcao="001500000000A";resp="O"
acao="A";vAcao="001600000000A";resp="O"
acao="A";vAcao="006700000000A";resp="O"
acao="A";vAcao="006A00000000A";resp="O"
acao="A";vAcao="006B00000000A";resp="O"
acao="A";vAcao="006C00000000A";resp="O"
acao="A";vAcao="006E00000000A";resp="O"
acao="A";vAcao="006F00000000A";resp="O"
acao="A";vAcao="007000000000A";resp="O"
acao="A";vAcao="007200000000A";resp="O"
acao="A";vAcao="007400000000A";resp="O"
acao="A";vAcao="007500000000A";resp="O"
acao="A";vAcao="007600000000A";resp="O"
acao="A";vAcao="007700000000A";resp="O"
acao="A";vAcao="007800000000A";resp="O"
acao="A";vAcao="007900000000A";resp="O"
acao="A";vAcao="007B00000000A";resp="O"
acao="A";vAcao="007E00000000A";resp="O"
acao="A";vAcao="008000000000A";resp="O"
acao="A";vAcao="008200000000A";resp="O"
acao="A";vAcao="008900000000A";resp="O"
acao="A";vAcao="008A00000000A";resp="O"
acao="A";vAcao="008E00000000A";resp="O"
acao="A";vAcao="008F00000000A";resp="O"
acao="A";vAcao="009800000000A";resp="O"
acao="A";vAcao="009B00000000A";resp="O"
acao="A";vAcao="009D00000000A";resp="O"
acao="A";vAcao="009F00000000A";resp="O"
acao="A";vAcao="00A000000000A";resp="O"
acao="A";vAcao="00AA00000000A";resp="O"
acao="A";vAcao="00AB00000000A";resp="O"
acao="A";vAcao="00AC00000000A";resp="O"
acao="A";vAcao="00B500000000A";resp="O"
acao="A";vAcao="00C000000000A";resp="O"
acao="A";vAcao="00C100000000A";resp="O"
acao="A";vAcao="00C200000000A";resp="O"
acao="A";vAcao="01AA00000000A";resp="O"
acao="A";vAcao="021100000000A";resp="O"
acao="A";vAcao="021200000000A";resp="O"
acao="A";vAcao="039100000000A";resp="O"
acao="A";vAcao="039C00000000A";resp="O"
acao="A";vAcao="01C100000000A";resp="O"
acao="A";vAcao="000500000000A";resp="O"
acao="A";vAcao="001400000000A";resp="O"
acao="A";vAcao="005300000000A";resp="O"
acao="A";vAcao="005C00000000A";resp="O"
acao="A";vAcao="008400000000A";resp="O"
acao="A";vAcao="001600000000A";resp="O"
acao="A";vAcao="00F300000000A";resp="O"
acao="A";vAcao="00F000000000A";resp="O"
acao="A";vAcao="01C200000000A";resp="O"
acao="A";vAcao="00EF00000000A";resp="O"
acao="A";vAcao="01C300000000A";resp="O"
acao="A";vAcao="01C400000000A";resp="O"
acao="A";vAcao="00EE00000000A";resp="O"
acao="A";vAcao="01C600000000A";resp="O"
acao="A";vAcao="01C500000000A";resp="O"
acao="A";vAcao="01D700000000A";resp="O"
acao="A";vAcao="00EC00000000A";resp="O"
acao="A";vAcao="01C700000000A";resp="O"
acao="A";vAcao="00ED00000000A";resp="O"
acao="A";vAcao="01DE00000000A";resp="O"
acao="A";vAcao="01DD00000000A";resp="O"
acao="A";vAcao="01E200000000A";resp="O"
acao="A";vAcao="01C800000000A";resp="O"
acao="A";vAcao="01E100000000A";resp="O"
acao="A";vAcao="01E000000000A";resp="O"
acao="A";vAcao="01C900000000A";resp="O"
acao="A";vAcao="01CA00000000A";resp="O"
acao="A";vAcao="00F400000000A";resp="O"

The log above always starts with -- and can have hundreads of acao="A";vAcao="XXXX00000000A";resp="O" lines.
Here is the props.conf that I'm using:

SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=--
disabled=false
TIME_PREFIX=\d+\=\"
TIME_FORMAT=%Y%m%d %H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD=30

And here is the transforms.conf that I'm using:

REGEX = acao=\"(.*?)\";vAcao=\"(.*?)\";resp=\"(.*?)\" 
FORMAT = acao::$1 vAcao::$2 resp::$3 
MV_ADD = true

Sometimes, the event is not broken correctly. It breaks on acao="A";vAcao="XXXX00000000A";resp="O" lines.
I thought that I should include the TRUNCATE option in my props.conf.
However, when I try to Distribute Bundle Configuration using the UI at the Master, I receive an error message saying that I can't use TRUNCATE at my stanza.
So now I'm a bit lost.
Do you guys have any ideias that can help me?

Od course I can go directly to the Indexers and put the TRUNCATE option "by hand", but since they will soon add more indexers, this will not be feasible without the Master's Distribute Bundle Configuration.

Thanks in advance!
Regards,
GMA

0 Karma
Highlighted

Re: Events not breaking correctly - using mv-add

SplunkTrust
SplunkTrust

From where you're ingesting this log? Is it a data monitoring setup on Search Head OR on a Forwarder? Which location you're putting the props.conf for the sourcetype (with TRUNCATE)?

0 Karma
Highlighted

Re: Events not breaking correctly - using mv-add

Communicator

Hi somesoni2. I'm getting this data on a Forwarder. /opt/splunk/etc/master-apps/app/local/props.conf is the path on the Master. When I use Truncate, I can't use the Distribute Bundle Configuration on Master's UI.

0 Karma
Highlighted

Re: Events not breaking correctly - using mv-add

SplunkTrust
SplunkTrust

Are you able to push the bundle from the command line/shell on the Indexer Master?? using http://docs.splunk.com/Documentation/Splunk/6.2.0/Indexer/Updatepeerconfigurations#3._Apply_the_bund...

0 Karma
Highlighted

Re: Events not breaking correctly - using mv-add

Legend

Your props.conf does not specify MAX_EVENTS - which defaults to 256. So, when you have an event with many, many lines, Splunk breaks even without a --

Add this to the props.conf

MAX_EVENTS = 1024

Or some other number that is larger than the maximum possible number of events. Setting TRUNCATE would not have solved this problem anyway.

View solution in original post

0 Karma