Events not breaking correctly - using mv-add

guimilare — Thu, 07 Jan 2016 18:23:38 GMT

Hello Splunkers.

I'm helping a client to find out why some of his events are not being broken correctly.
They are currently running a Search Head Cluster with 3 SHs, 2 Indexers, 1 Master Cluster and 1 License/Deployer.
Here is a example of log:

--
tstamp="20160105 23:59:39.893"
IdCmd=01
Port=01
tstampResp="20160105 23:59:40.390"
Cmd="XXXXXX"
tipoAcao=ABC
Pri=I
Rsgmt=H
mainkey=12345678
acao="A";vAcao="000100000000A";resp="O"
acao="A";vAcao="000200000000A";resp="O"
acao="A";vAcao="000300000000A";resp="O"
acao="A";vAcao="000400000000A";resp="O"
acao="A";vAcao="000500000000A";resp="O"
acao="A";vAcao="000600000000A";resp="O"
acao="A";vAcao="000700000000A";resp="O"
acao="A";vAcao="000800000000A";resp="O"
acao="A";vAcao="000900000000A";resp="O"
acao="A";vAcao="001000000000A";resp="O"
acao="A";vAcao="001100000000A";resp="O"
acao="A";vAcao="001200000000A";resp="O"
acao="A";vAcao="001300000000A";resp="O"
acao="A";vAcao="001400000000A";resp="O"
acao="A";vAcao="001500000000A";resp="O"
acao="A";vAcao="001600000000A";resp="O"
acao="A";vAcao="006700000000A";resp="O"
acao="A";vAcao="006A00000000A";resp="O"
acao="A";vAcao="006B00000000A";resp="O"
acao="A";vAcao="006C00000000A";resp="O"
acao="A";vAcao="006E00000000A";resp="O"
acao="A";vAcao="006F00000000A";resp="O"
acao="A";vAcao="007000000000A";resp="O"
acao="A";vAcao="007200000000A";resp="O"
acao="A";vAcao="007400000000A";resp="O"
acao="A";vAcao="007500000000A";resp="O"
acao="A";vAcao="007600000000A";resp="O"
acao="A";vAcao="007700000000A";resp="O"
acao="A";vAcao="007800000000A";resp="O"
acao="A";vAcao="007900000000A";resp="O"
acao="A";vAcao="007B00000000A";resp="O"
acao="A";vAcao="007E00000000A";resp="O"
acao="A";vAcao="008000000000A";resp="O"
acao="A";vAcao="008200000000A";resp="O"
acao="A";vAcao="008900000000A";resp="O"
acao="A";vAcao="008A00000000A";resp="O"
acao="A";vAcao="008E00000000A";resp="O"
acao="A";vAcao="008F00000000A";resp="O"
acao="A";vAcao="009800000000A";resp="O"
acao="A";vAcao="009B00000000A";resp="O"
acao="A";vAcao="009D00000000A";resp="O"
acao="A";vAcao="009F00000000A";resp="O"
acao="A";vAcao="00A000000000A";resp="O"
acao="A";vAcao="00AA00000000A";resp="O"
acao="A";vAcao="00AB00000000A";resp="O"
acao="A";vAcao="00AC00000000A";resp="O"
acao="A";vAcao="00B500000000A";resp="O"
acao="A";vAcao="00C000000000A";resp="O"
acao="A";vAcao="00C100000000A";resp="O"
acao="A";vAcao="00C200000000A";resp="O"
acao="A";vAcao="01AA00000000A";resp="O"
acao="A";vAcao="021100000000A";resp="O"
acao="A";vAcao="021200000000A";resp="O"
acao="A";vAcao="039100000000A";resp="O"
acao="A";vAcao="039C00000000A";resp="O"
acao="A";vAcao="01C100000000A";resp="O"
acao="A";vAcao="000500000000A";resp="O"
acao="A";vAcao="001400000000A";resp="O"
acao="A";vAcao="005300000000A";resp="O"
acao="A";vAcao="005C00000000A";resp="O"
acao="A";vAcao="008400000000A";resp="O"
acao="A";vAcao="001600000000A";resp="O"
acao="A";vAcao="00F300000000A";resp="O"
acao="A";vAcao="00F000000000A";resp="O"
acao="A";vAcao="01C200000000A";resp="O"
acao="A";vAcao="00EF00000000A";resp="O"
acao="A";vAcao="01C300000000A";resp="O"
acao="A";vAcao="01C400000000A";resp="O"
acao="A";vAcao="00EE00000000A";resp="O"
acao="A";vAcao="01C600000000A";resp="O"
acao="A";vAcao="01C500000000A";resp="O"
acao="A";vAcao="01D700000000A";resp="O"
acao="A";vAcao="00EC00000000A";resp="O"
acao="A";vAcao="01C700000000A";resp="O"
acao="A";vAcao="00ED00000000A";resp="O"
acao="A";vAcao="01DE00000000A";resp="O"
acao="A";vAcao="01DD00000000A";resp="O"
acao="A";vAcao="01E200000000A";resp="O"
acao="A";vAcao="01C800000000A";resp="O"
acao="A";vAcao="01E100000000A";resp="O"
acao="A";vAcao="01E000000000A";resp="O"
acao="A";vAcao="01C900000000A";resp="O"
acao="A";vAcao="01CA00000000A";resp="O"
acao="A";vAcao="00F400000000A";resp="O"

The log above always starts with -- and can have hundreads of acao="A";vAcao="XXXX00000000A";resp="O" lines.
Here is the props.conf that I'm using:

SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=--
disabled=false
TIME_PREFIX=\d+\=\"
TIME_FORMAT=%Y%m%d %H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD=30

And here is the transforms.conf that I'm using:

REGEX = acao=\"(.*?)\";vAcao=\"(.*?)\";resp=\"(.*?)\" 
FORMAT = acao::$1 vAcao::$2 resp::$3 
MV_ADD = true

Sometimes, the event is not broken correctly. It breaks on acao="A";vAcao="XXXX00000000A";resp="O" lines.
I thought that I should include the TRUNCATE option in my props.conf.
However, when I try to Distribute Bundle Configuration using the UI at the Master, I receive an error message saying that I can't use TRUNCATE at my stanza.
So now I'm a bit lost.
Do you guys have any ideias that can help me?

Od course I can go directly to the Indexers and put the TRUNCATE option "by hand", but since they will soon add more indexers, this will not be feasible without the Master's Distribute Bundle Configuration.

Thanks in advance!
Regards,
GMA

Re: Events not breaking correctly - using mv-add

somesoni2 — Thu, 07 Jan 2016 18:51:17 GMT

From where you're ingesting this log? Is it a data monitoring setup on Search Head OR on a Forwarder? Which location you're putting the props.conf for the sourcetype (with TRUNCATE)?

Re: Events not breaking correctly - using mv-add

guimilare — Thu, 07 Jan 2016 18:56:07 GMT

Hi somesoni2. I'm getting this data on a Forwarder. /opt/splunk/etc/master-apps/app/local/props.conf is the path on the Master. When I use Truncate, I can't use the Distribute Bundle Configuration on Master's UI.

Re: Events not breaking correctly - using mv-add

somesoni2 — Thu, 07 Jan 2016 19:15:11 GMT

Are you able to push the bundle from the command line/shell on the Indexer Master?? using http://docs.splunk.com/Documentation/Splunk/6.2.0/Indexer/Updatepeerconfigurations#3._Apply_the_bundle_to_the_peers

Re: Events not breaking correctly - using mv-add

lguinn2 — Thu, 07 Jan 2016 19:22:44 GMT

Your props.conf does not specify MAX_EVENTS - which defaults to 256. So, when you have an event with many, many lines, Splunk breaks even without a --

Add this to the props.conf

MAX_EVENTS = 1024

Or some other number that is larger than the maximum possible number of events. Setting TRUNCATE would not have solved this problem anyway.

topic Re: Events not breaking correctly - using mv-add in Getting Data In

Events not breaking correctly - using mv-add

Re: Events not breaking correctly - using mv-add

Re: Events not breaking correctly - using mv-add

Re: Events not breaking correctly - using mv-add

Re: Events not breaking correctly - using mv-add