Some events are merged when overiding multiple sourcetypes from single source

[Starlette]

I am forwarding a single source (file) from kiwisyslog with LFW to the indexer, so got 1 sourcetype [kiwisyslog]

Then I am overriding the hosts and sourcetypes.

props.conf

[kiwisyslog]
TRANSFORMS-hostoverride = asa_hostoverride

[kiwisyslog]
TRANSFORMS-sourcetypeoverride = cisco_asa

[kiwisyslog]
TRANSFORMS-sourcetypeoverride1 = cisco_fwsm

[kiwisyslog]
TRANSFORMS-sourcetypeoverride3 = named

[kiwisyslog]
TRANSFORMS-sourcetypeoverride4 = dhcp

transform.conf

[asa_hostoverride]
DEST_KEY = MetaData:Host
REGEX = \S+\t\S+\s(.*)\t+
FORMAT = host::$1

[cisco_asa]
DEST_KEY = MetaData:Sourcetype
REGEX = (%ASA)
FORMAT = sourcetype::cisco_firewall

[cisco_fwsm]
DEST_KEY = MetaData:Sourcetype
REGEX = (%FWSM)
FORMAT = sourcetype::cisco_firewall

[named]
DEST_KEY = MetaData:Sourcetype
REGEX = (\snamed)
FORMAT = sourcetype::named

[dhcp]
DEST_KEY = MetaData:Sourcetype
REGEX = (\sdhcpd)
FORMAT = sourcetype::dhcp

This works fine BUT for several events, events are merged! So I see multiple events with named sourcetype fi the 3 events below are merged as 1 event..

2010-07-03 09:40:29 Local7.Debug    15.0.5.3    Sat Jul  3 09:55:44 2010<010>jmpd[2534]: Invalid token length: 4, token: none.

2010-07-03 09:40:29 Local0.Warning  15.50.13.27 Jul  3 09:40:53 145.50.143.237 named[27677]: client 5.50.6.51#56994: RFC 1918 response from Internet for 83.251.139.10.in-addr.arpa

2010-07-03 09:40:29 Local4.Warning  1.12.7.130  %FWSM-4-106023: Deny tcp src HGR-RTR-xxxxxxx:145.21.136.78/53634 dst xxxxxxx:15.1.158.12/443 by access-group "xxxxxx_access_in" [0x0, 0x0]

( and the middle one is the right one)

I dont see a patern here,,,is there a limitation when overiding sourcetypes, or do I miss something here?

[ftk]

I suggest adding SHOULD_LINEMERGE=false in your props.conf stanzas. This is how I index my kiwisyslog files and I have no problems with line merging whatsoever.

props.conf:

[kiwisyslog]
TRANSFORMS-hostoverride = asa_hostoverride
SHOULD_LINEMERGE = false

[kiwisyslog]
TRANSFORMS-sourcetypeoverride = cisco_asa
SHOULD_LINEMERGE = false

[kiwisyslog]
TRANSFORMS-sourcetypeoverride1 = cisco_fwsm
SHOULD_LINEMERGE = false

[kiwisyslog]
TRANSFORMS-sourcetypeoverride3 = named
SHOULD_LINEMERGE = false

[kiwisyslog]
TRANSFORMS-sourcetypeoverride4 = dhcp
SHOULD_LINEMERGE = false

[Simeon · Accepted solution]

This sounds more like a line breaking problem instead of a host/source/sourcetypes labeling issue. For your events, they appear to be network device logs that typically resemble syslog. If you are certain these events are all single line "syslog" style, you can copy the syslog line breaking settings into your kiwisyslog sourcetype. This will properly break your events and the host/source/sourcetypes should then show up as expected.

[Starlette]

I will open a ticket cause it looks like 99 percent goes oke,,,and the other one is merging the 4 sourcetypes from time to time...sometimes 15 a 20 linebreakes

[Simeon]

I would not suspect this to be a performance problem in any way. The line breaking/merging functionality should not be affected by poor performance, as the system will just index or search at a slower rate when encountering resource issues. Are you positive you loaded the exact same set of data, in the exact same scenario as it gets input to your system? If you did that AND you set the correct line breaking (syslog settings) as suggested, then you should probably create a support case.

[Starlette]

The strange thing here is that most events are going well?
And If I load the kiwisyslog on a testsystem then it didnt happen?

So are you sure it isnt a performance issue here?