topic Re: Line Merge difficulties in Getting Data In

Line Merge difficulties

timrich66 — Thu, 07 Jan 2021 09:22:08 GMT

Hello helpful people,

I'm afraid I have an issue that is related to many questions already asked, but I have not been able to come up with a solution.

I have a log file that creates large events - more than 257 lines at a time.

To test the file, I took an extract and uploaded it manually. Using this file, I was able to create props.conf entry as shown below and the events ingested correctly, without breaking.

When I applied this to our clustered environment, the breaking has returned.

Events -

++++ information 2021-01-06 16:38:53 host = xxxx.xxxx.net process = 00002fa8 thread = 73ffe380 context = Server::calculate(), module Request failed with error(s): <?xml version='1.0'?>

Show all 257 lines

[031004] Variable has no value. [035006] Cannot have child <xxxxx[E.3] (B6I2)> (xxx) on link xxxxxxxxxxxxxxxxxxx (B6I1)> </clc:Error> </xxxx__xxxxx_xxxx_xxx_f123_2>

Show all 257 lines

props.conf

[source::///xxxx/Log/xxxxServer.log]
SHOULD_LINEMERGE=true
MAX_EVENTS=10000
TIME_PREFIX=\+\+\+\+ \w+

The reason I am using source and not sourcetype is because this source file is common to a number of environments and I am already changing sourcetype using props and transforms to determine the sourcetype per servername.

Thanks in advance for help - much appreciated.

Re: Line Merge difficulties

richgalloway — Thu, 07 Jan 2021 14:06:14 GMT

How did you apply the props.conf to your cluster? They should be installed on the indexers (pushed from the CM).

IMO, one should not be changing sourcetypes based on the server name. Source types refer to a kind of data. Kinds do not change because the server name changed. If you need to distinguish originating servers then use the host field.

Re: Line Merge difficulties

timrich66 — Thu, 07 Jan 2021 15:29:55 GMT

@richgalloway yes, props.conf is pushed from CM to indexers

The reason that the sourcetype is being set based on host name is because the sourcetype includes the environment - e.g. dev1, dev2, prod1, prod2 etc The source file has the same path and name on all servers. The consumers of the logs do not necessarily know which hosts make up which environment. Therefore, by including the environment in the sourcetype, the users can find their data more easily.

Line Merge difficulties

timrich66 — Fri, 15 Jan 2021 14:35:39 GMT

Hello All,

FYI - I have found the issue. Deliberate mistake? Maybe not, but it should have been obvious..

Original props.conf -

[source::///xxxx/Log/xxxxServer.log]
SHOULD_LINEMERGE=true
MAX_EVENTS=10000
TIME_PREFIX=\+\+\+\+ \w+

Working props.conf

[source::///xxxx/Log/xxxxServer.log]
SHOULD_LINEMERGE = true
MAX_EVENTS = 10000
TIME_PREFIX = \+\+\+\+ \w+

Yes, it was pesky spaces 🙂 Watch out for them! All now working as planned.