Getting Data In

Line Merge difficulties

timrich66
Communicator

Hello helpful people,

I'm afraid I have an issue that is related to many questions already asked, but I have not been able to come up with a solution.

I have a log file that creates large events - more than 257 lines at a time.

To test the file, I took an extract and uploaded it manually.  Using this file, I was able to create props.conf entry as shown below and the events ingested correctly, without breaking.

When I applied this to our clustered environment, the breaking has returned.

Events -

++++ information 2021-01-06 16:38:53 host = xxxx.xxxx.net process = 00002fa8 thread = 73ffe380 context = Server::calculate(), module Request failed with error(s): <?xml version='1.0'?>

 Show all 257 lines

[031004] Variable  has no value. [035006] Cannot have child &lt;xxxxx[E.3] (B6I2)&gt; (xxx) on link xxxxxxxxxxxxxxxxxxx  (B6I1)&gt; </clc:Error> </xxxx__xxxxx_xxxx_xxx_f123_2>

 Show all 257 lines

props.conf

[source::///xxxx/Log/xxxxServer.log]
SHOULD_LINEMERGE=true
MAX_EVENTS=10000
TIME_PREFIX=\+\+\+\+ \w+

The reason I am using source and not sourcetype is because this source file is common to a number of environments and I am already changing sourcetype using props and transforms to determine the sourcetype per servername.

Thanks in advance for help - much appreciated.

Labels (1)
0 Karma
1 Solution

timrich66
Communicator

Hello All,

FYI - I have found the issue.  Deliberate mistake?  Maybe not, but it should have been obvious..

Original props.conf - 

[source::///xxxx/Log/xxxxServer.log]
SHOULD_LINEMERGE=true
MAX_EVENTS=10000
TIME_PREFIX=\+\+\+\+ \w+

Working props.conf

[source::///xxxx/Log/xxxxServer.log]
SHOULD_LINEMERGE = true
MAX_EVENTS = 10000
TIME_PREFIX = \+\+\+\+ \w+

Yes, it was pesky spaces 🙂 Watch out for them!  All now working as planned.

View solution in original post

timrich66
Communicator

Hello All,

FYI - I have found the issue.  Deliberate mistake?  Maybe not, but it should have been obvious..

Original props.conf - 

[source::///xxxx/Log/xxxxServer.log]
SHOULD_LINEMERGE=true
MAX_EVENTS=10000
TIME_PREFIX=\+\+\+\+ \w+

Working props.conf

[source::///xxxx/Log/xxxxServer.log]
SHOULD_LINEMERGE = true
MAX_EVENTS = 10000
TIME_PREFIX = \+\+\+\+ \w+

Yes, it was pesky spaces 🙂 Watch out for them!  All now working as planned.

timrich66
Communicator

@richgalloway yes, props.conf is pushed from CM to indexers

The reason that the sourcetype is being set based on host name is because the sourcetype includes the environment - e.g. dev1, dev2, prod1, prod2 etc  The source file has the same path and name on all servers.  The consumers of the logs do not necessarily know which hosts make up which environment.  Therefore, by including the environment in the sourcetype, the users can find their data more easily.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

How did you apply the props.conf to your cluster?  They should be installed on the indexers (pushed from the CM).

IMO, one should not be changing sourcetypes based on the server name.  Source types refer to a kind of data.  Kinds do not change because the server name changed.  If you need to distinguish originating servers then use the host field.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Unlock What’s Next: The Splunk Cloud Platform at .conf25

In just a few days, Boston will be buzzing as the Splunk team and thousands of community members come together ...