Getting Data In

Line Merge difficulties

timrich66
Communicator

Hello helpful people,

I'm afraid I have an issue that is related to many questions already asked, but I have not been able to come up with a solution.

I have a log file that creates large events - more than 257 lines at a time.

To test the file, I took an extract and uploaded it manually.  Using this file, I was able to create props.conf entry as shown below and the events ingested correctly, without breaking.

When I applied this to our clustered environment, the breaking has returned.

Events -

++++ information 2021-01-06 16:38:53 host = xxxx.xxxx.net process = 00002fa8 thread = 73ffe380 context = Server::calculate(), module Request failed with error(s): <?xml version='1.0'?>

 Show all 257 lines

[031004] Variable  has no value. [035006] Cannot have child &lt;xxxxx[E.3] (B6I2)&gt; (xxx) on link xxxxxxxxxxxxxxxxxxx  (B6I1)&gt; </clc:Error> </xxxx__xxxxx_xxxx_xxx_f123_2>

 Show all 257 lines

props.conf

[source::///xxxx/Log/xxxxServer.log]
SHOULD_LINEMERGE=true
MAX_EVENTS=10000
TIME_PREFIX=\+\+\+\+ \w+

The reason I am using source and not sourcetype is because this source file is common to a number of environments and I am already changing sourcetype using props and transforms to determine the sourcetype per servername.

Thanks in advance for help - much appreciated.

Labels (1)
0 Karma
1 Solution

timrich66
Communicator

Hello All,

FYI - I have found the issue.  Deliberate mistake?  Maybe not, but it should have been obvious..

Original props.conf - 

[source::///xxxx/Log/xxxxServer.log]
SHOULD_LINEMERGE=true
MAX_EVENTS=10000
TIME_PREFIX=\+\+\+\+ \w+

Working props.conf

[source::///xxxx/Log/xxxxServer.log]
SHOULD_LINEMERGE = true
MAX_EVENTS = 10000
TIME_PREFIX = \+\+\+\+ \w+

Yes, it was pesky spaces 🙂 Watch out for them!  All now working as planned.

View solution in original post

timrich66
Communicator

Hello All,

FYI - I have found the issue.  Deliberate mistake?  Maybe not, but it should have been obvious..

Original props.conf - 

[source::///xxxx/Log/xxxxServer.log]
SHOULD_LINEMERGE=true
MAX_EVENTS=10000
TIME_PREFIX=\+\+\+\+ \w+

Working props.conf

[source::///xxxx/Log/xxxxServer.log]
SHOULD_LINEMERGE = true
MAX_EVENTS = 10000
TIME_PREFIX = \+\+\+\+ \w+

Yes, it was pesky spaces 🙂 Watch out for them!  All now working as planned.

timrich66
Communicator

@richgalloway yes, props.conf is pushed from CM to indexers

The reason that the sourcetype is being set based on host name is because the sourcetype includes the environment - e.g. dev1, dev2, prod1, prod2 etc  The source file has the same path and name on all servers.  The consumers of the logs do not necessarily know which hosts make up which environment.  Therefore, by including the environment in the sourcetype, the users can find their data more easily.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

How did you apply the props.conf to your cluster?  They should be installed on the indexers (pushed from the CM).

IMO, one should not be changing sourcetypes based on the server name.  Source types refer to a kind of data.  Kinds do not change because the server name changed.  If you need to distinguish originating servers then use the host field.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...