Multiple lines are getting stuck together at indexing

hharvey — Mon, 28 Sep 2020 09:56:59 GMT

I am indexing a file of single line log events and some lines are getting chunked together into one event. Trying to figure out how to correct it.

Here's some example log data that is getting indexed together in each event:

a1-server1 App-> 2011-10-05 12:35:23 - a1-server1 - [99.99.99.99] user3(Active Directory Users)[gg-group] - Key Exchange number 23 occured for user with NCIP 192.168.212.63 
a1-server1 App-> 2011-10-05 12:35:23 - a1-server1 - [99.99.99.99]  user4(Active Directory Users)[gg-group] - Key Exchange number 5 occured for user with NCIP 192.168.212.221 
a1-server1 App-> 2011-10-05 12:35:23 - a1-server1 - [99.99.99.99]  user5(Active Directory Users)[gg-group] - Key Exchange number 2 occured for user with NCIP 192.168.212.193 

a1-server1 App-> 2011-10-05 12:33:46 - a1-server1 - [99.99.99.99]  auser2(Active Directory Users)[gg-group] - Key Exchange number 29 occured for user with NCIP 192.168.212.139 
a1-server1 App-> 2011-10-05 12:33:46 - a1-server1 - [99.99.99.99]  auser1(Active Directory Users)[gg-group] - Key Exchange number 16 occured for user with NCIP 192.168.212.108 

a1-server1 App-> 2011-10-05 12:28:43 - a1-server1 - [99.99.99.99]  auser(Active Directory Users)[gg-group]  - Logout from 99.99.99.99 (session:ff6f38ff)
a1-server1 App-> 2011-10-05 12:28:43 - a1-server1 - [99.99.99.99]  auser(Active Directory Users)[gg-group]  - Closed connection after 12513 seconds, with 4502 bytes read (in 19 chunks) and 19391 bytes written (in 43 chunks)
a1-server1 App-> 2011-10-05 12:28:43 - a1-server1 - [99.99.99.99]  auser(Active Directory Users)[gg-group]  - Network Connect: ACL count = 833.
a1-server1 App-> 2011-10-05 12:28:43 - a1-server1 - [99.99.99.99]  auser(Active Directory Users)[gg-group] - Network Connect: Session ended for user with IP 192.168.212.21

Initially, in props.conf I tried setting line_merge to false:

[source::source-to-break]

SHOULD_LINEMERGE = false

When that didn't work I tried specifying a line break before the server name at the beginning of each event:

[source::source-to-break]

SHOULD_LINEMERGE = true

BREAK_ONLY_BEFORE = a[0-9]-server[0-9]\sApp[\w'-]>\s

But that also is not working. I'm beginning to think this has something to do w/ the time, but not sure.

Re: Multiple lines are getting stuck together at indexing

_d_ — Mon, 28 Sep 2020 09:57:08 GMT

Try the stanza below:
[source::source-to-break]
TIME_PREFIX = a\d+\-server\d+\s+App\-\>\s+
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 20
LINE_BREAKER = ([\r\n]+)(?=a\d+\-server\d+\s+App\-\>\s+\d+\-\d+\-\d+\s+)
SHOULD_LINEMERGE = false

This will attempt to break right before a1-server1, extract the timestamp and format it properly. If this works please do not forget to vote 🙂

Re: Multiple lines are getting stuck together at indexing

hharvey — Thu, 27 Oct 2011 18:56:54 GMT

took me a while to get this implemented, but I just did and it worked. thanks so much!

topic Re: Multiple lines are getting stuck together at indexing in Getting Data In

Multiple lines are getting stuck together at indexing

Re: Multiple lines are getting stuck together at indexing

Re: Multiple lines are getting stuck together at indexing