topic Re: line breaking is not working in Getting Data In

line breaking is not working

kml_uvce — Thu, 22 Dec 2011 06:55:04 GMT

I am getting multiple line for an event

11-12-21 04:09:01 172.27.70.10 GET /OTPAuthentication/Service.asmx -
443 - 72.246.96.53 Python-urllib/1.17 200 0 0 0
2011-12-21 04:09:01 172.27.70.10 GET / - 443 - 72.247.36.53 - 200 0 64 93
2011-12-21 04:09:01 172.27.70.10 GET / - 443 - 72.246.50.40 - 200 0 64 46
2011-12-21 04:09:01 172.27.70.10 GET / - 443 - 72.246.50.40 - 200 0 64
249
2011-12-21 04:09:02 172.27.70.10 GET /OTPAuthentication/Service.asmx -
443 - 72.246.50.40 Python-urllib/1.17 200 0 0 46
2011-12-21 04:09:02 172.27.70.10 GET / - 443 - 72.246.50.41 - 200 0 64 46
2011-12-21 04:09:02 172.27.70.10 GET / - 443 - 72.246.50.41 - 200 0 64
249
2011-12-21 04:09:02 172.27.70.10 GET /OTPAuthentication/Service.asmx -
443 - 72.246.50.41 Python-urllib/1.17 200 0 0 46
2011-12-21 04:09:02 172.27.70.10 GET /OTPAuthentication/Service.asmx -
443 - 72.247.36.57 Python-urllib/1.17 200 0 0 78
2011-12-21 04:09:02 172.27.70.10 GET / - 443 - 72.247.36.57 - 200 0 64
296
2011-12-21 04:09:02 172.27.70.10 GET / - 443 - 72.247.36.56 - 200 0 64
296
2011-12-21 04:09:02 172.27.70.10 GET /OTPAuthentication/Service.asmx -
443 - 72.246.96.54 Python-urllib/1.17 200 0 0 0
2011-12-21 04:09:02 172.27.70.10 GET /OTPAuthentication/Service.asmx -
443 - 72.247.36.56 Python-urllib/1.17 200 0 0 78
2011-12-21 04:09:02 172.27.70.10 GET / - 443 - 72.247.36.57 - 200 0 64 78
2011-12-21 04:09:02 172.27.70.10 GET / - 443 - 72.247.36.56 - 200 0 64 78
2011-12-21 04:10:01 172.27.70.10 GET / - 443 - 72.246.96.55 - 200 0 0 15
2011-12-21 04:10:01 172.27.70.10 GET / - 443 - 72.246.96.55 - 200 0 64 0
2011-12-21 04:10:01 172.27.70.10 GET / - 443 - 72.246.96.54 - 200 0 0 15
2011-12-21 04:10:01 172.27.70.10 GET /OTPAuthentication/Service.asmx -
443 - 72.246.96.55 Python-urllib/1.17 200 0 0 0
2011-12-21 04:10:01 172.27.70.10 GET / - 443 - 72.246.96.54 - 200 0 64 0
2011-12-21 04:10:01 172.27.70.10 GET / - 443 - 72.246.50.42 - 200 0 64 46
2011-12-21 04:10:01 172.27.70.10 GET / - 443 - 72.246.50.42 - 200 0 64
249
2011-12-21 04:10:01 172.27.70.10 GET / - 443 - 72.246.96.53 - 200 0 0 0
2011-12-21 04:10:01 172.27.70.10 GET / - 443 - 72.246.96.53 - 200 0 64 0
2011-12-21 04:10:01 172.27.70.10 GET /OTPAuthentication/Service.asmx -
443 - 72.246.96.54 Python-urllib/1.17 200 0 0 0
2011-12-21 04:10:01 172.27.70.10 GET /OTPAuthentication/Service.asmx -
443 - 72.246.96.53 Python-urllib/1.17 200 0 0 0
2011-12-21 04:10:01 172.27.70.10 GET / - 443 - 72.246.50.40 - 200 0 64 46
2011-12-21 04:10:01 172.27.70.10 GET / - 443 - 72.246.50.40 - 200 0 64

I want only one like;

11-12-21 04:09:01 172.27.70.10 GET /OTPAuthentication/Service.asmx -
443 - 72.246.96.53 Python-urllib/1.17 200 0 0 0

I am using line breaker in props.conf like this

[tms-iis]
REPORT-tms_iisfields = tms_iisfields
SHOULD_LINEMERGE = false
LINE_BREAKER= ([\r\n]+)\s+\d+\s+\d+\s+\d+\s+\d+

but its not working, please help me on this, and also for time format , what i need to write.

Re: line breaking is not working

kristian_kolb — Thu, 22 Dec 2011 08:25:48 GMT

Hi, I assume that you have tried without any special directives first, which should work fine for IIS logs. Did you also try BREAK_ONLY_BEFORE_DATE=true ?

Anyway, your regex for LINE_BREAKER seems to be wrong, see below for a more correct version.

([\r\n]+)20\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d\s

UPDATE: well that seems a bit odd. Did you try BREAK_ONLY_BEFORE_DATE=true instead of LINE_BREAKER? In any case, for the time extraction you should use;

MAX_TIMESTAMP_LOOKAHEAD=25
TIME_FORMAT=%Y-%m-%d %H:%M:%S

UPDATE 2:

Also, make sure that this is configured where the parsing takes place;

If you have a heavy forwarder, on the forwarder.
If you have a universal, lightweight or no forwarder, on the indexer.

Restart the splunkd after making the changes.

Please mark as answered a/o upvote if this solves your problem.

/Kristian

Re: line breaking is not working

kml_uvce — Thu, 22 Dec 2011 09:13:28 GMT

Still not working...

Re: line breaking is not working

kml_uvce — Thu, 22 Dec 2011 09:20:14 GMT

Still not working for
LINE_BREAKER=([\r\n]+)20\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d\s

Re: line breaking is not working

Drainy — Thu, 22 Dec 2011 10:39:42 GMT

Some other things to remember;

Restart after making changes (there is a search command that reloads the configs but experience has taught me that its not 100% reliable).

These changes will NOT affect any previously indexed events, only the newest ones coming in.

Re: line breaking is not working

kristian_kolb — Thu, 22 Dec 2011 11:14:56 GMT

You're absolutely right. A restart IS required, since these configs relate to INDEX-time operations. Search-related operations, such as field extractions can usually be activated with

| extract reload=t

Re: line breaking is not working

kml_uvce — Thu, 22 Dec 2011 13:01:03 GMT

For sometime I got the single line but again getting same multiline error

Re: line breaking is not working

kml_uvce — Mon, 28 Sep 2020 10:14:38 GMT

I am using this in props.conf

[tms-iis]
CHECK_FOR_HEADER = False
MAX_TIMESTAMP_LOOKAHEAD = 25
TIME_FORMAT = %Y-%m-%d %H:%M:%S
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)20/\d/\d-/\d/\d-/\d\d/\s/\d/\d:/\d/\d:/\d/\d/\s
REPORT-tms_iisfields = tms_iisfields

Re: line breaking is not working

kml_uvce — Mon, 28 Sep 2020 10:14:41 GMT

Also used SHOULD_LINEMERGE = true and BREAK_ONLY_BEFORE_DATE=true but was not working...

Re: line breaking is not working

_d_ — Thu, 22 Dec 2011 17:06:14 GMT

Try the one below - notice the positive lookahead after the capture group ([\r\n]+):

[tms-iis] CHECK_FOR_HEADER = False MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_PREFIX = ^ TIME_FORMAT = %Y-%m-%d %H:%M:%S SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{1,2}-\d{1,2)\s+\d{1,2}:\d{1,2}:\d{1,2}) REPORT-tms_iisfields = tms_iisfields

Note that this will keep years in 4 digit format.

Hope this helps.

> please upvote and accept answer if you find it useful - thanks!

Re: line breaking is not working

kml_uvce — Fri, 23 Dec 2011 06:41:33 GMT

I wrote this in props.conf in indexer side, is there any need to write same in props.conf in forwarder side also ?

Re: line breaking is not working

kristian_kolb — Fri, 23 Dec 2011 07:24:37 GMT

see update2 above. /k

Re: line breaking is not working

kml_uvce — Wed, 28 Dec 2011 08:48:04 GMT

This is still not working...

Re: line breaking is not working

kml_uvce — Mon, 28 Sep 2020 10:15:10 GMT

Hi Its not wokring for me, I am using universal forwarder and

In forwarder:
props.conf
[tms-iis]
SHOULD_LINEMERGE = False
CHECK_FOR_HEADER = false

inputs.conf
[monitor://c:\inetpub\logs\logfiles\W3SVC1]
disabled = 0
sourcetype = tms-iis
index = windows

outputs.conf
[tcpout]
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = _audit
forwardedindex.filter.disable = false

and in indexer side:
props.conf
[tms-iis]
pulldown_type = true
CHECK_FOR_HEADER = False
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{1,2}-\d{1,2)\s+\d{1,2}:\d{1,2}:\d{1,2})
REPORT-tms_iisfields = tms_iisfields

transforms.conf
[tms_iisfields]
DELIMS = " "
FIELDS = date, time, s-ip, cs-method, cs-uri-stem, cs-uri-query, s-port, cs-username, c-ip, cs-User-Agent, sc-status, sc-substatus, sc-win32-status, time-taken

Re: line breaking is not working

kml_uvce — Mon, 28 Sep 2020 10:15:13 GMT

This is in index

props.conf
[tms-iis]
pulldown_type = true
CHECK_FOR_HEADER = False
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{1,2}-\d{1,2)s+\d{1,2}:\d{1,2}:\d{1,2})
REPORT-tms_iisfields = tms_iisfields

Re: line breaking is not working

kml_uvce — Thu, 29 Dec 2011 12:05:21 GMT

Sorry line breaker is

LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{1,2}-\d{1,2}\s+\d{1,2}:\d{1,2}:\d{1,2})

Re: line breaking is not working

Drainy — Thu, 29 Dec 2011 13:06:08 GMT

@kml_uvce
I've just tested your data and Splunk should be logging that correctly as single line events by default.
I would suggest removing any definitions you have made for it and test re-indexing it again (Or at least how it will appear in your data).

Also, when using Splunk-Base there are three fields, Question, Answer and comment. The question is an issue a person has raised with their Splunk experience. If you have any updates or changes to this then it is best practice to click on the edit button and update your question under a heading at the bottom like. EDIT. This keeps the thread simple to follow and will get you better answers as people can nip in and quickly read the problem and steps you have tried.
Answers are for other users to post an answer that solves your problem or if you fix it you can also post and accept your own answer.
Comments are used to make comments on answers or questions, a lot of your posts should really just be comments 🙂

But back to the point, you should clear out your props and transforms for any definitions that could be affecting your data and allow new logs to re-index. Bear in mind that changes after restart will only affect NEW data. Stuff you have already indexed will not change.

EDIT: Oh, and if someone gives you an answer that is correct then click on the little tick to the left of their answer. This marks it as being right and will help others experiencing the same problems in the future to find your question and answers. (Don't forget to do this for older questions you've asked too!)

Re: line breaking is not working

suhprano — Fri, 13 Jan 2012 20:03:56 GMT

I had a similar problem but it got fixed after putting in a millisecond value in the timestamp.