line breaking is not working

[kml_uvce]

I am getting multiple line for an event

11-12-21 04:09:01 172.27.70.10 GET /OTPAuthentication/Service.asmx -
443 - 72.246.96.53 Python-urllib/1.17 200 0 0 0
2011-12-21 04:09:01 172.27.70.10 GET / - 443 - 72.247.36.53 - 200 0 64 93
2011-12-21 04:09:01 172.27.70.10 GET / - 443 - 72.246.50.40 - 200 0 64 46
2011-12-21 04:09:01 172.27.70.10 GET / - 443 - 72.246.50.40 - 200 0 64
249
2011-12-21 04:09:02 172.27.70.10 GET /OTPAuthentication/Service.asmx -
443 - 72.246.50.40 Python-urllib/1.17 200 0 0 46
2011-12-21 04:09:02 172.27.70.10 GET / - 443 - 72.246.50.41 - 200 0 64 46
2011-12-21 04:09:02 172.27.70.10 GET / - 443 - 72.246.50.41 - 200 0 64
249
2011-12-21 04:09:02 172.27.70.10 GET /OTPAuthentication/Service.asmx -
443 - 72.246.50.41 Python-urllib/1.17 200 0 0 46
2011-12-21 04:09:02 172.27.70.10 GET /OTPAuthentication/Service.asmx -
443 - 72.247.36.57 Python-urllib/1.17 200 0 0 78
2011-12-21 04:09:02 172.27.70.10 GET / - 443 - 72.247.36.57 - 200 0 64
296
2011-12-21 04:09:02 172.27.70.10 GET / - 443 - 72.247.36.56 - 200 0 64
296
2011-12-21 04:09:02 172.27.70.10 GET /OTPAuthentication/Service.asmx -
443 - 72.246.96.54 Python-urllib/1.17 200 0 0 0
2011-12-21 04:09:02 172.27.70.10 GET /OTPAuthentication/Service.asmx -
443 - 72.247.36.56 Python-urllib/1.17 200 0 0 78
2011-12-21 04:09:02 172.27.70.10 GET / - 443 - 72.247.36.57 - 200 0 64 78
2011-12-21 04:09:02 172.27.70.10 GET / - 443 - 72.247.36.56 - 200 0 64 78
2011-12-21 04:10:01 172.27.70.10 GET / - 443 - 72.246.96.55 - 200 0 0 15
2011-12-21 04:10:01 172.27.70.10 GET / - 443 - 72.246.96.55 - 200 0 64 0
2011-12-21 04:10:01 172.27.70.10 GET / - 443 - 72.246.96.54 - 200 0 0 15
2011-12-21 04:10:01 172.27.70.10 GET /OTPAuthentication/Service.asmx -
443 - 72.246.96.55 Python-urllib/1.17 200 0 0 0
2011-12-21 04:10:01 172.27.70.10 GET / - 443 - 72.246.96.54 - 200 0 64 0
2011-12-21 04:10:01 172.27.70.10 GET / - 443 - 72.246.50.42 - 200 0 64 46
2011-12-21 04:10:01 172.27.70.10 GET / - 443 - 72.246.50.42 - 200 0 64
249
2011-12-21 04:10:01 172.27.70.10 GET / - 443 - 72.246.96.53 - 200 0 0 0
2011-12-21 04:10:01 172.27.70.10 GET / - 443 - 72.246.96.53 - 200 0 64 0
2011-12-21 04:10:01 172.27.70.10 GET /OTPAuthentication/Service.asmx -
443 - 72.246.96.54 Python-urllib/1.17 200 0 0 0
2011-12-21 04:10:01 172.27.70.10 GET /OTPAuthentication/Service.asmx -
443 - 72.246.96.53 Python-urllib/1.17 200 0 0 0
2011-12-21 04:10:01 172.27.70.10 GET / - 443 - 72.246.50.40 - 200 0 64 46
2011-12-21 04:10:01 172.27.70.10 GET / - 443 - 72.246.50.40 - 200 0 64

I want only one like;

11-12-21 04:09:01 172.27.70.10 GET /OTPAuthentication/Service.asmx -
443 - 72.246.96.53 Python-urllib/1.17 200 0 0 0

I am using line breaker in props.conf like this

[tms-iis]
REPORT-tms_iisfields = tms_iisfields
SHOULD_LINEMERGE = false
LINE_BREAKER= ([\r\n]+)\s+\d+\s+\d+\s+\d+\s+\d+

but its not working, please help me on this, and also for time format , what i need to write.

[suhprano]

I had a similar problem but it got fixed after putting in a millisecond value in the timestamp.

[Drainy]

@kml_uvce
I've just tested your data and Splunk should be logging that correctly as single line events by default.
I would suggest removing any definitions you have made for it and test re-indexing it again (Or at least how it will appear in your data).

Also, when using Splunk-Base there are three fields, Question, Answer and comment. The question is an issue a person has raised with their Splunk experience. If you have any updates or changes to this then it is best practice to click on the edit button and update your question under a heading at the bottom like. EDIT. This keeps the thread simple to follow and will get you better answers as people can nip in and quickly read the problem and steps you have tried.
Answers are for other users to post an answer that solves your problem or if you fix it you can also post and accept your own answer.
Comments are used to make comments on answers or questions, a lot of your posts should really just be comments 🙂

But back to the point, you should clear out your props and transforms for any definitions that could be affecting your data and allow new logs to re-index. Bear in mind that changes after restart will only affect NEW data. Stuff you have already indexed will not change.

EDIT: Oh, and if someone gives you an answer that is correct then click on the little tick to the left of their answer. This marks it as being right and will help others experiencing the same problems in the future to find your question and answers. (Don't forget to do this for older questions you've asked too!)

[kml_uvce]

Sorry line breaker is

LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{1,2}-\d{1,2}\s+\d{1,2}:\d{1,2}:\d{1,2})

[kml_uvce]

This is in index

props.conf
[tms-iis]
pulldown_type = true
CHECK_FOR_HEADER = False
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{1,2}-\d{1,2)s+\d{1,2}:\d{1,2}:\d{1,2})
REPORT-tms_iisfields = tms_iisfields

[kml_uvce]

Hi Its not wokring for me, I am using universal forwarder and

In forwarder:
props.conf
[tms-iis]
SHOULD_LINEMERGE = False
CHECK_FOR_HEADER = false

inputs.conf
[monitor://c:\inetpub\logs\logfiles\W3SVC1]
disabled = 0
sourcetype = tms-iis
index = windows

outputs.conf
[tcpout]
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = _audit
forwardedindex.filter.disable = false

and in indexer side:
props.conf
[tms-iis]
pulldown_type = true
CHECK_FOR_HEADER = False
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{1,2}-\d{1,2)\s+\d{1,2}:\d{1,2}:\d{1,2})
REPORT-tms_iisfields = tms_iisfields

transforms.conf
[tms_iisfields]
DELIMS = " "
FIELDS = date, time, s-ip, cs-method, cs-uri-stem, cs-uri-query, s-port, cs-username, c-ip, cs-User-Agent, sc-status, sc-substatus, sc-win32-status, time-taken

[kml_uvce]

This is still not working...

[kml_uvce]

I wrote this in props.conf in indexer side, is there any need to write same in props.conf in forwarder side also ?

[kristian_kolb]

see update2 above. /k

[_d_]

Try the one below - notice the positive lookahead after the capture group ([\r\n]+):

[tms-iis]
CHECK_FOR_HEADER = False
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{1,2}-\d{1,2)\s+\d{1,2}:\d{1,2}:\d{1,2})
REPORT-tms_iisfields = tms_iisfields

Note that this will keep years in 4 digit format.

Hope this helps.

> please upvote and accept answer if you find it useful - thanks!

[kml_uvce]

Also used SHOULD_LINEMERGE = true and BREAK_ONLY_BEFORE_DATE=true but was not working...

[kml_uvce]

I am using this in props.conf

[tms-iis]
CHECK_FOR_HEADER = False
MAX_TIMESTAMP_LOOKAHEAD = 25
TIME_FORMAT = %Y-%m-%d %H:%M:%S
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)20/\d/\d-/\d/\d-/\d\d/\s/\d/\d:/\d/\d:/\d/\d/\s
REPORT-tms_iisfields = tms_iisfields

[kml_uvce]

For sometime I got the single line but again getting same multiline error

[Drainy]

Some other things to remember;

Restart after making changes (there is a search command that reloads the configs but experience has taught me that its not 100% reliable).

These changes will NOT affect any previously indexed events, only the newest ones coming in.

[kristian_kolb]

You're absolutely right. A restart IS required, since these configs relate to INDEX-time operations. Search-related operations, such as field extractions can usually be activated with

| extract reload=t

/k

[kml_uvce]

Still not working for
LINE_BREAKER=([\r\n]+)20\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d\s

[kml_uvce]

Still not working...

[kristian_kolb]

Hi, I assume that you have tried without any special directives first, which should work fine for IIS logs. Did you also try BREAK_ONLY_BEFORE_DATE=true ?

Anyway, your regex for LINE_BREAKER seems to be wrong, see below for a more correct version.

([\r\n]+)20\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d\s

UPDATE: well that seems a bit odd. Did you try BREAK_ONLY_BEFORE_DATE=true instead of LINE_BREAKER? In any case, for the time extraction you should use;

MAX_TIMESTAMP_LOOKAHEAD=25
TIME_FORMAT=%Y-%m-%d %H:%M:%S

UPDATE 2:

Also, make sure that this is configured where the parsing takes place;

If you have a heavy forwarder, on the forwarder.
If you have a universal, lightweight or no forwarder, on the indexer.

Restart the splunkd after making the changes.

Please mark as answered a/o upvote if this solves your problem.

/Kristian