topic Inconsistent line breaking in Security

Inconsistent line breaking

vanaepi — Tue, 21 May 2013 12:38:05 GMT

I am very new to Splunk and this may be a rookie question but I cannot find this anywhere. I have a custom generated demo log file with 1000 similarly structured lines. For some reason, Splunk reads in the first 257 lines as one event, and then the next 743 lines as separate events.

I figured it had something to do with either LINE _ BREAKER or MAX _ EVENTS but the strange thing is that it's not consistent. I would be less surprised if it grouped all my events per 257 in stead of just the first 257 and then single event per line from there on.

Could it perhaps have something to do with the fact that I don't have a timestamp added yet?

Re: Inconsistent line breaking

kristian_kolb — Tue, 21 May 2013 12:48:53 GMT

Yes. If there is no timestamp Splunk will have problems to automatically create/break events.

If you really want to index the events anyway you should set the following in props.conf;

[your sourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
DATETIME_CONFIG = CURRENT

Re: Inconsistent line breaking

jkat54 — Tue, 21 May 2013 14:32:44 GMT

Try the truncate option in the props.conf stanza for the sourcetype. Truncate is needed when the file you're consuming is abnormally large or the lines in the file are abnormally long (no line breaks etc). From the doc, "Change the default maximum line length (in bytes)"

For instance, if you were to consume the applicationHost.config files for IIS servers you could use the following stanza in props.conf:

[iisconfig]
SHOULD_LINEMERGE = True
MAX_EVENTS = 400960
LINE_BREAKER = </configuration>
NO_BINARY_CHECK = true
TRUNCATE = 1000000

And in inputs.conf:

[monitor://C:\Windows\System32\inetsrv\config\applicationHost.conf]
disabled = 0
host = hostname
index = main
sourcetype = iisconfig
crcSalt = <SOURCE>

Re: Inconsistent line breaking

jkat54 — Tue, 21 May 2013 18:30:47 GMT

Who down voted my solution? It would solve his problem if his issue was long lines. The issue that truncate solves can be explained the exact same way he explained his problem. So without a example of the data he's consuming both answers could be correct.