How to send events over long-lived TCP connection?

[giraffe]

Our application generates many small monitoring events. We are feeding these to Splunk by TCP. We would like to keep a single connection open between each of our processes and the Splunk TCP input. However, when we do this the events do not appear when sent over the TCP connection, but only when the TCP stream is closed. Then they all appear together as a single event.

How do we configure the TCP input to generate one event per line as they stream in, instead of treating all data received over the connection as a single event?

[Vladimir]

You can add timestamp/unix time to the begging of event line. For example:

1320663001,event1
1320663002,event2

In this case splunk will divide events

unix time is a "date +%s" command in linux systems, or $sTimeStamp = get-date ((Get-Date).touniversaltime()) -uformat "%s" in PowerShell

[Ayn · Accepted solution]

You need to tell Splunk when to break the incoming data into new events. Right now Splunk doesn't find anything to break the events on, so it waits until the end of the input stream. By default Splunk will create a new event when it encounters something it recognizes as a timestamp, or when the maximum defined number of lines in an event has been reached (default 256). These settings are specified in props.conf (docs: http://docs.splunk.com/Documentation/Splunk/latest/admin/propsconf - the settings you're looking for are MAX_EVENTS and BREAK_ONLY_BEFORE_DATE).

Could you paste some sample events? That way we could help with coming up with working settings for breaking your log stream into individual events.

[giraffe]

To answer my own question: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Wheretofindtheconfigurationfiles

[giraffe]

I've found a props.conf in etc/system/defaults. Can I create one in /etc/system/local? And if so, will the definitions in there be merged with the one in etc/system/defaults or will the defaults be ignored?