Issues creating a gateway to create splunk events from an integration platform

[Lowell]

We would like to be able to send splunk events from our integration platform, but the existing logging infrastructure is not ideal, so we've been looking for some better options. Essentially, the goal is to create simple service that would push message to splunk. For illustration, say a function like: writeSplunkEvent(message, source, sourcetype, host=myhostname, index=main)

The goal is to have full control over where the event ends up in splunk. Also, we want the results in splunk to be 1-for-1, in other words, one call to writeSplunkEvent should yield exactly one event in splunk.

So I came up with the following approach:

1. Use TCP socket to send the events to splunk.
2. Use a dynamic splunk header to assign the desired host, source, sourcetype, and index.
3. Disable line merging and use a hard-coded line breaker. (An idea borrowed from the wmi sourcetype.)

The problem I'm running into to is simple to explain. It doesn't work. The event never get indexed. It seems like this approach should work, so let me know if I'm missing something here or have another approach you think would work better for us.

Thanks!

---

Here is my splunk config:

inputs.conf:

[tcp://9797]
source = tcp-9797
sourcetype = tcp-9797

props.conf:

[tcp-9797]
LINE_BREAKER = ([\r\n]---+ ?MYCOMP-SPLUNK-TCP v\d{1,4} END\.OF\.EVENT ?---+[\r\n]+)
SHOULD_LINEMERGE = False
MAX_TIMESTAMP_LOOKAHEAD = 40
BREAK_ONLY_BEFORE_DATE = false

My writeSplunkEvent() service:

The java code used to write the message:

try {
    s = new Socket();
    s.connect(new InetSocketAddress("splunk.domain.com", 9797), 500);
    OutputStream raw = s.getOutputStream();
    OutputStream buffered = new BufferedOutputStream(raw);
    out = new OutputStreamWriter(buffered, "UTF-8");
    out.write(message);
    out.flush();
}
catch (Exception e) {
    throw new ServiceException(e);
}
finally {
    // Shutdown output stream.
    try {
    out.close();
    s.close();
    }
    catch (Exception e) {
        throw new ServiceException(e);
    }
}

The final message sent to splunk is in the form:

***SPLUNK*** index=test sourcetype=remote_spnk_message-tcp source=my.test.service
2010-10-01 17:21:19.123
<insert actual message here, which could be many lines long...>
...
----------MYCOMP-SPLUNK-TCP v1 END.OF.EVENT -----------

---

I thought that perhaps the issue was that the sourcetype assigned by the dynamic splunk head was being used to determine how the event was being handled in terms of line merging and event breaking, so I tried setting up the remote_spnk_message-tcp with the same SHOULD_LINEMERGE and LINE_BREAKER, but that didn't seem to work either. And if that did work, then I would have the issue where that config would have to established for all possible sourcetypes used by this gateway service...

[Lowell]

Here's the solution I came up with. I created by own "dynamic" header (which simply uses a bunch of transformers) because I couldn't get the built-in dynamic header to work for me in this case.

Each message is send to splunk on TCP port 9797. And the format of each message is like this:

SPLUNK-MESSAGE: sourcetype="mysourcetype" source="mysource" host="myhost.domain.com" index="myindex"
My multiline
message goes here...
---END-OF-MESSAGE---

Splunk configuration

Entry in inputs.conf

[tcp://9797]
source = tcp-9797
sourcetype = _generic_message

Entries in props.conf:

[_generic_message]
# Messages can optionally start with a header:
#   SPLUNK-MESSAGE: sourcetype="st" source="s" host="host" index="index"
# All messages must be followed by a tailer:
#   ---END-OF-MESSAGE---
SHOULD_LINEMERGE = False
LINE_BREAKER = ([\r\n]+---END-OF-MESSAGE---[\r\n]+)
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
# Handle the generic header (notice that order is important here:  drop-header must be last)
TRANSFORMS-aa_genmsg-header = genmsg-set-index, genmsg-set-host, genmsg-set-source, genmsg-set-sourcetype, genmsg-drop-header
TRUNCATE=1000000

Entries in transforms.conf:

### Generic message header line (for setting host/source/sourcetype/index)
[genmsg-set-sourcetype]
REGEX    = ^SPLUNK-MESSAGE:[^\r\n]*\bsourcetype="([^"\r\n]+)"
DEST_KEY = MetaData:Sourcetype
FORMAT   = sourcetype::$1

[genmsg-set-source]
REGEX    = ^SPLUNK-MESSAGE:[^\r\n]*\bsource="([^"\r\n]+)"
DEST_KEY = MetaData:Source
FORMAT   = source::$1

[genmsg-set-host]
REGEX    = ^SPLUNK-MESSAGE:[^\r\n]*\bhost="([^"\r\n]+)"
DEST_KEY = MetaData:Host
FORMAT   = host::$1

[genmsg-set-index]
REGEX    = ^SPLUNK-MESSAGE:[^\r\n]*\bindex="([^"\r\n]+)"
DEST_KEY = _MetaData:Index
FORMAT   = $1

[genmsg-drop-header]
REGEX    = (?s)^SPLUNK-MESSAGE:[^\r\n]+[\r\n]{1,2}(.*)$
DEST_KEY = _raw
FORMAT   = $1

[gkanapathy]

Well, a source props.conf stanza overrides a sourcetype one in case of conflict, but regardless of those settings (unless you have nullQueue or similar transforms going on) you should still get something indexed. You could specify linebreak settings in the source stanzas to solve the long-term problem, but first, you should just get the events indexed, even with timestamp and linebreak errors.

[southeringtonp]

Silly question but it has to be asked - Are you sure the event is actually being sent across the network? Can you see it with a sniffer?