Splunk Search

Issues creating a gateway to create splunk events from an integration platform

Super Champion

We would like to be able to send splunk events from our integration platform, but the existing logging infrastructure is not ideal, so we've been looking for some better options. Essentially, the goal is to create simple service that would push message to splunk. For illustration, say a function like: writeSplunkEvent(message, source, sourcetype, host=myhostname, index=main)

The goal is to have full control over where the event ends up in splunk. Also, we want the results in splunk to be 1-for-1, in other words, one call to writeSplunkEvent should yield exactly one event in splunk.

So I came up with the following approach:

  1. Use TCP socket to send the events to splunk.
  2. Use a dynamic splunk header to assign the desired host, source, sourcetype, and index.
  3. Disable line merging and use a hard-coded line breaker. (An idea borrowed from the wmi sourcetype.)

The problem I'm running into to is simple to explain. It doesn't work. The event never get indexed. It seems like this approach should work, so let me know if I'm missing something here or have another approach you think would work better for us.


Here is my splunk config:


source = tcp-9797
sourcetype = tcp-9797


LINE_BREAKER = ([\r\n]---+ ?MYCOMP-SPLUNK-TCP v\d{1,4} END\.OF\.EVENT ?---+[\r\n]+)

My writeSplunkEvent() service:

The java code used to write the message:

try {
    s = new Socket();
    s.connect(new InetSocketAddress("splunk.domain.com", 9797), 500);
    OutputStream raw = s.getOutputStream();
    OutputStream buffered = new BufferedOutputStream(raw);
    out = new OutputStreamWriter(buffered, "UTF-8");
catch (Exception e) {
    throw new ServiceException(e);
finally {
    // Shutdown output stream.
    try {
    catch (Exception e) {
        throw new ServiceException(e);

The final message sent to splunk is in the form:

***SPLUNK*** index=test sourcetype=remote_spnk_message-tcp source=my.test.service
2010-10-01 17:21:19.123
<insert actual message here, which could be many lines long...>
----------MYCOMP-SPLUNK-TCP v1 END.OF.EVENT -----------

I thought that perhaps the issue was that the sourcetype assigned by the dynamic splunk head was being used to determine how the event was being handled in terms of line merging and event breaking, so I tried setting up the remote_spnk_message-tcp with the same SHOULD_LINEMERGE and LINE_BREAKER, but that didn't seem to work either. And if that did work, then I would have the issue where that config would have to established for all possible sourcetypes used by this gateway service...

0 Karma

Super Champion

Here's the solution I came up with. I created by own "dynamic" header (which simply uses a bunch of transformers) because I couldn't get the built-in dynamic header to work for me in this case.

Each message is send to splunk on TCP port 9797. And the format of each message is like this:

SPLUNK-MESSAGE: sourcetype="mysourcetype" source="mysource" host="myhost.domain.com" index="myindex"
My multiline
message goes here...

Splunk configuration

Entry in inputs.conf

source = tcp-9797
sourcetype = _generic_message

Entries in props.conf:

# Messages can optionally start with a header:
#   SPLUNK-MESSAGE: sourcetype="st" source="s" host="host" index="index"
# All messages must be followed by a tailer:
#   ---END-OF-MESSAGE---
LINE_BREAKER = ([\r\n]+---END-OF-MESSAGE---[\r\n]+)
# Handle the generic header (notice that order is important here:  drop-header must be last)
TRANSFORMS-aa_genmsg-header = genmsg-set-index, genmsg-set-host, genmsg-set-source, genmsg-set-sourcetype, genmsg-drop-header

Entries in transforms.conf:

### Generic message header line (for setting host/source/sourcetype/index)
REGEX    = ^SPLUNK-MESSAGE:[^\r\n]*\bsourcetype="([^"\r\n]+)"
DEST_KEY = MetaData:Sourcetype
FORMAT   = sourcetype::$1

REGEX    = ^SPLUNK-MESSAGE:[^\r\n]*\bsource="([^"\r\n]+)"
DEST_KEY = MetaData:Source
FORMAT   = source::$1

REGEX    = ^SPLUNK-MESSAGE:[^\r\n]*\bhost="([^"\r\n]+)"
DEST_KEY = MetaData:Host
FORMAT   = host::$1

REGEX    = ^SPLUNK-MESSAGE:[^\r\n]*\bindex="([^"\r\n]+)"
DEST_KEY = _MetaData:Index
FORMAT   = $1

REGEX    = (?s)^SPLUNK-MESSAGE:[^\r\n]+[\r\n]{1,2}(.*)$
DEST_KEY = _raw
FORMAT   = $1
0 Karma

Splunk Employee
Splunk Employee

Well, a source props.conf stanza overrides a sourcetype one in case of conflict, but regardless of those settings (unless you have nullQueue or similar transforms going on) you should still get something indexed. You could specify linebreak settings in the source stanzas to solve the long-term problem, but first, you should just get the events indexed, even with timestamp and linebreak errors.

0 Karma


Silly question but it has to be asked - Are you sure the event is actually being sent across the network? Can you see it with a sniffer?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...