Splunk Search

Issues creating a gateway to create splunk events from an integration platform

Lowell
Super Champion

We would like to be able to send splunk events from our integration platform, but the existing logging infrastructure is not ideal, so we've been looking for some better options. Essentially, the goal is to create simple service that would push message to splunk. For illustration, say a function like: writeSplunkEvent(message, source, sourcetype, host=myhostname, index=main)

The goal is to have full control over where the event ends up in splunk. Also, we want the results in splunk to be 1-for-1, in other words, one call to writeSplunkEvent should yield exactly one event in splunk.

So I came up with the following approach:

  1. Use TCP socket to send the events to splunk.
  2. Use a dynamic splunk header to assign the desired host, source, sourcetype, and index.
  3. Disable line merging and use a hard-coded line breaker. (An idea borrowed from the wmi sourcetype.)

The problem I'm running into to is simple to explain. It doesn't work. The event never get indexed. It seems like this approach should work, so let me know if I'm missing something here or have another approach you think would work better for us.

Thanks!


Here is my splunk config:

inputs.conf:

[tcp://9797]
source = tcp-9797
sourcetype = tcp-9797

props.conf:

[tcp-9797]
LINE_BREAKER = ([\r\n]---+ ?MYCOMP-SPLUNK-TCP v\d{1,4} END\.OF\.EVENT ?---+[\r\n]+)
SHOULD_LINEMERGE = False
MAX_TIMESTAMP_LOOKAHEAD = 40
BREAK_ONLY_BEFORE_DATE = false

My writeSplunkEvent() service:

The java code used to write the message:

try {
    s = new Socket();
    s.connect(new InetSocketAddress("splunk.domain.com", 9797), 500);
    OutputStream raw = s.getOutputStream();
    OutputStream buffered = new BufferedOutputStream(raw);
    out = new OutputStreamWriter(buffered, "UTF-8");
    out.write(message);
    out.flush();
}
catch (Exception e) {
    throw new ServiceException(e);
}
finally {
    // Shutdown output stream.
    try {
    out.close();
    s.close();
    }
    catch (Exception e) {
        throw new ServiceException(e);
    }
}

The final message sent to splunk is in the form:

***SPLUNK*** index=test sourcetype=remote_spnk_message-tcp source=my.test.service
2010-10-01 17:21:19.123
<insert actual message here, which could be many lines long...>
...
----------MYCOMP-SPLUNK-TCP v1 END.OF.EVENT -----------


I thought that perhaps the issue was that the sourcetype assigned by the dynamic splunk head was being used to determine how the event was being handled in terms of line merging and event breaking, so I tried setting up the remote_spnk_message-tcp with the same SHOULD_LINEMERGE and LINE_BREAKER, but that didn't seem to work either. And if that did work, then I would have the issue where that config would have to established for all possible sourcetypes used by this gateway service...

0 Karma

Lowell
Super Champion

Here's the solution I came up with. I created by own "dynamic" header (which simply uses a bunch of transformers) because I couldn't get the built-in dynamic header to work for me in this case.

Each message is send to splunk on TCP port 9797. And the format of each message is like this:

SPLUNK-MESSAGE: sourcetype="mysourcetype" source="mysource" host="myhost.domain.com" index="myindex"
My multiline
message goes here...
---END-OF-MESSAGE---

Splunk configuration

Entry in inputs.conf

[tcp://9797]
source = tcp-9797
sourcetype = _generic_message

Entries in props.conf:

[_generic_message]
# Messages can optionally start with a header:
#   SPLUNK-MESSAGE: sourcetype="st" source="s" host="host" index="index"
# All messages must be followed by a tailer:
#   ---END-OF-MESSAGE---
SHOULD_LINEMERGE = False
LINE_BREAKER = ([\r\n]+---END-OF-MESSAGE---[\r\n]+)
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
# Handle the generic header (notice that order is important here:  drop-header must be last)
TRANSFORMS-aa_genmsg-header = genmsg-set-index, genmsg-set-host, genmsg-set-source, genmsg-set-sourcetype, genmsg-drop-header
TRUNCATE=1000000

Entries in transforms.conf:

### Generic message header line (for setting host/source/sourcetype/index)
[genmsg-set-sourcetype]
REGEX    = ^SPLUNK-MESSAGE:[^\r\n]*\bsourcetype="([^"\r\n]+)"
DEST_KEY = MetaData:Sourcetype
FORMAT   = sourcetype::$1

[genmsg-set-source]
REGEX    = ^SPLUNK-MESSAGE:[^\r\n]*\bsource="([^"\r\n]+)"
DEST_KEY = MetaData:Source
FORMAT   = source::$1

[genmsg-set-host]
REGEX    = ^SPLUNK-MESSAGE:[^\r\n]*\bhost="([^"\r\n]+)"
DEST_KEY = MetaData:Host
FORMAT   = host::$1

[genmsg-set-index]
REGEX    = ^SPLUNK-MESSAGE:[^\r\n]*\bindex="([^"\r\n]+)"
DEST_KEY = _MetaData:Index
FORMAT   = $1

[genmsg-drop-header]
REGEX    = (?s)^SPLUNK-MESSAGE:[^\r\n]+[\r\n]{1,2}(.*)$
DEST_KEY = _raw
FORMAT   = $1
0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Well, a source props.conf stanza overrides a sourcetype one in case of conflict, but regardless of those settings (unless you have nullQueue or similar transforms going on) you should still get something indexed. You could specify linebreak settings in the source stanzas to solve the long-term problem, but first, you should just get the events indexed, even with timestamp and linebreak errors.

0 Karma

southeringtonp
Motivator

Silly question but it has to be asked - Are you sure the event is actually being sent across the network? Can you see it with a sniffer?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...