topic Re: allowing TCP input to define the sourcetype implicitly in All Apps and Add-ons

allowing TCP input to define the sourcetype implicitly

acidkewpie — Fri, 17 Oct 2014 14:33:20 GMT

Hi, I'm looking to replicate the output from streamfwd with a separate script, and from what I understand of the Stream solution, it's entirely on the client to send fully formed messages to the server. But obviously streamfwd provides all the Proprietary metadata compared to just spitting a JSON structure into a TCP port which is what my script will need to do (I presume). A key thing to allow the data to be used as if it were actually from streamfwd is the setting of the sourcetype. How can I best allow these messages coming through a single TCP input to have a proper sourcetype as defined by the client? I believe it's possible for props.conf to mangle sourcetypes upon indexing, but I'd like to stay away from that if possible and let the client define it just like I believe streamfwd does. I noticed I could just put "sourcetype" as a field in the JSON payload, but this left the logs having two sourcetypes, the one set there AND "syslog" as defined on the TCP input itself.

Re: allowing TCP input to define the sourcetype implicitly

ndoshi — Fri, 17 Oct 2014 14:41:08 GMT

You can use props.conf

[source::stream]
TRANSFORM-mytype=set_sourcetype

transforms.conf

[set_sourcetype]
REGEX=< insert your regex>
FORMAT= sourcetype::<name of your sourcetype>
DEST_KEY = MetaData:Sourcetype

See docs:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

Re: allowing TCP input to define the sourcetype implicitly

acidkewpie — Fri, 17 Oct 2014 15:36:59 GMT

Thanks, is there a way to do this on the client side though?

Re: allowing TCP input to define the sourcetype implicitly

ndoshi — Fri, 17 Oct 2014 15:39:34 GMT

If you use a heavy forwarder, yes you can apply transforms. I would not recommend a heavy forwarder unless you have a good reason as the Universal Forwarder scales better. The Heavy Forwarder sends events while the UF sends blocks of data at a time making it perform better..

Re: allowing TCP input to define the sourcetype implicitly

acidkewpie — Fri, 17 Oct 2014 15:43:35 GMT

Well as above, this is a script sending data to a raw tcp input.