Hi, I'm looking to replicate the output from streamfwd with a separate script, and from what I understand of the Stream solution, it's entirely on the client to send fully formed messages to the server. But obviously streamfwd provides all the Proprietary metadata compared to just spitting a JSON structure into a TCP port which is what my script will need to do (I presume). A key thing to allow the data to be used as if it were actually from streamfwd is the setting of the sourcetype. How can I best allow these messages coming through a single TCP input to have a proper sourcetype as defined by the client? I believe it's possible for props.conf to mangle sourcetypes upon indexing, but I'd like to stay away from that if possible and let the client define it just like I believe streamfwd does. I noticed I could just put "sourcetype" as a field in the JSON payload, but this left the logs having two sourcetypes, the one set there AND "syslog" as defined on the TCP input itself.
You can use props.conf
[set_sourcetype] REGEX=< insert your regex> FORMAT= sourcetype::<name of your sourcetype> DEST_KEY = MetaData:Sourcetype
If you use a heavy forwarder, yes you can apply transforms. I would not recommend a heavy forwarder unless you have a good reason as the Universal Forwarder scales better. The Heavy Forwarder sends events while the UF sends blocks of data at a time making it perform better..