Hi, I'm looking to replicate the output from streamfwd with a separate script, and from what I understand of the Stream solution, it's entirely on the client to send fully formed messages to the server. But obviously streamfwd provides all the Proprietary metadata compared to just spitting a JSON structure into a TCP port which is what my script will need to do (I presume). A key thing to allow the data to be used as if it were actually from streamfwd is the setting of the sourcetype. How can I best allow these messages coming through a single TCP input to have a proper sourcetype as defined by the client? I believe it's possible for props.conf to mangle sourcetypes upon indexing, but I'd like to stay away from that if possible and let the client define it just like I believe streamfwd does. I noticed I could just put "sourcetype" as a field in the JSON payload, but this left the logs having two sourcetypes, the one set there AND "syslog" as defined on the TCP input itself.
You can use props.conf
REGEX=< insert your regex>
FORMAT= sourcetype::<name of your sourcetype>
DEST_KEY = MetaData:Sourcetype
If you use a heavy forwarder, yes you can apply transforms. I would not recommend a heavy forwarder unless you have a good reason as the Universal Forwarder scales better. The Heavy Forwarder sends events while the UF sends blocks of data at a time making it perform better..