topic Re: universal forwarder manipulate host and source via inputs.conf in Getting Data In

universal forwarder manipulate host and source via inputs.conf

vessev — Wed, 30 Sep 2020 04:55:39 GMT

I have a dedicated server which is running syslog-ng and a universal forwarder.

i want to set 3 things one of them dynamically:

# /opt/splunkforwarder/etc/system/local/inputs.conf
[monitor:///data/syslog-ng/logs/u514/cisco/ios/*/*.log]
sourcetype = syslog
source = syslog-ng:udp514
host_segment = 7

The problem is that i cannot set source and host_segment (or host_regex) at the same time.
Because the host_segment uses (why ever on earth i don't know) the source string.
Host segment defines the "7" position as the host variable.
So if i define the source by myselft host_segment will fail.

Is it possibile to have a manually created source field and a dynamically generated host field?

I could do this by creating a new props.conf and transforms.conf to manipulate the source segment. But i do not want this to be generally done... There are a few logs for which i do not want that.

Re: universal forwarder manipulate host and source via inputs.conf

DalJeanis — Fri, 10 Apr 2020 21:13:52 GMT

In the case of a monitor stanza, the source, by definition, is the name of the file it came from.

If changing the order of those lines in the monitor stanza doesn't work, then move the alteration of the source to a transforms stanza and use a regex to identify which ones do or do not qualify to become the new source.

That would look something like this. (Try this in a sandbox first, because it's all air code.):

in inputs.conf

[monitor:///data/syslog-ng/logs/u514/cisco/ios/*/*.log]
 sourcetype = syslog
 host_segment = 7

in props.conf

 [syslog]
 transforms-newsource1 = somestanza

in transforms.conf

[somestanza]
SOURCE_KEY = however you are deciding which ones to change 
REGEX = however you are deciding which ones to change 
DEST_KEY = MetaData:Source 
FORMAT = source::syslog-ng:udp514

You can do really tricky stuff, but you have to do it in the exact right order. Here's an old example.

https://answers.splunk.com/answers/511972/regex-to-match-two-fields-in-transformsconf.html

Re: universal forwarder manipulate host and source via inputs.conf

vessev — Fri, 17 Apr 2020 14:53:08 GMT

I thought about your idea before and this could work if there wouldn't be a problem with all other stuff which has the sourcetype [syslog].
As far as i know there is no possibility to divide INPUT Data in props.conf by two keys - sourcetype match AND host match. The only possibility would be to use the source in props.conf and manipulate via transform.conf.
If i want to use the source i have to specify all relevant information in the file path. For example:
/data/syslog/syslog-ng/u514/cisco/ios/ciscoswitch/20200417-cisco-ios.log and the take the information out of it that i want.
After thinking about those possibilities i decided to let go of this requirement.
I changed the folder path to include all important information and live with the fact that there is no easy solution for that. To manipulate 3 confs to achive this goal could backfire in a year or two.

Thanks for your suggestion - i check it as answer cause it is for all what i know the only possibilty to achive this. BR Michele

Re: universal forwarder manipulate host and source via inputs.conf

DalJeanis — Thu, 23 Apr 2020 22:06:28 GMT

@vessev -

That's probably a good decision. In essence, if you wanted to do that, you could build a synthetic field in multiple steps, but you are leaving land mines for a later admin... who might very well be yourself.

More importantly, overriding the "source" is not perhaps the best strategy.

Consider adding another index-time field that contains what you want: a tag, or another named field, or whatever. Think about the overhead, and make your best guess.

If you only added it to the ones that you wanted to change, the overhead cost might be limited.

Best wishes.

Re: universal forwarder manipulate host and source via inputs.conf

vessev — Mon, 27 Apr 2020 11:39:00 GMT

Thats a good idea thank you for that suggestion.

BR Michele