topic Re: How to configure props.conf to set specific Datetime-configs on specific sources/sourcetypes from specific hosts? in Getting Data In

How to configure props.conf to set specific Datetime-configs on specific sources/sourcetypes from specific hosts?

tweaktubbie — Tue, 29 Sep 2020 08:29:19 GMT

I have an issue with two servers with WebSphere logs that have an overriding different timezone setting in the jvm. Other servers don't have an issue. To prevent wrong interpretation of timestamps, I want to add to the props.conf something like:

using specific hosts [host::server1|server2] ór based on the logfile [source::/opt/data/WebSphere/logs/log.log]
DATETIME_CONFIG=NONE

But you can't of course somehow filter host AND source/sourcetype. Using only host would make other (performance/OS) events useless and unreliable; using only source would affect all the other logfiles from multiple other servers.

It's not only datetime related; I also'd like to be able to break_only_before/after some logfiles from specific host (think like, some end with <END>, other must be broken before datetime)

So in general is the question: how can you use some local props.conf settings for specific sources/sourcetypes on specific hosts?

Re: How to configure props.conf to set specific Datetime-configs on specific sources/sourcetypes from specific hosts?

javiergn — Thu, 21 Jan 2016 14:00:23 GMT

This is what I would do:

For those servers that have an issue modify your inputs.conf and rename the sourcetype on those stanzas currently reading your WebSphere logs to something like: MyPreviousSourceTypeName-Host1 and MyPreviousSourceTypeName-Host2
Use the new sourcetypes to filter out in your next hop (heavy forwarder or indexer) and do two things: apply the time configurations you want to apply and rename the sourcetype back to whatever it was before.

Let me know if that makes sense.

Some quick instructions on how to rename a sourcetype before indexing:

props.conf:

[MyPreviousSourceTypeName-Host1]
TRANSFORMS-wrongTimeSourcetypeFix1 = set_sourcetype_back_WebSphere

[MyPreviousSourceTypeName-Host2]
TRANSFORMS-wrongTimeSourcetypeFix2 = set_sourcetype_back_WebSphere

tranforms.conf:

[set_sourcetype_back_WebSphere] 
FORMAT= sourcetype::MyPreviousSourceTypeName
DEST_KEY = MetaData:Sourcetype

Re: How to configure props.conf to set specific Datetime-configs on specific sources/sourcetypes from specific hosts?

tweaktubbie — Fri, 22 Jan 2016 10:49:04 GMT

Ah, that indeed is a nice approach I hadn't thought of, also handy for other purposes!

Only one thing I forgot to mention which now comes to mind: I use a deployment server to use a generic configuration on all WebSphere servers. From what I read, precedence should work fine with your solution, if the default etc/apps/websphere app is overruled by inputs.conf with an alternative sourcetype in an app in /etc/apps/(anything starting with a-v) configured.

Thanks a lot for thinking along!