Unable to divert events at index time to different sourcetypes from single source

[Cuyose]

Log contains a hodgepodge of different logger events. Using transforms and props I am still unable to assign the sourcetypes to the events based on simple regex. This is just to test the ability to do this, but nothing I have tried is working. Everything is going into the default declared augustus sourcetype. This is all being run on a stand alone enterprise test machine.

props.conf

[sourcetype::augustus]
TRANSFORMS-change_sourcetype = mapped_events

[sourcetype::augustus_mapped]
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1

transforms.conf

[mapped_events]
REGEX = :\s
FORMAT = sourcetype::augustus_mapped
DEST_KEY = MetaData:Sourcetype

[Cuyose]

So the following worked with my example. This is just a test for functionality, so the sourcetype definitions aren't actually any different yet.

Not 100% sure what made this work, as I was positive I had this config earlier, however I think that the existence of the sourcetypes already available in the app, along with the transforms.conf FORMAT section including the sourcetype:: key did it.

The following configuration seems to work as intended.
Within the scope of the app: "all referenced sourcetypes were pre-created"

inputs.conf
[default]
index = iem_test

[monitor://C:\QV\Augustus.log]
disabled = false
sourcetype = augustus

props.conf
[augustus]
TRANSFORMS-change_sourcetype = mapped_events
DATETIME_CONFIG =
NO_BINARY_CHECK = true
category = Custom
disabled = false

[augustus_mapped]
DATETIME_CONFIG =
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1

transforms.conf
[mapped_events]
FORMAT = sourcetype::augustus_mapped
REGEX = DEBUG.*EXIT
DEST_KEY = MetaData:Sourcetype

[gcusello]

hi Cuyose,
at first in props.conf, you don't need to use sourcetype in stanza header

[augustus]
TRANSFORMS-change_sourcetype = mapped_events

then in transforms.conf you shoud modify the row with the ovverriding sourcetype in

[mapped_events]
REGEX = :\s
FORMAT = augustus_mapped
DEST_KEY = MetaData:Sourcetype

Anyway the problem probably is in the regex you used: could you share an example of the events to tranform?
Using your regex you're saying that you want to override all the events with sourcetype augustus where there's a colon followed by a space, is it what you want to transforms?

Bye.
Giuseppe

[Cuyose]

I have also tried the source:: setting in the props stanza and still no luck. Editing props to a simple text string in your example still does not work. it just creates a new sourcetype of Augustus. inputs.conf does not specify a default sourcetype for this input

inputs.conf

[default]
index = iem_test

[monitor://C:\QV\Augustus.log]

sample event
2018-06-29 12:17:37,608 DEBUG com.foo.augustus.daoimpl.ApplicationEventInformationDAOImpl - EXIT : create in 4 msec

[gcusello]

Sourcetype is the best way to identify a log.
But in stanza header you don't need to put sourcetype::
Anyway the problem is the regex, try a different one like this
REGEX = DEBUG.*EXIT
Bye.
Giuseppe

[Cuyose]

This is working now, however not entirely sure why. I deleted the previous inputs using the UI, then added the data with the ui, choosing manual sourcetype setting, rather than auto and it works. Not sure what the ui triggered to make this work

[Cuyose]

I spoke too soon, something still is not working, even though I am seeing sourcetype=augustus_mapped for some of the events, it is not a searchable field. when using sourctype=augustus_mapped, you get 0 results. Not sure why this is being so difficult

current props and transforms.

props
[augustus]
TRANSFORMS-change_sourcetype = mapped_events
DATETIME_CONFIG =
NO_BINARY_CHECK = true
category = Custom
disabled = false

[augustus_mapped]
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1

transforms

[mapped_events]
REGEX = DEBUG.*EXIT
WRITE_META = true
FORMAT= augustus_mapped
DEST_KEY = MetaData:Sourcetype

[gcusello]

try with this regex.

.*DEBUG.*EXIT.*

Another question: Where do you put props.conf and transforms.conf?
They must be on the Indexer not on the Forwarder.
Instead inputs.conf must be on Forwarder.

Bye.
Giuseppe

[Cuyose]

turns out you do need the sourcetype:: in the FORMAT for it to work. this now works and lets me search on the sourcetype

[mapped_events]
FORMAT = sourcetype::augustus_mapped
REGEX = DEBUG.*EXIT
DEST_KEY = MetaData:Sourcetype

[Cuyose]

Thanks for your quick responses. I removed any sourcetype:: in stanzas, and edit the regex per your example. This still is not working 😞

[gcusello]

Could you share props.conf and transforms.conf?
Bye.
Giuseppe