Sourcetytping and override source name on directory with multiple files

[Starlette]

Consider i have a directory like :

/mydir/file1.log
/mydir/file.2.log
/mydir/message_1234.trc

Now i want to end up with choosen sourcetypes and souce overrriding for getting rid of the 1234.
When I configured this according the docs it doenst work, the source is not overridden.
I am not sure if this is the best approach and if this will work due the loop in destkey :

inputs.conf
[monitor:///mydir]

props.conf

[source::.../file1.log]
sourcetype = mysourcetype1

[source::.../file2.log]
sourcetype = mysourcetype2

[source::.../*.trc]
sourcetype = temp_sourcetype

[temp_sourcetype]
TRANSFORMS-source_fix = sourceoverride

[mysourcetype3]
TRANSFORMS-fix_source = source_clean_roll_1

tranforms.conf

[temp_sourcetype]
DEST_KEY = MetaData:Sourcetype
SOURCE_KEY = MetaData:Source
REGEX = \d+\.trc
FORMAT = sourcetype::mysourcetype3


[source_clean_roll_1]
DEST_KEY   = MetaData:Source
SOURCE_KEY = MetaData:Source
REGEX = (.*)(message_)(\d+)(\.trc)
FORMAT = source::$1$2$4

[Starlette]

I am indeed checking this but thats a bit difficult :

/mydir/message_1234.trc

it would be handy if

/mydir/message_\d+\.trc 

was supported but that becomes a whitelist and that will exclude the *.log so it looks like I am in a lock

[kristian_kolb]

Hi Starlette,

It seems you have a bad reference between props and transforms. Your first transforms stanza should be named [sourceoverride] instead of [temp_sourcetype].

UPDATE: After looking a little more closely at what your trying to do; I'm not sure that you can parse your data twice with props -> transforms -> props -> transforms. I think, and I may be wrong here, that this transforms operations can only occur once, i.e. data comes in, goes through props, pops over to transforms, and goes into the index. You cannot go back for another round of parsing.

UPDATE 2: Well, going out on a limb here. I have not tried this configuration - but it looks like it could work. Sorry if it doesn't.

Wouldn't it be possible do some of the work in inputs.conf, i.e.

inputs.conf

[monitor:///mydir/*.trc]
sourcetype=blahblah

props.conf

[blahblah]
TRANSFORMS-asdf = trc_resourcing

transforms.conf

[trc_resourcing]
DEST_KEY=MetaData:Source
SOURCE_KEY=MetaData:Source
REGEX = (.*)(message_)(\d+)(\.trc)
FORMAT = source::$1$2$4

hope this helps,

Kristian

[Starlette]

I am indeed checking this but thats a bit difficult :

/mydir/message_1234.trc

it would be handy if

/mydir/message_\d+.trc

was supported but that becomes a whitelist and that will exclude the *.log so it looks like I am in a lock

[kristian_kolb]

Sorry - do you want a different sourcetype for each .trc-file? Hmm, yes. That's what you were trying to do in the first transform...

Will there be a unique sourcetype per file, or are there say a dozen different sourctypes you want the trc-files to be? Could you hardcode it in separate monitor stanzas?

[Starlette]

Hi Kristian,...
Then i ended up with 1 sourcetype, ( for more *.trc's)
So thats not ideal as well...thanks

[Starlette]

yes, thats what was thinking as well, then it might be not possible to handle this...