Getting Data In

Sourcetytping and override source name on directory with multiple files

Starlette
Contributor

Consider i have a directory like :

/mydir/file1.log
/mydir/file.2.log
/mydir/message_1234.trc

Now i want to end up with choosen sourcetypes and souce overrriding for getting rid of the 1234.
When I configured this according the docs it doenst work, the source is not overridden.
I am not sure if this is the best approach and if this will work due the loop in destkey :

inputs.conf
[monitor:///mydir]

props.conf

[source::.../file1.log]
sourcetype = mysourcetype1

[source::.../file2.log]
sourcetype = mysourcetype2

[source::.../*.trc]
sourcetype = temp_sourcetype

[temp_sourcetype]
TRANSFORMS-source_fix = sourceoverride

[mysourcetype3]
TRANSFORMS-fix_source = source_clean_roll_1

tranforms.conf

[temp_sourcetype]
DEST_KEY = MetaData:Sourcetype
SOURCE_KEY = MetaData:Source
REGEX = \d+\.trc
FORMAT = sourcetype::mysourcetype3


[source_clean_roll_1]
DEST_KEY   = MetaData:Source
SOURCE_KEY = MetaData:Source
REGEX = (.*)(message_)(\d+)(\.trc)
FORMAT = source::$1$2$4
Tags (2)
0 Karma

Starlette
Contributor

I am indeed checking this but thats a bit difficult :

/mydir/message_1234.trc

it would be handy if

/mydir/message_\d+\.trc 

was supported but that becomes a whitelist and that will exclude the *.log so it looks like I am in a lock

0 Karma

kristian_kolb
Ultra Champion

Hi Starlette,

It seems you have a bad reference between props and transforms. Your first transforms stanza should be named [sourceoverride] instead of [temp_sourcetype].

UPDATE: After looking a little more closely at what your trying to do; I'm not sure that you can parse your data twice with props -> transforms -> props -> transforms. I think, and I may be wrong here, that this transforms operations can only occur once, i.e. data comes in, goes through props, pops over to transforms, and goes into the index. You cannot go back for another round of parsing.

UPDATE 2: Well, going out on a limb here. I have not tried this configuration - but it looks like it could work. Sorry if it doesn't.

Wouldn't it be possible do some of the work in inputs.conf, i.e.

inputs.conf

[monitor:///mydir/*.trc]
sourcetype=blahblah

props.conf

[blahblah]
TRANSFORMS-asdf = trc_resourcing

transforms.conf

[trc_resourcing]
DEST_KEY=MetaData:Source
SOURCE_KEY=MetaData:Source
REGEX = (.*)(message_)(\d+)(\.trc)
FORMAT = source::$1$2$4

hope this helps,

Kristian

Starlette
Contributor

I am indeed checking this but thats a bit difficult :

/mydir/message_1234.trc

it would be handy if

/mydir/message_\d+.trc

was supported but that becomes a whitelist and that will exclude the *.log so it looks like I am in a lock

0 Karma

kristian_kolb
Ultra Champion

Sorry - do you want a different sourcetype for each .trc-file? Hmm, yes. That's what you were trying to do in the first transform...

Will there be a unique sourcetype per file, or are there say a dozen different sourctypes you want the trc-files to be? Could you hardcode it in separate monitor stanzas?

0 Karma

Starlette
Contributor

Hi Kristian,...
Then i ended up with 1 sourcetype, ( for more *.trc's)
So thats not ideal as well...thanks

0 Karma

Starlette
Contributor

yes, thats what was thinking as well, then it might be not possible to handle this...

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.