topic Re: Can't load my custom fields in Getting Data In

Can't load my custom fields

apare — Fri, 03 Feb 2012 19:29:24 GMT

I try to index my logs with a custom Source type. Here is my files :

Log file exemple :

#HeaderKey1: header value 1
#HeaderKey2: header value 2
id|severity|date|time|duration|description|uri|operation|collection|lang|format|size|source|errorcode|links|badlinks|collectionid|sourceid|printableuri
...

transforms.conf :

[my_log]
DELIMS="|"
FIELDS = "id", "severity", "date", "time", "duration", "description", "uri", "operation", "collection", "lang", "format", "size", "source", "errorcode", "links", "badlinks", "collectionid", "sourceid", "printableuri"

[my_log_comment]
REGEX = ^#
DEST_KEY = queue
FORMAT = nullQueue

props.conf :

[my_log]
EXTRACT-my_log = my_log_comment, my_log

The result is that all the line are indexed (header include) and the only field i got is _raw... I do not care about the information in the header; I don't want to index it or to take information from it. I only want to have all my log with all the fields.

What is wrong in my configuration? What i should do to index my custom log file correctly

thank you

Re: Can't load my custom fields

bojanz — Sun, 05 Feb 2012 13:13:24 GMT

Your props.conf file is wrong. It's not EXTRACT but TRANSFORMS.
The rest looks ok.

Re: Can't load my custom fields

Ayn — Sun, 05 Feb 2012 13:32:24 GMT

TRANSFORMS is for index-time creation of fields and doesn't support extracting fields using delimiters like you want to. bojanz is right however in that you shouldn't be using EXTRACT, as it is used for defining directly in props.conf how fields should be extracted using regexes. For extractions that use definitions in transforms.conf, you should use REPORT.

Re: Can't load my custom fields

apare — Mon, 28 Sep 2020 10:23:39 GMT

Thank you
there is my new props.conf
[ces_index_log]
EXTRACT-ces_index_log = (?\d*)|(?\w*)|(?\d{4}-\d{2}-\d{2})|(?\d{2}:\d{2}:\d{2})|(?\d*)|(?[^|])|(?[^|])|(?[^|])|(?[^|])|(?[^|])|(?[^|])|(?\d*)|(?[^|])|(?[^|])|(?[^|])|(?[^|])|(?\d*)|(?\d*)|(?[^|]*)