topic Re: How to parse and index a litmonth in other language? (ex: ago (agosto) is August in Spanish) in Getting Data In

How to parse and index a litmonth in other language? (ex: ago (agosto) is August in Spanish)

MacaVergara — Wed, 07 Oct 2015 21:42:20 GMT

The date I'm trying to index is in a field inside of each row within a log, and looks like this:

Time Field
ago 31,2015 02:01:18 PM

"ago" is for agosto (August in English). In other words "ago" is the litmonth for august in Spanish language.

Every time I'm trying to index a litmonth that is different from English language, it doesn't catch them.

I tried to configure the datetime.xml like this:

<define name="_litmonth2"  extract="litmonth">
     <text><![CDATA[(?<![\d\w])(ene|feb|mar|abr|may|jun|jul|ago|sep|oct|nov|dic)[a-z,\.;]*]]></text>
</define>

<define name="_otherdate" extract="litmonth, ignored_sep, day, zone, ignored_sep2, year">
     <text><![CDATA[(?<!\w|\d[:\.\-])]]></text>
     <use name="_litmonth2"/> 
         <text><![CDATA[([/\- ]) {0,2}]]></text>
         <use name="_day"/>
         <text><![CDATA[(?!:) {0,2}(?:\d\d:\d\d:\d\d(?:[\.\,]\d+)? {0,2}]]></text>
     <use name="_zone"/>
         <text><![CDATA[)?((?:\2|,) {0,2}]]></text>
         <use name="_year"/> 
         <text><![CDATA[)?(?!/|\w|\.\d)]]></text>
</define>

<timePatterns>
      <use name="_time"/>
      <use name="_hmtime"/>
      <use name="_hmtime"/>
      <use name="_dottime"/>
      <use name="_combdatetime"/>
      <use name="_utcepoch"/>
      <use name="_combdatetime2"/>
</timePatterns>
<datePatterns>
      <use name="_otherdate"/>
</datePatterns>

This actually catch the litmonth ago in the indexing time like the month number 8:

But when is indexed, then it shows this:

As you can see, it not the same date than before.

¿Any clues?

Re: How to parse and index a litmonth in other language? (ex: ago (agosto) is August in Spanish)

LukeMcfly3 — Fri, 05 Feb 2016 14:53:45 GMT

Hi MacaVergara

Have you found a solution for this.
I'm stuck on the same problem. I replicated your solution for german months

<datetime>

<define name="_year" extract="year">
    <text><![CDATA[(20\d\d|19\d\d|[901]\d(?!\d))]]></text>
</define>

<define name="_day"  extract="day">
    <text><![CDATA[(0?[1-9]|[12]\d|3[01])]]></text> 
</define>


 <define name="_litmonth2"  extract="litmonth">
      <text><![CDATA[(?<![\d\w])(Januar|Februar|März|April|Mai|Juni|Juli|August|September|Oktober|November|Dezember)[a-z,\.;]*]]></text>
 </define>


<define name="_irequestdate"  extract="day, litmonth, year">
     <text><![CDATA[(?<![\d\w])]]></text>
     <use name="_day"/>
     <text><![CDATA[\. ]]></text>
     <use name="_litmonth2"/>
     <text><![CDATA[ ]]></text>
         <use name="_year"/>     
</define>


<datePatterns>
      <use name="_irequestdate"/>
</datePatterns>

</datetime>

But I don't even get as far as to see the stamp beeing recognized correctly.

Re: How to parse and index a litmonth in other language? (ex: ago (agosto) is August in Spanish)

Jeremiah — Fri, 05 Feb 2016 17:19:15 GMT

Just to clarify, it catches the correct date in the preview, but not when you actually index the data?

Re: How to parse and index a litmonth in other language? (ex: ago (agosto) is August in Spanish)

LukeMcfly3 — Tue, 09 Feb 2016 08:59:48 GMT

Unfortunately It doesn't even recognize the correct date format in the preview.

Re: How to parse and index a litmonth in other language? (ex: ago (agosto) is August in Spanish)

jmallorquin — Tue, 09 Feb 2016 10:05:23 GMT

Hi,

Maybe a easy way to correct this problem is set 12 sedcmd with all possible months:

SEDCMD-AGOSTO = s/^ago/8/
SEDCMD-SEPTIEMBRE = s/^sep/9/
.....

Re: How to parse and index a litmonth in other language? (ex: ago (agosto) is August in Spanish)

LukeMcfly3 — Tue, 09 Feb 2016 13:26:37 GMT

Unfortunately Timestamp Extraction takes place before SED

Re: How to parse and index a litmonth in other language? (ex: ago (agosto) is August in Spanish)

jmallorquin — Tue, 09 Feb 2016 13:57:39 GMT

Thanks,

I answer without testing and now i can say that you are right.

Have you try a HF to do this change?

Regards,

Re: How to parse and index a litmonth in other language? (ex: ago (agosto) is August in Spanish)

benjamin1337 — Tue, 09 Feb 2016 15:47:04 GMT

You could try extracting the correct month in a new field (during index time) and then map that field onto _time (during search time).

So in props, you would have:

[testsourcetype]
TRANSFORMS-correctMonths = fixJanuar, fixFebruar, ...
EVAL-_time = strptime(correct_date,"%d.%m.%Y")

And in transforms.conf you would have:

[fixJanuar]
REGEX = (\d+). Januar (\d+)
FORMAT = correct_date::$1.01.$2
WRITE_META = true

[fixFebruar]
REGEX = (\d+). Februar (\d+)
FORMAT = correct_date::$1.02.$2
WRITE_META = true

...

This works fine for German timestamps in a format like: 03. Januar 2015

For Spanish, it needs to be adopted slightly (also to take time into consideration).

Best, Benjamin

Re: How to parse and index a litmonth in other language? (ex: ago (agosto) is August in Spanish)

LukeMcfly3 — Wed, 10 Feb 2016 22:09:41 GMT

Thank you very much Benjamin. This does in fact work. The timestamp won't be recognized in the preview but as soon as it ist indexed the correct time stamp is shown.

Re: How to parse and index a litmonth in other language? (ex: ago (agosto) is August in Spanish)

jeffland — Thu, 01 Oct 2020 10:57:38 GMT

Just to quickly leave a note here for anyone googling this: using INGEST_EVAL in transforms.conf and targeting _time with := to overwrite it, this makes it even better than EVAL-searchtime_things because the event will be saved with a correct timestamp, meaning you'll find your events where you expect.