Why is my datetime.xml configuration not extracting two separate timestamp formats from a single log file?

mookiie2005 — Tue, 24 Mar 2015 13:19:31 GMT

I am trying to extract two separate timestamp formats from a single log file. Here is a sample of the logfile:

[16 Mar 23:33:49.967] INFO com.hp.exstream.cc.server.core.phase.ExternalProgramPhase:pool-15-thread-14  - (See the entire program output in the file [var/jobs/5127/java.155941.out].)
[16 Mar 23:33:49.967] INFO com.hp.exstream.cc.server.core.phase.ExternalProgramPhase:pool-15-thread-14  - External program finished OK with return code [0].
[16 Mar 23:33:50.012] INFO com.hp.exstream.cc.server.core.job.JobDriver:pool-15-thread-14   - Execution of [SFTP to OPC] phase finished successfully.
[16 Mar 23:33:50.042] INFO com.hp.exstream.cc.server.core.job.JobDriver:pool-15-thread-14   - Starting execution of [SFTP hal to OPC] phase.
[16 Mar 23:33:50.044] INFO com.hp.exstream.cc.server.core.phase.ExternalProgramPhase:pool-15-thread-14  - Executing program [java] with arguments [-jar /opt/hp/ccdm/delivery-manager-2.3.0/command-center-2.3.0/lib/java/CCDeliveryClient.jar -c sftp -t put -u USWPLA  -p PLA2014highway -v cachi1-bh001.amer.zurich.corp -n 22 -l /opt/hp/ccdm/delivery-manager-2.3.0/command-center-2.3.0/var/jobs/5127 -r /PROD/PLA -x *.hal] in directory [/opt/hp/ccdm/delivery-manager-2.3.0/command-center-2.3.0/var/jobs/5127].
[16 Mar 23:33:56.121] INFO com.hp.exstream.cc.server.core.phase.ExternalProgramPhase:pool-15-thread-14  - Program output: 
(last 1024 chars) ... ession - is present.  The value is: *.hal
[INFO ]20150316@23:33:50,496:com.hpexstream.client.Sftp - Start FTP process.
[DEBUG]20150316@23:33:50,502:com.hpexstream.util.Find - Use file filter: WildcardFileFilter(*.hal)
[DEBUG]20150316@23:33:50,803:com.hpexstream.util.Find - Found file: VI0H150316.hal
[DEBUG]20150316@23:33:50,803:com.hpexstream.util.Find - Found file: VT0H150316.hal
[INFO ]20150316@23:33:50,803:com.hpexstream.client.Sftp - Found files count: 2
[INFO ]20150316@23:33:51,877:com.hpexstream.client.Sftp - Open SFTP channel
[INFO ]20150316@23:33:51,877:com.hpexstream.client.Sftp - Put local file: /opt/hp/ccdm/delivery-manager-2.3.0/command-center-2.3.0/var/jobs/5127/VI0H150316.hal to remote file: /PROD/PLA/VI0H150316.hal
[INFO ]20150316@23:33:55,694:com.hpexstream.client.Sftp - Put local file: /opt/hp/ccdm/delivery-manager-2.3.0/command-center-2.3.0/var/jobs/5127/VT0H150316.hal to remote file: /PROD/PLA/VT0H150316.hal

My props.conf looks like the below:

[datetime]
DATETIME_CONFIG = D:\Program Files\SplunkUniversalForwarder\etc\apps\FORMS\local\datetime.xml
LINE_BREAKER = ([\r\n])+(?=(\[\d{2}\s(\w{3})\s(\d{2}):(\d{2}):(\d{2}).(\d{3})]|\[(\d{4})(\d{2})(\d{2})@(\d{2}):(\d{2}):(\d{2})\,(\d{3})\]))
SHOULD_LINEMERGE=false

Here is my datetime.xml:

\[\d{2}\s(\w{3})\s(\d{2}):(\d{2}):(\d{2}).(\d{3})\]

\[(\d{4})(\d{2})(\d{2})@(\d{2}):(\d{2}):(\d{2})\,(\d{3})\]

Re: Why is my datetime.xml configuration not extracting two separate timestamp formats from a single log file?

cpetterborg — Wed, 25 Mar 2015 02:53:32 GMT

The problem is that you have square brackets around the date in your second format. Here is what you probably should have in your datetime.xml file:

<datetime>
<define name="_mydatetimeformat1" extract="day, month, hour, minute, second, subsecond">
    <text>\[\d{2}\s(\w{3})\s(\d{2}):(\d{2}):(\d{2}).(\d{3})\]</text>
</define>
<define name="_mydatetimeformat2" extract="year, month, day, hour, minute, second, subsecond">
     <text>(\d{4})(\d{2})(\d{2})@(\d{2}):(\d{2}):(\d{2})\,(\d{3})</text>
</define>
<timePatterns>
     <use name="_mydatetimeformat1"/>
     <use name="_mydatetimeformat2"/>
</timePatterns>
<datePatterns>
     <use name="_mydatetimeformat1"/>
     <use name="_mydatetimeformat2"/>
</datePatterns>
</datetime>

I think you may have a problem not having a year in the first date format.

Re: Why is my datetime.xml configuration not extracting two separate timestamp formats from a single log file?

nitesh218ss — Tue, 12 May 2015 05:44:45 GMT

Hi in my log event and filename date is not present i want give a fix date to log so what is do ?

topic Re: Why is my datetime.xml configuration not extracting two separate timestamp formats from a single log file? in Getting Data In

Why is my datetime.xml configuration not extracting two separate timestamp formats from a single log file?

Re: Why is my datetime.xml configuration not extracting two separate timestamp formats from a single log file?

Re: Why is my datetime.xml configuration not extracting two separate timestamp formats from a single log file?