Getting Data In
Highlighted

Why is my datetime.xml configuration not extracting two separate timestamp formats from a single log file?

Communicator

I am trying to extract two separate timestamp formats from a single log file. Here is a sample of the logfile:

[16 Mar 23:33:49.967] INFO com.hp.exstream.cc.server.core.phase.ExternalProgramPhase:pool-15-thread-14  - (See the entire program output in the file [var/jobs/5127/java.155941.out].)
[16 Mar 23:33:49.967] INFO com.hp.exstream.cc.server.core.phase.ExternalProgramPhase:pool-15-thread-14  - External program finished OK with return code [0].
[16 Mar 23:33:50.012] INFO com.hp.exstream.cc.server.core.job.JobDriver:pool-15-thread-14   - Execution of [SFTP to OPC] phase finished successfully.
[16 Mar 23:33:50.042] INFO com.hp.exstream.cc.server.core.job.JobDriver:pool-15-thread-14   - Starting execution of [SFTP hal to OPC] phase.
[16 Mar 23:33:50.044] INFO com.hp.exstream.cc.server.core.phase.ExternalProgramPhase:pool-15-thread-14  - Executing program [java] with arguments [-jar /opt/hp/ccdm/delivery-manager-2.3.0/command-center-2.3.0/lib/java/CCDeliveryClient.jar -c sftp -t put -u USWPLA  -p PLA2014highway -v cachi1-bh001.amer.zurich.corp -n 22 -l /opt/hp/ccdm/delivery-manager-2.3.0/command-center-2.3.0/var/jobs/5127 -r /PROD/PLA -x *.hal] in directory [/opt/hp/ccdm/delivery-manager-2.3.0/command-center-2.3.0/var/jobs/5127].
[16 Mar 23:33:56.121] INFO com.hp.exstream.cc.server.core.phase.ExternalProgramPhase:pool-15-thread-14  - Program output: 
(last 1024 chars) ... ession - is present.  The value is: *.hal
[INFO ]20150316@23:33:50,496:com.hpexstream.client.Sftp - Start FTP process.
[DEBUG]20150316@23:33:50,502:com.hpexstream.util.Find - Use file filter: WildcardFileFilter(*.hal)
[DEBUG]20150316@23:33:50,803:com.hpexstream.util.Find - Found file: VI0H150316.hal
[DEBUG]20150316@23:33:50,803:com.hpexstream.util.Find - Found file: VT0H150316.hal
[INFO ]20150316@23:33:50,803:com.hpexstream.client.Sftp - Found files count: 2
[INFO ]20150316@23:33:51,877:com.hpexstream.client.Sftp - Open SFTP channel
[INFO ]20150316@23:33:51,877:com.hpexstream.client.Sftp - Put local file: /opt/hp/ccdm/delivery-manager-2.3.0/command-center-2.3.0/var/jobs/5127/VI0H150316.hal to remote file: /PROD/PLA/VI0H150316.hal
[INFO ]20150316@23:33:55,694:com.hpexstream.client.Sftp - Put local file: /opt/hp/ccdm/delivery-manager-2.3.0/command-center-2.3.0/var/jobs/5127/VT0H150316.hal to remote file: /PROD/PLA/VT0H150316.hal

My props.conf looks like the below:

[datetime]
DATETIME_CONFIG = D:\Program Files\SplunkUniversalForwarder\etc\apps\FORMS\local\datetime.xml
LINE_BREAKER = ([\r\n])+(?=(\[\d{2}\s(\w{3})\s(\d{2}):(\d{2}):(\d{2}).(\d{3})]|\[(\d{4})(\d{2})(\d{2})@(\d{2}):(\d{2}):(\d{2})\,(\d{3})\]))
SHOULD_LINEMERGE=false

Here is my datetime.xml:

\[\d{2}\s(\w{3})\s(\d{2}):(\d{2}):(\d{2}).(\d{3})\]

\[(\d{4})(\d{2})(\d{2})@(\d{2}):(\d{2}):(\d{2})\,(\d{3})\]
0 Karma
Highlighted

Re: Why is my datetime.xml configuration not extracting two separate timestamp formats from a single log file?

SplunkTrust
SplunkTrust

The problem is that you have square brackets around the date in your second format. Here is what you probably should have in your datetime.xml file:

<datetime>
<define name="_mydatetimeformat1" extract="day, month, hour, minute, second, subsecond">
    <text>\[\d{2}\s(\w{3})\s(\d{2}):(\d{2}):(\d{2}).(\d{3})\]</text>
</define>
<define name="_mydatetimeformat2" extract="year, month, day, hour, minute, second, subsecond">
     <text>(\d{4})(\d{2})(\d{2})@(\d{2}):(\d{2}):(\d{2})\,(\d{3})</text>
</define>
<timePatterns>
     <use name="_mydatetimeformat1"/>
     <use name="_mydatetimeformat2"/>
</timePatterns>
<datePatterns>
     <use name="_mydatetimeformat1"/>
     <use name="_mydatetimeformat2"/>
</datePatterns>
</datetime>

I think you may have a problem not having a year in the first date format.

0 Karma
Highlighted

Re: Why is my datetime.xml configuration not extracting two separate timestamp formats from a single log file?

Communicator

Hi in my log event and filename date is not present i want give a fix date to log so what is do ?

0 Karma