topic Re: TIME_PREFIX in Splunk Search

TIME_PREFIX

surfjose — Mon, 17 Feb 2014 13:17:32 GMT

Hi
I have a log-file with diffrent time formats.
Is it possible to extract this diffrent timestamps with TIME_PREFIX in propts.conf?
Loglines:

<Feb 17, 2014 8:32:42 AM CET> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING> 
[EL Finest]: 2014-02-17 08:32:45.961--ServerSession(998916174)--Thread(Thread[[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads])--property=eclipselink.logging.level.sql; value=FINEST; translated value=FINEST
[EL Finer]: 2014-02-17 08:32:45.973--ServerSession(998916174)--Thread(Thread[[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads])--Searching for default mapping file in file:/opt/user_projects/domains/tdapp12/servers/tsapp12maportal01/tmp/_WL_user/clientportal/a7daku/APP-INF/lib/sympany.clientportal.jpa.jar
[EL Config]: 2014-02-17 08:32:45.982--ServerSession(998916174)--Thread(Thread[[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads])--The access type for the persistent class [class ch.sympany.clientportal.model.UserPreferences] is set to [FIELD].
2014-02-17 08:32:54,248 [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] DEBUG ch.sympany.clientportal.servlet.StartupServletContextListener  - StartupServletContextListener initialised.

Re: TIME_PREFIX

kristian_kolb — Mon, 17 Feb 2014 14:49:51 GMT

Well TIME_PREFIX is used to ensure that you find the start of the timestamp in the events, whereas TIME_FORMAT will allow you to specify how the timstamp is formatted.

In your case, you seem to be having three different time formats, but the first seems to be of less interest (service started?). The second two differ very little, and if you can make do without the millisecond precision, you could probably have something like this config (in props.conf);

[your_sourcetype]
MAX_TIMESTAMP_LOOKAHEAD = 50
TIME_FORMAT = %Y-%m-%d %H:%M:%S

Unfortunately, TIME_FORMAT does not take a regex as parameter value, otherwise it would have been easy to handle the dot/comma difference.

UPDATE:

Perhaps you could also add the TIME_PREFIX as such;

TIME_PREFIX = (:\s|<|^)

The TIME_PREFIX will match the first instance it finds of the pattern, and will start looking for a timestamp immediately after it (which must be formatted according to the TIME_FORMAT)

Not sure it will have any success on the first message.

Re: TIME_PREFIX

surfjose — Mon, 17 Feb 2014 15:18:34 GMT

It does not work.
It taks findes only the timestamp from this log entry:
2014-02-17 08:32:54,248 [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] DEBUG ch.sympany.clientportal.servlet.StartupServletContextListener - StartupServletContextListener initialised.

The best filter i have create is this one:

[my_sourcetype]
MAX_TIMESTAMP_LOOKAHEAD = 50
TIME_PREFIX = \[EL Finest]:|\[EL Finer]:|\[EL Config]: |\<

But the result is not 100%.

Any ideas in this way?