Getting Data In

Script Input - Timestamp field extraction from multi key value pair data-set (props.conf, inputs.conf)

Engager

Hello Team,

I am using a python script input method to extract data (meta-data and some raw data) from binary and txt files. My script is working fine, I can search extracted data from Splunk's search app.

Issue:- This script has 'ctime' field, I am trying to match this field to splunk timestamp or _time field but the matching is not working. I am getting timestamp field at Splunk end with NONE value but this suppose to be either ctime field value.

Following is the detail of configurations and data-set:-

1) props.conf:-

[waterwise]
SHOULD_LINEMERGE = false
CHARSET = UTF-8
LINE_BREAKER = ([\r\n])
TIMESTAMP_FIELDS = ctime
TIME_FORMAT = %m-%d-%Y %H:%M:%S %z

2) inputs.conf:-

[script://$SPLUNK_HOME\bin\scripts\waterwise-data.py]
disabled = false
interval = 300
source = waterwise
sourcetype = waterwise
index = main

3) Script out-put data-set (multi key value pair):-

sensor_data=361, sensor_data=361, sensor_data=360,location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\BTRY\STN_11_2011001232949_BTRY_600_15.txt, size=164, atime=06-04-2014 04:51:32, mtime=01-01-2011 15:40:14, ctime=01-01-2011 15:40:14 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=BTRY

sensor_data=361, sensor_data=361, sensor_data=361, location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\BTRY\STN_11_2011001234012_BTRY_600_15.txt, size=164, atime=06-04-2014 04:51:32, mtime=01-01-2011 15:50:42, ctime=01-01-2011 15:50:42 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=BTRY

sensor_data=359, sensor_data=359, location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\BTRY\STN_11_2011001235034_BTRY_600_15.txt, size=164, atime=06-04-2014 04:51:32, mtime=01-01-2011 16:01:01, ctime=01-01-2011 16:01:01 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=BTRY

location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\ORP\STN_11_2011001000056_ORP_SGT.raw, size=52, atime=06-03-2014 07:31:55, mtime=12-31-2010 16:01:32, ctime=12-31-2010 16:01:32 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=ORP

location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\ORP\STN_11_2011001000156_ORP_SGT.raw, size=52, atime=06-03-2014 07:31:55, mtime=12-31-2010 16:02:32, ctime=12-31-2010 16:02:32 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=ORP

Thanks in advance. I look forward to hearing from you team.

--

Rupesh

0 Karma

Path Finder

Based on the props.conf spec it doesn't look like the TIMESTAMPFIELDS is the correct declaration for you to use based on the sample output.
Further, it looks like the time stamp (crime) in the output is in the middle of the event. Again by the spec, Splunk will only examine the first 150 characters by default.
I would try adding TIME
PREFIX and MAXTIMESTAMPLOOKAHEAD values to props.conf.

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

Hope that helps

0 Karma

Engager

Thank you Alterdego for your reply. There was problem with time zone.

0 Karma

Communicator

I think the problem is time zone, "China Standard Time" is not the correct format of TZ , you can set TZ=CN or TZ=Asia/Shanghai in props.conf

for example :

TIME_FORMAT=%m-%d-%Y %H:%M:%S 
TIME_PREFIX=ctime= 
TZ=CN

Engager

Thank you Dmlee, yes 'TZ=CN' works for me.

0 Karma