Getting Data In

Script Input - Timestamp field extraction from multi key value pair data-set (props.conf, inputs.conf)

rupesh_kumar
Engager

Hello Team,

I am using a python script input method to extract data (meta-data and some raw data) from binary and txt files. My script is working fine, I can search extracted data from Splunk's search app.

Issue:- This script has 'ctime' field, I am trying to match this field to splunk timestamp or _time field but the matching is not working. I am getting timestamp field at Splunk end with NONE value but this suppose to be either ctime field value.

Following is the detail of configurations and data-set:-

1) props.conf:-

[waterwise]
SHOULD_LINEMERGE = false
CHARSET = UTF-8
LINE_BREAKER = ([\r\n])
TIMESTAMP_FIELDS = ctime
TIME_FORMAT = %m-%d-%Y %H:%M:%S %z

2) inputs.conf:-

[script://$SPLUNK_HOME\bin\scripts\waterwise-data.py]
disabled = false
interval = 300
source = waterwise
sourcetype = waterwise
index = main

3) Script out-put data-set (multi key value pair):-

sensor_data=361, sensor_data=361, sensor_data=360,location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\BTRY\STN_11_2011001232949_BTRY_600_15.txt, size=164, atime=06-04-2014 04:51:32, mtime=01-01-2011 15:40:14, ctime=01-01-2011 15:40:14 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=BTRY

sensor_data=361, sensor_data=361, sensor_data=361, location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\BTRY\STN_11_2011001234012_BTRY_600_15.txt, size=164, atime=06-04-2014 04:51:32, mtime=01-01-2011 15:50:42, ctime=01-01-2011 15:50:42 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=BTRY

sensor_data=359, sensor_data=359, location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\BTRY\STN_11_2011001235034_BTRY_600_15.txt, size=164, atime=06-04-2014 04:51:32, mtime=01-01-2011 16:01:01, ctime=01-01-2011 16:01:01 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=BTRY

location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\ORP\STN_11_2011001000056_ORP_SGT.raw, size=52, atime=06-03-2014 07:31:55, mtime=12-31-2010 16:01:32, ctime=12-31-2010 16:01:32 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=ORP

location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\ORP\STN_11_2011001000156_ORP_SGT.raw, size=52, atime=06-03-2014 07:31:55, mtime=12-31-2010 16:02:32, ctime=12-31-2010 16:02:32 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=ORP

Thanks in advance. I look forward to hearing from you team.

--

Rupesh

0 Karma

alterdego
Path Finder

Based on the props.conf spec it doesn't look like the TIMESTAMP_FIELDS is the correct declaration for you to use based on the sample output.
Further, it looks like the time stamp (crime) in the output is in the middle of the event. Again by the spec, Splunk will only examine the first 150 characters by default.
I would try adding TIME_PREFIX and MAX_TIMESTAMP_LOOKAHEAD values to props.conf.

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

Hope that helps

0 Karma

rupesh_kumar
Engager

Thank you Alterdego for your reply. There was problem with time zone.

0 Karma

dmlee
Communicator

I think the problem is time zone, "China Standard Time" is not the correct format of TZ , you can set TZ=CN or TZ=Asia/Shanghai in props.conf

for example :

TIME_FORMAT=%m-%d-%Y %H:%M:%S 
TIME_PREFIX=ctime= 
TZ=CN

rupesh_kumar
Engager

Thank you Dmlee, yes 'TZ=CN' works for me.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...