Getting Data In

Script Input - Timestamp field extraction from multi key value pair data-set (props.conf, inputs.conf)

rupesh_kumar
Engager

Hello Team,

I am using a python script input method to extract data (meta-data and some raw data) from binary and txt files. My script is working fine, I can search extracted data from Splunk's search app.

Issue:- This script has 'ctime' field, I am trying to match this field to splunk timestamp or _time field but the matching is not working. I am getting timestamp field at Splunk end with NONE value but this suppose to be either ctime field value.

Following is the detail of configurations and data-set:-

1) props.conf:-

[waterwise]
SHOULD_LINEMERGE = false
CHARSET = UTF-8
LINE_BREAKER = ([\r\n])
TIMESTAMP_FIELDS = ctime
TIME_FORMAT = %m-%d-%Y %H:%M:%S %z

2) inputs.conf:-

[script://$SPLUNK_HOME\bin\scripts\waterwise-data.py]
disabled = false
interval = 300
source = waterwise
sourcetype = waterwise
index = main

3) Script out-put data-set (multi key value pair):-

sensor_data=361, sensor_data=361, sensor_data=360,location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\BTRY\STN_11_2011001232949_BTRY_600_15.txt, size=164, atime=06-04-2014 04:51:32, mtime=01-01-2011 15:40:14, ctime=01-01-2011 15:40:14 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=BTRY

sensor_data=361, sensor_data=361, sensor_data=361, location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\BTRY\STN_11_2011001234012_BTRY_600_15.txt, size=164, atime=06-04-2014 04:51:32, mtime=01-01-2011 15:50:42, ctime=01-01-2011 15:50:42 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=BTRY

sensor_data=359, sensor_data=359, location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\BTRY\STN_11_2011001235034_BTRY_600_15.txt, size=164, atime=06-04-2014 04:51:32, mtime=01-01-2011 16:01:01, ctime=01-01-2011 16:01:01 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=BTRY

location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\ORP\STN_11_2011001000056_ORP_SGT.raw, size=52, atime=06-03-2014 07:31:55, mtime=12-31-2010 16:01:32, ctime=12-31-2010 16:01:32 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=ORP

location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\ORP\STN_11_2011001000156_ORP_SGT.raw, size=52, atime=06-03-2014 07:31:55, mtime=12-31-2010 16:02:32, ctime=12-31-2010 16:02:32 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=ORP

Thanks in advance. I look forward to hearing from you team.

--

Rupesh

0 Karma

alterdego
Path Finder

Based on the props.conf spec it doesn't look like the TIMESTAMP_FIELDS is the correct declaration for you to use based on the sample output.
Further, it looks like the time stamp (crime) in the output is in the middle of the event. Again by the spec, Splunk will only examine the first 150 characters by default.
I would try adding TIME_PREFIX and MAX_TIMESTAMP_LOOKAHEAD values to props.conf.

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

Hope that helps

0 Karma

rupesh_kumar
Engager

Thank you Alterdego for your reply. There was problem with time zone.

0 Karma

dmlee
Communicator

I think the problem is time zone, "China Standard Time" is not the correct format of TZ , you can set TZ=CN or TZ=Asia/Shanghai in props.conf

for example :

TIME_FORMAT=%m-%d-%Y %H:%M:%S 
TIME_PREFIX=ctime= 
TZ=CN

rupesh_kumar
Engager

Thank you Dmlee, yes 'TZ=CN' works for me.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Accelerate Splunk App Development with AI

  Learn how to automate the repetitive parts of your build cycle.   Watch On-Demand   Replay the Accelerate ...

Splunk Agent Observability: Measuring AI Value with Splunk Tokenomics on AWS

The AI landscape is moving faster than ever, and our latest 2026 State of Observability report provides the ...

Enterprise Security Watch Party: What’s New in 8.8 - Recap

Splunk Community HQ hosted a Splunk Enterprise Security Watch Party: What’s New In ES 8.8 last Thursday. I had ...