source type issues

[cnhn]

Hello All,

we have started working with splunk to deal with a pile of date. for that we have created a custom source type and put it in the props.conf file. it looks like this:

[mk_csv]

CHECK_FOR_HEADER = true

KV_MODE = none

NO_BINARY_CHECK = 1

SHOULD_LINEMERGE = False

TIME_FORMAT = %Y-%jT%H:%M:%S.%3N

pulldown_type = 1

TZ = UTC

that is in ./splunk/etc/system/local/props.conf and yes we restarted the server

So far so good. we added the source type to the data inputs. we built a new index mk_mission.

Now from the search window if I run a sourcetype="mk_csv" nothing shows up. however I do find that there is now a mk_csv-3 with 37 events in it, it even correctly displays the julian dates (the %j in the time_format)

last problem. I configured the data inputs to use julian time as seen above however everything indexed is showing up with the wrong dates, always a bit early

examples:

search index for sourcetype="mk_csv-3"

record returned:

4/13/12 6:32:52.138 PM with the time stamp from the entry as: 2012-104T18:32:52.138

that looks like the sourcetype in the props.conf is working correctly

search for:

index="mk_mission_gra_eng"

get:

4/6/12 9:11:09.467 PM from 2012-088T21:11:09.467

4/6/12 5:42:38.170 PM from 2012-081T17:42:38.170

4/6/12 4:05:10.097 PM from 2012-102T16:05:10.097

4/5/12 11:17:30.163 PM from 2012-101T23:17:30.163

as near as I can tell this is not what I would expect since we set the source type as the data input level.

[gkanapathy]

I generally would advise not to use CHECK_FOR_HEADER. As far as the time stamp issue, that's tricky to debug, but my guess is Splunk is for whatever reason not reading your Julian days and getting the date from the last-mod time on the files.

[cnhn]

how do you eliminate the static column headers from the csv files without CHECK_FOR_HEADERS?