[my sourcetype]
INDEXED_EXTRACTIONS = json
TIME_FORMAT = %s%3N
TIME_PREFIX = jenkinsTimestamp
SHOULD_LINEMERGE = false
description = JavaScript Object Notation
category = Structured
disabled = false
pulldown_type = true
TRUNCATE=200000
MAX_EVENTS=200000
some of my events are 110,000+ characters long. I tried TRUNCATE=0 and MAX_EVENTS=0 to no effect. some of my events log from the middle and others do not log in the end. Some are logged just fine. Examples below,
----not logging in the start event----
5/22/15
2:08:39.000 AM
---------not logging in the end------
5/22/15
2:08:03.000 AM
Any help is greatly appreciated
Im not sure indexed extractions are what you want here... Try adding the following and test this..
SHOULD_LINEMERGE = true
KV_MODE = json
BREAK_ONLY_BEFORE = ^\d\d\-\d\d\-\d\d
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 25
This is assuming your events starts with the date and time
{"building": false, "changeSet": {"items": [{"comment": "Revert \"set pos field can be overrided by logGroup dynamically \"\n", "paths": [{"editType": "edit", "file": "src/datamodel/ComponentData.php" }], "author": {"absoluteUrl": "http://thefactory.corp.xxx.com:9999/jenkins/user/prajakta", "fullName": "prajakta"}, "timestamp": 1432177477000, "id": "5f428d3cde806ec296538023dae94ae694bca747", "com mitId": "5f428d3cde806ec296538023dae94ae694bca747", "msg": "Revert \"set pos field can be overrided by logGroup dynamically \"", "date": "2015-05-21T03:04:37+0000 -0700", "affectedPaths": ["src/datamo del/ComponentData.php"]}], "kind": "git"}, "builtOn": "rhel6-gq1-factory-slave-029.infra", "description": null, ...
And jenkinsTimestamp is the timestamp I want splunk to pick up. and hence the
TIME_FORMAT = %s%3N
TIME_PREFIX = jenkinsTimestamp
I tried KV_MODE=json earlier and somehow that doesn't seem to work for me. I read that KV_MODE is only for search-time extractions and not index-time. Do you have any suggestions for this?
Another weird thing that I am noticing is that, it works fine when I analyze the logs when uploading it and using the same sourcetype, but it appears to not work after I upload it and start the search. This is not the 1st time I am seeing this, it happened to me once before. Is the log analyzer in splunk 6.2 not working properly?
I just tested the same sourcetype with splunk 6.1 and it works perfectly! Safe to assume that something is wrong with Splunk 6.2 indexing