Splunk not logging the large json events properly

hvaithia · ‎05-21-2015

[my sourcetype]
INDEXED_EXTRACTIONS = json
TIME_FORMAT = %s%3N
TIME_PREFIX = jenkinsTimestamp
SHOULD_LINEMERGE = false
description = JavaScript Object Notation
category = Structured
disabled = false
pulldown_type = true
TRUNCATE=200000
MAX_EVENTS=200000

some of my events are 110,000+ characters long. I tried TRUNCATE=0 and MAX_EVENTS=0 to no effect. some of my events log from the middle and others do not log in the end. Some are logged just fine. Examples below,
----not logging in the start event----
5/22/15
2:08:39.000 AM

ckground-color: #F0F8FF;} </style><h4>Global Patterns</h4><table><tr ><td><b>Previous Job:</b></td><td><b>Started by <a href='/yhudson/job/searchproducts_gsm_upload_assembly/481/'>searchproducts_gsm_upload_assembly #481</a></b></td></tr><tr class=\"aliceblue\"><td><b>Previous Job:</b></td><td><b>Started by <a href='/yhudson/job/searchproducts_gsm_fr_map_component/499/'>searchproducts_gsm_fr_map_component #499</a></b></td></tr><tr ><td><b>Build host:</b></td><td><b>Built on <a href='/yhudson/computer/rhel6-gq1-slave-03.adcd.infra/'>rhel6-gq1-slave-03.adcd.infra</a></b></td></tr></table>"}], "id": "2015-05-22_00-34-30", "keepLog": false, "url": "http://thefactory.corp.xxx.com:9999/jenkins/view/Search_GSM/job/searchproducts_gsm_upload_deploy_int...", "culprits": [{"absoluteUrl": "http://thefactory.corp.xxx.com:9999/jenkins/user/by-tortuga", "fullName": "by-tortuga"}], "result": "SUCCESS", "executor": null, "duration": 98065, "fullDisplayName": "http://thefactory.corp.xxx.com:9999/jenkins/view/Search_GSM/job/searchproducts_gsm_upload_deploy_int", "estimatedDuration": 180575}

---------not logging in the end------
5/22/15
2:08:03.000 AM

{"building": true, "changeSet": {"items": [{"comment": "[GSM-1393] Support 'style' attribute in the GSM CardModule structure.\n", "paths": [{"editType": "edit", "file": "components/common/src/test/java/com/xxx/search/middleware/cosmos/template/CardModuleTest.java"}, {"editType": "edit", "file": "components/common/src/main/java/com/xxx/search/middleware/cosmos/template/CardModule.java"}], "author": {"absoluteUrl": "http://thefactory.corp.xxx.com:9999/jenkins/user/whsu", "fullName": "whsu"}, "timestamp": 1432159954000, "id": "86b19401707141f58cd3d561158fee38f69f648e", "commitId": "86b19401707141f58cd3d561158fee38f69f648e", "msg": "[GSM-1393] Support 'style' attribute in the GSM CardModule structure.", "date": "2015-05-20T22:12:34+0000 -0700", "affectedPaths": ["components/common/src/main/java/com/xxx/search/middleware/cosmos/template/CardModule.java", "components/common/src/test/java/com/xxx/search/middleware/cosmos/template/CardModuleTest.java"]}, {"comment": "[GSM-1393] Support 'url' attribute in the GSM CardModule structure.\n", "paths": [{"editType": "edit", "file": "components/common/src/test/java/com/xxx/search/middleware/cosmos/template/CardModuleTest.java"}, {"editType": "edit", "file": "components/common/src/main/java/com/xxx/search/middleware/cosmos/template/CardModule.java"}], "author": {"absoluteUrl": "http://thefactory.corp.xxx.com:9999/jenkins/user/whsu", "fullName": "whsu"}, "timestamp": 1432181780000, "id": "828dd4633fd5ecc565b1395c59586889fb2f57c9", "commitId": "828dd4633fd5ecc565b1395c59586889fb2f57c9", "msg": "[GSM-1393] Support 'url' attribute in the GSM CardModule structure.", "date": "2015-05-21T04:16:20+0000 -0700", "affectedPaths": ["components/common/src/main/java/com/xxx/search/middleware/cosmos/template/CardModule.java", "components/common/src/test/java/com/xxx/search/middleware/cosmos/template/CardModuleTest.java"]}, {"comment": "[GSM-1393] Fix url variable in CardModule.java.\n", "paths": [{"editType": "edit", "file": "components/common/src/main/java/com/xxx/search/middleware/cosmos/template/CardModule.java"}], "author": {"absoluteUrl": "http://thefactory.corp.xxx.com:9999/jenkins/user/whsu", "fullName": "whsu"}, "timestamp": 1432182356000, "id": "a0d1e6e7d2b65d5020cc2292d2bcc2699b8c5470", "commitId": "a0d1e6

Any help is greatly appreciated

esix_splunk · ‎05-21-2015

Im not sure indexed extractions are what you want here... Try adding the following and test this..

SHOULD_LINEMERGE = true
KV_MODE = json
BREAK_ONLY_BEFORE = ^\d\d\-\d\d\-\d\d
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 25

This is assuming your events starts with the date and time

hvaithia · ‎05-22-2015

Hi @esix_splunk, the timestamp is not the start of the event. My event is typically starts with a """ {"building": """ like this,

{"building": false, "changeSet": {"items": [{"comment": "Revert \"set pos field can be overrided by logGroup dynamically \"\n", "paths": [{"editType": "edit", "file": "src/datamodel/ComponentData.php" }], "author": {"absoluteUrl": "http://thefactory.corp.xxx.com:9999/jenkins/user/prajakta", "fullName": "prajakta"}, "timestamp": 1432177477000, "id": "5f428d3cde806ec296538023dae94ae694bca747", "com mitId": "5f428d3cde806ec296538023dae94ae694bca747", "msg": "Revert \"set pos field can be overrided by logGroup dynamically \"", "date": "2015-05-21T03:04:37+0000 -0700", "affectedPaths": ["src/datamo del/ComponentData.php"]}], "kind": "git"}, "builtOn": "rhel6-gq1-factory-slave-029.infra", "description": null, ...

And jenkinsTimestamp is the timestamp I want splunk to pick up. and hence the
TIME_FORMAT = %s%3N
TIME_PREFIX = jenkinsTimestamp
I tried KV_MODE=json earlier and somehow that doesn't seem to work for me. I read that KV_MODE is only for search-time extractions and not index-time. Do you have any suggestions for this?

hvaithia · ‎05-22-2015

Another weird thing that I am noticing is that, it works fine when I analyze the logs when uploading it and using the same sourcetype, but it appears to not work after I upload it and start the search. This is not the 1st time I am seeing this, it happened to me once before. Is the log analyzer in splunk 6.2 not working properly?

hvaithia · ‎05-22-2015

I just tested the same sourcetype with splunk 6.1 and it works perfectly! Safe to assume that something is wrong with Splunk 6.2 indexing