Getting Data In

Splunk not logging the large json events properly

hvaithia
Path Finder

[my sourcetype]
INDEXED_EXTRACTIONS = json
TIME_FORMAT = %s%3N
TIME_PREFIX = jenkinsTimestamp
SHOULD_LINEMERGE = false
description = JavaScript Object Notation
category = Structured
disabled = false
pulldown_type = true
TRUNCATE=200000
MAX_EVENTS=200000

some of my events are 110,000+ characters long. I tried TRUNCATE=0 and MAX_EVENTS=0 to no effect. some of my events log from the middle and others do not log in the end. Some are logged just fine. Examples below,
----not logging in the start event----
5/22/15
2:08:39.000 AM

ckground-color: #F0F8FF;} </style><h4>Global Patterns</h4><table><tr ><td><b>Previous Job:</b></td><td><b>Started by <a href='/yhudson/job/searchproducts_gsm_upload_assembly/481/'>searchproducts_gsm_upload_assembly #481</a></b></td></tr><tr class=\"aliceblue\"><td><b>Previous Job:</b></td><td><b>Started by <a href='/yhudson/job/searchproducts_gsm_fr_map_component/499/'>searchproducts_gsm_fr_map_component #499</a></b></td></tr><tr ><td><b>Build host:</b></td><td><b>Built on <a href='/yhudson/computer/rhel6-gq1-slave-03.adcd.infra/'>rhel6-gq1-slave-03.adcd.infra</a></b></td></tr></table>"}], "id": "2015-05-22_00-34-30", "keepLog": false, "url": "http://thefactory.corp.xxx.com:9999/jenkins/view/Search_GSM/job/searchproducts_gsm_upload_deploy_int...", "culprits": [{"absoluteUrl": "http://thefactory.corp.xxx.com:9999/jenkins/user/by-tortuga", "fullName": "by-tortuga"}], "result": "SUCCESS", "executor": null, "duration": 98065, "fullDisplayName": "http://thefactory.corp.xxx.com:9999/jenkins/view/Search_GSM/job/searchproducts_gsm_upload_deploy_int", "estimatedDuration": 180575}

---------not logging in the end------
5/22/15
2:08:03.000 AM

{"building": true, "changeSet": {"items": [{"comment": "[GSM-1393] Support 'style' attribute in the GSM CardModule structure.\n", "paths": [{"editType": "edit", "file": "components/common/src/test/java/com/xxx/search/middleware/cosmos/template/CardModuleTest.java"}, {"editType": "edit", "file": "components/common/src/main/java/com/xxx/search/middleware/cosmos/template/CardModule.java"}], "author": {"absoluteUrl": "http://thefactory.corp.xxx.com:9999/jenkins/user/whsu", "fullName": "whsu"}, "timestamp": 1432159954000, "id": "86b19401707141f58cd3d561158fee38f69f648e", "commitId": "86b19401707141f58cd3d561158fee38f69f648e", "msg": "[GSM-1393] Support 'style' attribute in the GSM CardModule structure.", "date": "2015-05-20T22:12:34+0000 -0700", "affectedPaths": ["components/common/src/main/java/com/xxx/search/middleware/cosmos/template/CardModule.java", "components/common/src/test/java/com/xxx/search/middleware/cosmos/template/CardModuleTest.java"]}, {"comment": "[GSM-1393] Support 'url' attribute in the GSM CardModule structure.\n", "paths": [{"editType": "edit", "file": "components/common/src/test/java/com/xxx/search/middleware/cosmos/template/CardModuleTest.java"}, {"editType": "edit", "file": "components/common/src/main/java/com/xxx/search/middleware/cosmos/template/CardModule.java"}], "author": {"absoluteUrl": "http://thefactory.corp.xxx.com:9999/jenkins/user/whsu", "fullName": "whsu"}, "timestamp": 1432181780000, "id": "828dd4633fd5ecc565b1395c59586889fb2f57c9", "commitId": "828dd4633fd5ecc565b1395c59586889fb2f57c9", "msg": "[GSM-1393] Support 'url' attribute in the GSM CardModule structure.", "date": "2015-05-21T04:16:20+0000 -0700", "affectedPaths": ["components/common/src/main/java/com/xxx/search/middleware/cosmos/template/CardModule.java", "components/common/src/test/java/com/xxx/search/middleware/cosmos/template/CardModuleTest.java"]}, {"comment": "[GSM-1393] Fix url variable in CardModule.java.\n", "paths": [{"editType": "edit", "file": "components/common/src/main/java/com/xxx/search/middleware/cosmos/template/CardModule.java"}], "author": {"absoluteUrl": "http://thefactory.corp.xxx.com:9999/jenkins/user/whsu", "fullName": "whsu"}, "timestamp": 1432182356000, "id": "a0d1e6e7d2b65d5020cc2292d2bcc2699b8c5470", "commitId": "a0d1e6

Any help is greatly appreciated

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Im not sure indexed extractions are what you want here... Try adding the following and test this..

SHOULD_LINEMERGE = true
KV_MODE = json
BREAK_ONLY_BEFORE = ^\d\d\-\d\d\-\d\d
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 25

This is assuming your events starts with the date and time

0 Karma

hvaithia
Path Finder

Hi @esix_splunk, the timestamp is not the start of the event. My event is typically starts with a """ {"building": """ like this,

{"building": false, "changeSet": {"items": [{"comment": "Revert \"set pos field can be overrided by logGroup dynamically \"\n", "paths": [{"editType": "edit", "file": "src/datamodel/ComponentData.php" }], "author": {"absoluteUrl": "http://thefactory.corp.xxx.com:9999/jenkins/user/prajakta", "fullName": "prajakta"}, "timestamp": 1432177477000, "id": "5f428d3cde806ec296538023dae94ae694bca747", "com mitId": "5f428d3cde806ec296538023dae94ae694bca747", "msg": "Revert \"set pos field can be overrided by logGroup dynamically \"", "date": "2015-05-21T03:04:37+0000 -0700", "affectedPaths": ["src/datamo del/ComponentData.php"]}], "kind": "git"}, "builtOn": "rhel6-gq1-factory-slave-029.infra", "description": null, ...


And jenkinsTimestamp is the timestamp I want splunk to pick up. and hence the
TIME_FORMAT = %s%3N
TIME_PREFIX = jenkinsTimestamp
I tried KV_MODE=json earlier and somehow that doesn't seem to work for me. I read that KV_MODE is only for search-time extractions and not index-time. Do you have any suggestions for this?

0 Karma

hvaithia
Path Finder

Another weird thing that I am noticing is that, it works fine when I analyze the logs when uploading it and using the same sourcetype, but it appears to not work after I upload it and start the search. This is not the 1st time I am seeing this, it happened to me once before. Is the log analyzer in splunk 6.2 not working properly?

0 Karma

hvaithia
Path Finder

I just tested the same sourcetype with splunk 6.1 and it works perfectly! Safe to assume that something is wrong with Splunk 6.2 indexing

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...