About vegerlandecs

vegerlandecs · ‎09-30-2020

@mitag Because your file start with square brackets Splunk is probably thinking it's a single event. If you force it to understand that the line-breaking includes a newline character followed by '{' it will ignore the '[]' as the first level. The way to do this is to create a sourcetype with the right configs - field extraction done in search-time so you need this config in both forwarders, indexers and search heads. [your_sourcetype] SHOULD_LINEMERGE = false LINE_BREAKER = ([\n\r]+)\{ TRUNCATE = 10000 TIME_PREFIX = \"optional_timestamp\"\s*:\s*\" TIME_FORMAT = %Y-%m-%d %H:%M MAX_TIMESTAMP_LOOKAHEAD = 16 KV_MODE=json Note you will need to adjust the options as per the different logs types/formats. See the line-breaking config test here: https://regex101.com/r/66ufQK/1

vegerlandecs · ‎09-30-2020

@vj5 SEDCMD is the kind of option that is not processed by universal forwarders. ref: https://wiki.splunk.com/Community:HowIndexingWorks, very last image shows as it's part of the typing processor - which only enterprise installations (HF and IDX) will have. Also, Splunk uses PCRE notation, so \u is not supported. From the snippets it isn't very clear to me what are you trying to SED, but consider this replacement regex as a start: SEDCMD-1_unjsonify = s/log:\s+?{.*?{(.*?)}/\1/g

vegerlandecs · ‎09-25-2019

That's great - do you happen to have reference for this? I need to prove with oficial reference that this is how it works 🙂

vegerlandecs · ‎01-22-2019

This message is telling you that queue is full. If you increase the parsing queue it will take more memory but will handle a bit better the events before sending them through. That alone, however, may not fix the problem, since the reason why it's it's likely to be the throughput (network) in limits.conf, which is 256 kbps by default. input flow > output flow = queueing -> delay In a more constant flow case, it's easy to predict queueing, problem is calculating quality of service when it has surges (for context: https://en.wikipedia.org/wiki/Queueing_theory) How to increase thruput: https://docs.splunk.com/Documentation/Splunk/7.2.3/Admin/Limitsconf#.5Bthruput.5D parsing queue can be changed to something like this (server.conf): [queue=parsingQueue] maxSize = 2MB Edit: forgot to add that if your indexers are rejecting inputs, you need to increase capacity, either by changing to high speed disks or adding new indexers depending on where the bottleneck is - I suggest comparing queueing graphs to disk/cpu utilisation on the indexers. There may be ways to workaround it before you add capacity, like playing with queues and etc. However, I'm afraid that it will be best if you engage support directly in this case.

vegerlandecs · ‎10-27-2017

I finally found what was wrong. The output was being generated like this: echo '[' > $OUTPUT_FILENAME echo ' ' >> $OUTPUT_FILENAME echo ' "datetime":"'$(date --rfc-3339=seconds)'",' >> $OUTPUT_FILENAME echo ' "user": "'$username'",' >> $OUTPUT_FILENAME echo ' "environment_category": "'$environment_category'",' >> $OUTPUT_FILENAME echo ' "release_type": "'$release_type'",' >> $OUTPUT_FILENAME echo ' "environment_frontend": "'$environment_frontend'",' >> $OUTPUT_FILENAME echo ' "environment_backend": "'$environment_backend'",' >> $OUTPUT_FILENAME echo ' "release_version": "'$release_version'",' >> $OUTPUT_FILENAME echo ' "branch_version": "'$branch_version'",' >> $OUTPUT_FILENAME echo ' "status": "'$status'",' >> $OUTPUT_FILENAME echo ' "next_planned_release_version": "'$next_planned_release_version'",' >> $OUTPUT_FILENAME echo ' "next_planned_branch_version": "'$next_planned_branch_version'",' >> $OUTPUT_FILENAME echo ' "comment": "'$comment'"' >> $OUTPUT_FILENAME echo ' ' >> $OUTPUT_FILENAME echo ']' >> $OUTPUT_FILENAME Replaced with echo ' "datetime":"'$(date --rfc-3339=seconds)'", "user":"'$username'", "environment_category":"'$environment_category'", "release_type":"'$release_type'", "environment_frontend": "'$environment_frontend'", "environment_backend": "'$environment_backend'", "release_version": "'$release_version'", "branch_version": "'$branch_version'", "status": "'$status'", "next_planned_release_version": "'$next_planned_release_version'", "next_planned_branch_version": "'$next_planned_branch_version'", "comment": "'$comment'"' >> $OUTPUT_FILENAME Not looking good for humans now, but apparently Splunk didn't like the line breaking (possibly didn't care about square brackets ) Now, why json files were indexed fine after restarting Splunk but not the following files during runtime, the question remains.

vegerlandecs · ‎10-25-2017

Hi, I'm getting errors with parsing of json files in the universal forwarder. I'm generating json outputs - a new file is generated every time a run a routine. Output has the below: [ { "datetime":"2017-10-25 14:33:16+01:00", "user":"", "category":"ST", "type":"ABC", "frontend":"3.0", "backend":"", "r_version":"", "b_version":"", "status":"R", "next_planned_r_version":"", "next_planned_b_version":"", "comment":"" } ] Splunk forwarder gives me the following log entries in splunkd.log: 10-25-2017 14:33:16.273 +0100 ERROR JsonLineBreaker - JSON StreamId:16742053991537090041 had parsing error:Unexpected character: ':' - data_source="/root/status-update/environment_health_status_50.json", data_host="hostxyz", data_sourcetype="_json" The line above repeats about the same number of lines with ":" in the output. Then lines below: 10-25-2017 14:33:16.273 +0100 ERROR JsonLineBreaker - JSON StreamId:16742053991537090041 had parsing error:Unexpected character: '}' - data_source="/root/status-update/environment_health_status_50.json", data_host="hostxyz", data_sourcetype="_json" 10-25-2017 14:33:16.273 +0100 ERROR JsonLineBreaker - JSON StreamId:16742053991537090041 had parsing error:Unexpected character: ']' - data_source="/root/status-update/environment_health_status_50.json", data_host="hostxyz", data_sourcetype="_json" I've tried universal forwarders versions 7.0 and 6.5.3. I've been trying to isolated the root cause but had no luck with that - even without changing anything. Sometimes it goes fine, but mostly it doesn't. If I stop splunk, erase fishbucket and start it again, it will ingest all files just fine. However, when I run my test afterwards that is creating new files, it will fail. (or not, as I explained). monitor in inputs.conf: [monitor:///root/status-update/environment_health_status_*.json] index=dev_test sourcetype=_json _json stanza on the forwarder by using btool: PS: I haven't made any config in props.conf, only inputs. [_json] ANNOTATE_PUNCT = True AUTO_KV_JSON = true BREAK_ONLY_BEFORE = BREAK_ONLY_BEFORE_DATE = True CHARSET = UTF-8 DATETIME_CONFIG = /etc/datetime.xml HEADER_MODE = INDEXED_EXTRACTIONS = json KV_MODE = none LEARN_MODEL = true LEARN_SOURCETYPE = true LINE_BREAKER_LOOKBEHIND = 100 MATCH_LIMIT = 100000 MAX_DAYS_AGO = 2000 MAX_DAYS_HENCE = 2 MAX_DIFF_SECS_AGO = 3600 MAX_DIFF_SECS_HENCE = 604800 MAX_EVENTS = 256 MAX_TIMESTAMP_LOOKAHEAD = 128 MUST_BREAK_AFTER = MUST_NOT_BREAK_AFTER = MUST_NOT_BREAK_BEFORE = SEGMENTATION = indexing SEGMENTATION-all = full SEGMENTATION-inner = inner SEGMENTATION-outer = outer SEGMENTATION-raw = none SEGMENTATION-standard = standard SHOULD_LINEMERGE = True TRANSFORMS = TRUNCATE = 10000 category = Structured description = JavaScript Object Notation format. For more information, visit http://json.org/ detect_trailing_nulls = false maxDist = 100 priority = pulldown_type = true sourcetype =

Posts	6
Solutions	1
Karma Given	5
Karma Received	2
Member Since	‎06-09-2017

Online Status	Offline
Date Last Visited	‎09-30-2020 05:03 AM

JSON parsing error in the universal forwarder

Re: ingesting a large list of JSONs as separate ev...

Re: JSON parsing error in the universal forwarder

Re: Splunk Cloud HEC (http event collector) - Does...

Re: Could not send data to output queue (parsingQu...

Re: JSON parsing error in the universal forwarder

JSON parsing error in the universal forwarder