About King_Of_Shawn

King_Of_Shawn · ‎01-23-2023

I agree about your first 1, but for 2, I want to confirm: "Transfer encoding is a mechanism used to encode data in transit", I think you mean that for the CherryPy of Splunk, if I transmit a piece of compressed data, what it sees must be the size of the decompressed data, right? because for me (the client of HEC), the size of my request should be 49039690 bytes, and according to the relevant regulations of RFC, if gzip is used, Content-Length should be the payload length encoded by gzip, about 46.77 MiB, not the limit of 800MB in Splunk. So Splunk limits the actual data size (after decompression)? Instead of the requested Content-Length?

King_Of_Shawn · ‎01-23-2023

Hi @PickleRick thank you, please refer my reply above, in fact, the gzip not one local file, that's one remote resource from another request's response, and i'm ensure the gzip stream is good, the data is line json after decompression, looks like: {...} {...} {...} ...... so i think HEC endpoint "/services/collector/raw" should be able to parse it directly and correctly.

King_Of_Shawn · ‎01-23-2023

Hi @jamie00171 thanks for reply, i will show you the demo: import requests import gzip import ndjson import functools from io import BytesIO, SEEK_SET, SEEK_END # About 50MB after gzip compression, and about 900MB after decompression url = "https://{{fqdn}}/packages/{{uuid}}" payload = {} headers = { 'Authorization': 'Bearer {{token}}', "Accept-Encoding": "gzip", 'Connection': 'keep-alive' } def stream_transfer(): with requests.request("GET", url, headers=headers, data=payload, stream=True) as r: print(r.headers) r1 = requests.post("http://10.64.21.32:8088/services/collector/raw", headers={ "Authorization": "Splunk f98f1101-e880-49e9-88ac-d08e8ce0c1e5", "Content-Encoding": "gzip", "Transfer-Encoding": "chunked", "Connection": "keep-alive", # "Content-Length": r.headers["Content-Length"] }, data=r.raw.stream(1024*1024, decode_content=False)) if __name__ == "__main__": stream_transfer() and i also show you the headers in response, the content length is in it, {'Date': 'Tue, 24 Jan 2023 01:46:16 GMT', 'Content-Type': 'text/plain; charset=UTF-8', 'Content-Length': '49039690', 'Connection': 'keep-alive', 'Content-Encoding': 'gzip', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'} then it will raise this like i said above: <!doctype html><html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><title>413 Content-Length of 838889996 too large (maximum is 838860800)</title></head><body><h1>Content-Length of 838889996 too large (maximum is 838860800)</h1><p>The request your client sent was too large.</p></body></html>

King_Of_Shawn · ‎01-23-2023

Hi guys, Happy New Year, i do some code testing with the Splunk HEC, now i need to transfer some large volum data with gzip compressed. 1. first i find one limit in $SPLUNK_HOME$/etc/system/default/limits.conf [http_input] max_content_length = <integer> * The maximum length, in bytes, of HTTP request content that is accepted by the HTTP Event Collector server. * Default: 838860800 (~ 800 MB) but i It is found that this value seems to calculate the size after decompression, because i have one test file about 50MiB, it's far less than 800MB, but when i sending request, Splunk raise the: <!doctype html><html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><title>413 Content-Length of 838889996 too large (maximum is 838860800)</title></head><body><h1>Content-Length of 838889996 too large (maximum is 838860800)</h1><p>The request your client sent was too large.</p></body></html> 2. the 2nd limit i find at $SPLUNK_HOME$/etc/apps/splunk_httpinput/local/inputs.conf [http] maxEventSize = <positive integer>[KB|MB|GB] * The maximum size of a single HEC (HTTP Event Collector) event. * HEC disregards and triggers a parsing error for events whose size is greater than 'maxEventSize'. * Default: 5MB i think this limit is set at only one event size? if i send batch events in one request by "/services/collector", so this limit will apply to every event in the batch events, right? Are there any relevant experts to help confirm this behavior? if need more details feel free to let me know, Many thanks!

King_Of_Shawn · ‎10-26-2021

Hi, friend. In fact, I have encountered a new question, If I need to upgrade the APP later, do I need to resubmit the new version for Splunk Cloud review?

King_Of_Shawn · ‎10-12-2021

Hi, If I pass the Splunk Cloud review in US site, do I need to send review request again on other sites such as AU site?

King_Of_Shawn · ‎10-10-2021

Is that the only way? And Is trial license of Splunk Cloud account OK? Thanks for your reply.

King_Of_Shawn · ‎10-09-2021

I'm a Splunk Add-on/App developer, and the Splunk-App I developed has passed the app inspect and runs well on Splunk Enterprise, but how can I get Splunk Cloud certification flag? The flag like this picture: Only customers can send Splunk Cloud App request? Really? I think It's too inconvenient, as a developer, I can't submit my own review request...

King_Of_Shawn · ‎09-09-2021

Thank you for your reply. I have one more question. If i restart the Splunk server in the middle of the running of the modular input script, is there a default checkpoint mechanism for Splunk to restore the last unfinished trigger when the Splunk server start again? or just start a new processing at the next trigger time?

King_Of_Shawn · ‎09-09-2021

I have a problem similar to this: Scripted Input timeout . In modular input, i use python script to collect data, and in most cases, the single collection takes 10 minutes. But my interval is set to 5 minutes. Splunk will run the first one and wait until it finishes to start the next one? Right? And the missed scheduling will be directly ignored?

Posts	10
Solutions	0
Karma Given	3
Karma Received	0
Member Since	‎09-09-2021

Online Status	Offline
Date Last Visited	‎02-06-2023 06:28 AM

Splunk HTTP Event Collector's: What is the definit...

How can App's developer send Splunk Cloud App requ...

Input script processing time greater than trigger ...

Re: Splunk HTTP Event Collector's definition of si...

Re: Splunk HTTP Event Collector's definition of si...

Re: Splunk HTTP Event Collector's definition of si...

Splunk HTTP Event Collector's: What is the definit...

Re: How can App's developer send Splunk Cloud App ...

Re: How can App's developer send Splunk Cloud App ...

Re: How can App's developer send Splunk Cloud App ...

How can App's developer send Splunk Cloud App requ...

Re: Input script processing time greater than trig...

Input script processing time greater than trigger ...