topic Re: Is it possible to remove unnecessary JSON wrapping before it's ingested to save license? in Getting Data In

Is it possible to remove unnecessary JSON wrapping before it's ingested to save license?

michael_sleep — Thu, 02 Feb 2023 15:16:46 GMT

Hey there, we have a large volume (about 500-600gb) of data coming in daily but about 200gb of this is a JSON wrapper from Amazon Firehose. The data essentially looks like this:

{ "message": "ACTUAL_DATA_WE_WANT", "logGroup": "/use1/prod/eks/primary/containers", "logStream": "fluent-bit/cross-services/settings-7dbb9dbdb4-qjz5b/settings-api/81d3685eaaeae0effab5931590784016ce75a8171ad7e3e76152e30bd732a739", "timestamp": 1675349068034 }

As you can see, ACTUAL_DATA_WE_WANT is what we need. This contains everything including timestamp and application information. The JSON wrapper is added by Firehose and makes up at least 250 bytes of every event.

Is it possible to remove all of this unnecessary data so that we can save ingestion for more useful things? I have heard that the SEDCMD can do this but it is resource intensive and we ingest almost a billion events a day.

Re: Is it possible to remove unnecessary JSON wrapping before it's ingested to save license?

richgalloway — Thu, 02 Feb 2023 15:37:46 GMT

Usually, this is done with SEDCMD. The resource use depends on the efficiency of the regex used. Test the regex on regex101.com and evaluate the resource usage on your dev/test instances.

Another option is to use Cribl to remove the unwanted bytes.

Re: Is it possible to remove unnecessary JSON wrapping before it's ingested to save license?

isoutamo — Thu, 02 Feb 2023 16:29:27 GMT

As you have pure json event you probably could try INGEST_EVAL with json_extract? https://docs.splunk.com/Documentation/Splunk/9.0.3/SearchReference/JSONFunctions#json_extract.28.26lt.3Bjson.26gt.3B.2C_.26lt.3Bpaths.26gt.3B.29