topic Re: Filtering the request POST in Rest API in Getting Data In

Filtering the request POST in Rest API

nalia_v — Wed, 06 Nov 2019 09:28:20 GMT

I apologize if somewhere there is already this topic on the portal.
If there is, please click on the link.

Question
There is a rest api request by POST method
There is a rest api request using the post method, which accesses the URL and picks up the log in the format JSON.
JSON log itself is VERY large and voluminous.
When collecting, the forwarder and its turn begins to flow memory and CPU.
The problem is that the log in the response is very large, but the log has unique ID and time fields.

[{"ID":"65426","DATE":"2019-11-05T12:49:02+03:00"

How can I configure / build a post request with filtering by timestamp or ID field ?
That is, if by timestamp, the request would take logs only for the current day and increment the data.
Or compared the ID field.
How to specify these settings in the filter through the addon RestAPI ?

Re: Filtering the request POST in Rest API

DavidHourani — Wed, 06 Nov 2019 10:57:56 GMT

Hi @nalia_v, what kind of logs are we talking about and whats the API you're trying to fetch from ?

From what I understand you're trying to read data via REST and push it into Splunk ?

Re: Filtering the request POST in Rest API

arjunpkishore5 — Wed, 06 Nov 2019 11:40:39 GMT

Hi @nalia_v

Agree to what @DavidHourani mentioned. Could you please clarify more on this.

Are you trying to load data from an external API into Splunk? If yes, you would have to look into the external systems's REST API documentation

Are you trying to use's Splunk's REST API to query data? If yes, please provide a sample of the POST request that you are making and some sample data.

Re: Filtering the request POST in Rest API

nalia_v — Wed, 06 Nov 2019 12:51:59 GMT

Hi arjunpkishore5.
higher answer.

Re: Filtering the request POST in Rest API

nalia_v — Wed, 06 Nov 2019 12:52:47 GMT

The moderator is still checking my answer ))

Re: Filtering the request POST in Rest API

nalia_v — Wed, 30 Sep 2020 02:49:54 GMT

I am trying to upload data through an addon RestAPI from the portal Bitrix24.
Data regarding the activity of user actions - added / deleted directory, file ... and some other actions.

{"result":[{"ID":"65426","DATE":"2019-11-05T12:49:02+03:00","USER_ID":"16707","IP_ADDRESS":"XXX.XXX.XXX.XXX","USER_AGENT":"Mozilla\/5.0 (Windows NT 6.1; Win64; x64; rv:70.0) Gecko\/20100101 Firefox\/70.0","TYPE":"dir","ACTION":"create","OBJECT_ID":"406544","ENTITY_NAME":"\u0420\u0435\u043c\u043e\u043d\u0442 \u0438 \u0432\u0441\u0451 \u0447\u0442\u043e \u0441 \u043d\u0438\u043c \u0441\u0432\u044f\u0437\u0430\u043d\u043e","ENTITY_SIZE":"0","ENTITY_PATH":"\u0422\u0430\u0442\u044c\u044f\u043d\u0430 \u041c\u043e\u0442\u044b\u043b\u044c\/","ENTITY_VERSION":"","ENTITY_NAME_NEW":"","ENTITY_VERSION_NEW":"","ENTITY_PATH_NEW":""},
{"ID":"65425","DATE":"2019-11-05T12:48:37+03:00","USER_ID":"17071","IP_ADDRESS":"XXX.XXX.XXX.XXX","USER_AGENT":"Mozilla\/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident\/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)","TYPE":"file","ACTION":"upload","OBJECT_ID":"406543","ENTITY_NAME":"\u042f \u043f\u043e\u0434\u0430\u0440\u044e \u0442\u0435\u0431\u0435 \u041a\u0440\u044b\u043b\u044c\u044f. \u041a\u043d\u0438\u0433\u0430 File_name.pdf","ENTITY_SIZE":"2604457","ENTITY_PATH":"\u0425\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435 \u0434\u043b\u044f \u0412\u0435\u0431-\u043c\u0435\u0441\u0441\u0435\u043d\u0434\u0436\u0435\u0440\u0430\/35793\/","ENTITY_VERSION":"1","ENTITY_NAME_NEW":"","ENTITY_VERSION_NEW":"","ENTITY_PATH_NEW":""},
{"ID":"65424","DATE":"2019-11-05T12:47:46+03:00","USER_ID":"16707","IP_ADDRESS":"XXX.XXX.XXX.XXX","USER_AGENT":"Mozilla\/5.0 (Windows NT 6.1; Win64; x64; rv:70.0) Gecko\/20100101 Firefox\/70.0","TYPE":"file","ACTION":"view","OBJECT_ID":"80519","ENTITY_NAME":"File_name88.pdf","ENTITY_SIZE":"469506","ENTITY_PATH":"\u0410\u043b\u0435\u043a\u0441\u0435\u0439 \u0414\u0430\u043d\u0438\u043b\u0438\u043d\/\u0417\u0430\u0433\u0440\u0443\u0436\u0435\u043d\u043d\u044b\u0435 \u0444\u0430\u0439\u043b\u044b\/","ENTITY_VERSION":"1","ENTITY_NAME_NEW":"","ENTITY_VERSION_NEW":"","ENTITY_PATH_NEW":""}

The POST request itself is a normal URL with parameters in the line userID and Token by which the connection is made.
I can’t drop it here, because it contains confidential data.
Our corporate developers wrote specially api (on the Bitrix24 portal) to upload such data.
Fine ! the slank takes them, but here it takes away ALL the data at once. And there is a lot of data for different dates.
Also, our developers of the Bitrix24 portal have provided fields by which you can filter the request. But in which fields to specify them in the addon RestAPI settings.
Query Parameters for Filtering Data (They are the same fields in the event.)
ID
date_from
date_to
type
action
limit
offset

I think the most basic fields by which you can filter with melon incrementation are:
ID
date_from
date_to

But where to specify them in the add api addon is unclear.
And if you rely on a time stamp, then how to increase the day