topic Re: Curl command data collection in Getting Data In

Curl command data collection

MichaelBs — Fri, 28 Jun 2024 16:48:35 GMT

I am using a curl command to get data from an api endpoint, the data comes as a single event but I want to be able to store each event as the events come through. I want to get a timechart from that

Re: Curl command data collection

PickleRick — Fri, 28 Jun 2024 16:54:09 GMT

If you mean that you want to ingest data available over some HTTP endpoint, you need to either have a scripted or modular input polling said endpoint or have an external script pulling the data periodically and either writing to file (from which you'd ingest with normal monitor input) or push to HEC endpoint - these are the most straightforward options.

If I remember correctly, Add-on Builder can be used to make such polling input for external HTTP sources.

Re: Curl command data collection

tscroggins — Sun, 30 Jun 2024 17:42:16 GMT

Hi @MichaelBs,

If you're using , the command should automatically convert a body containing an array/list into separate events. The RIPEstat Looking Glass API returns a single object and multiple rrcs items in the data field:

| curl url="https://stat.ripe.net/data/looking-glass/data.json?resource=1.1.1.1"

{ "messages": [ [ "info", "IP address (1.1.1.1) has been converted to its encompassing routed prefix (1.1.1.0/24)" ] ], "see_also": [], "version": "2.1", "data_call_name": "looking-glass", "data_call_status": "supported", "cached": false, "data": { "rrcs": [ ... ], "query_time": "2024-06-30T17:24:44", "latest_time": "2024-06-30T17:24:29", "parameters": { "resource": "1.1.1.0/24", "look_back_limit": 86400, "cache": null } }, "query_id": "20240630172444-e3bf9bf6-dd38-4cff-aa4b-e78b33f1a2c3", "process_time": 70, "server_id": "app111", "build_version": "live.2024.6.24.207", "status": "ok", "status_code": 200, "time": "2024-06-30T17:24:44.525141" }

You return rrcs items as individual events with various combinations of spath, mvexpand, eval, etc.:

| fields data | spath input=data path="rrcs{}" output=rrcs | fields rrcs | mvexpand rrcs | eval rrc=spath(rrcs, "rrc"), location=spath(rrcs, "location"), peers=spath(rrcs, "peers{}") | fields rrc location peers | mvexpand peers | spath input=peers | fields - peers

For experimentation, I recommend storing the data in a lookup file to limit the number of calls you make to stat.ripe.net.

First search:

| curl url="https://stat.ripe.net/data/looking-glass/data.json?resource=1.1.1.1" | outputlookup ripenet_looking_glass.csv

Subsequent searches:

| inputlookup ripenet_looking_glass.csv | fields data ``` ... ```

Re: Curl command data collection

MichaelBs — Mon, 01 Jul 2024 09:44:25 GMT

I get you but I want to create a timechart per the events or data coming through

Re: Curl command data collection

tscroggins — Sun, 14 Jul 2024 15:15:10 GMT

Hi @MichaelBs,

After receiving the data, you can use timechart as you normally would. Do you have specific questions about timechart using the sample data provided?