Curl command data collection

[MichaelBs]

I am using a curl command to get data from an api endpoint, the data comes as a single event but I want to be able to store each event as the events come through. I want to get a timechart from that

[tscroggins]

Hi @MichaelBs,

If you're using Curl search , the command should automatically convert a body containing an array/list into separate events. The RIPEstat Looking Glass API returns a single object and multiple rrcs items in the data field:

| curl url="https://stat.ripe.net/data/looking-glass/data.json?resource=1.1.1.1"

{
    "messages": [
        [
            "info",
            "IP address (1.1.1.1) has been converted to its encompassing routed prefix (1.1.1.0/24)"
        ]
    ],
    "see_also": [],
    "version": "2.1",
    "data_call_name": "looking-glass",
    "data_call_status": "supported",
    "cached": false,
    "data": {
        "rrcs": [
            ...
        ],
        "query_time": "2024-06-30T17:24:44",
        "latest_time": "2024-06-30T17:24:29",
        "parameters": {
            "resource": "1.1.1.0/24",
            "look_back_limit": 86400,
            "cache": null
        }
    },
    "query_id": "20240630172444-e3bf9bf6-dd38-4cff-aa4b-e78b33f1a2c3",
    "process_time": 70,
    "server_id": "app111",
    "build_version": "live.2024.6.24.207",
    "status": "ok",
    "status_code": 200,
    "time": "2024-06-30T17:24:44.525141"
}

You return rrcs items as individual events with various combinations of spath, mvexpand, eval, etc.:

| fields data
| spath input=data path="rrcs{}" output=rrcs
| fields rrcs
| mvexpand rrcs
| eval rrc=spath(rrcs, "rrc"), location=spath(rrcs, "location"), peers=spath(rrcs, "peers{}")
| fields rrc location peers
| mvexpand peers
| spath input=peers
| fields - peers

For experimentation, I recommend storing the data in a lookup file to limit the number of calls you make to stat.ripe.net.

First search:

| curl url="https://stat.ripe.net/data/looking-glass/data.json?resource=1.1.1.1"
| outputlookup ripenet_looking_glass.csv

Subsequent searches:

| inputlookup ripenet_looking_glass.csv
| fields data
``` ... ```

[MichaelBs]

I get you but I want to create a timechart per the events or data coming through

[tscroggins]

Hi @MichaelBs,

After receiving the data, you can use timechart as you normally would. Do you have specific questions about timechart using the sample data provided?

[PickleRick]

If you mean that you want to ingest data available over some HTTP endpoint, you need to either have a scripted or modular input polling said endpoint or have an external script pulling the data periodically and either writing to file (from which you'd ingest with normal monitor input) or push to HEC endpoint - these are the most straightforward options.

If I remember correctly, Add-on Builder can be used to make such polling input for external HTTP sources.