API Query Dataset

API Query Dataset refers to converting an HTTP JSON API into a dataset through API queries.

Creating a Dataset

Please follow the steps below to create an API query dataset.

Creation Entry

On the Dataset interface, click "New Dataset" and select "API Query".

Supported API Request Configurations

Request methods support both GET and POST.

1. Request Header: By configuring the Header, you can pass custom headers in the HTTP request. For example, you can commonly set Content-Type to application/json, or configure Authorization as Basic SOME-BASE64 to support simple authentication features.

2. Query Parameters: Generally used to specify the object to query or to configure pagination. For example, some endpoints use ?compainId=123 to specify the campaign ID, and you can combine this with the pagination configuration below to implement pagination. For instance, if an endpoint supports pagination like ?page=1, you can use the pagination parameter PAGE specified in the pagination settings to achieve this. Define ?page={{$PAGE}} to paginate.

3. Request Body Definition: Similar to query parameters, you can also specify the request body required for a specific query.

4. Pagination Settings: Two pagination methods are supported—pagination by page number, such as ?page=1&n=10, and pagination by offset, such as offset=20&limit=10.

5. Authentication Methods: Supports Basic authentication with username and password, or OAuth2 authorization code mode.

JSON Field Splitting

After completing the data acquisition configuration, you need to configure JSON parsing. The syntax for JSON parsing is consistent with the syntax for JSON splitting, supporting the splitting of first-level JSON. Each split field must have an alias, and aliases cannot be duplicated. The types of split fields only support number, text, and time.

After completing the field splitting configuration, you can preview the split results on the first page. If the split results do not meet your expectations, you can go back to the previous page to modify the field splitting configuration. If the results are as expected, you can directly save them as a dataset.

By default, API datasets are not imported into the engine. If you need to perform computational analysis, you can set the dataset to be imported into the engine and configure an update schedule when creating or modifying the dataset.