--- url: /v6.3/en-us/connecting-ext-api.md --- # API Data Source When users work with EXT API data, they can bring it into HENGSHI SENSE through an API Data Source.\ An API Data Source defines a unified ingestion specification that supports a large number of APIs lacking native compute capabilities. Out-of-the-box connectors for many common APIs are provided, while custom development lets you add internal enterprise APIs or any public API, enabling rapid ingestion. API Data Sources make it easy to scale external API ingestion from 1 to N, continuously strengthening HENGSHI SENSE’s data-integration power. ![](/images/data-connection/connecting-ext-api-001.png) ## Built-in API Data Sources ### Mingdao Cloud Data Source Application data on Mingdao Cloud can be connected to HENGSHI SENSE with a few simple steps: 1. Select **API** as the data source when creating a new data connection.\ ![](/images/data-connection/connecting-ext-api-001.png) 2. Enter a name for the new connection and set **API name** to `mingdaoyun`.\ ![](/images/data-connection/connecting-ext-api-mingdao.png) 3. Fill in the Mingdao Cloud configuration information under **API Config**. Example: ```json { "appKey":"ab1axxxxxxxx", "secretKey":"68b7eqbp7r1p5ZeAbw60aXcG0h8Gxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "sign":"NWRiODkxZmQ0NzljNzcxZDhiMDZjMzE5ZDxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "apiUrl": "https://api.mingdao.com/v2/open" } ``` **Mingdao Cloud API Config Description** | Field | Required | Description | | --------- | -------- | --------------------------------------------------------------------------------------------- | | appKey | Yes | AppKey from your Mingdao Cloud application | | secretKey | Yes | secretKey from your Mingdao Cloud application | | sign | Yes | sign from your Mingdao Cloud application | | apiUrl | No | For private deployment, provide the private URL; for public deployment, leave empty or use the public Mingdao Cloud URL | 4. After successful validation, click **Save** to complete the data connection. You can now view and use Mingdao Cloud data within the system.\ ![](/images/data-connection/connecting-ext-api-mingdao-path-data.png) ### FTP Data Source The system supports FTP data sources. In HENGSHI SENSE, files under a specified FTP path can be read through simple configuration. The configuration steps are as follows. 1. Select **API Data Source** when creating a new data connection.\ ![](/images/data-connection/connecting-ext-api-ftp.png) 2. Enter the new connection name and choose `ftp` as the API name. 3. Fill in the FTP API Config information. Example: ```json { "host": "10.10.x.y", "username": "xxxxxxx", "password": "xxxxxxx", "directory": "/opt/ftp", "encoding": "utf8", "passive": true } ``` **FTP Data Source API Config Description** | Field | Required | Description | | --------- | -------- | ------------------------------- | | host | Yes | FTP server address | | username | Yes | FTP login username | | password | Yes | FTP login password | | directory | No | FTP mapped directory | | encoding | No | File encoding | | passive | No | Whether to use passive mode | 4. After successful validation, click **Save** to complete the data connection creation. The connected data can then be viewed and used in the system.\ ![](/images/data-connection/connecting-ext-api-ftp-path-data.png) > \[!TIP] Tips > > 1. FTP data sources can only parse CSV and Excel files into table format; other file types are not supported. > > 2. For Excel files in FTP data sources, data can only be previewed after creating a new Dataset; direct preview in the data connection is not supported. ### Other Built-in Data Sources In addition to the FTP and Mingdao Cloud mentioned above, the system also supports API data sources such as `S3`, `Wangdiantong`, `WeCom`, `Shimo Docs`, `Jushuitan`, `E-Cang Technology`, and `Wai Qin`. After configuring the API config, users can bring the data into the HENGSHI SENSE platform. ![](/images/data-connection/connecting-ext-api-apiName.png) #### `API Config` Configuration Examples Below are the `API Config` examples for several built-in API data sources: **S3 Data Source `API Config`** ```json { "access_key_id": "AKIAWBC5HCDHELXXXXXXXX", "secret_key_id": "67U4ZvclZxkqiM9owAVTEootwyxxxxxxxx", "region": "cn-north-x", "useVirtualFolderDelimiter": false, "bucket": "", "path": "" } ``` **WangDianTong Data Source `API Config`** ```json { "appsecret": "3a5737fc6bc2d6970f790308xxxxxxxx", "sid": "xxxxxxxx", "appkey": "xxxxxxxx-ot", "env": "production", "customizedStartTime": "2022-01-01 00:00:00", "addHSTimeColumn": true, "tableConfig": { "/stockin_order_query.php": { "timeWindow": 60, "timeWindowLimit": 50 } } } ``` **WeCom Data Source `API Config`** ```json { "corpid": "wwcXXXXXXXXXX", "corpsecret": "XGGBxxxxxxxxxxxxxxxxxxxxxxxx", "customizedStartTime": "2022-09-01 00:00:00", "addHSTimeColumn": false, "templates": [ "ZLyhvnbAYmJnwLXyyxxxxxxxxxxxxxxxxxxxx1", "ZLyhvnbAYmJnwLXyyxxxxxxxxxxxxxxxxxxxx2" ] } ``` **Shimo Data Source `API Config`** ```json { "sid": "s%3A7db3e9d7bab34969a6d592xxxxxxxxxxxx.hnmK3aq2yOYYJB9exxxxxxx", "addHSTimeColumn": true, "customCacheMaxLifetime": 5 } ``` **Jushuitan Data Source `API Config`** ```json { "app_key": "b0b7d1db226d4216xxxxxxxxxxxxxxxx", "app_secret": "99c4cef262f34ca88xxxxxxxxxxxx", "access_token": "b7e3b1e24e174593xxxxxxxxxxx", "env": "test", "customizedStartTime": "2022-11-15 00:00:00", "addHSTimeColumn": true, "tableConfig": { "/open/orders/single/query": { "timeWindow": 30, "timeWindowLimit": 3 } } } ``` **ECCANG Data Source `API Config`** ```json { "app_key": "e4bd91dcxxxxxxxx", "service_id": "xxxxxxxx", "secret_key": "bab4d5xxxxxxxx" } ``` **Wai Qin Data Source `API Config`** ```json { "openId": "76776371xxxxxxxx", "appKey": "u7To2bdLxxxxxxxx", "customFormList": [ "74842108646xxxxxxxx1", "74842108646xxxxxxxx2" ], "aiProductIdentificationResultQueryStartTime": "2025-01-01 00:00:00" } ``` ## Metric API Data Source A Metric API Data Source connects a third-party metrics platform to HENGSHI SENSE. A [Dynamic Domain](business-indicator-domain.md#create-a-dynamic-domain) reads its measures and dimensions, and an App Creation Dashboard queries measure data through the connection. This connection has no database table tree and cannot create a regular Dataset. ### Create a Metric API Data Source 1. Open `Data Connections`, click `New Data Connection`, and select `Metric API Data Source`. 2. Enter a connection name. If the Groovy script needs connection parameters, provide the third-party platform address, endpoint paths, and authentication values in `API Config`; otherwise, leave it blank or enter `{}`. 3. Paste the Groovy adapter for the third-party platform into `Custom Script`. 4. Click `Verify`. Save the connection after verification succeeds. 5. To share the connection with other users, configure `Metric view permission` from the connection's three-dot menu. > \[!CAUTION] Attention > Page-entered Groovy scripts require `ENABLE_GROOVY_SCRIPT`. Restart the service after enabling it. Scripts managed on the server through `EXT_API_PATH` do not use this switch. The script sends external HTTP requests, so restrict target addresses, validate certificates, and keep authentication values and full responses out of logs. The Groovy script parses `API Config`, so its fields can follow the third-party platform contract. The following example contains no real address or credential: ```json { "baseUrl": "https://metrics.example.com", "metricsPath": "/api/metrics", "analysisContextPath": "/api/metrics/context", "dataPath": "/api/metrics/query", "token": "" } ``` Do not place real credentials in shared documentation, script comments, or debug logs. ### Groovy Script Contract A Metric API Data Source uses its own script contract. It does not use `getApiName`, `getPathTables`, or `fetchTableData` from a regular custom API data source. Implement the methods needed by the third-party platform: | Method | Purpose | | --- | --- | | `setOptions(String)` | Parse `API Config` | | `fetchMetrics()` / `fetchMetrics(JSONObject)` | Return measures and dimensions | | `fetchMetricsPage(JSONObject)` | Return measures and dimensions with a total count | | `queryMeasureAnalysisContext(JSONObject)` | Return candidate measures, compatible measures, and analyzable dimensions | | `buildChartRequest(options)` | Convert chart options to a third-party query request | | `fetchData(JSONObject)` | Return chart schema and data rows | | `fetchFunctions()` | Return calculation functions supported by the third-party platform; optional | | `testConnection()` | Verify the address, authentication, and measure endpoint; optional. When omitted, the system calls `fetchMetrics()` | The redacted sample below assumes that each endpoint returns a `data` object. The measure endpoint uses `metrics`, `dimensions`, and `total`; the analysis-context endpoint uses `candidateMetrics`, `metrics`, `unionDims`, and `dims`; the data endpoint uses `schema`, `rows`, and `total`. Map different response fields in the script. ```groovy import com.alibaba.fastjson.JSON import com.alibaba.fastjson.JSONArray import com.alibaba.fastjson.JSONObject import com.hengshi.nangaparbat.dto.PageDto import com.hengshi.nangaparbat.model.DatasetField import com.hengshi.nangaparbat.model.Type import java.net.HttpURLConnection import java.net.URL class MetricApiProviderExample { private JSONObject config = new JSONObject() def setOptions(String optionsText) { config = JSON.parseObject(optionsText ?: "{}") ?: new JSONObject() } def fetchMetrics() { return fetchMetricsPage(null).data } def fetchMetrics(JSONObject request) { return fetchMetricsPage(request).data } def fetchMetricsPage(JSONObject request) { JSONObject data = responseData(postJson( config.getString("metricsPath") ?: "/api/metrics", request ?: new JSONObject())) def fields = [] fields.addAll(toFields(data.getJSONArray("metrics"), DatasetField.CategoryEnum.measure)) fields.addAll(toFields(data.getJSONArray("dimensions"), DatasetField.CategoryEnum.field)) PageDto page = new PageDto() page.setData(fields) Long total = data.getLong("total") page.setTotalHits(total == null ? fields.size() : total) return page } def queryMeasureAnalysisContext(JSONObject request) { JSONObject data = responseData(postJson( config.getString("analysisContextPath") ?: "/api/metrics/context", request ?: new JSONObject())) return [ candidateMeasures : toFieldMaps(data.getJSONArray("candidateMetrics"), "measure"), compatibleMeasures: toFieldMaps(data.getJSONArray("metrics"), "measure"), unionFields : toFieldMaps(data.getJSONArray("unionDims"), "field"), intersectionFields: toFieldMaps(data.getJSONArray("dims"), "field") ] } def buildChartRequest(options) { // Map dimensions, measures, filters, sorting, and pagination here when field names differ. return JSON.parseObject(JSON.toJSONString(options ?: [:])) } def fetchData(JSONObject request) { JSONObject data = responseData(postJson( config.getString("dataPath") ?: "/api/metrics/query", request ?: new JSONObject())) return JSON.toJSONString([ schema : toFieldMaps(data.getJSONArray("schema"), null), data : data.getJSONArray("rows") ?: [], totalHits: data.getLong("total") ]) } def fetchFunctions() { return ["sum", "avg", "countDistinct"] } def testConnection() { return fetchMetrics() != null } private JSONObject postJson(String path, Object body) { String baseUrl = config.getString("baseUrl") if (!baseUrl) { throw new IllegalArgumentException("baseUrl is required") } String url = baseUrl.replaceAll('/+$', '') + '/' + path.replaceAll('^/+', '') HttpURLConnection connection = (HttpURLConnection) new URL(url).openConnection() connection.setRequestMethod("POST") connection.setDoOutput(true) connection.setConnectTimeout(30000) connection.setReadTimeout(30000) connection.setRequestProperty("Content-Type", "application/json;charset=UTF-8") connection.setRequestProperty("Accept", "application/json") String token = config.getString("token") if (token) { connection.setRequestProperty("Authorization", "Bearer " + token) } connection.outputStream.withCloseable { output -> output.write(JSON.toJSONString(body ?: [:]).getBytes("UTF-8")) } int responseCode = connection.getResponseCode() def stream = responseCode >= 400 ? connection.getErrorStream() : connection.getInputStream() String responseText = stream == null ? "" : stream.getText("UTF-8") connection.disconnect() if (responseCode >= 400) { throw new IllegalStateException("Metric API request failed: HTTP " + responseCode) } return JSON.parseObject(responseText) } private JSONObject responseData(JSONObject response) { return response?.getJSONObject("data") ?: response ?: new JSONObject() } private List toFields(JSONArray items, DatasetField.CategoryEnum category) { return (items ?: []).collect { JSONObject item -> DatasetField field = new DatasetField() field.setFieldName(item.getString("code") ?: item.getString("fieldName")) field.setLabel(item.getString("name") ?: item.getString("label")) field.setCategory(category) field.setType(toType(item.getString("type"))) return field } } private List toFieldMaps(JSONArray items, String category) { return (items ?: []).collect { JSONObject item -> def field = [ fieldName: item.getString("code") ?: item.getString("fieldName"), label : item.getString("name") ?: item.getString("label"), type : toType(item.getString("type")).name() ] if (category) { field.category = category } return field } } private Type.TypeName toType(String rawType) { switch ((rawType ?: "string").toLowerCase()) { case ["number", "integer", "long", "decimal", "double", "float"]: return Type.TypeName.number case ["date", "datetime", "timestamp", "time"]: return Type.TypeName.date case ["boolean", "bool"]: return Type.TypeName.bool case "json": return Type.TypeName.json default: return Type.TypeName.string } } } ``` ## Custom API Data Source > \[!CAUTION] Attention > Starting from version 5.1, for security reasons, the system disables the Groovy script feature filled in on the page by default. > To use a custom API data source (via Groovy script entered on the page), add the following to the configuration file: > > ```properties > ENABLE_GROOVY_SCRIPT=true > ``` > > After the configuration is complete, restart the service for it to take effect. Groovy scripts uploaded to the server (configured via the `EXT_API_PATH` variable) are not affected. In addition to the built-in API data sources, the system supports user-defined API data sources, which can be implemented using Groovy scripts. 1. Select the API data source when creating a new data connection. ![](/images/data-connection/connecting-ext-api-groovy.png) 2. Enter the new connection name and select `Custom` for the API name. 3. Fill in the API Config information. **Note: This configuration needs to be parsed in the script via code**. If no configuration is required, it is recommended to leave it empty as `{}`. 4. Write the Groovy script for the custom API data source according to the specifications and place it in `Custom Script`. ![](/images/data-connection/connecting-ext-api-groovy-script.png) ## Custom API Data Source Script Specification When connecting a custom data source, implement the following four interfaces in the Groovy script: * **getAPIName**: Returns the API name. * **setOptions**: Handles authentication and authorization configuration. * **fetchPathTables**: Retrieves the data source directory structure. * **fetchTableData**: Fetches data for the table node. ##### Function getAPIName The function `getAPIName` is used to return the API name. Each API data-source name must be unique; duplicates will cause conflicts.\ This is a class-level static function and returns a string value. ```groovy static getApiName() { return "TrivalApiDemo" } ``` ##### Function setOptions The `setOptions` function is used to persist configuration related to authentication and authorization.\ If no external network requests are involved, it can be left empty.\ This is an `object` function; its parameter is a JSON string containing the information the user filled in when adding the data connection. ```groovy def setOptions(String options) { if (!options?.trim()) { return } def json = jsonSlurper.parseText(options) if (json.cityCodeURL?.trim()) { config.cityCodeURL = json.cityCodeURL } if (json.weatherURL?.trim()) { config.weatherURL = json.weatherURL } } ``` ##### Function getPathTables The `getPathTables` function returns the directory tree, listing the entire catalog/schema/table structure of the data source. Sample code: ```groovy def getPathTables() { def tableTree = [ [ pathType: "table", name: "table1", tableType: "table"], [ pathType: "path", name: "schema2", children: [ [ pathType: "table", name: "table3", tableType: "view"], [ pathType: "table", name: "table4", tableType: "table", remarks: "table alias 4"] ] ] ] return new JsonBuilder(tableTree).toString() } ``` Structure definition: * `pathType` values: `table` (leaf node representing a table) or `path` (intermediate node representing a schema or catalog) * `name` is the table name or schema/catalog name * `tableType` is only meaningful when `pathType` is `table`; usually set to `table` (alternative: `view`, rarely used) * When `pathType` is `path`, the node is not a table but an intermediate directory such as a schema or catalog; it may contain a `children` array * `remarks` is an optional description for the current node With the schema defined above, tables can be retrieved from the data connection. ##### Function fetchTableData The `fetchTableData` function is used to retrieve the data of a leaf node, i.e., a table node. Sample code is as follows. ```groovy def fetchTableData(List tablePath) { def tableName = tablePath.last() def schemaData = [ schema: [ [fieldName: "field1", label: "Field Alias 1", type: "string"], [fieldName: "field2", label: null, type: "string"], [fieldName: "field3", type: "number"], [fieldName: "field4", type: "date"], [fieldName: "field5", type: "time"]], data: [ ["a", "b", 10, "2022-01-01", "2022-01-01 23:59:59"], ["c", "d", 20.20, "2019-12-39", "2019-12-39 23:59:59"] ] ] return new JsonBuilder(schemaData).toString() } ``` Schema structure definition: * `fieldName`: Field name, must be unique within the table to distinguish different fields. * `type`: Field type; commonly used types are: string, number, date, time, json, bool. * `label`: Field alias. Data structure definition: * Each element in `data` is an array; the number of elements must match the schema, and the data types must correspond one-to-one. ## Custom API Data Source Code Example Below is a Java-based custom data source for a weather API. The content of the `weather.groovy` file is as follows: ```groovy //Weather.groovy import com.hengshi.nangaparbat.dto.DatasetResultDto import com.hengshi.nangaparbat.model.PathTableNode import static com.hengshi.nangaparbat.model.Dataset.Field import static com.hengshi.nangaparbat.model.Type.TypeName import groovy.json.JsonSlurper class Provider { static getApiName() { return "Weather API" } def setOptions(String options) { if (!options?.trim()) { return } def json = jsonSlurper.parseText(options) if (json.cityCodeURL?.trim()) { config.cityCodeURL = json.cityCodeURL } if (json.weatherURL?.trim()) { config.weatherURL = json.weatherURL } } def getPathTables() { def txt = config.cityCodeURL.toURL().text def arr = jsonSlurper.parseText(txt) return arr.collect{ obj -> def node = PathTableNode.createTableNode(obj.city_code) node.setRemarks(obj.city_name) return node } } def fetchTableData(List tablePath) { def url = config.weatherURL + tablePath.last() def txt = url.toURL().text; def result = new DatasetResultDto() def schema = [] def data = [] def row = [] def arr = jsonSlurper.parseText(txt) arr.data.each{ k, v -> def field = new Field(); field.setFieldName(k) field.setType(TypeName.string) if (k.startsWith("pm")) { field.setType(TypeName.integer) } schema.add(field) row.add(v); } result.setSchema(schema) data.add(row) result.setData(data) return result } def jsonSlurper = new JsonSlurper() def config = [ cityCodeURL: "https://github.com/baichengzhou/weather.api/raw/master/src/main/resources/citycode-2019-08-23.json", weatherURL: "http://t.weather.itboy.net/api/weather/city/" ] } ``` Code explanation: * `DatasetResultDto` defines the return result of `fetchTableData`, consisting of `schema` and `data`, written via `setData` and `setSchema`. * `PathTableNode` defines the return of `getPathTables`, consisting of `pathType`, `name`, `children`, and `tableType`, also written via their respective setters. * `schema` is composed of multiple `Field`s, each consisting of `fieldName`, `type`, and `label`. * The `type` is an enum defined by `TypeName`. * [CityCode](https://github.com/baichengzhou/weather.api/raw/master/src/main/resources/citycode-2019-08-23.json) defines the directory structure. * [Weather Site](http://t.weather.itboy.net/api/weather/city/101120201) is the source of weather data, recording weather information for various cities.