Script

HENGSHI SENSE provides self-service data processing capabilities, enabling data cleaning, transformation, and merging. However, experienced analysts often prefer to write their own code for data processing. To meet the needs of advanced analysts, HENGSHI SENSE offers the Script module, allowing analysts to write and execute SQL code for data cleaning, data exploration, data modeling, machine learning, data visualization, and other data processing tasks.

The Script module is an open workspace available to users with the Data Management Role. Users can create notes, execute notes, view note execution results, and also edit and run notes created by other users.

Notes

Notes are the basic unit for running script tasks, and each note operates independently. A note includes paragraphs (written code), execution plans, runtime environment settings, and more.

The general workflow for notes is as follows:

1. Create a note
2. Set up the note's runtime environment
3. Create paragraphs, write code, and implement data processing functions
4. Execute the note and view the results.

The following example demonstrates the general workflow of working with notes.

Note Example

1. In the script module, click "Create Note," enter the note name, and complete the note creation.

   

2. On the note page, click "Settings" to enter the Runtime Environment Settings page. Add the required data connections and transaction commit methods for code execution, as shown below.

   

3. Click the "+" icon to create a new paragraph, select the language, runtime environment, and default schema.

   

4. Write code in the paragraph, use Test Execution to check the results, or click Submit Execution.

   

5. Refer to steps 3 and 4 to create other paragraphs. Once all paragraphs are created, you can click Run All to manually execute the note, or set up an Execution Schedule to run at a specified time as needed.

The example above briefly introduces the note workflow. The following sections will provide a detailed explanation of various concepts and common operations for script notes.

Paragraph

A paragraph is the smallest unit for implementing data processing functions in a script. Each note can consist of one or more paragraphs. Paragraphs support operations such as creating, configuring, editing, deleting, and executing.

Create and Delete

Creating a paragraph includes two methods: creating a blank paragraph and uploading a file.

• Create a blank paragraph: Click the "+" icon to add a new blank paragraph, where users can write their own code.

• Upload a text file: Click the upload file icon to upload a text file into the paragraph.

To discard a paragraph, click the delete button on the right to remove it.

Configure Paragraph

Configuring a paragraph includes setting the language, runtime environment, and default execution path Schema.

• Language: Refers to the scripting language used in the paragraph. Currently, only SQL is supported.

  • SQL: Allows you to write code in SQL. The code must comply with the syntax requirements of the selected connection type.

• Runtime Environment: Refers to the environment in which the current notebook runs. You can select from the list of valid connections in Notebook Settings.

• Default Schema: Refers to the Schema that will be used by default if no Schema is specified in the paragraph. This is the specific path for paragraph execution.

Edit Paragraph

Paragraphs can be edited directly, or you can click the button on the right to open and edit in full-screen mode. The full-screen mode displays all tables and their field names in the runtime environment, making it convenient for users to write and debug code.

Paragraph Execution

After the code in a paragraph is written, it needs to be executed to view the results. Paragraphs support both test execution and submit execution.

• Test Execution

  Test execution runs the code to check whether the results are correct, but does not submit the results to the database.

• Submit Execution

  Submit execution runs the code and directly submits the execution results to the database.

Both test execution and submit execution will display a preview of the results and an execution log.

• Preview Results

  Preview results are only available for SQL language paragraphs. When the code contains standard output statements such as show or select, the corresponding output information will be displayed in the preview results. In other cases, the preview results page will not display any data.

• Execution Log

  The execution log shows the running status of the paragraph, which can help users debug the paragraph and locate issues.

  

Note Settings

Note settings include runtime environment settings and runtime transaction commit settings.

Runtime Environment

The runtime environment refers to the data connection used by the paragraph during execution. Click the Add button, and select the required data connection on the pop-up data connection page. The data connection will be displayed in the authorized connection list.

If you have RW permission for the connection, the connection status will be shown as valid; otherwise, the connection status will be invalid.

• Authorization

  In the authorized connection list, open the three-dot menu of a connection and select Authorize. This allows you to authorize the connection to all users of the script. All users of the script can use this connection to run the notebook, regardless of whether they have RW permission.

• Revoke Authorization

  After authorization, all users can execute the notebook, which may lead to data leakage risks. If you do not want to authorize other users, you can revoke authorization after each notebook execution. This way, other users will not be able to use the connection to execute the notebook. After clicking Revoke Authorization, the connection status will become invalid, and no user will be able to use the connection.

• Delete

  In the authorized connection list, open the three-dot menu of a connection and select Delete. The connection will disappear from the authorized connection list. If any paragraph is using this connection, an error will be reported and a prompt message will be displayed when attempting to delete.

Transaction Submission

When running a notebook, you can choose between submitting transactions as a whole or submitting transactions by paragraph.

• Submit Transactions as a Whole

  When running the notebook, all paragraphs are submitted as a single transaction. If any paragraph fails, all paragraphs will be rolled back.

• Submit Transactions by Paragraph

  When running the notebook, each paragraph is submitted as an individual transaction. If a paragraph fails, only that paragraph will be rolled back, while other transactions will execute and commit as normal.

Execution

After completing the paragraph in the note, you need to execute it. Each paragraph can be executed individually, or executed together with the note. Note execution includes executing all and execution plans.

Run All

Run All refers to manual execution. Clicking this button will execute all paragraphs in the current notebook, and the execution results will be submitted to the database.

Execution Plan

In addition to manual execution, a notebook can be set to run according to a scheduled execution plan. Click the execution plan icon in the upper right corner to enter the execution plan settings page, where you can configure the basic information, scheduling information, and alert information for the plan.

• Basic Information: Set the number of retry attempts upon execution failure and the task priority.
• Scheduling Information:
  • Set the task's scheduled execution time; multiple schedules can be configured.
  • Set task dependencies; multiple prerequisite tasks can be configured.
  • Set dependency wait time.
• Alert Information: Enable failure alerts. When a task fails to execute, an email notification will be sent to the recipients.

• After the notebook enables the scheduled execution plan, you can view the execution status in Task Management.