Use the test set to complete the pre-launch effectiveness evaluation of 3Chat Agent

[tang.jing]

Function Overview

The Test Set feature in 3Chat helps you batch-create, import, and evaluate test cases to quickly assess the agent’s performance in different scenarios. You can create test sets by adding questions or uploading files, then run tests to generate reports for further scoring and analysis of the responses. This feature improves the efficiency of managing and optimizing the agent’s quality.

Quick Operation Guide

Accessing the Function Page

1. Log in to the 3Chat Agent admin backend.
2. In the left navigation bar, click: 3Chat Agent → Testing → Test Set to enter the Test Set page.

Create a New Test Set

When you first use the Test Set feature, you will enter the initialization page, where you can start adding test sets and test questions.
You can add test sets in two ways:

1. Click “Add Directly” to manually input questions
2. Alternatively, you can import files in CSV or XLSX formats.

Use Direct Add: Manually Input the Question

1. Click the ‘Add Directly’ card to display the add window. The default name is ‘Test Set 1’, and you can also rename it as needed.

   

2. Enter the questions you want to include in the test set, and click ‘Add Question’ to input multiple questions. You can modify or delete your questions at any time.

   

3. You can also batch copy and paste questions that have line breaks from Word or Excel. You can add up to 50 questions at a time. If you exceed 50 questions, the system will automatically limit and retain only the first 50 questions.

4. Once you have entered the questions, click ‘Save’ to create the test set.

Using Upload File: Batch Import Questions

1. Click the ‘Upload File’ card to display the add popup. The default name is ‘Test Set 1’, and you can also modify the name as needed.

2.Click or drag the file to upload. Please upload files in supported formats: .xls, .xlsx, .xlsm, .csv. Ensure that the questions are placed in the first column. We will retrieve the first 50 rows of content from the first column and add them as test questions.

3.You can also click ‘Clear File’ to reselect the file you want to upload. Only one file can be uploaded at a time. Once the file is successfully uploaded, click the ‘Upload’ button to create the test set.

Run Tests and Generate Results
After creating the test set, the system will automatically run the added questions and batch request 3Chat Agent to provide single-turn responses.

---

View Answer Details
1.Once you enter the test set page, you will be able to see the list of test sets that are running, including the questions, response status, scores, and score reasons. You can click on each test item to view its detailed information. The details include your test question, the agent’s specific response to the question, and the guidance used to answer the question.
The response status includes “Answered” and “Not Answered”, which is determined by the system.

• Answered: The agent has provided a valid response to the question.

• Not Answered: The agent failed to provide a valid response to the question.

2.Your test questions may involve different scenarios and could trigger handover to human or AI task actions. In such cases, you will see a small note below the response indicating that the question can successfully trigger a handover or execute a task in the production environment. However, in the test set environment, these actions will not be actually carried out.

Update Answer
1.Batch Operations: You can select one or multiple test items for batch editing of answers or deleting questions. Test items that have already been completed allow you to edit them, while unfinished ones are only viewable.

2.If you’re not satisfied with the agent’s response, you can also click the Refresh button at the top of the detail panel to try updating the response.

Rate the Answer

1. Select an Answer Rating for the Response: In the answer detail panel, choose “Good,” “Acceptable,” or “Poor” to assess the quality of the Agent’s response. You can also add a rating comment (optional, not required).

2.If you select “Good” or “Acceptable”, you will not be required to select a scoring reason.If you select “Poor”, click “Select Scoring Reason” to choose a reason for the negative score and further improve the response.

3.The available scoring reasons include: not using relevant content, incorrect content usage, incorrect tone of response, response length too long or too short, failure to reflect role identity, failure to follow communication style, failure to adhere to handover to human agent conditions, failure to follow condition restrictions, failure to correctly trigger the task, task not correctly executed, and other.
Select the scoring reason you find appropriate, as it is closely related to the subsequent answer improvement actions. After selecting the scoring reason, click “Improve Answer” to view improvement suggestions.

4.Improve Answer: Different scoring reasons correspond to different improvement suggestions. You can also click “Create or Update AI Task” or “Create or Update Guidance” to navigate to the AI Task or Guidance function pages, where you can further adjust configurations to improve the agent’s responses.

Add Questions to the Test Set
If you wish to add new questions to the current test set, click the “Add Question” button. You can add new questions to the test set either by directly inputting them or uploading a file.

Filter Questions and Answers of Interest
Filter: Customize the test set list display by using different filtering conditions. You can filter the test set list by setting answer status, answer score, and scoring reasons, using either single or multiple filtering criteria.

Add More Test Sets
1.If you want to create a new test set or switch to another test set, click to expand the left-side test set panel to switch, view, or manage other test sets.

2.Click the “+” button to create a new test set, either by directly inputting questions or uploading a file. You can also manage existing test sets, including renaming the test set, deleting the test set, and exporting the test set report.

Export Test Set

Export Test Set Report
In the left-side test set list panel, click “…” → “Export Report”, or use the “Export” button at the top of the test set list to export the current test set report. After clicking, the report will be automatically added to the download list and exported in CSV format.

Frequently Asked Questions

1.How to batch add questions?
You can choose the “Direct Add” method to copy and paste questions or the “Upload File” method to batch import questions.

2.Which file formats are supported for uploading?
Supported file formats include .xls, .xlsx, .xlsm, and .csv.

3.Can multiple test items be edited at the same time?
Yes, you can batch update the answers of test items or batch delete the selected test items.