XSearch · Label Augmentation with Assertion

LLM Prompt for Label Augmentation

The following prompt is used for GPT "o4-mini-2025-04-16".

You are an expert in language {e.g. Python}.

Inputs:
comment: str, code: str

Outputs (JSON only):

        {
          "COMMENT_CONCEPTS": [
            {
              "concept_i": str
            }
            ...
          ],
          "STEPWISE_DESCS": [
            {
              "desc_of_step_j": {
                "description": str,
                "code span": str
              }
            }
            ...
          ],
          "ALIGNMENT_MAP": [
            {
              "concept_i": [
                "desc_of_step_j",
                "desc_of_step_k"
              ],
              ...
            }
          ]
        }

Chain-of-thought steps:

Step 1: Pick the important comment tokens by analyzing the syntactic and semantic dependencies, then output COMMENT_CONCEPTS.
Step 2: Summarize the code into a step-wise description (e.g., desc_of_step_1, desc_of_step_2) and output STEPWISE_DESCS.
Step 3: Map desc_of_step_i to concept_j, locate the segment from desc_of_step_j, and return the comment-to-code mapping as ALIGNMENT_MAP.

Constraints:

Number of concepts must be ≥ 2
One concept_i can align with multiple desc_of_step_j
Output must be valid JSON — no explanations or extra keys
All words and formatting must appear exactly as in the original data

Assertion Feedback (Optional):
Your previous response has validation errors {error messages from assertion checks}, please retry.

List of Assertions

Assertion Category	Assertion Error Message
Formatting	Return must be in JSON format
	Miss any of the `"STEPWISE_DESCS"`, `"COMMENT_CONCEPTS"`, `"ALIGNMENT_MAP"` in return
	`STEPWISE_DESCS` must be a list
	`STEPWISE_DESCS[{idx}]` must be a dict with one key
	`STEPWISE_DESCS[{idx}]['{key}']` must be a dict
	`STEPWISE_DESCS[{idx}]['{key}']` missing 'desc' or 'code'
	`STEPWISE_DESCS[{idx}]['{key}']['desc']` and `['code']` must be strings
	`COMMENT_CONCEPTS` must be a list
	`COMMENT_CONCEPTS[{idx}]` must be a dict with one key
	`COMMENT_CONCEPTS[{idx}]['{key}']` must be a string
	`ALIGNMENT_MAP` must be a list
	`ALIGNMENT_MAP[{idx}]` must be a dict with one key
No Hallucination	Comment words not found in original comment
	Code segment not found in original code
	`ALIGNMENT_MAP[{idx}]['{concept}']` uses unknown step key `'{desc_key}'`
	`ALIGNMENT_MAP[{idx}]['{concept}']` uses unknown code step key `'{desc_key}'`
Concept Diversity	Must return at least 2 concepts

Assertion Trigger Frequency

Annotation Failed Attempt Statistics

	Python	PHP	Java	Javascript	Ruby	Go
No Retries	142720 (72.98%)	131348 (58.56%)	103389 (73.42%)	38566 (71.83%)	15751 (63.05%)	70829 (67.92%)
1 retry	39838 (20.37%)	51657 (23.03%)	28196 (20.02%)	10446 (19.46%)	6113 (24.47%)	22994 (22.05%)
2 retries	7339 (3.75%)	24133 (10.76%)	5550 (3.94%)	1947 (3.63%)	1458 (5.84%)	7048 (6.76%)
3 retries	2214 (1.13%)	7185 (3.20%)	1507 (1.07%)	670 (1.25%)	524 (2.10%)	1614 (1.55%)
4 retries	943 (0.48%)	2690 (1.20%)	561 (0.40%)	370 (0.69%)	245 (0.98%)	577 (0.55%)
5 retries	2500 (1.28%)	7292 (3.25%)	1619 (1.15%)	1689 (3.15%)	892 (3.57%)	1220 (1.17%)