Moodle Coderunner

CodeRunner Question Generator (Minimal Pipeline)

Generate Moodle CodeRunner questions. Two modes depending on your environment.

Mode Detection

Before doing anything else, determine your mode:

Run this command:

echo "jobe-check"

• If the command succeeds (you can execute bash): use FULL PIPELINE MODE (Sections 3-9). Jobe computes all expected output via curl. This is the preferred mode.
• If the command fails (no bash access, e.g. Claude.ai or ChatGPT): use FALLBACK MODE (see box below). You must mentally trace expected output yourself.

> FALLBACK MODE (no bash access) > > When you cannot run bash commands (Claude.ai, ChatGPT, API without tools): > 1. Generate the same JSON structure as Section 6, but you MUST include "expected" in each test case by carefully tracing the solution through the test code. > 2. Trace character-by-character. Pay special attention to: trailing spaces, newlines, floating-point precision, array formatting. > 3. Apply all per-language rules from Section 2 and auto-fix rules from Section 8 manually. > 4. Wrap in XML using the template from Section 7. > 5. Add this warning to the user: "These questions were generated WITHOUT Jobe server validation. Before using them, import into Moodle and ensure Validate on save is enabled (it is by default). Moodle will run the solution against all test cases on import and flag any mismatches." > 6. All other sections (question design, type rules, XML template, checklist) still apply.

Full Pipeline mode is strongly preferred. It eliminates the #1 source of errors (hallucinated expected output, which causes 35% of all failures).

---

1. Question Type Reference

Choose the type FIRST -- it determines everything else.

| Type | Student Writes | Test Mechanism | Stdin? | Reliability | |------|---------------|----------------|--------|-------------| | java_class | A class | Test code calls methods | No | HIGH | | java_method | Static method(s) ONLY | Test code calls methods | No | HIGH | | java_program | Full program with main | Empty test, stdin input | Yes | LOW -- EOF issues | | python3 | Function(s) or class | Test code calls with print() | No | HIGH | | python3_w_input | Full program | Empty test, stdin input | Yes | MEDIUM -- EOF issues | | c_function | Function(s) + headers | Test code has main() | No | HIGH | | c_program | Full program with main | Empty test, stdin input | Yes | MEDIUM | | cpp_function | Function(s) + headers | Test code has main() | No | MEDIUM -- Werror | | cpp_program | Full program with main | Empty test, stdin input | Yes | MEDIUM -- Werror | | nodejs | Function(s) | Test code uses console.log() | No | HIGH |

Default choice: Prefer java_class > java_method > java_program. Prefer function types over program types -- they avoid stdin/EOF problems entirely.

---

2. Per-Language Rules

These rules are derived from 1000+ validated questions. Every rule exists because questions failed without it.

Java

| Rule | Applies To | What To Do | |------|-----------|------------| | Scanner EOF guard | java_program | ALWAYS call hasNextLine()/hasNextInt() before EVERY Scanner read. #1 Java failure cause. | | Class name | java_program | Class MUST be named Answer | | Method-only solution | java_method | Solution MUST be ONLY the method(s) -- NO class wrapper, NO main(), NO import statements. The buildSource step wraps it in public class Answer { main() { testcode } }. Including public class or main() causes compilation errors. | | Class qualifier in tests | java_class | Test code runs in a separate __Tester__ class. Call static methods as ClassName.method(). Create objects as ClassName obj = new ClassName(); obj.method(). |

Python

| Rule | Applies To | What To Do | |------|-----------|------------| | EOF guard | python3_w_input | Use import sys; data = sys.stdin.read().strip() with if data: guard. Never bare input(). | | Function-only solution | python3 | Solution is function/class definitions only. Test code calls them with print(). |

C

| Rule | Applies To | What To Do | |------|-----------|------------| | -Werror | ALL C types | Jobe compiles with -Werror. ALL warnings are fatal. No unused variables, no implicit declarations. | | #include in answer | c_function | ALL headers (`, , , , ) MUST be in the solution, not just test code. | | scanf return check | cprogram | Always: if (scanf("%d", &n) == 1) { ... }. Unchecked scanf on empty stdin = uninitialized variable. | | Test code wrapper | cfunction | Test code MUST include #include and int main(void) { ... return 0; } | | Array printing | ALL C types | Use if (i) printf(" "); printf("%d", arr[i]); -- NOT printf("%d ", arr[i]) (trailing space fails). | | Function-only solution | c_function` | Solution is function(s) with #include headers only. NO main(). Test code provides main(). |

C++

| Rule | Applies To | What To Do | |------|-----------|------------| | -Werror | ALL C++ types | Same as C -- all warnings fatal. | | size_t for .size() | ALL C++ types | for (size_t i = 0; i , `, , etc. in the solution, not just test code. | | Test code wrapper | cpp_function | Test code MUST include #include , using namespace std;, and int main() { ... } | | Array printing | ALL C++ types | if (i) cout ` tag should NOT include a trailing newline.

Escaping for curl -d

The sourcecode and input fields are inside a JSON string inside a bash command. You MUST properly escape:

• Backslashes: \ becomes \\
• Double quotes: " becomes \"
• Newlines: actual newlines become \n
• Tabs: actual tabs become \t
• Dollar signs in bash: use single-quoted heredoc to avoid shell expansion

Recommended pattern: Write the JSON payload to a temp file, then curl with --data @file:

# Write payload to temp file (avoids all escaping issues)
cat > /tmp/jobe_payload.json Penalty Regime:

You have a total of 3 attempts without any penalties.
Starting from the 4th attempt onwards, a penalty of 10% will be applied.

Test Case Requirements

• Minimum 3 test cases
• First test case: visible (useasexample="1", display SHOW)
• Remaining test cases: hidden (useasexample="0", display HIDE)
• Each test MUST produce DIFFERENT output (prevents hard-coding)
• Include at least one edge case (empty input, zero, negative, boundary)
• For program types with stdin: include empty stdin test case
• For partial credit (allornothing="0"): marks MUST sum to 1.0
• Do NOT include expected output in the JSON -- Jobe will compute it

---

6. Workflow (Full Pipeline Mode)

This workflow assumes you passed the mode detection check and can run bash commands. If you cannot, follow Fallback Mode described at the top.

Step 1: Gather Requirements

Ask the user for:

• Language: Java, Python, C, C++, Node.js (and specific type if they have a preference)
• Topic: What the questions should be about
• Count: How many questions (default: 5)
• Difficulty: easy / medium / hard
• Grading mode: all-or-nothing (default) or partial credit

Step 2: Generate JSON for All Questions

For each question, generate a JSON object with this structure:

{
  "name": "Micro-Assessment - Descriptive Title",
  "qtype": "java_class",
  "question_text": "TitleInstructions with examples...Penalty Regime:You have a total of 3 attempts without any penalties.Starting from the 4th attempt onwards, a penalty of 10% will be applied.",
  "feedback": "Explanation of the correct approach.",
  "solution": "complete model solution code",
  "preload": "skeleton code for students",
  "test_cases": [
    {"testcode": "System.out.println(obj.method(arg));", "stdin": "", "visible": true},
    {"testcode": "System.out.println(obj.method(arg2));", "stdin": "", "visible": false},
    {"testcode": "System.out.println(obj.method(edge));", "stdin": "", "visible": false}
  ]
}

Do NOT include expected in test_cases. Jobe will compute it.

For program types (javaprogram, python3winput, cprogram, cpp_program):

• testcode is empty string ""
• stdin contains the input data

Step 3: Validate Each Test Case on Jobe

For each question, for each test case:

1. Build the full source code using the assembly rules from Section 4
2. Determine the language_id from Section 3
3. Determine the filename from Section 4
4. Write the Jobe payload to a temp file
5. Call Jobe via curl
6. Check the outcome:

• If outcome is 15: strip the trailing newline from stdout and save it as the expected output for this test case
• If outcome is NOT 15: the test case failed -- see Step 4

Example -- validating a python3 question:

# Question: def add(a, b): return a + b
# Test case: print(add(2, 3))
# Built source: "def add(a, b):\n    return a + b\n\nprint(add(2, 3))"

cat > /tmp/jobe_payload.json  /tmp/jobe_payload.json ` value is the stdout captured from Jobe (with trailing newline stripped).

### Step 6: Write and Confirm

1. Write the XML to a file named `{module}_{topic}_coderunner.xml`
2. Report to the user:
   - Number of questions generated
   - Number that passed validation
   - Any that failed and were excluded
   - The file path

---

## 7. XML Template

```xml

  
    $course$/Category Name
  

  
  
    QUESTION_NAME
    
    
    1
    0
    0
    
    QTYPE
    0
    1
    0, 0, 0, 10, 20, ...
    0
    0
    0
    18
    100
    
    
    
    
    
    
    
    
    1
    
    
    
    
    
    
    
    
    
    1
    1
    None
    0
    {}
    0
    
    
    0
    0
    10240
    
    
    1
    0
    
    
      
      
        TESTCODE
        STDIN
        EXPECTED_FROM_JOBE
        
        DISPLAY_VALUE
      
    
  

Field Values per Test Case

| Field | Visible test (first) | Hidden test (remaining) | |-------|---------------------|------------------------| | useasexample | 1 | 0 | | display | SHOW | HIDE | | mark | 1.0000000 (all-or-nothing) | 1.0000000 (all-or-nothing) |

For partial credit: distribute marks so they sum to 1.0 (e.g., 4 tests = 0.2500000 each).

Optional Feature Tags

| Feature | Tag | Values | Effect | |---------|-----|--------|--------| | Precheck | 1 | 0/1 | Students can test before submitting | | Give up | 1 | 0/1 | Students can reveal model answer | | CPU limit | 5 | seconds | Enforce time limit | | Memory limit | 64 | MB | Enforce memory limit |

---

8. Auto-Fix Rules to Apply Before Jobe

Before sending code to Jobe, apply these fixes to the solution and test code. These catch the most common AI generation mistakes.

Solution Fixes

| # | Fix | Applies To | What To Do | |---|-----|-----------|------------| | 1 | Strip class/main/imports | java_method | Remove any public class wrapper, main() method, and import statements. javamethod solutions must be bare static methods only. | | 2 | Wrap in public class | java_class | If solution has no class keyword, wrap in public class Answer { ... } | | 3 | Rename class to Answer | java_program | If solution has a class not named Answer, rename it | | 4 | Add Scanner hasNext guards | java_program | Wrap scanner.nextInt() with scanner.hasNextInt() ? scanner.nextInt() : 0 and scanner.nextLine() with scanner.hasNextLine() ? scanner.nextLine() : "" | | 5 | Remove accidental main() | c_function, cpp_function | Strip int main(...) block from function-type solutions | | 6 | Add missing main() | c_program, cpp_program | Append int main(void) { return 0; } if no main() exists | | 7 | Auto-close braces | java_*, c_*, cpp_*, nodejs | Count { vs } and append missing closing braces | | 8 | Wrap input() in try/except | python3_w_input | If solution uses bare input() without sys.stdin or try, wrap in try: ... except EOFError: pass | | 9 | Add missing C headers | c_function, c_program | Detect function usage and add missing #include: stdio.h, string.h, stdlib.h, math.h, ctype.h, limits.h | | 10 | Add missing C++ headers | cpp_function, cpp_program | Detect usage and add: iostream, vector, string, algorithm, map, set, stack, queue, sstream | | 11 | Fix int to sizet | cpp_function, cpp_program | Rewrite for (int i = 0; i + int main(void) { ... return 0; } | | 2 | Wrap C++ test code | cpp_function | If test code lacks int main, wrap with #include + using namespace std; + int main() { ... } | | 3 | Fix unmatched parens | All types | If closing parens exceed opening parens, trim trailing ) |

Apply these fixes programmatically before building the source and calling Jobe.

---

9. Complete Worked Example

Here is a complete end-to-end example generating one Python question.

9a. Generate JSON

{
  "name": "Micro-Assessment - Sum of Even Numbers",
  "qtype": "python3",
  "question_text": "Sum of Even NumbersWrite a function sum_evens(nums) that takes a list of integers and returns the sum of all even numbers in the list.Examplesum_evens([1, 2, 3, 4, 5, 6]) -> 12Penalty Regime:You have a total of 3 attempts without any penalties.Starting from the 4th attempt onwards, a penalty of 10% will be applied.",
  "feedback": "Use a list comprehension or loop to filter even numbers (n % 2 == 0) and sum them. The built-in sum() function works well with a generator expression.",
  "solution": "def sum_evens(nums):\n    return sum(n for n in nums if n % 2 == 0)",
  "preload": "def sum_evens(nums):\n    # Write your code here\n    pass",
  "test_cases": [
    {"testcode": "print(sum_evens([1, 2, 3, 4, 5, 6]))", "stdin": "", "visible": true},
    {"testcode": "print(sum_evens([]))", "stdin": "", "visible": false},
    {"testcode": "print(sum_evens([1, 3, 5]))", "stdin": "", "visible": false},
    {"testcode": "print(sum_evens([-2, -1, 0, 1, 2]))", "stdin": "", "visible": false}
  ]
}

9b. Validate Test Case 1 on Jobe

Build source (python3 = solution + "\n\n" + testcode):

def sum_evens(nums):
    return sum(n for n in nums if n % 2 == 0)

print(sum_evens([1, 2, 3, 4, 5, 6]))

cat > /tmp/jobe_payload.json ` tag uses descriptive title starting with `Micro-Assessment -`
- [ ] Correct `` for the task
- [ ] `` contains complete, compilable model solution
- [ ] `` for java_method is raw method(s) only (no class wrapper)
- [ ] `` provides compilable skeleton
- [ ] `` explains correct approach
- [ ] `1`
- [ ] `0, 0, 0, 10, 20, ...`
- [ ] Penalty regime text in ``
- [ ] Self-contained (no lecture references)
- [ ] 3+ test cases: 1 visible, 2+ hidden, edge cases included
- [ ] Each test produces different output
- [ ] No floating-point exact comparisons
- [ ] Marks sum to 1.0 (if partial credit)
- [ ] Language-specific rules from Section 2 followed
- [ ] Auto-fix rules from Section 8 applied before Jobe calls
- [ ] **ALL test cases returned outcome 15 from Jobe** (Full Pipeline mode)
- [ ] **ALL expected values are Jobe stdout (never hand-written)** (Full Pipeline mode)
- [ ] If Fallback Mode: warned user to validate on import with `Validate on save` enabled

---

## 11. File Naming

- Questions: `{module}_{topic}_coderunner.xml`
- Example: `oop1_polymorphism_coderunner.xml`

---

## 12. Moodle Import

1. Course > Question bank > Import
2. Format: Moodle XML
3. Upload `.xml` file
4. Import and review

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [danielcregg](https://github.com/danielcregg)
- **Source:** [danielcregg/coderunner-skill](https://github.com/danielcregg/coderunner-skill)
- **License:** MIT

Install and usage instructions live in the source repository linked above.