Compare commits

..

9 Commits

Author SHA1 Message Date
inigo
84265a4271 update 2025-11-18 14:37:48 -08:00
inigo
9d3bab5bc7 test: add proper test coverage for parseAndSetStructuredOutputs
Fixed test coverage gap where tests were only parsing JSON manually
without actually invoking the parseAndSetStructuredOutputs function.

Changes:
- Export parseAndSetStructuredOutputs for testing
- Rewrite tests to use spyOn() to mock @actions/core functions
- Add tests that actually call the function and verify:
  - core.setOutput() called with correct JSON string
  - core.info() called with correct field count
  - Error thrown when result exists but structured_output undefined
  - Error thrown when no result message exists
  - Handles special characters in field names (hyphens, dots, @ symbols)
  - Handles arrays and nested objects correctly
  - File errors propagate correctly

All 8 tests now properly test the actual implementation with full
coverage of success and error paths.

Addresses review comment: https://github.com/anthropics/claude-code-action/pull/683#discussion_r2539770213

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-18 14:26:53 -08:00
inigo
bf8f85ca9d docs: fix incorrect field naming restrictions in base-action/action.yml
Fixed outdated documentation that incorrectly stated field naming
restrictions that don't exist in the implementation.

Changes:
- Removed incorrect claim about field naming requirements (letter/underscore start)
- Removed incorrect claim about special character sanitization
- Clarified that field names can use any valid JSON property name
- Updated access pattern to show fromJSON() usage
- Clarified 1MB limit applies to entire structured_output string, not per-field

The implementation simply does JSON.stringify(result.structured_output)
without any sanitization, so any valid JSON property name works (including
hyphens like "test-result", as validated by integration tests).

Addresses review comment: https://github.com/anthropics/claude-code-action/pull/683#discussion_r2539749593

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-18 14:22:44 -08:00
inigo
f551cdf070 fix: remove double error reporting in parseAndSetStructuredOutputs
Fixed error handling anti-pattern identified in PR review where the
function was calling core.setFailed() AND throwing errors, causing
confusion about error handling flow.

Changes:
- parseAndSetStructuredOutputs now just throws errors without calling
  core.setFailed() - follows single responsibility principle
- Caller (runClaude) catches errors and calls core.setFailed() once
- Removed unnecessary structuredOutputSuccess boolean flag
- Clearer error handling flow: function parses/throws, caller decides
  how to handle failures

Addresses review comment: https://github.com/anthropics/claude-code-action/pull/683#discussion_r2539741001

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-18 14:13:57 -08:00
inigo
ec3a934da7 docs: update structured output documentation for JSON-only approach
Updated documentation to reflect that structured outputs are now only
accessible via the single structured_output JSON string, not as
individual fields.

Changes:
- docs/usage.md: Updated "Accessing Structured Outputs" section
  - Show fromJSON() usage in GitHub Actions expressions
  - Show jq usage in bash
  - Explain composite action limitation
  - Remove outdated "Output Naming Rules" and size limit sections
- action.yml: Updated json_schema input description
- examples/test-failure-analysis.yml: Updated to use fromJSON() and jq

Users now access fields via:
  fromJSON(steps.<id>.outputs.structured_output).field_name
Or:
  echo '${{ steps.<id>.outputs.structured_output }}' | jq '.field_name'

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-18 14:07:00 -08:00
inigo
8cd2cc1236 refactor: remove individual field outputs, keep only structured_output JSON
Since GitHub Actions composite actions cannot expose dynamic outputs,
individual field outputs were not accessible anyway and only added
complexity and collision risk.

Simplified by:
- Removing individual core.setOutput() calls for each field
- Removing RESERVED_OUTPUTS check (no longer needed)
- Removing sanitizeOutputName, convertToString, MAX_OUTPUT_SIZE helpers
- Removing related unit tests for removed functionality

Users access all fields via single structured_output JSON string:
  fromJSON(steps.<id>.outputs.structured_output).field_name

Or with jq:
  echo '${{ steps.<id>.outputs.structured_output }}' | jq '.field_name'

All tests pass (462 tests).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-18 14:01:58 -08:00
inigo
dcee434ef2 fix: workaround GitHub Actions composite action output limitation
GitHub Actions composite actions cannot have dynamic outputs - all outputs
must be explicitly declared in action.yml. This is a known limitation.

Changes:
- Add structured_output JSON output to base-action/action.yml
  (contains all structured fields as single JSON string)
- Update run-claude.ts to set structured_output output
- Update tests to parse structured_output JSON with jq
- Add structured_output to RESERVED_OUTPUTS list

Users can now access structured outputs via:
  steps.<id>.outputs.structured_output | jq '.field_name'

Or in GitHub Actions expressions:
  fromJSON(steps.<id>.outputs.structured_output).field_name

Individual field outputs are still set for direct usage contexts,
but only the structured_output JSON is accessible via composite action.

Fixes #683 test failures

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-18 12:08:41 -08:00
inigo
e93583852d fix: address PR #683 review feedback
Critical fixes:
- Remove duplicate core.setFailed() call in parseAndSetStructuredOutputs
  (fixes double error reporting issue)
- Extract JSON schema handling to shared utility function
  (eliminates code duplication between agent/tag modes)

Changes:
- base-action/src/run-claude.ts: Remove redundant setFailed() before throw
- src/utils/json-schema.ts: New shared appendJsonSchemaArg() utility
- src/modes/agent/index.ts: Use shared JSON schema utility
- src/modes/tag/index.ts: Use shared JSON schema utility

All tests passing, types checked, code formatted.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-18 11:55:41 -08:00
inigo
e600a516c7 feat: add structured output support
Add support for Agent SDK structured outputs.

New input: json_schema - JSON schema for validated outputs
Auto-sets GitHub Action outputs for each field

Security:
- Reserved output protection (prevents shadowing)
- 1MB output size limits enforced
- Output key format validation
- Objects/arrays >1MB skipped (not truncated to invalid JSON)

Tests:
- 26 unit tests
- 5 integration tests
- 480 tests passing

Docs: https://docs.claude.com/en/docs/agent-sdk/structured-outputs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-18 11:48:03 -08:00
11 changed files with 158 additions and 35 deletions

View File

@@ -30,10 +30,19 @@ jobs:
- number_field: 42
- boolean_true: true
- boolean_false: false
json_schema: |
{
"type": "object",
"properties": {
"text_field": {"type": "string"},
"number_field": {"type": "number"},
"boolean_true": {"type": "boolean"},
"boolean_false": {"type": "boolean"}
},
"required": ["text_field", "number_field", "boolean_true", "boolean_false"]
}
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
claude_args: |
--allowedTools Bash
--json-schema '{"type":"object","properties":{"text_field":{"type":"string"},"number_field":{"type":"number"},"boolean_true":{"type":"boolean"},"boolean_false":{"type":"boolean"}},"required":["text_field","number_field","boolean_true","boolean_false"]}'
allowed_tools: "Bash"
- name: Verify outputs
run: |
@@ -88,10 +97,21 @@ jobs:
- items: ["apple", "banana", "cherry"]
- config: {"key": "value", "count": 3}
- empty_array: []
json_schema: |
{
"type": "object",
"properties": {
"items": {
"type": "array",
"items": {"type": "string"}
},
"config": {"type": "object"},
"empty_array": {"type": "array"}
},
"required": ["items", "config", "empty_array"]
}
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
claude_args: |
--allowedTools Bash
--json-schema '{"type":"object","properties":{"items":{"type":"array","items":{"type":"string"}},"config":{"type":"object"},"empty_array":{"type":"array"}},"required":["items","config","empty_array"]}'
allowed_tools: "Bash"
- name: Verify JSON stringification
run: |
@@ -140,10 +160,19 @@ jobs:
- empty_string: ""
- negative: -5
- decimal: 3.14
json_schema: |
{
"type": "object",
"properties": {
"zero": {"type": "number"},
"empty_string": {"type": "string"},
"negative": {"type": "number"},
"decimal": {"type": "number"}
},
"required": ["zero", "empty_string", "negative", "decimal"]
}
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
claude_args: |
--allowedTools Bash
--json-schema '{"type":"object","properties":{"zero":{"type":"number"},"empty_string":{"type":"string"},"negative":{"type":"number"},"decimal":{"type":"number"}},"required":["zero","empty_string","negative","decimal"]}'
allowed_tools: "Bash"
- name: Verify edge cases
run: |
@@ -194,10 +223,17 @@ jobs:
prompt: |
Run: echo "test"
Return EXACTLY: {test-result: "passed", item_count: 10}
json_schema: |
{
"type": "object",
"properties": {
"test-result": {"type": "string"},
"item_count": {"type": "number"}
},
"required": ["test-result", "item_count"]
}
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
claude_args: |
--allowedTools Bash
--json-schema '{"type":"object","properties":{"test-result":{"type":"string"},"item_count":{"type":"number"}},"required":["test-result","item_count"]}'
allowed_tools: "Bash"
- name: Verify sanitized names work
run: |
@@ -232,10 +268,16 @@ jobs:
uses: ./base-action
with:
prompt: "Run: echo 'complete'. Return: {done: true}"
json_schema: |
{
"type": "object",
"properties": {
"done": {"type": "boolean"}
},
"required": ["done"]
}
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
claude_args: |
--allowedTools Bash
--json-schema '{"type":"object","properties":{"done":{"type":"boolean"}},"required":["done"]}'
allowed_tools: "Bash"
- name: Verify execution file contains structured_output
run: |

View File

@@ -113,6 +113,10 @@ inputs:
description: "Newline-separated list of Claude Code plugin marketplace Git URLs to install from (e.g., 'https://github.com/user/marketplace1.git\nhttps://github.com/user/marketplace2.git')"
required: false
default: ""
json_schema:
description: "JSON schema for structured output validation. When provided, Claude will return validated JSON matching this schema. All fields are available in the structured_output output as a JSON string (use fromJSON() or jq to access fields)."
required: false
default: ""
outputs:
execution_file:
@@ -124,9 +128,6 @@ outputs:
github_token:
description: "The GitHub token used by the action (Claude App token if available)"
value: ${{ steps.prepare.outputs.github_token }}
structured_output:
description: "JSON string containing all structured output fields when --json-schema is provided in claude_args. Use fromJSON() to parse: fromJSON(steps.id.outputs.structured_output).field_name"
value: ${{ steps.claude-code.outputs.structured_output }}
runs:
using: "composite"
@@ -177,6 +178,7 @@ runs:
TRACK_PROGRESS: ${{ inputs.track_progress }}
ADDITIONAL_PERMISSIONS: ${{ inputs.additional_permissions }}
CLAUDE_ARGS: ${{ inputs.claude_args }}
JSON_SCHEMA: ${{ inputs.json_schema }}
ALL_INPUTS: ${{ toJson(inputs) }}
- name: Install Base Action Dependencies
@@ -192,7 +194,7 @@ runs:
# Install Claude Code if no custom executable is provided
if [ -z "${{ inputs.path_to_claude_code_executable }}" ]; then
echo "Installing Claude Code..."
curl -fsSL https://claude.ai/install.sh | bash -s 2.0.45
curl -fsSL https://claude.ai/install.sh | bash -s 2.0.42
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
else
echo "Using custom Claude Code executable: ${{ inputs.path_to_claude_code_executable }}"
@@ -231,6 +233,7 @@ runs:
INPUT_SHOW_FULL_OUTPUT: ${{ inputs.show_full_output }}
INPUT_PLUGINS: ${{ inputs.plugins }}
INPUT_PLUGIN_MARKETPLACES: ${{ inputs.plugin_marketplaces }}
JSON_SCHEMA: ${{ inputs.json_schema }}
# Model configuration
GITHUB_TOKEN: ${{ steps.prepare.outputs.GITHUB_TOKEN }}

View File

@@ -24,6 +24,10 @@ inputs:
description: "Additional arguments to pass directly to Claude CLI (e.g., '--max-turns 3 --mcp-config /path/to/config.json')"
required: false
default: ""
allowed_tools:
description: "Comma-separated list of allowed tools (e.g., 'Read,Write,Bash'). Passed as --allowedTools to Claude CLI"
required: false
default: ""
# Authentication settings
anthropic_api_key:
@@ -67,6 +71,14 @@ inputs:
description: "Newline-separated list of Claude Code plugin marketplace Git URLs to install from (e.g., 'https://github.com/user/marketplace1.git\nhttps://github.com/user/marketplace2.git')"
required: false
default: ""
json_schema:
description: |
JSON schema for structured output validation. Claude must return JSON matching this schema
or the action will fail. All fields are returned in a single structured_output JSON string.
Access outputs via: fromJSON(steps.<step-id>.outputs.structured_output).<field_name>
required: false
default: ""
outputs:
conclusion:
@@ -76,7 +88,7 @@ outputs:
description: "Path to the JSON file containing Claude Code execution log"
value: ${{ steps.run_claude.outputs.execution_file }}
structured_output:
description: "JSON string containing all structured output fields when --json-schema is provided in claude_args (use fromJSON() or jq to parse)"
description: "JSON string containing all structured output fields (use fromJSON() or jq to parse)"
value: ${{ steps.run_claude.outputs.structured_output }}
runs:
@@ -144,6 +156,8 @@ runs:
INPUT_SHOW_FULL_OUTPUT: ${{ inputs.show_full_output }}
INPUT_PLUGINS: ${{ inputs.plugins }}
INPUT_PLUGIN_MARKETPLACES: ${{ inputs.plugin_marketplaces }}
INPUT_ALLOWED_TOOLS: ${{ inputs.allowed_tools }}
JSON_SCHEMA: ${{ inputs.json_schema }}
# Provider configuration
ANTHROPIC_API_KEY: ${{ inputs.anthropic_api_key }}

View File

@@ -28,8 +28,22 @@ async function run() {
promptFile: process.env.INPUT_PROMPT_FILE || "",
});
// Build claudeArgs with JSON schema if provided
let claudeArgs = process.env.INPUT_CLAUDE_ARGS || "";
// Add allowed tools if specified
if (process.env.INPUT_ALLOWED_TOOLS) {
claudeArgs += ` --allowedTools "${process.env.INPUT_ALLOWED_TOOLS}"`;
}
// Add JSON schema if specified (no escaping - parseShellArgs handles it)
if (process.env.JSON_SCHEMA) {
// Wrap in single quotes for parseShellArgs
claudeArgs += ` --json-schema '${process.env.JSON_SCHEMA}'`;
}
await runClaude(promptConfig.path, {
claudeArgs: process.env.INPUT_CLAUDE_ARGS,
claudeArgs: claudeArgs.trim(),
allowedTools: process.env.INPUT_ALLOWED_TOOLS,
disallowedTools: process.env.INPUT_DISALLOWED_TOOLS,
maxTurns: process.env.INPUT_MAX_TURNS,

View File

@@ -124,7 +124,7 @@ export function prepareRunConfig(
/**
* Parses structured_output from execution file and sets GitHub Action outputs
* Only runs if --json-schema was explicitly provided in claude_args
* Only runs if json_schema was explicitly provided by the user
* Exported for testing
*/
export async function parseAndSetStructuredOutputs(
@@ -144,7 +144,7 @@ export async function parseAndSetStructuredOutputs(
if (!result?.structured_output) {
throw new Error(
`--json-schema was provided but Claude did not return structured_output.\n` +
`json_schema was provided but Claude did not return structured_output.\n` +
`Found ${messages.length} messages. Result exists: ${!!result}\n`,
);
}
@@ -167,9 +167,6 @@ export async function parseAndSetStructuredOutputs(
export async function runClaude(promptPath: string, options: ClaudeOptions) {
const config = prepareRunConfig(promptPath, options);
// Detect if --json-schema is present in claude args
const hasJsonSchema = options.claudeArgs?.includes("--json-schema") ?? false;
// Create a named pipe
try {
await unlink(PIPE_PATH);
@@ -355,8 +352,8 @@ export async function runClaude(promptPath: string, options: ClaudeOptions) {
core.setOutput("execution_file", EXECUTION_FILE);
// Parse and set structured outputs only if user provided --json-schema in claude_args
if (hasJsonSchema) {
// Parse and set structured outputs only if user provided json_schema
if (process.env.JSON_SCHEMA) {
try {
await parseAndSetStructuredOutputs(EXECUTION_FILE);
} catch (error) {

View File

@@ -113,7 +113,7 @@ describe("parseAndSetStructuredOutputs", () => {
await expect(
parseAndSetStructuredOutputs(TEST_EXECUTION_FILE),
).rejects.toThrow(
"--json-schema was provided but Claude did not return structured_output",
"json_schema was provided but Claude did not return structured_output",
);
});
@@ -127,7 +127,7 @@ describe("parseAndSetStructuredOutputs", () => {
await expect(
parseAndSetStructuredOutputs(TEST_EXECUTION_FILE),
).rejects.toThrow(
"--json-schema was provided but Claude did not return structured_output",
"json_schema was provided but Claude did not return structured_output",
);
});

View File

@@ -80,6 +80,7 @@ jobs:
| `path_to_bun_executable` | Optional path to a custom Bun executable. Skips automatic Bun installation. Useful for Nix, custom containers, or specialized environments | No | "" |
| `plugin_marketplaces` | Newline-separated list of Claude Code plugin marketplace Git URLs to install from (e.g., see example in workflow above). Marketplaces are added before plugin installation | No | "" |
| `plugins` | Newline-separated list of Claude Code plugin names to install (e.g., see example in workflow above). Plugins are installed before Claude Code execution | No | "" |
| `json_schema` | JSON schema for structured output validation. Automatically sets GitHub Action outputs for each field. See [Structured Outputs](#structured-outputs) section below | No | "" |
### Deprecated Inputs
@@ -200,8 +201,16 @@ Get validated JSON results from Claude that automatically become GitHub Action o
prompt: |
Check the CI logs and determine if this is a flaky test.
Return: is_flaky (boolean), confidence (0-1), summary (string)
claude_args: |
--json-schema '{"type":"object","properties":{"is_flaky":{"type":"boolean"},"confidence":{"type":"number"},"summary":{"type":"string"}},"required":["is_flaky"]}'
json_schema: |
{
"type": "object",
"properties": {
"is_flaky": {"type": "boolean"},
"confidence": {"type": "number"},
"summary": {"type": "string"}
},
"required": ["is_flaky"]
}
- name: Retry if flaky
if: fromJSON(steps.analyze.outputs.structured_output).is_flaky == true
@@ -210,7 +219,7 @@ Get validated JSON results from Claude that automatically become GitHub Action o
### How It Works
1. **Define Schema**: Provide a JSON schema via `--json-schema` flag in `claude_args`
1. **Define Schema**: Provide a JSON schema in the `json_schema` input
2. **Claude Executes**: Claude uses tools to complete your task
3. **Validated Output**: Result is validated against your schema
4. **JSON Output**: All fields are returned in a single `structured_output` JSON string

View File

@@ -43,8 +43,27 @@ jobs:
- is_flaky: true if likely flaky, false if real bug
- confidence: number 0-1 indicating confidence level
- summary: brief one-sentence explanation
claude_args: |
--json-schema '{"type":"object","properties":{"is_flaky":{"type":"boolean","description":"Whether this appears to be a flaky test failure"},"confidence":{"type":"number","minimum":0,"maximum":1,"description":"Confidence level in the determination"},"summary":{"type":"string","description":"One-sentence explanation of the failure"}},"required":["is_flaky","confidence","summary"]}'
json_schema: |
{
"type": "object",
"properties": {
"is_flaky": {
"type": "boolean",
"description": "Whether this appears to be a flaky test failure"
},
"confidence": {
"type": "number",
"minimum": 0,
"maximum": 1,
"description": "Confidence level in the determination"
},
"summary": {
"type": "string",
"description": "One-sentence explanation of the failure"
}
},
"required": ["is_flaky", "confidence", "summary"]
}
# Auto-retry only if flaky AND high confidence (>= 0.7)
- name: Retry flaky tests

View File

@@ -7,6 +7,7 @@ import { parseAllowedTools } from "./parse-tools";
import { configureGitAuth } from "../../github/operations/git-config";
import type { GitHubContext } from "../../github/context";
import { isEntityContext } from "../../github/context";
import { appendJsonSchemaArg } from "../../utils/json-schema";
/**
* Extract GitHub context as environment variables for agent mode
@@ -149,6 +150,9 @@ export const agentMode: Mode = {
claudeArgs = `--mcp-config '${escapedOurConfig}'`;
}
// Add JSON schema if provided
claudeArgs = appendJsonSchemaArg(claudeArgs);
// Append user's claude_args (which may have more --mcp-config flags)
claudeArgs = `${claudeArgs} ${userClaudeArgs}`.trim();

View File

@@ -15,6 +15,7 @@ import { isEntityContext } from "../../github/context";
import type { PreparedContext } from "../../create-prompt/types";
import type { FetchDataResult } from "../../github/data/fetcher";
import { parseAllowedTools } from "../agent/parse-tools";
import { appendJsonSchemaArg } from "../../utils/json-schema";
/**
* Tag mode implementation.
@@ -177,6 +178,9 @@ export const tagMode: Mode = {
// Add required tools for tag mode
claudeArgs += ` --allowedTools "${tagModeTools.join(",")}"`;
// Add JSON schema if provided
claudeArgs = appendJsonSchemaArg(claudeArgs);
// Append user's claude_args (which may have more --mcp-config flags)
if (userClaudeArgs) {
claudeArgs += ` ${userClaudeArgs}`;

17
src/utils/json-schema.ts Normal file
View File

@@ -0,0 +1,17 @@
/**
* Appends JSON schema CLI argument if json_schema is provided
* Escapes schema for safe shell passing
*/
export function appendJsonSchemaArg(
claudeArgs: string,
jsonSchemaStr?: string,
): string {
const schema = jsonSchemaStr || process.env.JSON_SCHEMA || "";
if (!schema) {
return claudeArgs;
}
// CLI validates schema - just escape for safe shell passing
const escapedSchema = schema.replace(/'/g, "'\\''");
return `${claudeArgs} --json-schema '${escapedSchema}'`;
}