LLM Expert Review Flow

このドキュメントは、LLM Labo の教材PRに入れる専門レビュー手順です。目的は、ChatGPT級のLLMシステムを扱う説明が、学習用の小さな実装と現実の工程を混同しないようにすることです。

Codex が継続的に利用する repo-local agent definition は .agents/llm-ai-expert-reviewer.md にあります。新しい Codex セッションでは、ルートの AGENTS.md からこの定義を参照します。

Reviewer role

LLM/AI Expert Reviewer subagent は、LLM curriculum PR をレビューする独立レビュワーです。主担当の実装者とは別視点で、次を確認します。

• 「いま動くもの」と「これから作るもの」が明確に分かれているか。
• base LM、post-training、product layer、agent layer が混同されていないか。
• ChatGPT級という表現が、実装済み能力を過大に見せていないか。
• major claim に一次情報、公式ドキュメント、代表的論文への導線があるか。
• data、evaluation、safety、serving、observability が抜け落ちていないか。

Invocation prompt

PRで ChatGPT path、LLM curriculum、RAG、tool use、agent、post-training の説明を変更した場合は、次のような subagent review を走らせます。

実行時に subagent を使える場合は、.agents/llm-ai-expert-reviewer.md の全文を reviewer prompt として渡します。subagent を使えない場合は、同じ定義を rubric としてローカルレビューします。

You are the LLM/AI Expert Reviewer subagent for this PR.
Review the changed docs and site navigation for technical realism.
Focus on whether the path presents a credible sequence toward a ChatGPT-like LLM system for a hands-on learning lab.
Check for missing stages, overclaims, unclear runnable/planned status, missing evaluation or safety gates, and weak source grounding.
Do not edit files.
Return findings ordered by severity, with concrete recommendations and file/line references where possible.
Also propose any review-flow checklist updates.

Severity rubric

Checklist

• Every Runnable now stage has a command, source file, and test.
• Every Planned stage is visibly labeled as planned in docs and site UI.
• The path separates base LM pretraining, chat runtime, SFT, preference/alignment, evaluation, serving, RAG, tool use, and agent control.
• The page states the scale boundary: tiny lab demonstrates mechanisms, not production ChatGPT capability.
• Data pipeline coverage includes cleaning, deduplication, splits, mixture design, and contamination risk.
• Post-training coverage includes SFT plus preference/alignment, not SFT alone.
• Product layer coverage includes message roles, context window, conversation history, API/streaming boundary, and logging.
• RAG coverage includes retrieval evaluation, grounding checks, and prompt-injection isolation.
• Tool/agent coverage includes schema validation, permissions, side effects, retry limits, and timeout handling.
• Navigation shows layer, chapter order, and implementation status.
• Major technical claims cite primary sources, official docs, or clearly marked concept notes.

Required response to review

After the subagent returns findings, the PR author must do one of the following for each P1/P2 item.

• Apply the fix.
• Downgrade the claim and label the section as planned or concept-only.
• Leave a short rationale in the PR body explaining why the issue is intentionally deferred.

The PR is not considered ready until the reviewer findings and the author response are reflected in the PR body or the docs.