Initial commit: copilot-skill-factory v0.1.0
Meta-plugin for GitHub Copilot CLI that watches workflows and auto-generates reusable agent skills. - postToolUse hook: fingerprints and logs tool calls per session - sessionEnd hook: sliding-window pattern detection across sessions - sessionStart hook: injects context about proposed skills - /skill-factory slash command for reviewing/managing skills - 13 passing tests validating observer, analyzer, and edge cases Inspired by hermes-skill-factory (Romanescu11/hermes-skill-factory)
This commit is contained in:
@@ -0,0 +1,4 @@
|
|||||||
|
node_modules/
|
||||||
|
.data/
|
||||||
|
*.log
|
||||||
|
.DS_Store
|
||||||
@@ -0,0 +1,194 @@
|
|||||||
|
# 🏭 Copilot Skill Factory
|
||||||
|
|
||||||
|
> **The meta-plugin that creates skills.**
|
||||||
|
> Watches your GitHub Copilot CLI workflows and automatically turns them into reusable agent skills.
|
||||||
|
|
||||||
|
Built for [GitHub Copilot CLI](https://docs.github.com/copilot/concepts/agents/about-copilot-cli) (v1.0.52+).
|
||||||
|
|
||||||
|
Inspired by [hermes-skill-factory](https://github.com/Romanescu11/hermes-skill-factory) for [Hermes Agent](https://github.com/NousResearch/hermes-agent).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What It Does
|
||||||
|
|
||||||
|
Every time you solve a problem with Copilot CLI, you're performing a workflow worth repeating. Skill Factory watches silently via hooks, detects patterns, and proposes reusable skills.
|
||||||
|
|
||||||
|
```
|
||||||
|
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
🏭 SKILL FACTORY — New Pattern Detected
|
||||||
|
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
|
||||||
|
You've repeated this workflow in 3 sessions:
|
||||||
|
|
||||||
|
1. bash (git commands)
|
||||||
|
2. grep (search code)
|
||||||
|
3. edit (apply changes)
|
||||||
|
4. bash (run tests)
|
||||||
|
|
||||||
|
Proposed Skill: git-edit-test
|
||||||
|
Status: Ready for review
|
||||||
|
|
||||||
|
Use /skill-factory review to inspect it.
|
||||||
|
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
```
|
||||||
|
|
||||||
|
## How It Works
|
||||||
|
|
||||||
|
### Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────────────────────────────────┐
|
||||||
|
│ Copilot CLI Session │
|
||||||
|
│ │
|
||||||
|
│ postToolUse ──┐ sessionEnd ──┐ │
|
||||||
|
│ ▼ ▼ │
|
||||||
|
│ ┌─────────────┐ ┌─────────────┐ │
|
||||||
|
│ │ observer.js │ │ analyzer │ │
|
||||||
|
│ │ (logs tools)│ │ (detects │ │
|
||||||
|
│ └──────┬───────┘ │ patterns) │ │
|
||||||
|
│ │ └──────┬──────┘ │
|
||||||
|
│ ▼ ▼ │
|
||||||
|
│ ┌─────────────────────────────┐ │
|
||||||
|
│ │ pattern-db.json │ │
|
||||||
|
│ │ (cross-session patterns) │ │
|
||||||
|
│ └─────────────┬───────────────┘ │
|
||||||
|
│ ▼ │
|
||||||
|
│ ┌─────────────────────────────┐ │
|
||||||
|
│ │ proposed/<name>/SKILL.md │ │
|
||||||
|
│ │ (auto-generated skills) │ │
|
||||||
|
│ └─────────────────────────────┘ │
|
||||||
|
└──────────────────────────────────────────┘
|
||||||
|
|
||||||
|
sessionStart ──► Injects notification about new skills
|
||||||
|
|
||||||
|
/skill-factory ──► User reviews & manages skills
|
||||||
|
```
|
||||||
|
|
||||||
|
### Plugin Components
|
||||||
|
|
||||||
|
| Component | File | Purpose |
|
||||||
|
|---|---|---|
|
||||||
|
| **Manifest** | `plugin.json` | Plugin metadata (name, version, hooks path) |
|
||||||
|
| **Hooks config** | `hooks/hooks.json` | Defines `postToolUse`, `sessionEnd`, `sessionStart` hooks |
|
||||||
|
| **Observer** | `bin/observer.js` | Main hook handler — logs, analyzes, notifies |
|
||||||
|
| **Skill** | `skills/skill-factory/SKILL.md` | `/skill-factory` slash command for user interaction |
|
||||||
|
|
||||||
|
### Detection Algorithm
|
||||||
|
|
||||||
|
1. **postToolUse hook** — Every successful tool call is fingerprinted (tool name + normalized args) and logged to a per-session JSONL file
|
||||||
|
2. **sessionEnd hook** — Analyzes the session using sliding windows (3-8 consecutive tool calls). Matching sequences across 2+ sessions are flagged as "patterns"
|
||||||
|
3. **Auto-generation** — Patterns seen in 2+ different sessions get a proposed `SKILL.md` generated automatically
|
||||||
|
4. **sessionStart hook** — Next session, the agent is notified about pending proposals via `additionalContext`
|
||||||
|
5. **User review** — The `/skill-factory` skill lets the user review, accept, edit, or dismiss proposals
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
|
||||||
|
- [GitHub Copilot CLI](https://docs.github.com/copilot/concepts/agents/about-copilot-cli) v1.0.52+ installed and authenticated
|
||||||
|
- Node.js 18+ (for the hook scripts)
|
||||||
|
|
||||||
|
### Via Copilot CLI plugin install (recommended)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
copilot plugin install droideparanoico/copilot-skill-factory
|
||||||
|
```
|
||||||
|
|
||||||
|
Then install the companion skill:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mkdir -p ~/.copilot/skills/skill-factory
|
||||||
|
cp skills/skill-factory/SKILL.md ~/.copilot/skills/skill-factory/
|
||||||
|
```
|
||||||
|
|
||||||
|
### Manual installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Clone the repo
|
||||||
|
git clone https://github.com/droideparanoico/copilot-skill-factory.git
|
||||||
|
cd copilot-skill-factory
|
||||||
|
|
||||||
|
# Install plugin
|
||||||
|
mkdir -p ~/.copilot/plugins/skill-factory
|
||||||
|
cp plugin.json ~/.copilot/plugins/skill-factory/
|
||||||
|
cp -r hooks/ ~/.copilot/plugins/skill-factory/
|
||||||
|
cp -r bin/ ~/.copilot/plugins/skill-factory/
|
||||||
|
chmod +x ~/.copilot/plugins/skill-factory/bin/observer.js
|
||||||
|
|
||||||
|
# Install companion skill
|
||||||
|
mkdir -p ~/.copilot/skills/skill-factory
|
||||||
|
cp skills/skill-factory/SKILL.md ~/.copilot/skills/skill-factory/
|
||||||
|
```
|
||||||
|
|
||||||
|
Then restart Copilot CLI.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
Once installed, Skill Factory runs silently in the background. Use the slash command to interact:
|
||||||
|
|
||||||
|
| Command | What it does |
|
||||||
|
|---|---|
|
||||||
|
| `/skill-factory` | Show available commands |
|
||||||
|
| `/skill-factory review` | Review proposed skills (accept/edit/dismiss) |
|
||||||
|
| `/skill-factory propose` | Force-analyze current session patterns |
|
||||||
|
| `/skill-factory list` | List all generated skills |
|
||||||
|
| `/skill-factory status` | Show detection statistics |
|
||||||
|
| `/skill-factory save <name>` | Save with custom name |
|
||||||
|
| `/skill-factory dismiss <name>` | Dismiss a proposal |
|
||||||
|
| `/skill-factory clear` | Reset all data |
|
||||||
|
|
||||||
|
You can also tell Copilot naturally:
|
||||||
|
- *"Review my skill factory proposals"*
|
||||||
|
- *"Save this workflow as a skill"*
|
||||||
|
- *"Turn this into a reusable skill"*
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Skill Factory stores data in:
|
||||||
|
|
||||||
|
```
|
||||||
|
~/.copilot/skills/skill-factory/.data/
|
||||||
|
├── pattern-db.json # Cross-session pattern database
|
||||||
|
└── sessions/
|
||||||
|
└── <session-id>.jsonl # Per-session tool call logs (kept for last 20 sessions)
|
||||||
|
```
|
||||||
|
|
||||||
|
Proposed skills are stored in:
|
||||||
|
|
||||||
|
```
|
||||||
|
~/.copilot/skills/skill-factory/proposed/
|
||||||
|
└── <skill-name>/
|
||||||
|
├── SKILL.md # Generated skill definition
|
||||||
|
└── .proposal.json # Detection metadata
|
||||||
|
```
|
||||||
|
|
||||||
|
## Limitations
|
||||||
|
|
||||||
|
- **Not real-time**: Patterns are analyzed at session end, not during the session
|
||||||
|
- **Heuristic-based**: Detection uses fingerprint similarity — similar but not identical workflows may not match
|
||||||
|
- **No cross-device sync**: Pattern database is local to each machine
|
||||||
|
- **Windows**: PowerShell hooks are not yet implemented (contributions welcome!)
|
||||||
|
|
||||||
|
## Roadmap
|
||||||
|
|
||||||
|
- [ ] Real-time pattern suggestion during sessions
|
||||||
|
- [ ] Cross-device sync via GitHub Gist
|
||||||
|
- [ ] Pattern confidence scoring
|
||||||
|
- [ ] Automatic skill refinement (merge similar patterns)
|
||||||
|
- [ ] Web dashboard for pattern visualization
|
||||||
|
- [ ] Export/import pattern database
|
||||||
|
|
||||||
|
## Comparison with hermes-skill-factory
|
||||||
|
|
||||||
|
| Feature | hermes-skill-factory | copilot-skill-factory |
|
||||||
|
|---|---|---|
|
||||||
|
| Platform | Hermes Agent | GitHub Copilot CLI |
|
||||||
|
| Observation | In-process (Python plugin) | Hooks (external Node.js) |
|
||||||
|
| Detection | AI-assisted pattern analysis | Heuristic fingerprint matching |
|
||||||
|
| Skill format | SKILL.md + plugin.py | SKILL.md (Copilot agent skill) |
|
||||||
|
| User interaction | Inline prompt during session | Slash command + sessionStart context |
|
||||||
|
| Slash commands | `/skill-factory propose`, etc. | `/skill-factory review`, etc. |
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
MIT
|
||||||
Executable
+526
@@ -0,0 +1,526 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
|
||||||
|
/**
|
||||||
|
* copilot-skill-factory — Observer
|
||||||
|
*
|
||||||
|
* Handles three hook events dispatched by Copilot CLI:
|
||||||
|
* postToolUse — logs every successful tool call to a per-session JSONL file
|
||||||
|
* sessionEnd — analyzes the session log, updates the pattern database
|
||||||
|
* sessionStart — injects additionalContext about recently detected patterns
|
||||||
|
*
|
||||||
|
* Input: JSON payload on stdin (camelCase format from hooks.json)
|
||||||
|
* Output: JSON to stdout for hooks that support output (sessionStart → additionalContext)
|
||||||
|
*/
|
||||||
|
|
||||||
|
const fs = require('fs');
|
||||||
|
const path = require('path');
|
||||||
|
const crypto = require('crypto');
|
||||||
|
|
||||||
|
// ── helpers ────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
function readStdin() {
|
||||||
|
try {
|
||||||
|
const raw = fs.readFileSync(0, 'utf8');
|
||||||
|
if (!raw.trim()) return null;
|
||||||
|
return JSON.parse(raw);
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function ensureDir(dir) {
|
||||||
|
fs.mkdirSync(dir, { recursive: true });
|
||||||
|
}
|
||||||
|
|
||||||
|
function dataDir() {
|
||||||
|
const fromEnv = process.env.COPILOT_PLUGIN_DATA || process.env.CLAUDE_PLUGIN_DATA;
|
||||||
|
if (fromEnv && fromEnv.trim() && !fromEnv.trim().startsWith('${')) return fromEnv;
|
||||||
|
// Fallback: use plugin root directory
|
||||||
|
const root = process.env.PLUGIN_ROOT || process.env.COPILOT_PLUGIN_ROOT || process.env.CLAUDE_PLUGIN_ROOT || __dirname;
|
||||||
|
return path.join(root, '.data');
|
||||||
|
}
|
||||||
|
|
||||||
|
function skillsDir() {
|
||||||
|
const home = process.env.HOME || process.env.USERPROFILE || '/tmp';
|
||||||
|
return path.join(home, '.copilot', 'skills', 'skill-factory');
|
||||||
|
}
|
||||||
|
|
||||||
|
function dbPath() {
|
||||||
|
return path.join(dataDir(), 'pattern-db.json');
|
||||||
|
}
|
||||||
|
|
||||||
|
function sessionLogPath(sessionId) {
|
||||||
|
return path.join(dataDir(), 'sessions', `${sessionId}.jsonl`);
|
||||||
|
}
|
||||||
|
|
||||||
|
function loadDb() {
|
||||||
|
try {
|
||||||
|
return JSON.parse(fs.readFileSync(dbPath(), 'utf8'));
|
||||||
|
} catch {
|
||||||
|
return { patterns: {}, sessions: {}, meta: { version: 1, createdAt: new Date().toISOString() } };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function saveDb(db) {
|
||||||
|
ensureDir(dataDir());
|
||||||
|
fs.writeFileSync(dbPath(), JSON.stringify(db, null, 2));
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── tool-call fingerprinting ───────────────────────────────────────────────
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Creates a fingerprint for a tool call — captures the "what" without the
|
||||||
|
* session-specific parameters (file paths, timestamps, exact content).
|
||||||
|
* This lets us match similar tool calls across sessions.
|
||||||
|
*/
|
||||||
|
function fingerprint(toolName, toolArgs) {
|
||||||
|
const normalized = normalizeArgs(toolName, toolArgs);
|
||||||
|
return `${toolName}:${hash(normalized)}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function normalizeArgs(toolName, toolArgs) {
|
||||||
|
if (!toolArgs || typeof toolArgs !== 'object') return '{}';
|
||||||
|
|
||||||
|
const args = { ...toolArgs };
|
||||||
|
|
||||||
|
// Strip file paths (replace with placeholder based on extension)
|
||||||
|
const fileFields = ['path', 'filePath', 'file_path', 'file', 'output', 'target'];
|
||||||
|
for (const f of fileFields) {
|
||||||
|
if (args[f] && typeof args[f] === 'string') {
|
||||||
|
const ext = path.extname(args[f]);
|
||||||
|
args[f] = `<file${ext}>`;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Strip command content (replace with command length)
|
||||||
|
if (args.command && typeof args.command === 'string') {
|
||||||
|
args.command = `<cmd:${args.command.length}>`;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Strip content/patch bodies (replace with length)
|
||||||
|
const contentFields = ['content', 'new_string', 'old_string', 'new_str', 'old_str',
|
||||||
|
'patch', 'text', 'body', 'message', 'code', 'script'];
|
||||||
|
for (const f of contentFields) {
|
||||||
|
if (args[f] && typeof args[f] === 'string') {
|
||||||
|
args[f] = `<content:${args[f].length}>`;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Simplify arrays
|
||||||
|
for (const [k, v] of Object.entries(args)) {
|
||||||
|
if (Array.isArray(v)) args[k] = `<array:${v.length}>`;
|
||||||
|
}
|
||||||
|
|
||||||
|
return JSON.stringify(args, Object.keys(args).sort());
|
||||||
|
}
|
||||||
|
|
||||||
|
function hash(str) {
|
||||||
|
return crypto.createHash('md5').update(str).digest('hex').slice(0, 12);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── postToolUse handler ────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
function handlePostToolUse(input) {
|
||||||
|
const { sessionId, toolName, toolArgs } = input;
|
||||||
|
|
||||||
|
// Skip noisy/redundant tools
|
||||||
|
const skipTools = ['todo_write', 'update_todo', 'TodoWrite', 'ask_user', 'AskUserQuestion'];
|
||||||
|
if (skipTools.includes(toolName)) return;
|
||||||
|
|
||||||
|
const fp = fingerprint(toolName, toolArgs);
|
||||||
|
const entry = {
|
||||||
|
ts: new Date().toISOString(),
|
||||||
|
sessionId,
|
||||||
|
toolName,
|
||||||
|
fingerprint: fp,
|
||||||
|
// Store minimal args for context (no full content)
|
||||||
|
argsSummary: summarizeArgs(toolName, toolArgs)
|
||||||
|
};
|
||||||
|
|
||||||
|
ensureDir(path.dirname(sessionLogPath(sessionId)));
|
||||||
|
fs.appendFileSync(sessionLogPath(sessionId), JSON.stringify(entry) + '\n');
|
||||||
|
}
|
||||||
|
|
||||||
|
function summarizeArgs(toolName, toolArgs) {
|
||||||
|
if (!toolArgs || typeof toolArgs !== 'object') return {};
|
||||||
|
|
||||||
|
const summary = {};
|
||||||
|
|
||||||
|
// Capture file pattern hints
|
||||||
|
const fileFields = ['path', 'filePath', 'file_path', 'file'];
|
||||||
|
for (const f of fileFields) {
|
||||||
|
if (toolArgs[f] && typeof toolArgs[f] === 'string') {
|
||||||
|
summary.fileExt = path.extname(toolArgs[f]) || undefined;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Capture command category hints
|
||||||
|
if (toolArgs.command && typeof toolArgs.command === 'string') {
|
||||||
|
const cmd = toolArgs.command.trim();
|
||||||
|
const firstWord = cmd.split(/\s+/)[0];
|
||||||
|
summary.commandCategory = firstWord;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Tool-specific hints
|
||||||
|
switch (toolName) {
|
||||||
|
case 'bash':
|
||||||
|
case 'Bash':
|
||||||
|
case 'terminal':
|
||||||
|
summary.commandCategory = summary.commandCategory || 'shell';
|
||||||
|
break;
|
||||||
|
case 'edit':
|
||||||
|
case 'Edit':
|
||||||
|
case 'write':
|
||||||
|
case 'Write':
|
||||||
|
case 'create':
|
||||||
|
case 'Create':
|
||||||
|
summary.operation = 'write';
|
||||||
|
break;
|
||||||
|
case 'read_file':
|
||||||
|
case 'Read':
|
||||||
|
case 'view':
|
||||||
|
case 'View':
|
||||||
|
summary.operation = 'read';
|
||||||
|
break;
|
||||||
|
case 'grep':
|
||||||
|
case 'Grep':
|
||||||
|
case 'search':
|
||||||
|
summary.operation = 'search';
|
||||||
|
break;
|
||||||
|
case 'web_fetch':
|
||||||
|
case 'WebFetch':
|
||||||
|
case 'web_search':
|
||||||
|
case 'WebSearch':
|
||||||
|
summary.operation = 'web';
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
return summary;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── sessionEnd handler (pattern analysis) ──────────────────────────────────
|
||||||
|
|
||||||
|
function handleSessionEnd(input) {
|
||||||
|
const { sessionId, reason } = input;
|
||||||
|
|
||||||
|
// Skip failed/aborted sessions
|
||||||
|
if (reason === 'error' || reason === 'abort') return;
|
||||||
|
|
||||||
|
const logPath = sessionLogPath(sessionId);
|
||||||
|
if (!fs.existsSync(logPath)) return;
|
||||||
|
|
||||||
|
const entries = [];
|
||||||
|
try {
|
||||||
|
const lines = fs.readFileSync(logPath, 'utf8').trim().split('\n');
|
||||||
|
for (const line of lines) {
|
||||||
|
if (line.trim()) entries.push(JSON.parse(line));
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (entries.length < 3) return; // Too short to be a pattern
|
||||||
|
|
||||||
|
const db = loadDb();
|
||||||
|
|
||||||
|
// Extract sequences (sliding window of 3-8 consecutive tool calls)
|
||||||
|
for (let windowSize = 3; windowSize <= Math.min(8, entries.length); windowSize++) {
|
||||||
|
for (let i = 0; i <= entries.length - windowSize; i++) {
|
||||||
|
const window = entries.slice(i, i + windowSize);
|
||||||
|
const seqKey = window.map(e => e.fingerprint).join('→');
|
||||||
|
|
||||||
|
if (!db.patterns[seqKey]) {
|
||||||
|
db.patterns[seqKey] = {
|
||||||
|
sequence: window.map(e => ({
|
||||||
|
toolName: e.toolName,
|
||||||
|
fingerprint: e.fingerprint,
|
||||||
|
argsSummary: e.argsSummary
|
||||||
|
})),
|
||||||
|
sessions: [],
|
||||||
|
totalOccurrences: 0,
|
||||||
|
firstSeen: new Date().toISOString(),
|
||||||
|
lastSeen: new Date().toISOString(),
|
||||||
|
proposedSkill: null
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const pattern = db.patterns[seqKey];
|
||||||
|
if (!pattern.sessions.includes(sessionId)) {
|
||||||
|
pattern.sessions.push(sessionId);
|
||||||
|
}
|
||||||
|
pattern.totalOccurrences++;
|
||||||
|
pattern.lastSeen = new Date().toISOString();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Mark session as analyzed
|
||||||
|
db.sessions[sessionId] = {
|
||||||
|
analyzedAt: new Date().toISOString(),
|
||||||
|
toolCount: entries.length,
|
||||||
|
reason
|
||||||
|
};
|
||||||
|
|
||||||
|
saveDb(db);
|
||||||
|
|
||||||
|
// Generate proposed skills for patterns seen in 2+ sessions
|
||||||
|
generateProposedSkills(db);
|
||||||
|
|
||||||
|
// Clean up session log (keep last 10 for future reference)
|
||||||
|
cleanupOldSessions(db);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── skill generation ───────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
function generateProposedSkills(db) {
|
||||||
|
const candidates = [];
|
||||||
|
|
||||||
|
for (const [key, pattern] of Object.entries(db.patterns)) {
|
||||||
|
// Only propose patterns seen in 2+ different sessions
|
||||||
|
if (pattern.sessions.length >= 2 && !pattern.proposedSkill) {
|
||||||
|
candidates.push({ key, pattern });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (candidates.length === 0) return;
|
||||||
|
|
||||||
|
// Sort by session count (most repeated first), take top 3
|
||||||
|
candidates.sort((a, b) => b.pattern.sessions.length - a.pattern.sessions.length);
|
||||||
|
const topCandidates = candidates.slice(0, 3);
|
||||||
|
|
||||||
|
for (const { key, pattern } of topCandidates) {
|
||||||
|
const skillName = generateSkillName(pattern);
|
||||||
|
const skillMd = generateSkillMd(skillName, pattern);
|
||||||
|
|
||||||
|
// Write proposed SKILL.md
|
||||||
|
const skillDir = path.join(skillsDir(), 'proposed', skillName);
|
||||||
|
ensureDir(skillDir);
|
||||||
|
fs.writeFileSync(path.join(skillDir, 'SKILL.md'), skillMd);
|
||||||
|
|
||||||
|
// Write proposal metadata
|
||||||
|
fs.writeFileSync(path.join(skillDir, '.proposal.json'), JSON.stringify({
|
||||||
|
patternKey: key,
|
||||||
|
detectedAt: new Date().toISOString(),
|
||||||
|
sessionCount: pattern.sessions.length,
|
||||||
|
totalOccurrences: pattern.totalOccurrences,
|
||||||
|
status: 'proposed'
|
||||||
|
}, null, 2));
|
||||||
|
|
||||||
|
pattern.proposedSkill = skillName;
|
||||||
|
}
|
||||||
|
|
||||||
|
saveDb(db);
|
||||||
|
}
|
||||||
|
|
||||||
|
function generateSkillName(pattern) {
|
||||||
|
// Derive a name from the dominant tool types in the sequence
|
||||||
|
const toolCounts = {};
|
||||||
|
for (const step of pattern.sequence) {
|
||||||
|
toolCounts[step.toolName] = (toolCounts[step.toolName] || 0) + 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const dominant = Object.entries(toolCounts)
|
||||||
|
.sort((a, b) => b[1] - a[1])
|
||||||
|
.slice(0, 2)
|
||||||
|
.map(([name]) => simplifyToolName(name));
|
||||||
|
|
||||||
|
const name = dominant.join('-') || 'generic-workflow';
|
||||||
|
return name.slice(0, 40).replace(/[^a-z0-9-]/g, '-').replace(/-+/g, '-');
|
||||||
|
}
|
||||||
|
|
||||||
|
function simplifyToolName(toolName) {
|
||||||
|
const map = {
|
||||||
|
'bash': 'shell', 'Bash': 'shell', 'terminal': 'shell',
|
||||||
|
'edit': 'edit', 'Edit': 'edit', 'write': 'write', 'Write': 'write',
|
||||||
|
'create': 'create', 'Create': 'create',
|
||||||
|
'read_file': 'read', 'Read': 'read', 'view': 'view',
|
||||||
|
'grep': 'search', 'Grep': 'search',
|
||||||
|
'web_search': 'web-search', 'WebSearch': 'web-search',
|
||||||
|
'web_fetch': 'web-fetch', 'WebFetch': 'web-fetch',
|
||||||
|
'patch': 'patch', 'apply_patch': 'patch',
|
||||||
|
'git': 'git', 'task': 'task', 'Task': 'task',
|
||||||
|
'glob': 'find', 'Glob': 'find',
|
||||||
|
'ask_user': 'ask', 'AskUserQuestion': 'ask',
|
||||||
|
'notebookedit': 'notebook', 'NotebookEdit': 'notebook'
|
||||||
|
};
|
||||||
|
return map[toolName] || toolName.toLowerCase().replace(/_/g, '-');
|
||||||
|
}
|
||||||
|
|
||||||
|
function generateSkillMd(skillName, pattern) {
|
||||||
|
const tools = pattern.sequence.map(s => `\`${s.toolName}\``);
|
||||||
|
const uniqueTools = [...new Set(tools)];
|
||||||
|
|
||||||
|
const steps = pattern.sequence.map((s, i) => {
|
||||||
|
const context = s.argsSummary.fileExt
|
||||||
|
? ` (typically with ${s.argsSummary.fileExt} files)`
|
||||||
|
: s.argsSummary.commandCategory
|
||||||
|
? ` (typically ${s.argsSummary.commandCategory} commands)`
|
||||||
|
: '';
|
||||||
|
return `${i + 1}. **\`${s.toolName}\`**${context}`;
|
||||||
|
});
|
||||||
|
|
||||||
|
return `---
|
||||||
|
name: ${skillName}
|
||||||
|
description: >
|
||||||
|
Auto-detected workflow: ${uniqueTools.join(', ')}.
|
||||||
|
Detected across ${pattern.sessions.length} sessions (${pattern.totalOccurrences} occurrences).
|
||||||
|
Triggers when you follow a similar pattern of tool calls.
|
||||||
|
argument-hint: 'Describe what you want to do with this workflow'
|
||||||
|
---
|
||||||
|
|
||||||
|
# ${skillName}
|
||||||
|
|
||||||
|
> 🤖 **Auto-generated by Skill Factory**
|
||||||
|
> Detected across **${pattern.sessions.length} sessions** with **${pattern.totalOccurrences} occurrences**.
|
||||||
|
> First seen: ${pattern.firstSeen} | Last seen: ${pattern.lastSeen}
|
||||||
|
|
||||||
|
## Detected Pattern
|
||||||
|
|
||||||
|
This skill was automatically detected from your repeated tool-call sequences:
|
||||||
|
|
||||||
|
${steps.join('\n')}
|
||||||
|
|
||||||
|
## When to Activate
|
||||||
|
|
||||||
|
Use this skill when you're about to perform a workflow involving ${uniqueTools.join(', ')}.
|
||||||
|
|
||||||
|
## Workflow
|
||||||
|
|
||||||
|
${pattern.sequence.map((s, i) => `${i + 1}. Run \`${s.toolName}\` to ${describeToolAction(s)}`).join('\n')}
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- This is a **proposed** skill — review and customize it before relying on it.
|
||||||
|
- Edit this file to add specific instructions, pitfalls, and examples.
|
||||||
|
- Once reviewed, move it from \`proposed/\` to the main skills directory to make it active.
|
||||||
|
|
||||||
|
---
|
||||||
|
*Generated by [copilot-skill-factory](https://github.com/droideparanoico/copilot-skill-factory)*
|
||||||
|
`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function describeToolAction(step) {
|
||||||
|
const { toolName, argsSummary } = step;
|
||||||
|
const ext = argsSummary.fileExt ? ` (${argsSummary.fileExt} files)` : '';
|
||||||
|
const cmd = argsSummary.commandCategory ? ` (${argsSummary.commandCategory})` : '';
|
||||||
|
|
||||||
|
switch (toolName) {
|
||||||
|
case 'bash': case 'Bash': case 'terminal':
|
||||||
|
return `execute shell commands${cmd}${ext}`;
|
||||||
|
case 'edit': case 'Edit':
|
||||||
|
return `edit files${ext}`;
|
||||||
|
case 'write': case 'Write': case 'create': case 'Create':
|
||||||
|
return `create/write files${ext}`;
|
||||||
|
case 'read_file': case 'Read': case 'view': case 'View':
|
||||||
|
return `read files${ext}`;
|
||||||
|
case 'grep': case 'Grep':
|
||||||
|
return `search code${ext}`;
|
||||||
|
case 'web_search': case 'WebSearch':
|
||||||
|
return `search the web`;
|
||||||
|
case 'web_fetch': case 'WebFetch':
|
||||||
|
return `fetch web content`;
|
||||||
|
case 'patch': case 'apply_patch':
|
||||||
|
return `apply patches${ext}`;
|
||||||
|
case 'glob': case 'Glob':
|
||||||
|
return `find files`;
|
||||||
|
case 'task': case 'Task':
|
||||||
|
return `delegate to subagent`;
|
||||||
|
default:
|
||||||
|
return `perform operation`;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── sessionStart handler (inject context about detected skills) ─────────────
|
||||||
|
|
||||||
|
function handleSessionStart(input) {
|
||||||
|
const db = loadDb();
|
||||||
|
|
||||||
|
// Find proposed skills the user hasn't reviewed yet
|
||||||
|
let proposedSkills = [];
|
||||||
|
try {
|
||||||
|
const proposedDir = path.join(skillsDir(), 'proposed');
|
||||||
|
if (fs.existsSync(proposedDir)) {
|
||||||
|
proposedSkills = fs.readdirSync(proposedDir, { withFileTypes: true })
|
||||||
|
.filter(d => d.isDirectory())
|
||||||
|
.map(d => d.name);
|
||||||
|
}
|
||||||
|
} catch {}
|
||||||
|
|
||||||
|
// Find patterns ripe for proposal (seen in 2+ sessions, no skill yet)
|
||||||
|
const ripePatterns = Object.entries(db.patterns)
|
||||||
|
.filter(([, p]) => p.sessions.length >= 2 && !p.proposedSkill)
|
||||||
|
.map(([, p]) => p);
|
||||||
|
|
||||||
|
const totalProposed = proposedSkills.length;
|
||||||
|
const totalRipe = ripePatterns.length;
|
||||||
|
|
||||||
|
if (totalProposed === 0 && totalRipe === 0) {
|
||||||
|
// Nothing to report — silent
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Emit additionalContext so the agent knows about skill factory
|
||||||
|
let context = '';
|
||||||
|
|
||||||
|
if (totalProposed > 0) {
|
||||||
|
context += `🏭 **Skill Factory** has ${totalProposed} proposed skill(s) ready for review.\n`;
|
||||||
|
context += `Use \`/skill-factory review\` to review them.\n`;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (totalRipe > 0) {
|
||||||
|
context += `📊 **Skill Factory** detected ${totalRipe} repeatable workflow pattern(s) across your sessions.\n`;
|
||||||
|
context += `Use \`/skill-factory propose\` to see what can be turned into a skill.\n`;
|
||||||
|
}
|
||||||
|
|
||||||
|
// sessionStart command hooks can inject additionalContext via stdout
|
||||||
|
process.stdout.write(JSON.stringify({ additionalContext: context }));
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── cleanup ────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
function cleanupOldSessions(db) {
|
||||||
|
const maxSessions = 20;
|
||||||
|
const sessions = Object.keys(db.sessions);
|
||||||
|
if (sessions.length <= maxSessions) return;
|
||||||
|
|
||||||
|
// Keep newest, delete oldest session logs
|
||||||
|
const toDelete = sessions
|
||||||
|
.sort((a, b) => (db.sessions[b].analyzedAt || '').localeCompare(db.sessions[a].analyzedAt || ''))
|
||||||
|
.slice(maxSessions);
|
||||||
|
|
||||||
|
for (const sid of toDelete) {
|
||||||
|
const logPath = sessionLogPath(sid);
|
||||||
|
try { fs.unlinkSync(logPath); } catch {}
|
||||||
|
delete db.sessions[sid];
|
||||||
|
}
|
||||||
|
saveDb(db);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── main ───────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
const eventType = process.argv[2];
|
||||||
|
if (!eventType) {
|
||||||
|
console.error('Usage: observer.js <postToolUse|sessionEnd|sessionStart>');
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
|
||||||
|
const input = readStdin();
|
||||||
|
if (!input) {
|
||||||
|
// No input — exit silently
|
||||||
|
process.exit(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
switch (eventType) {
|
||||||
|
case 'postToolUse':
|
||||||
|
handlePostToolUse(input);
|
||||||
|
break;
|
||||||
|
case 'sessionEnd':
|
||||||
|
handleSessionEnd(input);
|
||||||
|
break;
|
||||||
|
case 'sessionStart':
|
||||||
|
handleSessionStart(input);
|
||||||
|
break;
|
||||||
|
default:
|
||||||
|
console.error(`Unknown event: ${eventType}`);
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
{
|
||||||
|
"version": 1,
|
||||||
|
"hooks": {
|
||||||
|
"postToolUse": [
|
||||||
|
{
|
||||||
|
"type": "command",
|
||||||
|
"matcher": "",
|
||||||
|
"bash": "root=\"${PLUGIN_ROOT:-${COPILOT_PLUGIN_ROOT:-.}}\"; node \"$root/bin/observer.js\" postToolUse",
|
||||||
|
"timeoutSec": 5
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"sessionEnd": [
|
||||||
|
{
|
||||||
|
"type": "command",
|
||||||
|
"bash": "root=\"${PLUGIN_ROOT:-${COPILOT_PLUGIN_ROOT:-.}}\"; node \"$root/bin/observer.js\" sessionEnd",
|
||||||
|
"timeoutSec": 30
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"sessionStart": [
|
||||||
|
{
|
||||||
|
"type": "command",
|
||||||
|
"bash": "root=\"${PLUGIN_ROOT:-${COPILOT_PLUGIN_ROOT:-.}}\"; node \"$root/bin/observer.js\" sessionStart",
|
||||||
|
"timeoutSec": 10
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
Executable
+41
@@ -0,0 +1,41 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
PLUGIN_NAME="copilot-skill-factory"
|
||||||
|
PLUGIN_DIR="${HOME}/.copilot/plugins/${PLUGIN_NAME}"
|
||||||
|
SKILL_DIR="${HOME}/.copilot/skills/skill-factory"
|
||||||
|
|
||||||
|
echo "🏭 Installing ${PLUGIN_NAME}..."
|
||||||
|
|
||||||
|
# Get script directory
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
|
# Install plugin
|
||||||
|
echo " → Installing plugin to ${PLUGIN_DIR}"
|
||||||
|
mkdir -p "${PLUGIN_DIR}"
|
||||||
|
cp "${SCRIPT_DIR}/plugin.json" "${PLUGIN_DIR}/"
|
||||||
|
cp -r "${SCRIPT_DIR}/hooks" "${PLUGIN_DIR}/"
|
||||||
|
cp -r "${SCRIPT_DIR}/bin" "${PLUGIN_DIR}/"
|
||||||
|
chmod +x "${PLUGIN_DIR}/bin/observer.js"
|
||||||
|
|
||||||
|
# Install companion skill
|
||||||
|
echo " → Installing skill to ${SKILL_DIR}"
|
||||||
|
mkdir -p "${SKILL_DIR}"
|
||||||
|
cp "${SCRIPT_DIR}/skills/skill-factory/SKILL.md" "${SKILL_DIR}/"
|
||||||
|
|
||||||
|
# Verify Node.js is available
|
||||||
|
if command -v node &>/dev/null; then
|
||||||
|
echo " ✓ Node.js $(node --version) detected"
|
||||||
|
else
|
||||||
|
echo " ⚠ Node.js not found — hooks require Node.js 18+"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "✅ ${PLUGIN_NAME} installed successfully!"
|
||||||
|
echo ""
|
||||||
|
echo "Next steps:"
|
||||||
|
echo " 1. Restart Copilot CLI"
|
||||||
|
echo " 2. Work normally — Skill Factory watches silently"
|
||||||
|
echo " 3. After a few sessions, run /skill-factory review"
|
||||||
|
echo ""
|
||||||
|
echo "Data stored at: ${SKILL_DIR}/.data/"
|
||||||
+26
@@ -0,0 +1,26 @@
|
|||||||
|
{
|
||||||
|
"name": "copilot-skill-factory",
|
||||||
|
"description": "Meta-plugin that watches your Copilot CLI workflows and automatically proposes reusable agent skills. Detects repeatable tool-call patterns across sessions and generates SKILL.md files — just like hermes-skill-factory for Hermes Agent.",
|
||||||
|
"version": "0.1.0",
|
||||||
|
"author": {
|
||||||
|
"name": "Skill Factory"
|
||||||
|
},
|
||||||
|
"homepage": "https://github.com/droideparanoico/copilot-skill-factory",
|
||||||
|
"repository": "https://github.com/droideparanoico/copilot-skill-factory",
|
||||||
|
"license": "MIT",
|
||||||
|
"keywords": [
|
||||||
|
"skill-factory",
|
||||||
|
"meta-skill",
|
||||||
|
"skill-generator",
|
||||||
|
"automation",
|
||||||
|
"productivity",
|
||||||
|
"workflow"
|
||||||
|
],
|
||||||
|
"category": "productivity",
|
||||||
|
"tags": [
|
||||||
|
"skill-generation",
|
||||||
|
"automation",
|
||||||
|
"meta"
|
||||||
|
],
|
||||||
|
"hooks": "hooks/hooks.json"
|
||||||
|
}
|
||||||
Executable
+91
@@ -0,0 +1,91 @@
|
|||||||
|
---
|
||||||
|
name: skill-factory
|
||||||
|
description: >
|
||||||
|
Meta-skill for managing automatically detected workflows. Review proposed skills,
|
||||||
|
inspect detected patterns, and turn your repeated tool-call sequences into
|
||||||
|
reusable Copilot CLI agent skills. Use /skill-factory review to see what's ready.
|
||||||
|
argument-hint: '[review|propose|list|status|save <name>|dismiss <name>|clear]'
|
||||||
|
---
|
||||||
|
|
||||||
|
# 🏭 Skill Factory
|
||||||
|
|
||||||
|
> **The meta-skill that creates skills.**
|
||||||
|
> Automatically watches your workflows and turns them into reusable agent skills.
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
|
||||||
|
When the user types `/skill-factory` (with or without arguments), follow this guide:
|
||||||
|
|
||||||
|
### `/skill-factory review`
|
||||||
|
Show all proposed skills ready for review. For each one:
|
||||||
|
1. Read the `SKILL.md` in `~/.copilot/skills/skill-factory/proposed/<name>/SKILL.md`
|
||||||
|
2. Read the `.proposal.json` for metadata (session count, occurrences)
|
||||||
|
3. Present a summary to the user and ask if they want to:
|
||||||
|
- **Accept** — move to `~/.copilot/skills/<name>/` (makes it active)
|
||||||
|
- **Edit** — open the SKILL.md for editing first
|
||||||
|
- **Dismiss** — archive to `~/.copilot/skills/skill-factory/dismissed/`
|
||||||
|
- **Skip** — leave as-is for later
|
||||||
|
|
||||||
|
### `/skill-factory propose`
|
||||||
|
Force analysis of current session patterns. Read the pattern database:
|
||||||
|
- `~/.copilot/skills/skill-factory/.data/pattern-db.json` (or in the plugin's data directory)
|
||||||
|
- Show the top 5 most frequent patterns (by session count)
|
||||||
|
- For each, describe the tool-call sequence and ask if the user wants to generate a skill
|
||||||
|
|
||||||
|
### `/skill-factory list`
|
||||||
|
List all skills generated by Skill Factory (both proposed and accepted):
|
||||||
|
- `~/.copilot/skills/skill-factory/proposed/` — proposed
|
||||||
|
- `~/.copilot/skills/<name>/` — accepted (look for `.factory.json` marker)
|
||||||
|
|
||||||
|
### `/skill-factory status`
|
||||||
|
Show Skill Factory statistics:
|
||||||
|
- Number of sessions tracked
|
||||||
|
- Number of patterns detected
|
||||||
|
- Number of proposed/accepted/dismissed skills
|
||||||
|
- Read from `~/.copilot/skills/skill-factory/.data/pattern-db.json`
|
||||||
|
|
||||||
|
### `/skill-factory save <name>`
|
||||||
|
Save the last proposal with a custom name. Used when the user wants to rename a proposed skill.
|
||||||
|
|
||||||
|
### `/skill-factory dismiss <name>`
|
||||||
|
Dismiss a proposed skill. Moves it to `~/.copilot/skills/skill-factory/dismissed/`.
|
||||||
|
|
||||||
|
### `/skill-factory clear`
|
||||||
|
Clear the pattern database and all proposed skills. Asks for confirmation first.
|
||||||
|
|
||||||
|
## Data Locations
|
||||||
|
|
||||||
|
| Data | Path |
|
||||||
|
|---|---|
|
||||||
|
| Pattern database | `~/.copilot/skills/skill-factory/.data/pattern-db.json` |
|
||||||
|
| Proposed skills | `~/.copilot/skills/skill-factory/proposed/<name>/` |
|
||||||
|
| Accepted skills | `~/.copilot/skills/<name>/` (with `.factory.json` marker) |
|
||||||
|
| Dismissed skills | `~/.copilot/skills/skill-factory/dismissed/<name>/` |
|
||||||
|
| Session logs | `~/.copilot/skills/skill-factory/.data/sessions/<id>.jsonl` |
|
||||||
|
|
||||||
|
## Accepting a Skill
|
||||||
|
|
||||||
|
When the user accepts a proposed skill:
|
||||||
|
1. Read the `SKILL.md` from `proposed/<name>/`
|
||||||
|
2. Create `~/.copilot/skills/<name>/SKILL.md` with the content
|
||||||
|
3. Create `~/.copilot/skills/<name>/.factory.json` with:
|
||||||
|
```json
|
||||||
|
{ "generatedBy": "skill-factory", "generatedAt": "<ISO timestamp>", "patternKey": "<key>" }
|
||||||
|
```
|
||||||
|
4. Remove from `proposed/<name>/`
|
||||||
|
5. Tell the user the skill is now active and can be invoked with `/<name>`
|
||||||
|
|
||||||
|
## Dismissing a Skill
|
||||||
|
|
||||||
|
When the user dismisses a proposed skill:
|
||||||
|
1. Move `proposed/<name>/` to `dismissed/<name>/`
|
||||||
|
2. Update the pattern in the DB to mark it as dismissed
|
||||||
|
|
||||||
|
## Important Notes
|
||||||
|
|
||||||
|
- The plugin (`copilot-skill-factory`) handles automatic pattern detection via hooks.
|
||||||
|
- This skill handles user interaction — reviewing, accepting, editing, dismissing.
|
||||||
|
- Always confirm before deleting or moving files.
|
||||||
|
- If the data directory doesn't exist yet, tell the user the plugin needs to run for at least one session first.
|
||||||
|
- Proposed skills are **suggestions** — they may need editing before they're useful.
|
||||||
|
- Encourage the user to customize accepted skills with specific instructions, pitfalls, and examples.
|
||||||
@@ -0,0 +1,356 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Test harness for copilot-skill-factory observer.js
|
||||||
|
*
|
||||||
|
* Simulates hook payloads to verify:
|
||||||
|
* 1. postToolUse logging works
|
||||||
|
* 2. sessionEnd analysis detects patterns
|
||||||
|
* 3. sessionStart injects context
|
||||||
|
* 4. Cross-session pattern detection triggers skill generation
|
||||||
|
*/
|
||||||
|
|
||||||
|
const { execFileSync, execSync } = require('child_process');
|
||||||
|
const fs = require('fs');
|
||||||
|
const path = require('path');
|
||||||
|
|
||||||
|
const OBSERVER = path.join(__dirname, 'bin', 'observer.js');
|
||||||
|
const DATA_DIR = path.join(__dirname, '.test-data');
|
||||||
|
|
||||||
|
// Override data directory for tests
|
||||||
|
process.env.COPILOT_PLUGIN_DATA = DATA_DIR;
|
||||||
|
|
||||||
|
// Also override HOME for skills dir
|
||||||
|
process.env.HOME = DATA_DIR;
|
||||||
|
|
||||||
|
let passed = 0;
|
||||||
|
let failed = 0;
|
||||||
|
|
||||||
|
function test(name, fn) {
|
||||||
|
try {
|
||||||
|
fn();
|
||||||
|
console.log(` ✓ ${name}`);
|
||||||
|
passed++;
|
||||||
|
} catch (e) {
|
||||||
|
console.log(` ✗ ${name}: ${e.message}`);
|
||||||
|
failed++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function runObserver(event, payload) {
|
||||||
|
return execFileSync('node', [OBSERVER, event], {
|
||||||
|
input: JSON.stringify(payload),
|
||||||
|
encoding: 'utf8',
|
||||||
|
env: { ...process.env, COPILOT_PLUGIN_DATA: DATA_DIR, HOME: DATA_DIR },
|
||||||
|
timeout: 5000
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function cleanup() {
|
||||||
|
try { fs.rmSync(DATA_DIR, { recursive: true, force: true }); } catch {}
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log('\n🏭 Skill Factory — Test Suite\n');
|
||||||
|
|
||||||
|
// ── postToolUse tests ──────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
console.log('postToolUse:');
|
||||||
|
cleanup();
|
||||||
|
|
||||||
|
const SESSION_A = 'test-session-aaa';
|
||||||
|
const SESSION_B = 'test-session-bbb';
|
||||||
|
|
||||||
|
test('logs a tool call', () => {
|
||||||
|
runObserver('postToolUse', {
|
||||||
|
sessionId: SESSION_A,
|
||||||
|
timestamp: Date.now(),
|
||||||
|
cwd: '/test/project',
|
||||||
|
toolName: 'bash',
|
||||||
|
toolArgs: { command: 'git status', description: 'check git status' }
|
||||||
|
});
|
||||||
|
|
||||||
|
const logPath = path.join(DATA_DIR, 'sessions', `${SESSION_A}.jsonl`);
|
||||||
|
if (!fs.existsSync(logPath)) throw new Error('Session log not created');
|
||||||
|
|
||||||
|
const content = fs.readFileSync(logPath, 'utf8').trim();
|
||||||
|
if (!content) throw new Error('Session log is empty');
|
||||||
|
|
||||||
|
const entry = JSON.parse(content.split('\n')[0]);
|
||||||
|
if (entry.toolName !== 'bash') throw new Error(`Expected bash, got ${entry.toolName}`);
|
||||||
|
if (!entry.fingerprint) throw new Error('Missing fingerprint');
|
||||||
|
if (entry.argsSummary.commandCategory !== 'git') throw new Error(`Expected git category, got ${entry.argsSummary.commandCategory}`);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('fingerprints are stable for same tool+args', () => {
|
||||||
|
const fp1 = runObserverAndGetFingerprint('bash', { command: 'git diff' });
|
||||||
|
const fp2 = runObserverAndGetFingerprint('bash', { command: 'git diff --stat' });
|
||||||
|
// Different command lengths → different fingerprints
|
||||||
|
if (fp1 === fp2) throw new Error('Different commands should have different fingerprints');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('fingerprints match for similar file operations', () => {
|
||||||
|
const fp1 = runObserverAndGetFingerprint('edit', { path: '/a/b/file1.py', old_string: 'abc' });
|
||||||
|
const fp2 = runObserverAndGetFingerprint('edit', { path: '/x/y/file2.py', old_string: 'xyz' });
|
||||||
|
// Same tool, file extension normalized, content normalized → same fingerprint
|
||||||
|
if (fp1 !== fp2) throw new Error('Similar edits should have same fingerprint');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('skips noisy tools (todo, ask)', () => {
|
||||||
|
runObserver('postToolUse', {
|
||||||
|
sessionId: SESSION_A,
|
||||||
|
timestamp: Date.now(),
|
||||||
|
cwd: '/test',
|
||||||
|
toolName: 'todo_write',
|
||||||
|
toolArgs: { todos: [] }
|
||||||
|
});
|
||||||
|
|
||||||
|
const logPath = path.join(DATA_DIR, 'sessions', `${SESSION_A}.jsonl`);
|
||||||
|
const content = fs.readFileSync(logPath, 'utf8').trim();
|
||||||
|
const todoEntries = content.split('\n').filter(l => l.includes('todo_write'));
|
||||||
|
if (todoEntries.length > 0) throw new Error('todo_write should be skipped');
|
||||||
|
});
|
||||||
|
|
||||||
|
function runObserverAndGetFingerprint(toolName, toolArgs) {
|
||||||
|
const tmpSession = 'tmp-' + Math.random().toString(36).slice(2, 8);
|
||||||
|
runObserver('postToolUse', {
|
||||||
|
sessionId: tmpSession,
|
||||||
|
timestamp: Date.now(),
|
||||||
|
cwd: '/test',
|
||||||
|
toolName,
|
||||||
|
toolArgs
|
||||||
|
});
|
||||||
|
|
||||||
|
const logPath = path.join(DATA_DIR, 'sessions', `${tmpSession}.jsonl`);
|
||||||
|
const entry = JSON.parse(fs.readFileSync(logPath, 'utf8').trim());
|
||||||
|
return entry.fingerprint;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── sessionEnd tests ───────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
console.log('\nsessionEnd:');
|
||||||
|
cleanup();
|
||||||
|
|
||||||
|
test('analyzes session and creates pattern DB', () => {
|
||||||
|
// Simulate a full session with a repeated workflow
|
||||||
|
const sessionId = 'test-session-1';
|
||||||
|
const workflow = [
|
||||||
|
{ toolName: 'bash', toolArgs: { command: 'git pull' } },
|
||||||
|
{ toolName: 'grep', toolArgs: { pattern: 'TODO' } },
|
||||||
|
{ toolName: 'edit', toolArgs: { path: 'src/app.py', old_string: 'old', new_string: 'new' } },
|
||||||
|
{ toolName: 'bash', toolArgs: { command: 'pytest' } },
|
||||||
|
];
|
||||||
|
|
||||||
|
for (const call of workflow) {
|
||||||
|
runObserver('postToolUse', {
|
||||||
|
sessionId,
|
||||||
|
timestamp: Date.now(),
|
||||||
|
cwd: '/test/project',
|
||||||
|
...call
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
runObserver('sessionEnd', {
|
||||||
|
sessionId,
|
||||||
|
timestamp: Date.now(),
|
||||||
|
cwd: '/test/project',
|
||||||
|
reason: 'complete'
|
||||||
|
});
|
||||||
|
|
||||||
|
// Check DB was created
|
||||||
|
const dbPath = path.join(DATA_DIR, 'pattern-db.json');
|
||||||
|
if (!fs.existsSync(dbPath)) throw new Error('Pattern DB not created');
|
||||||
|
|
||||||
|
const db = JSON.parse(fs.readFileSync(dbPath, 'utf8'));
|
||||||
|
if (Object.keys(db.patterns).length === 0) throw new Error('No patterns detected');
|
||||||
|
|
||||||
|
if (!db.sessions[sessionId]) throw new Error('Session not recorded in DB');
|
||||||
|
if (db.sessions[sessionId].toolCount !== 4) throw new Error(`Expected 4 tools, got ${db.sessions[sessionId].toolCount}`);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('detects cross-session patterns and generates skills', () => {
|
||||||
|
cleanup();
|
||||||
|
|
||||||
|
// Session 1: Same workflow
|
||||||
|
const workflow = [
|
||||||
|
{ toolName: 'bash', toolArgs: { command: 'npm install' } },
|
||||||
|
{ toolName: 'edit', toolArgs: { path: 'src/index.ts', old_string: 'a', new_string: 'b' } },
|
||||||
|
{ toolName: 'bash', toolArgs: { command: 'npm test' } },
|
||||||
|
];
|
||||||
|
|
||||||
|
// Run 2 sessions with the same workflow
|
||||||
|
for (let s = 1; s <= 2; s++) {
|
||||||
|
const sid = `cross-session-${s}`;
|
||||||
|
for (const call of workflow) {
|
||||||
|
runObserver('postToolUse', {
|
||||||
|
sessionId: sid,
|
||||||
|
timestamp: Date.now(),
|
||||||
|
cwd: '/test/project',
|
||||||
|
...call
|
||||||
|
});
|
||||||
|
}
|
||||||
|
runObserver('sessionEnd', {
|
||||||
|
sessionId: sid,
|
||||||
|
timestamp: Date.now(),
|
||||||
|
cwd: '/test/project',
|
||||||
|
reason: 'complete'
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// After 2 sessions with same pattern, a skill should be proposed
|
||||||
|
const proposedDir = path.join(DATA_DIR, '.copilot', 'skills', 'skill-factory', 'proposed');
|
||||||
|
if (!fs.existsSync(proposedDir)) throw new Error('Proposed directory not created');
|
||||||
|
|
||||||
|
const proposed = fs.readdirSync(proposedDir, { withFileTypes: true })
|
||||||
|
.filter(d => d.isDirectory())
|
||||||
|
.map(d => d.name);
|
||||||
|
|
||||||
|
if (proposed.length === 0) throw new Error('No skills proposed after cross-session patterns');
|
||||||
|
|
||||||
|
// Verify SKILL.md was generated
|
||||||
|
const skillMd = path.join(proposedDir, proposed[0], 'SKILL.md');
|
||||||
|
if (!fs.existsSync(skillMd)) throw new Error('SKILL.md not generated');
|
||||||
|
|
||||||
|
const content = fs.readFileSync(skillMd, 'utf8');
|
||||||
|
if (!content.includes('Auto-generated by Skill Factory')) throw new Error('SKILL.md missing auto-gen marker');
|
||||||
|
if (!content.includes('2 sessions')) throw new Error('SKILL.md should mention 2 sessions');
|
||||||
|
|
||||||
|
console.log(` → Generated skill: ${proposed[0]}`);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('does not generate skills for single-session patterns', () => {
|
||||||
|
cleanup();
|
||||||
|
|
||||||
|
const sid = 'single-session';
|
||||||
|
const workflow = [
|
||||||
|
{ toolName: 'bash', toolArgs: { command: 'echo hello' } },
|
||||||
|
{ toolName: 'read_file', toolArgs: { path: 'test.txt' } },
|
||||||
|
{ toolName: 'edit', toolArgs: { path: 'test.txt', old_string: 'a', new_string: 'b' } },
|
||||||
|
];
|
||||||
|
|
||||||
|
for (const call of workflow) {
|
||||||
|
runObserver('postToolUse', { sessionId: sid, timestamp: Date.now(), cwd: '/test', ...call });
|
||||||
|
}
|
||||||
|
runObserver('sessionEnd', { sessionId: sid, timestamp: Date.now(), cwd: '/test', reason: 'complete' });
|
||||||
|
|
||||||
|
const proposedDir = path.join(DATA_DIR, '.copilot', 'skills', 'skill-factory', 'proposed');
|
||||||
|
const hasProposed = fs.existsSync(proposedDir) && fs.readdirSync(proposedDir).length > 0;
|
||||||
|
if (hasProposed) throw new Error('Should not propose skills from a single session');
|
||||||
|
});
|
||||||
|
|
||||||
|
// ── sessionStart tests ─────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
console.log('\nsessionStart:');
|
||||||
|
cleanup();
|
||||||
|
|
||||||
|
test('is silent when no proposals exist', () => {
|
||||||
|
const output = runObserver('sessionStart', {
|
||||||
|
sessionId: 'start-test',
|
||||||
|
timestamp: Date.now(),
|
||||||
|
cwd: '/test',
|
||||||
|
source: 'new'
|
||||||
|
});
|
||||||
|
|
||||||
|
if (output.trim()) throw new Error('Should be silent with no proposals');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('injects context when proposals exist', () => {
|
||||||
|
cleanup();
|
||||||
|
|
||||||
|
// Create a fake proposed skill
|
||||||
|
const proposedDir = path.join(DATA_DIR, '.copilot', 'skills', 'skill-factory', 'proposed', 'test-skill');
|
||||||
|
fs.mkdirSync(proposedDir, { recursive: true });
|
||||||
|
fs.writeFileSync(path.join(proposedDir, 'SKILL.md'), '# Test\n');
|
||||||
|
|
||||||
|
const output = runObserver('sessionStart', {
|
||||||
|
sessionId: 'start-test-2',
|
||||||
|
timestamp: Date.now(),
|
||||||
|
cwd: '/test',
|
||||||
|
source: 'new'
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!output.trim()) throw new Error('Should inject context when proposals exist');
|
||||||
|
const parsed = JSON.parse(output.trim());
|
||||||
|
if (!parsed.additionalContext) throw new Error('Missing additionalContext');
|
||||||
|
if (!parsed.additionalContext.includes('Skill Factory')) throw new Error('Context should mention Skill Factory');
|
||||||
|
if (!parsed.additionalContext.includes('/skill-factory review')) throw new Error('Context should suggest /skill-factory review');
|
||||||
|
});
|
||||||
|
|
||||||
|
// ── Edge cases ─────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
console.log('\nEdge cases:');
|
||||||
|
cleanup();
|
||||||
|
|
||||||
|
test('handles empty stdin gracefully', () => {
|
||||||
|
try {
|
||||||
|
execFileSync('node', [OBSERVER, 'postToolUse'], {
|
||||||
|
input: '',
|
||||||
|
encoding: 'utf8',
|
||||||
|
env: { ...process.env, COPILOT_PLUGIN_DATA: DATA_DIR, HOME: DATA_DIR },
|
||||||
|
timeout: 3000
|
||||||
|
});
|
||||||
|
// Should exit 0
|
||||||
|
} catch (e) {
|
||||||
|
if (e.status !== 0) throw new Error(`Expected exit 0, got ${e.status}`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('handles malformed JSON gracefully', () => {
|
||||||
|
try {
|
||||||
|
execFileSync('node', [OBSERVER, 'postToolUse'], {
|
||||||
|
input: '{not valid json',
|
||||||
|
encoding: 'utf8',
|
||||||
|
env: { ...process.env, COPILOT_PLUGIN_DATA: DATA_DIR, HOME: DATA_DIR },
|
||||||
|
timeout: 3000
|
||||||
|
});
|
||||||
|
} catch (e) {
|
||||||
|
// Should still exit 0 (hook failures are silent)
|
||||||
|
if (e.status !== 0) throw new Error(`Expected exit 0, got ${e.status}`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('sessionEnd with fewer than 3 tools creates no patterns', () => {
|
||||||
|
cleanup();
|
||||||
|
const sid = 'short-session';
|
||||||
|
|
||||||
|
runObserver('postToolUse', { sessionId: sid, timestamp: Date.now(), cwd: '/test', toolName: 'bash', toolArgs: { command: 'ls' } });
|
||||||
|
runObserver('postToolUse', { sessionId: sid, timestamp: Date.now(), cwd: '/test', toolName: 'read_file', toolArgs: { path: 'test.txt' } });
|
||||||
|
runObserver('sessionEnd', { sessionId: sid, timestamp: Date.now(), cwd: '/test', reason: 'complete' });
|
||||||
|
|
||||||
|
const dbPath = path.join(DATA_DIR, 'pattern-db.json');
|
||||||
|
let patternCount = 0;
|
||||||
|
if (fs.existsSync(dbPath)) {
|
||||||
|
const db = JSON.parse(fs.readFileSync(dbPath, 'utf8'));
|
||||||
|
patternCount = Object.keys(db.patterns).length;
|
||||||
|
}
|
||||||
|
if (patternCount > 0) throw new Error('Sessions with <3 tools should not create patterns');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('session cleanup keeps only last 20 sessions', () => {
|
||||||
|
cleanup();
|
||||||
|
|
||||||
|
// Create 25 sessions
|
||||||
|
for (let i = 1; i <= 25; i++) {
|
||||||
|
const sid = `bulk-session-${i}`;
|
||||||
|
runObserver('postToolUse', { sessionId: sid, timestamp: Date.now(), cwd: '/test', toolName: 'bash', toolArgs: { command: `cmd-${i}` } });
|
||||||
|
runObserver('postToolUse', { sessionId: sid, timestamp: Date.now(), cwd: '/test', toolName: 'grep', toolArgs: { pattern: `pat-${i}` } });
|
||||||
|
runObserver('postToolUse', { sessionId: sid, timestamp: Date.now(), cwd: '/test', toolName: 'edit', toolArgs: { path: 'file.py', old_string: 'a', new_string: 'b' } });
|
||||||
|
runObserver('sessionEnd', { sessionId: sid, timestamp: Date.now(), cwd: '/test', reason: 'complete' });
|
||||||
|
}
|
||||||
|
|
||||||
|
const dbPath = path.join(DATA_DIR, 'pattern-db.json');
|
||||||
|
const db = JSON.parse(fs.readFileSync(dbPath, 'utf8'));
|
||||||
|
const sessionCount = Object.keys(db.sessions).length;
|
||||||
|
|
||||||
|
if (sessionCount > 20) throw new Error(`Session cleanup failed: ${sessionCount} > 20`);
|
||||||
|
});
|
||||||
|
|
||||||
|
// ── Summary ────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
console.log(`\n${'─'.repeat(50)}`);
|
||||||
|
console.log(`Results: ${passed} passed, ${failed} failed`);
|
||||||
|
console.log(`${'─'.repeat(50)}\n`);
|
||||||
|
|
||||||
|
// Cleanup
|
||||||
|
cleanup();
|
||||||
|
|
||||||
|
process.exit(failed > 0 ? 1 : 0);
|
||||||
Reference in New Issue
Block a user