FINDS - TESTS · Slate 3, Test 2 — Book to Skill (find #20) on Scaling People Ch. 5
Date: 2026-11-20 · Tester: FINDS - TESTS subagent · Subject: book-to-skill, Finds Library find #20 (github.com/virgiliojr94/book-to-skill)
1. Source limits — read this first
This test ran under a hard source constraint, and its results must be read inside it:
- There is no digital copy of Scaling People in the workspace, and none was downloaded. Ros reads a physical copy. Per instructions, no book text was fetched, bought, or reconstructed from memory.
- The genuine chapter text available is smaller than the brief assumed. Of the two
uploaded photographs, one (
5AF95873…jpg) is the book's front cover — title, subtitle, author, no chapter text. The other (522F3008…jpg) is a two-page spread: the Ch. 5 title page and page 349, the chapter's opening page. So the total real source is the title page + one page of prose (341 words / 409 tokens), and p. 349 ends mid-sentence. Everything below is an excerpt-scale test. - The BOOK - Scaling People chat has no reader Q&A yet (Ros has just started the
chapter), so the interrogation questions were derived from the transcribed page plus
the chapter's announced agenda, and are labelled as such in
interrogation.md. - The chapter's announced agenda (from the BOOK chat): hypothesis-based coaching → hard feedback → informal feedback culture → formal reviews → compensation → high performers → low performers → managing managers → managing out / firing / layoffs. Only the coaching framing on p. 349 was testable.
Deliverables in this folder: transcription.md (faithful transcription, excerpt-only,
verified against enlarged crops of the photograph), skills/scaling-people/ (the
produced skill), interrogation.md (full transcript + scores), extraction/ (the
tool's own extraction output), repo/ (the cloned tool).
2. Tool provenance and upkeep
| Repo | github.com/virgiliojr94/book-to-skill, MIT-licensed (converter only) |
| Version tested | v2.0.1 — HEAD 57f3db5, committed 2026-11-19, one day before this test |
| History | 286 commits on master; first commit 2026-04-30; 7 release tags (v1.0.0 → v2.0.1) |
| Shape | Two halves: a deterministic Python extractor (scripts/extract.py → book_to_skill/) and a spec-driven generator — the repo's 895-line SKILL.md that an agent follows |
| Upkeep signals | Active: current release, 1,100+ pytest tests claimed, security hardening (prompt-injection sanitization of sources, generated-skill scanner), multilingual READMEs, sponsor-funded solo maintenance |
The claims under test, in the tool's own words (README, "What it generates"; docs/performance.md): a generated SKILL.md of ~4,000 tokens, chapter files of ~1,000 tokens each loaded on demand, and 24×–51× fewer tokens than dumping the book into context, measured on real books of 119K–256K tokens.
3. Conversion record — the tool's own process, followed
Mode 1 (Full Conversion), Steps 0–10 of the repo's SKILL.md:
- Extraction (Step 2, the tool's code, unmodified):
python3 repo/scripts/extract.py transcription.md --mode text→extraction/full_text.txt+metadata.json(984–992 estimated tokens by the tool's own heuristic; 1,191 by tiktoken cl100k — the heuristic undercounts ~17% here). Markdown input needs no external extractors, as documented. - Step 3 structure analysis corrected the tool. The extractor's structural chapter
detection reported "3 chapters" — it counted the transcription's three
##headings (title page / p. 349 / provenance notes) as chapters. The agent-side Step 3 analysis reduced this to the truth: one chapter (Ch. 5), opening page only. On a real book file this failure mode is less likely, but it is a real partial-input behavior: the deterministic half cannot recognize an excerpt; the spec relies on the agent half to catch it, and the spec's own "confirm the extraction is the document you asked for" check is what surfaces it. - Generation (Steps 6–9, agent following the spec): produced the prescribed layout —
SKILL.md(front-loaded Core Frameworks, Chapter Index, Topic Index, Scope & Limits),chapters/ch05-feedback-and-performance-mechanisms.md(Step 7 template, text-book variant, study depth),glossary.md,patterns.md,cheatsheet.md. Framework names preserved exactly ("extreme coach," "forgot-to coach," "Feedback is a gift"); no raw passages copied into chapter/pattern/cheatsheet files (one exception flagged in §5). - Step 9.5 scan + validation:
tools/scan_generated_skill.py→ passed (no injection/authority patterns).tools/validate_skill.pyon the generated SKILL.md → passed (no host-breaking issues).
Adaptations, all forced by the test setup and recorded here:
- Input document: the tool demands a book file; it was given transcription.md — the
faithful transcription including its provenance header and reader-annotation notes,
because the excerpt-only marking had to travel with the text. The generator worked
from the chapter-text portion.
- Interactive prompts resolved by test fiat: Step 1.5 book type = text (management
prose — the documented answer for this genre); Step 4 purpose = all-of-the-above →
DEPTH=study; Step 5 slug = scaling-people (title form; the spec's
author-concept alternative would be hughes-johnson-feedback).
- Destination redirected: the spec's default install root is ~/.agents/skills/. The
skill was written to this test folder instead, so the test installs nothing into Ros's
live agent environment. Layout inside the folder is exactly as prescribed.
- Step 10 cleanup deferred: the extraction workdir was kept (inside the test folder)
as evidence instead of being deleted.
- Step 11 (publish to GitHub) skipped: optional, needs user consent, and the tool's
own copyright policy requires skills from third-party copyrighted books to stay
private regardless.
4. Token measurements
Tokenizer: tiktoken 0.14.0, cl100k_base (o200k_base within ±1% on every file; the tool's claims are presumably in Claude tokens, so read these as close approximations, not exact Claude counts).
| Artifact | Tokens (cl100k) | Reference point |
|---|---|---|
| Book text only (title page + p. 349) | 409 | the actual source signal |
Full transcription.md (incl. provenance/annotations) |
1,157 | what was fed to the extractor |
Extractor's full_text.txt |
1,191 | tool's own heuristic said ~984–992 |
Generated SKILL.md (body excl. frontmatter: 1,176) |
1,249 | claim: ~4,000 |
| Generated chapter file ch05 | 1,189 | claim: ~1,000/chapter; spec budget (text+study): 1,000–1,800 |
glossary.md |
321 | spec cap 1,500 |
patterns.md |
348 | README guide ~2,000 |
cheatsheet.md |
335 | spec cap 1,200 |
| Whole generated package | 3,442 | — |
What CAN be concluded: - The generator's budgets hold even on a one-page input: chapter file lands inside its prescribed band (1,189 vs. 1,000–1,800), SKILL.md body is far under the 4,000 cap, all supporting files under their caps. Nothing padded to hit a number — the spec's density-over-completeness rule was followable. - The pipeline degrades gracefully on partial input if the generating agent scopes it honestly: the produced skill declares its one-page coverage and lists the uncovered agenda topics by name.
What CANNOT be concluded: - The ~4,000-token SKILL.md figure is untested. Ours is 1,249 tokens because 409 tokens of source cannot support 4,000 tokens of distilled frameworks. The claim is about the steady state for real books; this run neither confirms nor refutes it. - The 24×–51× savings claim is untestable at this scale — and inverts. The claim's regime is 119K–256K-token books vs. a ~5K loaded core. Here, answering from the skill loads 1,249 tokens (SKILL.md) or 2,438 (with the chapter) against a 409-token source: the skill is 3–6× more expensive than the raw text, because fixed scaffolding (usage instructions, indexes, scope notes) dwarfs a one-page source. Any reuse of the headline savings for excerpt-scale inputs would be wrong. - Chapter-file fidelity under real compression (a ~1,000-token file standing in for a ~10,000-token chapter) is untested — with a 409-token source there was almost nothing to drop.
5. Generation-fidelity notes (self-audit of the produced skill)
Checked claim-by-claim against p. 349 after generation. Two deviations found, both small, both recorded rather than silently fixed after the fact:
- Cheatsheet embellishment: the diagnostic row "'Feedback is a gift' said as a slogan before a harsh verdict → slogan, not service" invents a scenario (harsh verdicts) that is not on the page. It is plausible practitioner synthesis of the kind the spec's cheatsheet section invites ("tells & smells"), but strictly it adds content the author didn't write.
- One verbatim quote: the glossary's "Iterative process" entry quotes the page's defining sentence directly (~20 words), against the letter of the spec's Quality Rule #7 ("never copy raw book text"). Defensible as a definition, but a deviation.
Everything else in the generated files traces to the page, including the negative space: the files state three times (chapter banner, Worked Example, Scope & Limits) that the Bascobert story is truncated.
6. Interrogation results
Full transcript and scoring in interrogation.md. Summary:
| Q | Type | Result |
|---|---|---|
| Q1 coaching-continuum poles | excerpt | fidelity full, coverage full |
| Q2 "feedback is a gift" qualification | excerpt | fidelity full, coverage full |
| Q3 Bascobert story | excerpt + truncation trap | full — and the skill refused to invent the story's ending |
| Q4 formal review process | agenda, not covered | clean abstention (used the one genuine mention accurately) |
| Q5 managing out | agenda, not covered | clean abstention |
| Q6 upward feedback | outside chapter | clean abstention |
Skill vs. raw-transcription control on Q1–Q3: no factual divergence; the skill's
answers are compressed re-voicings whose every checkable claim traces to the page.
Caveats (also in interrogation.md): same-agent simulation rather than a fresh-agent
test; abstentions lean on the Scope & Limits the generator wrote for this partial input;
fidelity was easy to achieve with only 409 tokens of source.
7. Effort log
Single session, 2026-11-20, ~40 minutes of agent work: photo transcription with
enlarged-crop verification (~10 min — the slow step, and the one that caught the
cover-vs-page discrepancy); repo clone (first attempt interrupted, recovered by fetch)
+ docs and the 895-line generator spec (~10 min); extraction, generation of 5 files,
scan/validate, token accounting (~10 min); interrogation + self-audit + this report
(~10 min). No book content was downloaded at any point. Tokenizer installed into a
throwaway venv (/tmp/tok-venv) after the system pip refused (PEP 668).
8. Draft verdict
The converter works as prescribed and behaves well on partial input — but this test validates the plumbing, not the headline economics. Structurally valid output (scanner- and validator-clean), budgets respected, faithful answers on covered content, and — the behavior that matters most for a study tool — clean abstentions with no confabulation on the chapter's uncovered agenda topics, including refusing to complete a story the source cuts off mid-sentence. Against that: the extractor alone misreads an excerpt as "3 chapters" (the agent step must catch it), the generator can drift into small embellishments in the cheatsheet layer, and the ~4K-token / 24×–51× claims are book-scale claims that a 409-token excerpt cannot test — at this scale the skill costs more tokens than the source.
Reuse proposal: Reuse later, with a defined re-run. The tool fits a real gap for Ros — books he is actively studying — but only when a legitimate digital text exists (his own notes, or a text he has the rights to process; the README itself notes a photographed book needs OCR first, and output must stay private). For Scaling People specifically, he is reading a physical copy, so there is no lawful full text to feed it today.
Re-run trigger: if/when a legitimate full text of Chapter 5 (or the book) exists in the workspace, re-run this test at full scale — that run can test the things this one could not: chapter-file fidelity under real 10:1 compression, the ~4K SKILL.md figure, per-query token cost vs. context dump, and abstention behavior when the skill's source is large enough that its gaps are non-obvious.
Test-record block (for FINDS - LIBRARY)
- Subject: Book to Skill — find #20 (virgiliojr94/book-to-skill, v2.0.1)
- What ran: Full Conversion (Mode 1) of the tool's own pipeline — Python extractor + agent generation per the repo's SKILL.md spec — on the only genuine chapter text available: a faithful transcription of Scaling People Ch. 5's title page and opening page (p. 349), from Ros's photographs of his physical copy. Then a 6-question interrogation (3 excerpt-answerable incl. a truncated-story trap, 2 uncovered agenda topics, 1 off-chapter) with a raw-transcription control on the answerable three.
- When: 2026-11-20
- Outcome: Pipeline completed; generated skill passed the tool's own security scan and host validator. Interrogation: full fidelity on covered questions, clean abstentions on all three uncovered ones, no confabulation. Token claims (~4K SKILL.md, 24×–51× savings) not testable at excerpt scale — generated SKILL.md is 1,249 tokens against a 409-token source. Extractor alone mis-segmented the excerpt as "3 chapters"; agent Step 3 corrected it.
- Output reference: ~/workspace/finds-tests/book-to-skill-2026-11-20/ (report.md, transcription.md, interrogation.md, skills/scaling-people/, extraction/)
- Reuse status: Reuse later — adopt when a legitimate digital text of a book Ros is studying exists; re-run the full test if a legitimate full Chapter 5 text ever becomes available.