← All tests

Second Brain cluster — Lorem vs obsidian-skills vs Open Notebook

Slate 3: three candidate Second Brain data layers on identical real material — the 475-book Personal Library as a Markdown folder + the Mindprint research corpus

2026-11-20FINDS - TESTSFind #71 (Lorem) · #28 (obsidian-skills) · #17 (Open Notebook)

Screenshots & outputs

Report

FINDS - TESTS Slate 3, Test 6 — The Second Brain Cluster

Date: 2026-11-20 · Format: pair-format, three subjects, same real material for all three Subjects: (a) Lorem — Finds Library find #71 · (b) kepano/obsidian-skills — find #28 · (c) Open Notebook — find #17 Purpose: evidence for Ros's still-open Second Brain decision (module set + where the data lives). This report informs that decision; it does not make it. Working folder: ~/workspace/finds-tests/second-brain-cluster-2026-11-20/ — per-subject scorecards: lorem/scorecard.md, obsidian/scorecard.md, open-notebook/scorecard.md. Nothing in Ros's live apps or data was modified; all work ran on copies inside the test folder.


1. Shared material — construction and reconciliation

Library export (library-md/). The live Personal Library catalog was read through the app's own listbooks action (never the database directly) and exported as one Markdown note per book: YAML frontmatter with exactly the fields the catalog really has (title, author, status, category, date_added, source, source_link, cover) and the book's notes as the body, organised into folders by status, the way a person would keep them.

  • Ground truth (ground-truth.json): 475 books — Want to read 279 / Owned 159 / Reading 37. The brief's remembered counts (280/159/36) were off by one in two statuses; the live numbers above are the ones every subject was scored against.
  • Field reality: author is null on 59 books; notes present on 453; cover URLs on 375. Authors with ≥3 entries: James Hollis 9, Chase Hughes 8, Dorie Clark 5, Byung-Chul Han 5, Robert Greene 4, and eight authors at 3 (Yalom, Kissinger, Бородатюк, The School of Life, Easter, Plokhy, Harris, Russell). One real-world nuance the tests surfaced: "Hollis" appears 10× — 9× "James Hollis" + 1× "Dr. James Hollis".
  • Reconciliation: 475 notes created = 475 catalog records. Every subject's ingest was checked against this.

Research corpus (research/). Five real documents (~4,950 words) assembled from Ros's own Second Brain / Mindprint material in the workspace: (01) the 2026-11-19 design principle ("don't adopt a system — observe how your mind works…"), (02) the Mindprint product idea, (03) the full Mindprint proposal (canonical text), (04) the proposal PDF text, (05) the Lorem candidate note. A canonical grounded synthesis (claims cited [01]–[05], including the caveat that the corpus argues one side) was written once and used as the reference standard for judging T3 output.

Fixed tasks (identical for all three): T1 ingest the 475-note folder without mangling, counts verified · T2 answer three fixed questions from the data (counts per status; name the Reading books; all books by an author with ≥3 entries) · T3 add the corpus and produce a synthesis answering "What do the sources say about letting a person's own categories drive the system, vs adopting a ready-made system?", judged on groundedness · T4 data-ownership reality (physical location, format, exportability, lock-in/telemetry/account — verified, not marketed) · T5 agent operability (files/API/CLI vs GUI-only).

Environment honesty. Headless Linux VM, no GPU, no Docker. Desktop apps ran extracted under Xvfb with sandboxed HOMEs inside the test folder and were driven via Chromium DevTools Protocol. Open Notebook ran from source (SurrealDB binary + API + worker) with local Ollama models — the no-API-key path — because no LLM key exists in this environment. Where a task could not run, the scorecard says NOT RUN and why; nothing was faked.


2. Per-subject results

(a) Lorem — find #71 · lorem/scorecard.md

Lorem 0.8.4 (lorem.md by ANZI @anjihhg; release 2026-11-16, actively maintained; free, closed-source app, open data format).

  • T1 works. The plain folder opened as a workspace; the app's index held 478 entries (475 notes + 3 directories); after the full session every book file was byte-identical (app adds only small JSON sidecars). Its native route also verified: a .database.json collection rendered the library as a table — "475 items", all columns populated.
  • T2 works. From the app's own index: 279/159/37 exact; the 37 Reading titles exact; authors ≥3 exact. In the GUI, search "Hollis" returned exactly the 10 Hollis entries (frontmatter is indexed). No natural-language query — search + collection views are the query surface.
  • T3 partial. Corpus live-indexed into the workspace (478→485 files); the synthesis was produced by the agent writing a grounded, cited .md into the workspace — the vendor's documented agent workflow — and it appears in-app. Lorem's built-in AI chat was NOT RUN: it needs a user-supplied LLM API key, and none exists here.
  • T4 works — strongest of the three. Notes are plain .md in a folder the user picks; the search index is a rebuildable SQLite in the app's config dir; no account for local use; iCloud sign-in is optional sync only. Observed factually: first run performed an update check and generated a "staging user ID"; vendor claims no tracking of note content, and nothing in this run contradicted that.
  • T5 works. Agents operate on the files directly and the app picks changes up live (proven end-to-end). A built-in MCP server exists (toggle + generated config, port 33011) — present but not exercised. No public CLI.

(b) kepano/obsidian-skills — find #28 · obsidian/scorecard.md

Repo now lives at obsidianmd/obsidian-skills (the kepano URL redirects), HEAD 462a27f, last commit 2026-11-19, MIT. Exercised with Obsidian 1.10.6. The pack is agent skills (instructions) for Markdown vaults, Bases views, the Obsidian CLI, and headless Sync/Publish.

  • T1 works. A vault is just a folder: the shared export was copied in unchanged and opened directly; book files byte-identical afterwards (app adds only .obsidian/). No import, no conversion.
  • T2 works (one version gate). Following the bases skill, an agent authored Library.base (global folder filter, 2 formulas, 5 views). In-app: All Books = 475 results, Reading Now = 37 results; the filter logic independently evaluated in Python matched ground truth exactly, including the 13 authors at ≥3 (answered via the By Author grouping — Bases has no aggregate-count filter). The kanban view renders "Unknown view type: kanban" on Obsidian 1.10.6; the skill itself documents kanban as needing 1.14+, so this is a version gate, not an authoring error.
  • T3 works. Synthesis note authored per the obsidian-markdown skill with wikilinks as citations; verified in reading view: all 12 wikilinks resolve to corpus documents, 0 unresolved — a machine-checkable citation graph.
  • T4 works (strong). Plain .md + frontmatter in a user folder; no account for local vaults; the account/paid gates sit exactly where the skill documents them (headless Sync/Publish via ob). Skills are open source; the Obsidian app is proprietary freeware.
  • T5 works (one rig caveat). The skills made the file-level work succeed first try — this subject is agent tooling. obsidian-cli was NOT RUN: it needs the installer-registered obsidian shim plus a running app, which the extracted-AppImage rig doesn't provide. ob not run (account-gated by design).

(c) Open Notebook — find #17 · open-notebook/scorecard.md

lfovo/open-notebook, MIT, HEAD 2026-11-18, ~42k stars, actively maintained (a critical RCE in old versions was fixed in v1.8.4 — self-hosters must track security releases). Ran from source: SurrealDB + FastAPI + worker + Ollama (nomic-embed-text, qwen2.5:1.5b).

  • T1 works, with friction — and a caveat that defines this subject. Ingest is a copy into the app's database via REST API (/api/sources/json); there is no folder concept. 475/475 books + 5/5 research docs landed, status "completed", full text verbatim (476 library-side records — the one duplicate, "Scaling People" ×2, was my own pre-batch manual probe, disclosed in the scorecard). First batch hit 8 HTTP 500s (DB write-conflicts under parallel submission); sequential retry landed all 8.
  • T2 fails. With embeddings completed for 476/481 sources, the three fixed questions were put to both Q&A surfaces. Chat: Q1 no answer in 900 s; Q2 hit the product's own 180 s LLM timeout (HTTP 502); Q3 no answer in 900 s even after raising the server timeout to 1200 s. Ask pipeline (/api/search/ask): all three timed out at 600 s. The non-AI search is exact where the question is textual ("Hollis" → exactly 10 records) but status: "Reading" returns a 50-cap with false positives — status is unstructured text, and there is no structured query over the catalog. Architectural point, independent of the slow local model: a RAG pipeline only shows the model retrieved chunks, so catalog-complete aggregates (exact counts, complete lists) are not something this shape can guarantee; a hosted model would fix the timeouts, not the completeness. Scored for the deployment actually run — the local, no-key one.
  • T3 partial. The fixed question ran end-to-end on the Research notebook (61 s) and produced a fluent synthesis that fails groundedness: invented sentences presented in quotation marks as source quotes; a citation to note:randomstring, a note that does not exist; none of the corpus's actual content (observe-first, common denominators, codify categories/nomenclature/routing phrases, Mindprint Audit) appears; and the conclusion is generic both-sides "find a balance" filler where the corpus argues one side. Caveats recorded in the scorecard: 1.5B local model, only 1/5 sources embedded at chat time (all 5 were passed as selected context).
  • T4 partial. Ownership is real — self-hosted, MIT, no account, no telemetry observed, models local — but the notes live inside SurrealDB (86 MB of RocksDB incl. embeddings), not as files. The source folder becomes a stale snapshot the moment anything is edited in-app; leaving means an export project via the API, not picking up a folder.
  • T5 works — strongest API surface of the three. Everything in this test was done API-only: notebooks, 481 ingests, model registration, embedding jobs, search, chat. Caveats: bulk ingest needs pacing/retries; background work needs the separate worker process; in this VM's proxy environment the worker's HTTP client broke on NO_PROXY IPv6 entries until proxy vars were stripped from its env; one VM restart mid-run required manual service restarts (data survived intact).

3. Head-to-head

(a) Lorem #71 (b) obsidian-skills #28 (c) Open Notebook #17
T1 Ingest works — folder opened as-is, files byte-identical works — vault = folder, byte-identical works — full copy into DB; retry friction
T2 Query works — index/search exact on all 3 questions works — agent-authored Bases views exact; kanban version-gated fails — no Q&A request completed locally; text search exact, no structured query
T3 Research partial — grounded synthesis via agent-written note; built-in AI chat NOT RUN (needs key) works — cited synthesis, 12/12 wikilinks resolve partial — ran, but fabricated quotes + phantom citation + filler conclusion
T4 Ownership works — plain .md folder; rebuildable index; no account works — plain .md folder; gates only on Sync/Publish partial — self-hosted & account-free, but data lives in SurrealDB, export is a project
T5 Agent operability works — files live-watched; MCP server present, untested; no CLI works — skills are agent instructions; CLI NOT RUN (rig), ob account-gated works — full REST API drove the entire test
Where the data lives after Your folder Your folder Its database
Upkeep v0.8.4, 2026-11-16, active HEAD 2026-11-19, active, official Obsidian org HEAD 2026-11-18, active; security releases matter (RCE fix v1.8.4)
Cost/account Free, closed-source app Skills MIT; Obsidian free (personal), paid Sync/Publish optional MIT, self-host; you bring models (local or API key)

4. What this means for "where does the data live" — evidence only

  • Two of the three subjects leave the data home exactly where the Second Brain design principle wants it: a plain folder of Markdown files that any tool can read, with the app as a replaceable layer on top. Lorem and the Obsidian route both passed the byte-identical test after a full session — adopting either does not move or transform the data.
  • Open Notebook inverts that: the folder becomes an import source and the database becomes the home. That is a one-way door of a different kind — leaving, or letting another tool work on the notes, means going through its API/export. Its Q&A strengths (T5) are real but sit on top of that inversion, and on a local no-key deployment its Q&A over a real-size catalog did not complete (T2) and did not stay grounded (T3).
  • On the query axis, the two file-based subjects answer catalog questions exactly because the data stays structured text an agent can read, filter, and count (Lorem's index; Bases' declarative views). The database/RAG subject answers topical questions in principle but cannot promise catalog-complete answers — and in this run couldn't return them at all.
  • The agent axis cuts across the data axis: the strongest agent surfaces were the skills pack (instructions that make any agent competent on a vault) and Open Notebook's REST API. Lorem's agent story is files-first with an MCP server still untested.
  • Nothing here decides the module set. The open items the evidence leaves for Ros: Lorem is closed-source with an iCloud-centric sync story and an untested MCP server; the Obsidian route presumes Obsidian as the human app; Open Notebook would need a hosted model and an export discipline to be safe as anything more than a sidecar.

5. Effort log

  • Material build: live catalog pull via the app's own action; ground truth computed from the export itself; corpus assembled from workspace documents (incl. pdftotext of the proposal PDF).
  • (a) Lorem: .deb extracted, Electron app run under Xvfb with sandboxed HOME, driven via CDP. Native GTK folder chooser could not be driven reliably under bare Xvfb (rig limitation, recorded); workspace opened via the app's own persisted-workspace route.
  • (b) Obsidian: AppImage extracted, Xvfb, sandboxed HOME, vault pre-registered; .base authored from the skill doc and validated in Python + in-app screenshots.
  • (c) Open Notebook: no Docker in the VM, so source install per the project's dev docs (SurrealDB binary, uv venv, worker). Ollama's own pull path stalled in this network environment; models were fetched via manual OCI download into Ollama's on-disk layout (one blob arrived truncated and was re-fetched and size-verified). Two environment faults diagnosed and worked around: the worker's HTTP client failing on NO_PROXY IPv6 entries ("Invalid port: ':1]'"), and a mid-run VM restart that wiped /tmp and all processes (stack restarted; DB data intact — the real database lives in surreal_data/mydatabase.db/). Embeddings for 476/481 sources completed before the T2 chat attempts.

6. Draft test-record blocks (for filing to the Finds Library)

Find #71 — Lorem (lorem.md, by ANZI) What ran: Slate 3 test 6 (Second Brain cluster), 2026-11-20. Lorem 0.8.4 opened the real 475-book Personal Library export as a workspace (index 478 entries; book files byte-identical after the session) and via its native .database.json collection ("475 items" table). Fixed questions answered exactly from its index (279/159/37; 37 Reading titles; authors ≥3 exact; GUI search "Hollis" = 10/10). Research corpus indexed live; grounded synthesis placed in the workspace by agent file-write (the vendor's documented workflow); built-in AI chat NOT RUN (requires a user-supplied LLM key). Data stays plain .md in a user folder; no account; MCP server present, untested. Outcome: T1 works · T2 works · T3 partial · T4 works · T5 works. Output: ~/workspace/finds-tests/second-brain-cluster-2026-11-20/lorem/scorecard.md (+ report.md in the test folder root). Reuse proposal: Adopt — scoped to the Second Brain data-layer question. Open items: closed-source app; iCloud-centric sync; MCP untested.

Find #28 — kepano/obsidian-skills What ran: Slate 3 test 6 (Second Brain cluster), 2026-11-20. Repo (now obsidianmd/obsidian-skills, HEAD 2026-11-19) exercised with Obsidian 1.10.6: the 475-note export opened as a vault unchanged (byte-identical after session); an agent authored Library.base per the bases skill — in-app All Books = 475 results, Reading Now = 37 results, logic independently verified exact vs ground truth; kanban view version-gated (needs Obsidian 1.14+). Synthesis note authored per obsidian-markdown with wikilink citations — 12/12 resolve in-app. obsidian-cli NOT RUN (needs installer-registered shim; extracted-AppImage rig), ob headless not run (account/paid-gated, as the skill documents). Outcome: T1 works · T2 works · T3 works · T4 works · T5 works (with the two NOT RUN caveats above). Output: ~/workspace/finds-tests/second-brain-cluster-2026-11-20/obsidian/scorecard.md (+ report.md). Reuse proposal: Reuse later — scoped to the data-layer question: matches Lorem on ownership, beats it on agent-authorable views, but presumes Obsidian as the human app, which is Ros's open decision.

Find #17 — Open Notebook (lfovo/open-notebook) What ran: Slate 3 test 6 (Second Brain cluster), 2026-11-20. Ran from source (SurrealDB + API + worker, local Ollama models — the no-key path). All 475 book notes + 5 research docs ingested via REST API (full text verbatim; 8 initial write-conflict 500s landed on retry). T2: embeddings completed for 476/481 sources, but no Q&A request over the library completed — chat timed out / hit the product's 180 s LLM timeout on all three fixed questions, the ask pipeline timed out on all three at 600 s; plain text search is exact ("Hollis" 10/10), and there is no structured catalog query. T3: synthesis produced but ungrounded (fabricated quotes, a citation to a nonexistent note, generic conclusion vs a one-sided corpus). Data lives in SurrealDB, not files; the whole test was driven API-only. Outcome: T1 works · T2 fails · T3 partial · T4 partial · T5 works. Output: ~/workspace/finds-tests/second-brain-cluster-2026-11-20/open-notebook/scorecard.md (+ report.md, raw chat logs in open-notebook/chat-logs/). Reuse proposal: Skip — scoped to the Second Brain data-layer question: it replaces the file home with its database, against the design principle; its API surface could suit a future sidecar Q&A module over an export, but not the data layer itself.