Implement the 030-bulk-import-sources feature that adds a one-click bulk import button to load all bestiary sources at once, with real-time progress feedback in the side panel and a toast notification when the panel is closed, plus completion/failure reporting with auto-dismiss on success and persistent display on partial failure, while also hardening the bestiary normalizer to handle variable stat blocks (spell summons with special AC/HP/CR) and skip malformed monster entries gracefully

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

specs/030-bulk-import-sources/plan.md

@@ -5,7 +5,7 @@
 
 ## Summary
 
-Add a "Bulk Import All Sources" button to the top bar that opens the stat block side panel with a bulk import prompt. The user confirms a base URL, and the app fetches all ~104 bestiary source files concurrently, normalizes each, and caches them in IndexedDB. Progress is shown via a counter and progress bar in the side panel; if the panel is closed mid-import, a lightweight toast notification takes over progress display.
+Add a "Bulk Import All Sources" button to the top bar that opens the stat block side panel with a bulk import prompt. The user confirms a base URL, and the app fetches all bestiary source files concurrently, normalizes each, and caches them in IndexedDB. Progress is shown via a counter and progress bar in the side panel; if the panel is closed mid-import, a lightweight toast notification takes over progress display.
 
 ## Technical Context
 
@@ -17,7 +17,7 @@ Add a "Bulk Import All Sources" button to the top bar that opens the stat block
 **Project Type**: Web application (React SPA)
 **Performance Goals**: Non-blocking async import; UI remains responsive during ~12.5 MB download
 **Constraints**: All fetches fire concurrently (browser connection pooling); no third-party toast library
-**Scale/Scope**: ~104 sources, ~12.5 MB total data
+**Scale/Scope**: All sources from bestiary index (currently ~102–104), ~12.5 MB total data
 
 ## Constitution Check
 

specs/030-bulk-import-sources/spec.md

@@ -9,7 +9,7 @@
 
 ### User Story 1 - Bulk Load All Sources (Priority: P1)
 
-The user wants to pre-load all 102 bestiary sources at once so that every creature's stat block is instantly available without per-source fetch prompts during gameplay. They click an import button (Import icon) in the top bar, which opens the stat block side panel. The panel shows a description of what will happen, a pre-filled base URL they can edit, and a "Load All" confirmation button. On confirmation, the app fetches all source files concurrently, normalizes them, and caches them in IndexedDB. Already-cached sources are skipped.
+The user wants to pre-load all bestiary sources at once so that every creature's stat block is instantly available without per-source fetch prompts during gameplay. They click an import button (Import icon) in the top bar, which opens the stat block side panel. The panel shows a description of what will happen (including the dynamic source count from the bestiary index), a pre-filled base URL they can edit, and a "Load All" confirmation button. On confirmation, the app fetches all source files concurrently, normalizes them, and caches them in IndexedDB. Already-cached sources are skipped.
 
 **Why this priority**: This is the core feature — enabling one-click loading of the entire bestiary is the primary user value.
 
@@ -18,7 +18,7 @@ The user wants to pre-load all 102 bestiary sources at once so that every creatu
 **Acceptance Scenarios**:
 
 1. **Given** no sources are cached, **When** the user clicks the import button in the top bar, **Then** the stat block side panel opens showing a descriptive explanation, an editable pre-filled base URL, and a "Load All" button.
-2. **Given** the side panel is showing the bulk import prompt, **When** the user clicks "Load All", **Then** the app fires fetch requests for all 102 sources concurrently (appending `bestiary-{sourceCode}.json` to the base URL), normalizes each response, and caches results in IndexedDB.
+2. **Given** the side panel is showing the bulk import prompt, **When** the user clicks "Load All", **Then** the app fires fetch requests for all sources concurrently (appending `bestiary-{sourceCode}.json` to the base URL), normalizes each response, and caches results in IndexedDB.
 3. **Given** some sources are already cached, **When** the user initiates a bulk import, **Then** already-cached sources are skipped and only uncached sources are fetched.
 4. **Given** the user has edited the base URL, **When** they click "Load All", **Then** the app uses their custom base URL for all fetches.
 5. **Given** all fetches complete successfully, **When** the operation finishes, **Then** all creature stat blocks are immediately available for lookup without additional fetch prompts.
@@ -52,7 +52,7 @@ If the user closes the side panel while a bulk import is still in progress, a pe
 
 1. **Given** a bulk import is in progress, **When** the user closes the side panel, **Then** a toast notification appears at the bottom-center of the screen showing the progress counter and progress bar.
 2. **Given** the toast is visible, **When** all sources finish loading successfully, **Then** the toast shows "All sources loaded" and auto-dismisses after a few seconds.
-3. **Given** the toast is visible, **When** some sources fail to load, **Then** the toast shows "Loaded 99/102 sources (3 failed)" (with actual counts) and remains visible until the user dismisses it.
+3. **Given** the toast is visible, **When** some sources fail to load, **Then** the toast shows "Loaded N/T sources (F failed)" with actual counts (e.g., "Loaded 99/102 sources (3 failed)") and remains visible until the user dismisses it.
 
 ---
 
@@ -67,7 +67,7 @@ On completion, the user sees a clear success or partial-failure message. Partial
 **Acceptance Scenarios**:
 
 1. **Given** all sources load successfully, **When** the operation completes, **Then** the side panel (if open) or toast (if panel closed) shows "All sources loaded".
-2. **Given** some sources fail to load, **When** the operation completes, **Then** the message shows "Loaded X/102 sources (Y failed)" with accurate counts.
+2. **Given** some sources fail to load, **When** the operation completes, **Then** the message shows "Loaded X/T sources (Y failed)" with accurate counts (where T is the total number of sources in the index).
 3. **Given** completion message is in the toast, **When** the result is success, **Then** the toast auto-dismisses after a few seconds.
 4. **Given** completion message is in the toast, **When** the result is partial failure, **Then** the toast stays visible until manually dismissed.
 
@@ -76,8 +76,8 @@ On completion, the user sees a clear success or partial-failure message. Partial
 ### Edge Cases
 
 - What happens when the user clicks "Load All" while a bulk import is already in progress? The button should be disabled during an active import.
-- What happens when all 102 sources are already cached? The operation should complete immediately and report "All sources loaded" (0 fetches needed).
-- What happens when the network is completely unavailable? All fetches fail and the result shows "Loaded 0/102 sources (102 failed)".
+- What happens when all sources are already cached? The operation should complete immediately and report "All sources loaded" (0 fetches needed).
+- What happens when the network is completely unavailable? All fetches fail and the result shows "Loaded 0/T sources (T failed)" where T is the total source count.
 - What happens when the user navigates away or refreshes during import? Partially completed caches persist; the user can re-run to pick up remaining sources.
 - What happens if the base URL is empty or invalid? The "Load All" button should be disabled when the URL field is empty.
 
@@ -86,7 +86,7 @@ On completion, the user sees a clear success or partial-failure message. Partial
 ### Functional Requirements
 
 - **FR-001**: System MUST display an import button (Lucide Import icon) in the top bar that opens the stat block side panel with the bulk import prompt.
-- **FR-002**: System MUST show a descriptive text explaining the bulk import operation, including approximate data volume (~12.5 MB) and number of sources (102).
+- **FR-002**: System MUST show a descriptive text explaining the bulk import operation, including approximate data volume (~12.5 MB) and the dynamic number of sources (derived from the bestiary index at runtime).
 - **FR-003**: System MUST pre-fill a base URL (`https://raw.githubusercontent.com/5etools-mirror-3/5etools-src/main/data/bestiary/`) that the user can edit.
 - **FR-004**: System MUST construct individual fetch URLs by appending `bestiary-{sourceCode}.json` to the base URL for each source.
 - **FR-005**: System MUST fire all fetch requests concurrently (browser handles connection pooling).
@@ -109,7 +109,7 @@ On completion, the user sees a clear success or partial-failure message. Partial
 
 ### Measurable Outcomes
 
-- **SC-001**: Users can load all 102 bestiary sources with a single confirmation action.
+- **SC-001**: Users can load all bestiary sources with a single confirmation action.
 - **SC-002**: Users see real-time progress during the bulk import (counter updates as each source completes).
 - **SC-003**: Users can close the side panel during import without losing progress visibility (toast appears).
 - **SC-004**: Already-cached sources are skipped, reducing redundant data transfer on repeat imports.
@@ -118,7 +118,7 @@ On completion, the user sees a clear success or partial-failure message. Partial
 
 ## Assumptions
 
-- The existing bestiary index contains all 102 source codes needed to construct fetch URLs.
+- The existing bestiary index contains all source codes needed to construct fetch URLs (the count is dynamic, currently ~102–104).
 - The existing normalization pipeline handles all source file formats without modification.
 - The existing per-source fetch-and-cache logic serves as the reference implementation for individual source loading.
 - The base URL default matches the pattern already used for single-source fetches.

specs/030-bulk-import-sources/tasks.md

@@ -17,9 +17,12 @@
 
 **Purpose**: Adapter helpers and core hook that all user stories depend on
 
-- [ ] T001 Add `getAllSourceCodes()` helper to `apps/web/src/adapters/bestiary-index-adapter.ts` that returns all source codes from the bestiary index's `sources` object as a `string[]`
-- [ ] T002 Add `getBulkFetchUrl(baseUrl: string, sourceCode: string)` helper to `apps/web/src/adapters/bestiary-index-adapter.ts` that constructs `{baseUrl}bestiary-{sourceCode.toLowerCase()}.json` (ensure trailing slash normalization on baseUrl)
-- [ ] T003 Create `apps/web/src/hooks/use-bulk-import.ts` — the `useBulkImport` hook managing `BulkImportState` (status, total, completed, failed) with a `startImport(baseUrl: string, fetchAndCacheSource, isSourceCached, refreshCache)` method that: filters out already-cached sources via `isSourceCached`, fires all remaining fetches concurrently via `Promise.allSettled()`, increments completed/failed counters as each settles, calls `refreshCache()` once when all settle, and transitions status to `complete` or `partial-failure`
+- [x] T001 Add `getAllSourceCodes()` helper to `apps/web/src/adapters/bestiary-index-adapter.ts` that returns all source codes from the bestiary index's `sources` object as a `string[]`
+- [x] T002 Refactor `getDefaultFetchUrl` in `apps/web/src/adapters/bestiary-index-adapter.ts` to accept an optional `baseUrl` parameter: `getDefaultFetchUrl(sourceCode: string, baseUrl?: string): string`. When `baseUrl` is provided, construct `{baseUrl}bestiary-{sourceCode.toLowerCase()}.json` (with trailing-slash normalization). When omitted, use the existing hardcoded default. Update existing call sites (no behavior change for current callers).
+- [x] T003 Create `apps/web/src/hooks/use-bulk-import.ts` — the `useBulkImport` hook managing `BulkImportState` with a `startImport(baseUrl: string, fetchAndCacheSource, isSourceCached, refreshCache)` method. State shape: `{ status: 'idle' | 'loading' | 'complete' | 'partial-failure', total: number, completed: number, failed: number }` where `total` = ALL sources in the index (including pre-cached), `completed` = successfully loaded (fetched + pre-cached/skipped), `failed` = fetch failures. On start: set `total` to `getAllSourceCodes().length`, immediately count already-cached sources into `completed`, fire remaining fetches concurrently via `Promise.allSettled()`, increment completed/failed as each settles, call `refreshCache()` once when all settle, transition status to `complete` (failed === 0) or `partial-failure`.
+
+- [x] T003a [P] Write tests in `apps/web/src/__tests__/bestiary-index-helpers.test.ts` for: `getAllSourceCodes()` returns all keys from the index's `sources` object; `getDefaultFetchUrl(sourceCode, baseUrl)` constructs correct URL with trailing-slash normalization and lowercases the source code in the filename.
+- [x] T003b [P] Write tests in `apps/web/src/__tests__/use-bulk-import.test.ts` for: starts with `idle` status; skips already-cached sources (counts them into `completed`); increments `completed` on successful fetch; increments `failed` on rejected fetch; transitions to `complete` when all succeed; transitions to `partial-failure` when any fetch fails; all-cached edge case immediately transitions to `complete` with 0 fetches; calls `refreshCache()` exactly once when all settle.
 
 **Checkpoint**: Foundation ready — user story implementation can begin
 
@@ -33,10 +36,10 @@
 
 ### Implementation for User Story 1
 
-- [ ] T004 [US1] Create `apps/web/src/components/bulk-import-prompt.tsx` — component showing descriptive text ("Load stat block data for all 102 sources at once. This will download approximately 12.5 MB..."), an editable Input pre-filled with `https://raw.githubusercontent.com/5etools-mirror-3/5etools-src/main/data/bestiary/`, and a "Load All" Button (disabled when URL is empty or import is active). On click, calls a provided `onStartImport(baseUrl)` callback.
-- [ ] T005 [US1] Add Import button (Lucide `Import` icon) to `apps/web/src/components/action-bar.tsx` in the top bar area. Clicking it calls a new `onBulkImport` callback prop.
-- [ ] T006 [US1] Add bulk import mode to `apps/web/src/components/stat-block-panel.tsx` — when a new `bulkImportMode` prop is true, render `BulkImportPrompt` instead of the normal stat block or source fetch prompt content. Pass through `onStartImport` and bulk import state props.
-- [ ] T007 [US1] Wire bulk import in `apps/web/src/App.tsx` — add `bulkImportMode` state, pass `onBulkImport` to ActionBar (sets mode + opens panel), pass `bulkImportMode` and `useBulkImport` state to StatBlockPanel, call `useBulkImport.startImport()` with `fetchAndCacheSource`, `isSourceCached`, and `refreshCache` from `useBestiary`.
+- [x] T004 [US1] Create `apps/web/src/components/bulk-import-prompt.tsx` — component showing descriptive text ("Load stat block data for all {totalSources} sources at once. This will download approximately 12.5 MB..." where `totalSources` is derived from `getAllSourceCodes().length`), an editable Input pre-filled with `https://raw.githubusercontent.com/5etools-mirror-3/5etools-src/main/data/bestiary/`, and a "Load All" Button (disabled when URL is empty/whitespace-only or import is active). On click, calls a provided `onStartImport(baseUrl)` callback.
+- [x] T005 [US1] Add Import button (Lucide `Import` icon) to `apps/web/src/components/action-bar.tsx` in the top bar area. Clicking it calls a new `onBulkImport` callback prop.
+- [x] T006 [US1] Add bulk import mode to `apps/web/src/components/stat-block-panel.tsx` — when a new `bulkImportMode` prop is true, render `BulkImportPrompt` instead of the normal stat block or source fetch prompt content. Pass through `onStartImport` and bulk import state props.
+- [x] T007 [US1] Wire bulk import in `apps/web/src/App.tsx` — add `bulkImportMode` state, pass `onBulkImport` to ActionBar (sets mode + opens panel), pass `bulkImportMode` and `useBulkImport` state to StatBlockPanel, call `useBulkImport.startImport()` with `fetchAndCacheSource`, `isSourceCached`, and `refreshCache` from `useBestiary`.
 
 **Checkpoint**: User Story 1 fully functional — Import button opens panel, "Load All" fetches all sources, cached sources skipped
 
@@ -50,7 +53,7 @@
 
 ### Implementation for User Story 2
 
-- [ ] T008 [US2] Add progress display to `apps/web/src/components/bulk-import-prompt.tsx` — when import status is `loading`, replace the "Load All" button area with a text counter ("Loading sources... {completed}/{total}") and a Tailwind-styled progress bar (`<div>` with percentage width based on `(completed + failed) / total`). The component receives `BulkImportState` as a prop.
+- [x] T008 [US2] Add progress display to `apps/web/src/components/bulk-import-prompt.tsx` — when import status is `loading`, replace the "Load All" button area with a text counter ("Loading sources... {completed}/{total}") and a Tailwind-styled progress bar (`<div>` with percentage width based on `(completed + failed) / total`). The component receives `BulkImportState` as a prop.
 
 **Checkpoint**: User Stories 1 AND 2 work together — full import flow with live progress
 
@@ -64,8 +67,8 @@
 
 ### Implementation for User Story 3
 
-- [ ] T009 [P] [US3] Create `apps/web/src/components/toast.tsx` — lightweight toast component using `ReactDOM.createPortal` to `document.body`. Renders at bottom-center with fixed positioning. Shows: message text, optional progress bar, optional dismiss button (X). Accepts `onDismiss` callback. Styled with Tailwind (dark background, rounded, shadow, z-50).
-- [ ] T010 [US3] Wire toast visibility in `apps/web/src/App.tsx` — show the toast when bulk import status is `loading` AND the stat block panel is closed (or `bulkImportMode` is false). Derive toast message from `BulkImportState`: "Loading sources... {completed}/{total}" with progress value `(completed + failed) / total`. Hide toast when panel reopens in bulk import mode.
+- [x] T009 [P] [US3] Create `apps/web/src/components/toast.tsx` — lightweight toast component using `ReactDOM.createPortal` to `document.body`. Renders at bottom-center with fixed positioning. Shows: message text, optional progress bar, optional dismiss button (X). Accepts `onDismiss` callback. Styled with Tailwind (dark background, rounded, shadow, z-50).
+- [x] T010 [US3] Wire toast visibility in `apps/web/src/App.tsx` — show the toast when bulk import status is `loading` AND the stat block panel is closed (or `bulkImportMode` is false). Derive toast message from `BulkImportState`: "Loading sources... {completed}/{total}" with progress value `(completed + failed) / total`. Hide toast when panel reopens in bulk import mode.
 
 **Checkpoint**: User Story 3 functional — closing panel during import shows toast with progress
 
@@ -79,9 +82,9 @@
 
 ### Implementation for User Story 4
 
-- [ ] T011 [US4] Add completion states to `apps/web/src/components/bulk-import-prompt.tsx` — when status is `complete`, show "All sources loaded" success message. When status is `partial-failure`, show "Loaded {completed}/{total + completed} sources ({failed} failed)" message. Include a "Done" button to reset bulk import mode.
-- [ ] T012 [US4] Add completion behavior to toast in `apps/web/src/App.tsx` — when status is `complete`, show "All sources loaded" toast with `autoDismissMs` (e.g., 3000ms) and auto-hide via `setTimeout`. When status is `partial-failure`, show count message with dismiss button, no auto-dismiss. On dismiss, reset bulk import state to `idle`.
-- [ ] T013 [US4] Add auto-dismiss support to `apps/web/src/components/toast.tsx` — accept optional `autoDismissMs` prop. When set, start a `setTimeout` on mount that calls `onDismiss` after the delay. Clear timeout on unmount.
+- [x] T011 [US4] Add completion states to `apps/web/src/components/bulk-import-prompt.tsx` — when status is `complete`, show "All sources loaded" success message. When status is `partial-failure`, show "Loaded {completed}/{total} sources ({failed} failed)" message. Include a "Done" button to reset bulk import mode.
+- [x] T012 [US4] Add completion behavior to toast in `apps/web/src/App.tsx` — when status is `complete`, show "All sources loaded" toast with `autoDismissMs` (e.g., 3000ms) and auto-hide via `setTimeout`. When status is `partial-failure`, show count message with dismiss button, no auto-dismiss. On dismiss, reset bulk import state to `idle`.
+- [x] T013 [US4] Add auto-dismiss support to `apps/web/src/components/toast.tsx` — accept optional `autoDismissMs` prop. When set, start a `setTimeout` on mount that calls `onDismiss` after the delay. Clear timeout on unmount.
 
 **Checkpoint**: All user stories complete — full flow with progress, toast, and completion reporting
 
@@ -91,9 +94,9 @@
 
 **Purpose**: Edge cases and cleanup
 
-- [ ] T014 Disable Import button in `apps/web/src/components/action-bar.tsx` while bulk import status is `loading` to prevent double-trigger
-- [ ] T015 Handle all-cached edge case in `apps/web/src/hooks/use-bulk-import.ts` — if all sources are already cached (0 to fetch), immediately transition to `complete` status without firing any fetches
-- [ ] T016 Run `pnpm check` (knip + format + lint + typecheck + test) and fix any issues
+- [x] T014 Disable Import button in `apps/web/src/components/action-bar.tsx` while bulk import status is `loading` to prevent double-trigger
+- [x] T015 Handle all-cached edge case in `apps/web/src/hooks/use-bulk-import.ts` — if all sources are already cached (0 to fetch), immediately transition to `complete` status without firing any fetches
+- [x] T016 Run `pnpm check` (knip + format + lint + typecheck + test) and fix any issues
 
 ---
 
@@ -118,6 +121,7 @@
 ### Parallel Opportunities
 
 - T001 and T002 can run in parallel (different functions, same file)
+- T003a and T003b can run in parallel with each other and with T003 (different files)
 - T009 (toast component) can run in parallel with T004–T008 (different files)
 - T011 and T013 can run in parallel (different files)
 
@@ -149,7 +153,7 @@ Task: T009 "Create toast.tsx" (US3) — runs in parallel, different file
 2. Complete Phase 2: US1 — Import button, prompt, fetch logic
 3. Complete Phase 3: US2 — Progress counter + bar in panel
 4. **STOP and VALIDATE**: Full import flow works with progress feedback
-5. This delivers the core user value with 10 tasks (T001–T008)
+5. This delivers the core user value with tasks T001–T008 (plus T003a, T003b tests)
 
 ### Incremental Delivery
 
@@ -168,4 +172,4 @@ Task: T009 "Create toast.tsx" (US3) — runs in parallel, different file
 - US1 and US2 share `bulk-import-prompt.tsx` — US2 extends the component from US1
 - US3's toast component (T009) is fully independent and can be built any time
 - US4 adds completion behavior to both panel (from US1) and toast (from US3)
-- No test tasks included — not explicitly requested in spec
+- Test tasks (T003a, T003b) cover foundational helpers and hook logic