initiative/specs/030-bulk-import-sources/spec.md

Feature Specification: Bulk Import All Sources

Feature Branch: 030-bulk-import-sources Created: 2026-03-10 Status: Draft Input: User description: "Add a Bulk Import All Sources feature to the on-demand bestiary system"

User Scenarios & Testing (mandatory)

User Story 1 - Bulk Load All Sources (Priority: P1)

The user wants to pre-load all bestiary sources at once so that every creature's stat block is instantly available without per-source fetch prompts during gameplay. They click an import button (Import icon) in the top bar, which opens the stat block side panel. The panel shows a description of what will happen (including the dynamic source count from the bestiary index), a pre-filled base URL they can edit, and a "Load All" confirmation button. On confirmation, the app fetches all source files concurrently, normalizes them, and caches them in IndexedDB. Already-cached sources are skipped.

Why this priority: This is the core feature — enabling one-click loading of the entire bestiary is the primary user value.

Independent Test: Can be fully tested by clicking the import button, confirming the load, and verifying that all sources become available for stat block lookups.

Acceptance Scenarios:

1. Given no sources are cached, When the user clicks the import button in the top bar, Then the stat block side panel opens showing a descriptive explanation, an editable pre-filled base URL, and a "Load All" button.
2. Given the side panel is showing the bulk import prompt, When the user clicks "Load All", Then the app fires fetch requests for all sources concurrently (appending bestiary-{sourceCode}.json to the base URL), normalizes each response, and caches results in IndexedDB.
3. Given some sources are already cached, When the user initiates a bulk import, Then already-cached sources are skipped and only uncached sources are fetched.
4. Given the user has edited the base URL, When they click "Load All", Then the app uses their custom base URL for all fetches.
5. Given all fetches complete successfully, When the operation finishes, Then all creature stat blocks are immediately available for lookup without additional fetch prompts.

---

User Story 2 - Progress Feedback in Side Panel (Priority: P1)

While the bulk import is in progress, the user sees a text counter ("Loading sources... 34/102") and a progress bar in the side panel, giving them confidence the operation is proceeding.

Why this priority: Without progress feedback, the user has no way to know if the operation is working or stalled, especially for a ~12.5 MB download.

Independent Test: Can be tested by initiating a bulk import and observing the counter and progress bar update as sources complete.

Acceptance Scenarios:

1. Given a bulk import is in progress, When the user views the side panel, Then they see a text counter showing completed/total (e.g., "Loading sources... 34/102") and a visual progress bar.
2. Given sources complete at different times, When each source finishes loading, Then the counter and progress bar update immediately.

---

User Story 3 - Toast Notification on Panel Close (Priority: P2)

If the user closes the side panel while a bulk import is still in progress, a persistent toast notification appears at the bottom-center of the screen showing the same progress text and progress bar, so the user can continue using the app without losing visibility into the import status.

Why this priority: Allows the user to multitask while the import runs, which is important for a potentially long operation. Lower priority because the side panel already shows progress.

Independent Test: Can be tested by starting a bulk import, closing the side panel, and verifying the toast appears with progress information.

Acceptance Scenarios:

1. Given a bulk import is in progress, When the user closes the side panel, Then a toast notification appears at the bottom-center of the screen showing the progress counter and progress bar.
2. Given the toast is visible, When all sources finish loading successfully, Then the toast shows "All sources loaded" and auto-dismisses after a few seconds.
3. Given the toast is visible, When some sources fail to load, Then the toast shows "Loaded N/T sources (F failed)" with actual counts (e.g., "Loaded 99/102 sources (3 failed)") and remains visible until the user dismisses it.

---

User Story 4 - Completion and Failure Reporting (Priority: P2)

On completion, the user sees a clear success or partial-failure message. Partial failures report how many sources succeeded and how many failed, so the user knows if they need to retry.

Why this priority: Essential for the user to know the outcome, but slightly lower than the core loading and progress functionality.

Independent Test: Can be tested by simulating network failures for some sources and verifying the correct counts appear.

Acceptance Scenarios:

1. Given all sources load successfully, When the operation completes, Then the side panel (if open) or toast (if panel closed) shows "All sources loaded".
2. Given some sources fail to load, When the operation completes, Then the message shows "Loaded X/T sources (Y failed)" with accurate counts (where T is the total number of sources in the index).
3. Given completion message is in the toast, When the result is success, Then the toast auto-dismisses after a few seconds.
4. Given completion message is in the toast, When the result is partial failure, Then the toast stays visible until manually dismissed.

---

Edge Cases

• What happens when the user clicks "Load All" while a bulk import is already in progress? The button should be disabled during an active import.
• What happens when all sources are already cached? The operation should complete immediately and report "All sources loaded" (0 fetches needed).
• What happens when the network is completely unavailable? All fetches fail and the result shows "Loaded 0/T sources (T failed)" where T is the total source count.
• What happens when the user navigates away or refreshes during import? Partially completed caches persist; the user can re-run to pick up remaining sources.
• What happens if the base URL is empty or invalid? The "Load All" button should be disabled when the URL field is empty.

Requirements (mandatory)

Functional Requirements

• FR-001: System MUST display an import button (Lucide Import icon) in the top bar that opens the stat block side panel with the bulk import prompt.
• FR-002: System MUST show a descriptive text explaining the bulk import operation, including approximate data volume (~12.5 MB) and the dynamic number of sources (derived from the bestiary index at runtime).
• FR-003: System MUST pre-fill a base URL (https://raw.githubusercontent.com/5etools-mirror-3/5etools-src/main/data/bestiary/) that the user can edit.
• FR-004: System MUST construct individual fetch URLs by appending bestiary-{sourceCode}.json to the base URL for each source.
• FR-005: System MUST fire all fetch requests concurrently (browser handles connection pooling).
• FR-006: System MUST skip sources that are already cached in IndexedDB.
• FR-007: System MUST normalize fetched data using the existing normalization pipeline before caching.
• FR-008: System MUST show a text counter ("Loading sources... N/T") and progress bar during the operation.
• FR-009: System MUST show a toast notification with progress when the user closes the side panel during an active import.
• FR-010: System MUST auto-dismiss the success toast after a few seconds.
• FR-011: System MUST keep the partial-failure toast visible until the user dismisses it.
• FR-012: The toast system MUST be a lightweight custom component — no third-party toast library.
• FR-013: The bulk import MUST run asynchronously and not block the rest of the app.
• FR-014: The user MUST explicitly provide/confirm the URL before any fetches occur (app never auto-fetches copyrighted content).

Key Entities

• Bulk Import Operation: Tracks total sources, completed count, failed count, and current status (idle/loading/complete/partial-failure).
• Toast Notification: Lightweight UI element at bottom-center of screen with text, optional progress bar, and optional dismiss button.

Success Criteria (mandatory)

Measurable Outcomes

• SC-001: Users can load all bestiary sources with a single confirmation action.
• SC-002: Users see real-time progress during the bulk import (counter updates as each source completes).
• SC-003: Users can close the side panel during import without losing progress visibility (toast appears).
• SC-004: Already-cached sources are skipped, reducing redundant data transfer on repeat imports.
• SC-005: The rest of the app remains fully interactive during the import operation.
• SC-006: Users receive clear outcome reporting distinguishing full success from partial failure.

Assumptions

• The existing bestiary index contains all source codes needed to construct fetch URLs (the count is dynamic, currently ~102–104).
• The existing normalization pipeline handles all source file formats without modification.
• The existing per-source fetch-and-cache logic serves as the reference implementation for individual source loading.
• The base URL default matches the pattern already used for single-source fetches.

Dependencies

• Feature 029 (on-demand bestiary): bestiary cache, bestiary index adapter, normalization pipeline, bestiary hook.