initiative/specs/030-bulk-import-sources/spec.md

# Feature Specification: Bulk Import All Sources

**Feature Branch**: `030-bulk-import-sources`
**Created**: 2026-03-10
**Status**: Draft
**Input**: User description: "Add a Bulk Import All Sources feature to the on-demand bestiary system"

## User Scenarios & Testing *(mandatory)*

### User Story 1 - Bulk Load All Sources (Priority: P1)

The user wants to pre-load all 102 bestiary sources at once so that every creature's stat block is instantly available without per-source fetch prompts during gameplay. They click an import button (Import icon) in the top bar, which opens the stat block side panel. The panel shows a description of what will happen, a pre-filled base URL they can edit, and a "Load All" confirmation button. On confirmation, the app fetches all source files concurrently, normalizes them, and caches them in IndexedDB. Already-cached sources are skipped.

**Why this priority**: This is the core feature — enabling one-click loading of the entire bestiary is the primary user value.

**Independent Test**: Can be fully tested by clicking the import button, confirming the load, and verifying that all sources become available for stat block lookups.

**Acceptance Scenarios**:

1. **Given** no sources are cached, **When** the user clicks the import button in the top bar, **Then** the stat block side panel opens showing a descriptive explanation, an editable pre-filled base URL, and a "Load All" button.
2. **Given** the side panel is showing the bulk import prompt, **When** the user clicks "Load All", **Then** the app fires fetch requests for all 102 sources concurrently (appending `bestiary-{sourceCode}.json` to the base URL), normalizes each response, and caches results in IndexedDB.
3. **Given** some sources are already cached, **When** the user initiates a bulk import, **Then** already-cached sources are skipped and only uncached sources are fetched.
4. **Given** the user has edited the base URL, **When** they click "Load All", **Then** the app uses their custom base URL for all fetches.
5. **Given** all fetches complete successfully, **When** the operation finishes, **Then** all creature stat blocks are immediately available for lookup without additional fetch prompts.

---

### User Story 2 - Progress Feedback in Side Panel (Priority: P1)

While the bulk import is in progress, the user sees a text counter ("Loading sources... 34/102") and a progress bar in the side panel, giving them confidence the operation is proceeding.

**Why this priority**: Without progress feedback, the user has no way to know if the operation is working or stalled, especially for a ~12.5 MB download.

**Independent Test**: Can be tested by initiating a bulk import and observing the counter and progress bar update as sources complete.

**Acceptance Scenarios**:

1. **Given** a bulk import is in progress, **When** the user views the side panel, **Then** they see a text counter showing completed/total (e.g., "Loading sources... 34/102") and a visual progress bar.
2. **Given** sources complete at different times, **When** each source finishes loading, **Then** the counter and progress bar update immediately.

---

### User Story 3 - Toast Notification on Panel Close (Priority: P2)

If the user closes the side panel while a bulk import is still in progress, a persistent toast notification appears at the bottom-center of the screen showing the same progress text and progress bar, so the user can continue using the app without losing visibility into the import status.

**Why this priority**: Allows the user to multitask while the import runs, which is important for a potentially long operation. Lower priority because the side panel already shows progress.

**Independent Test**: Can be tested by starting a bulk import, closing the side panel, and verifying the toast appears with progress information.

**Acceptance Scenarios**:

1. **Given** a bulk import is in progress, **When** the user closes the side panel, **Then** a toast notification appears at the bottom-center of the screen showing the progress counter and progress bar.
2. **Given** the toast is visible, **When** all sources finish loading successfully, **Then** the toast shows "All sources loaded" and auto-dismisses after a few seconds.
3. **Given** the toast is visible, **When** some sources fail to load, **Then** the toast shows "Loaded 99/102 sources (3 failed)" (with actual counts) and remains visible until the user dismisses it.

---

### User Story 4 - Completion and Failure Reporting (Priority: P2)

On completion, the user sees a clear success or partial-failure message. Partial failures report how many sources succeeded and how many failed, so the user knows if they need to retry.

**Why this priority**: Essential for the user to know the outcome, but slightly lower than the core loading and progress functionality.

**Independent Test**: Can be tested by simulating network failures for some sources and verifying the correct counts appear.

**Acceptance Scenarios**:

1. **Given** all sources load successfully, **When** the operation completes, **Then** the side panel (if open) or toast (if panel closed) shows "All sources loaded".
2. **Given** some sources fail to load, **When** the operation completes, **Then** the message shows "Loaded X/102 sources (Y failed)" with accurate counts.
3. **Given** completion message is in the toast, **When** the result is success, **Then** the toast auto-dismisses after a few seconds.
4. **Given** completion message is in the toast, **When** the result is partial failure, **Then** the toast stays visible until manually dismissed.

---

### Edge Cases

- What happens when the user clicks "Load All" while a bulk import is already in progress? The button should be disabled during an active import.
- What happens when all 102 sources are already cached? The operation should complete immediately and report "All sources loaded" (0 fetches needed).
- What happens when the network is completely unavailable? All fetches fail and the result shows "Loaded 0/102 sources (102 failed)".
- What happens when the user navigates away or refreshes during import? Partially completed caches persist; the user can re-run to pick up remaining sources.
- What happens if the base URL is empty or invalid? The "Load All" button should be disabled when the URL field is empty.

## Requirements *(mandatory)*

### Functional Requirements

- **FR-001**: System MUST display an import button (Lucide Import icon) in the top bar that opens the stat block side panel with the bulk import prompt.
- **FR-002**: System MUST show a descriptive text explaining the bulk import operation, including approximate data volume (~12.5 MB) and number of sources (102).
- **FR-003**: System MUST pre-fill a base URL (`https://raw.githubusercontent.com/5etools-mirror-3/5etools-src/main/data/bestiary/`) that the user can edit.
- **FR-004**: System MUST construct individual fetch URLs by appending `bestiary-{sourceCode}.json` to the base URL for each source.
- **FR-005**: System MUST fire all fetch requests concurrently (browser handles connection pooling).
- **FR-006**: System MUST skip sources that are already cached in IndexedDB.
- **FR-007**: System MUST normalize fetched data using the existing normalization pipeline before caching.
- **FR-008**: System MUST show a text counter ("Loading sources... N/T") and progress bar during the operation.
- **FR-009**: System MUST show a toast notification with progress when the user closes the side panel during an active import.
- **FR-010**: System MUST auto-dismiss the success toast after a few seconds.
- **FR-011**: System MUST keep the partial-failure toast visible until the user dismisses it.
- **FR-012**: The toast system MUST be a lightweight custom component — no third-party toast library.
- **FR-013**: The bulk import MUST run asynchronously and not block the rest of the app.
- **FR-014**: The user MUST explicitly provide/confirm the URL before any fetches occur (app never auto-fetches copyrighted content).

### Key Entities

- **Bulk Import Operation**: Tracks total sources, completed count, failed count, and current status (idle/loading/complete/partial-failure).
- **Toast Notification**: Lightweight UI element at bottom-center of screen with text, optional progress bar, and optional dismiss button.

## Success Criteria *(mandatory)*

### Measurable Outcomes

- **SC-001**: Users can load all 102 bestiary sources with a single confirmation action.
- **SC-002**: Users see real-time progress during the bulk import (counter updates as each source completes).
- **SC-003**: Users can close the side panel during import without losing progress visibility (toast appears).
- **SC-004**: Already-cached sources are skipped, reducing redundant data transfer on repeat imports.
- **SC-005**: The rest of the app remains fully interactive during the import operation.
- **SC-006**: Users receive clear outcome reporting distinguishing full success from partial failure.

## Assumptions

- The existing bestiary index contains all 102 source codes needed to construct fetch URLs.
- The existing normalization pipeline handles all source file formats without modification.
- The existing per-source fetch-and-cache logic serves as the reference implementation for individual source loading.
- The base URL default matches the pattern already used for single-source fetches.

## Dependencies

- Feature 029 (on-demand bestiary): bestiary cache, bestiary index adapter, normalization pipeline, bestiary hook.