From c44c0e5569b756c36190e196b4edd69854aa93cc Mon Sep 17 00:00:00 2001 From: rektide Date: Mon, 9 Feb 2026 21:29:10 -0500 Subject: [PATCH 1/5] Add documentation for config sourcing and dynamic configs - doc/discovery/config-sourcing.md: Complete analysis of c12's configuration pipeline, showing how defu merges configs from multiple sources with mermaid diagrams - doc/discovery/dynamic-configs.md: Exploration of dynamic configuration sources including drop-in directories, provider pattern design, and layering provenance --- doc/discovery/config-sourcing.md | 394 ++++++++++++++++++++++ doc/discovery/dynamic-configs.md | 538 +++++++++++++++++++++++++++++++ 2 files changed, 932 insertions(+) create mode 100644 doc/discovery/config-sourcing.md create mode 100644 doc/discovery/dynamic-configs.md diff --git a/doc/discovery/config-sourcing.md b/doc/discovery/config-sourcing.md new file mode 100644 index 0000000..8f03dd1 --- /dev/null +++ b/doc/discovery/config-sourcing.md @@ -0,0 +1,394 @@ +# Configuration Sourcing in c12 + +This document explains how c12 layers together configuration from various sources, using `defu` for deep merging. + +## Overview + +c12's configuration loading follows a multi-stage pipeline that sources configuration from multiple locations, merges them using `unjs/defu`, and applies transformations. + +## Execution Pipeline + +```mermaid +flowchart TD + Start[Start: loadConfig] --> Normalize[Normalize options] + Normalize --> SetupMerger[Setup merger
(defu or custom)] + SetupMerger --> LoadEnv{dotenv enabled?} + LoadEnv -->|yes| LoadDotenv[Load .env files] + LoadDotenv --> MainConfig + LoadEnv -->|no| MainConfig[Load main config file
via resolveConfig] + MainConfig --> LoadRC{rcFile enabled?} + LoadRC -->|yes| LoadRcFiles[Load RC files
cwd, workspace, home] + LoadRcFiles --> MergeRC[Merge RC sources
with defu] + LoadRC -->|no| PackageJson + MergeRC --> PackageJson{packageJson enabled?} + PackageJson -->|yes| LoadPkg[Read package.json] + LoadPkg --> MergePkg[Merge package.json values
with defu] + PackageJson -->|no| ResolveFuncs + MergePkg --> ResolveFuncs[Resolve functions
in rawConfigs] + ResolveFuncs --> Combine{Main config is array?} + Combine -->|yes| UseArray[Use array directly
no merging] + Combine -->|no| MergeSources[Merge sources with defu
overrides → main → rc →
packageJson → defaultConfig] + MergeSources --> Extend{extends enabled?} + Extend -->|yes| ProcessExtends[Process extends
recursively] + ProcessExtends --> MergeLayers[Merge extended
layers with defu] + Extend -->|no| ApplyDefaults + MergeLayers --> ApplyDefaults + ApplyDefaults --> ApplyDefaults{defaults provided?} + ApplyDefaults -->|yes| MergeDefaults[Merge defaults
with defu] + ApplyDefaults -->|no| Cleanup + MergeDefaults --> Cleanup{omit$Keys enabled?} + Cleanup -->|yes| RemoveDollar[Remove $ prefixed keys] + Cleanup -->|no| Verify + RemoveDollar --> Verify{configFileRequired?} + Verify -->|yes| CheckExists[Check file exists
or throw error] + Verify -->|no| Return + CheckExists --> Return[Return resolved config] +``` + +## Main Config Loading Flow + +```mermaid +flowchart TD + subgraph LoadConfig [loadConfig function] + direction TB + A[Normalized options] --> B[Load dotenv] + B --> C[Load main config] + C --> D[Load RC files] + D --> E[Load package.json] + E --> F[Resolve config functions] + F --> G[Merge all sources] + G --> H[Process extends] + H --> I[Apply defaults] + I --> J[Final cleanup] + J --> K[Return resolved config] + end +``` + +## Config Sources Priority + +When all sources are present, c12 merges them in this order (highest to lowest priority): + +```mermaid +flowchart LR + overrides[overrides
Highest Priority] --> main[main config file] + main --> rc[RC files] + rc --> packageJson[package.json] + packageJson --> defaultConfig[defaultConfig
Lowest Priority] +``` + +The merge happens at `src/loader.ts:158-163`: + +```typescript +r.config = _merger( + configs.overrides, + configs.main, + configs.rc, + configs.packageJson, + configs.defaultConfig, +) as T; +``` + +## Where `defu` is Used + +`defu` (or a custom merger) is used at several points in the pipeline: + +### 1. Main Merger Setup +**Location**: `src/loader.ts:71` +```typescript +const _merger = options.merger || defu; +``` + +### 2. RC File Merging +**Location**: `src/loader.ts:128` +RC files from cwd, workspace, and home are merged: +```typescript +rawConfigs.rc = _merger({} as T, ...rcSources); +``` + +### 3. package.json Value Merging +**Location**: `src/loader.ts:140` +Multiple keys from package.json are merged: +```typescript +rawConfigs.packageJson = _merger({} as T, ...values); +``` + +### 4. Main Source Merging +**Location**: `src/loader.ts:158-163` +All primary config sources are merged: +```typescript +r.config = _merger( + configs.overrides, + configs.main, + configs.rc, + configs.packageJson, + configs.defaultConfig, +) as T; +``` + +### 5. Extended Layers Merging +**Location**: `src/loader.ts:171` +After processing `extends`, all layers are merged into the main config: +```typescript +r.config = _merger(r.config, ...r.layers!.map((e) => e.config)) as T; +``` + +### 6. Defaults Application +**Location**: `src/loader.ts:194` +Default config has the lowest priority: +```typescript +r.config = _merger(r.config, options.defaults) as T; +``` + +### 7. Environment-Specific Config Merging +**Location**: `src/loader.ts:418` (in `resolveConfig`) +Env-specific config overrides the base config: +```typescript +res.config = _merger(envConfig, res.config); +``` + +### 8. Meta Merging +**Location**: `src/loader.ts:423` (in `resolveConfig`) +Meta from source options and config are merged: +```typescript +res.meta = defu(res.sourceOptions!.meta, res.config!.$meta) as MT; +``` + +### 9. Source Overrides Merging +**Location**: `src/loader.ts:428` (in `resolveConfig`) +Per-source overrides are applied: +```typescript +res.config = _merger(res.sourceOptions!.overrides, res.config) as T; +``` + +## resolveConfig: Loading Individual Config Layers + +The `resolveConfig` function handles loading individual configuration files (including extended configs): + +```mermaid +flowchart TD + Start[resolveConfig start] --> CustomResolver{custom resolver?} + CustomResolver -->|yes| TryCustom[Try custom resolver] + TryCustom --> HasResult{result?} + HasResult -->|yes| ReturnCustom[Return custom result] + HasResult -->|no| GigetCheck + CustomResolver -->|no| GigetCheck{giget URI?} + GigetCheck -->|yes| Download[Download with giget
to local path] + Download --> NpmCheck + GigetCheck -->|no| NpmCheck{npm package?} + NpmCheck -->|yes| ResolvePkg[Resolve npm package] + ResolvePkg --> LocalImport + NpmCheck -->|no| LocalImport[Import from local FS] + LocalImport --> GetExt{has extension?} + GetExt -->|no| UseDir[Treat as directory
use configFile name] + GetExt -->|yes| TryResolve + UseDir --> TryResolve[Try resolve with
multiple paths] + TryResolve --> FileExists{file exists?} + FileExists -->|no| ReturnEmpty[Return empty config] + FileExists -->|yes| CheckFormat + CheckFormat{Async loader?} + CheckFormat -->|yes| ParseAsync[Parse with
confbox parsers] + CheckFormat -->|no| ImportModule[Import module
with jiti fallback] + ImportModule --> IsFunction{is function?} + ParseAsync --> IsFunction + IsFunction -->|yes| CallFunction[Call with context] + IsFunction -->|no| EnvCheck + CallFunction --> EnvCheck + EnvCheck{envName set?} + EnvCheck -->|yes| MergeEnv[Merge env-specific
config with defu] + EnvCheck -->|no| MergeMeta + MergeEnv --> MergeMeta[Merge meta with defu] + MergeMeta --> SourceOverrides{source overrides?} + SourceOverrides -->|yes| ApplyOverrides[Apply with defu] + SourceOverrides -->|no| NormalizePaths + ApplyOverrides --> NormalizePaths[Normalize paths] + NormalizePaths --> ReturnResult[Return resolved config] +``` + +## Environment-Specific Configuration + +Each config layer can define environment-specific overrides: + +```mermaid +flowchart LR + Config[Config object] --> HasEnv{Has envName?} + HasEnv -->|no| Skip[Skip env merging] + HasEnv -->|yes| CheckKeys{Has $
or $env.?} + CheckKeys -->|yes| ExtractEnv[Extract env config] + ExtractEnv --> MergeEnv[Merge env config
over base with defu] + CheckKeys -->|no| Skip + MergeEnv --> Skip + Skip --> Next[Continue pipeline] +``` + +The lookup order for env-specific config is (per `src/loader.ts:413-415`): +1. `config.$` (e.g., `$production`) +2. `config.$env.` (e.g., `$env.staging`) + +## RC File Loading + +RC files are loaded from multiple locations (if `globalRc` is enabled): + +```mermaid +flowchart TD + Start[Load RC files] --> Cwd[Load from cwd] + Cwd --> Workspace{globalRc enabled?} + Workspace -->|yes| FindWorkspace[Find workspace dir] + FindWorkspace --> LoadWorkspace[Load from workspace] + Workspace -->|no| Home + LoadWorkspace --> Home{globalRc enabled?} + Home -->|yes| LoadHome[Load from user home
via rc9.readUser] + Home -->|no| Merge + LoadHome --> Merge[Merge all RC sources
with defu] + Merge --> End[Return merged RC config] +``` + +RC file loading uses the `rc9` package, which reads from: +1. `cwd/.rc` +2. Workspace root `.rc` (if `globalRc`) +3. User home directory `.rc` (if `globalRc`) + +## Extended Configuration Processing + +The `extends` feature allows configs to inherit from other configs: + +```mermaid +flowchart TD + Start[extendConfig] --> FindExtends{Has extends key?} + FindExtends -->|no| End[Return] + FindExtends -->|yes| ExtractSources[Extract extend sources] + ExtractSources --> LoopSources[For each source] + LoopSources --> CheckFormat{Format?} + CheckFormats -->|{source, options}| Extract2[Extract source/options] + CheckFormats -->|[source, options]| Extract2 + CheckFormats -->|string| ResolveSource + Extract2 --> ResolveSource + ResolveSource --> RemoteCheck{Remote URI?} + RemoteCheck -->|yes| Download[Download with giget] + RemoteCheck -->|no| NpmCheck + Download --> NpmCheck{npm package?} + NpmCheck -->|yes| ResolvePkg[Resolve package] + NpmCheck -->|no| LocalPath + ResolvePkg --> LocalPath[Use local path] + LocalPath --> ResolveConfig2[Call resolveConfig] + ResolveConfig2 --> RecursiveExtend[Recursive extendConfig
on base] + RecursiveExtend --> PushLayer[Push to _layers array] + PushLayer --> NextSource{More sources?} + NextSource -->|yes| LoopSources + NextSource -->|no| MergeLayers[Merge layers
with defu] + MergeLayers --> End +``` + +## dotenv Integration + +Environment variables are loaded before any config files (per `src/loader.ts:94-100`): + +```mermaid +flowchart TD + Start[setupDotenv] --> LoadFiles[Load .env files] + LoadFiles --> ParseFiles[Parse with
node:util.parseEnv] + ParseFiles --> FileRefs{expandFileReferences?} + FileRefs -->|yes| ExpandFiles[Read _FILE vars
from disk] + FileRefs -->|no| Interpolate + ExpandFiles --> Interpolate{interpolate?} + Interpolate -->|yes| ExpandVars[Expand ${VAR}
references] + Interpolate -->|no| ApplyToEnv + ExpandVars --> ApplyToEnv[Apply to process.env] + ApplyToEnv --> End[Return] +``` + +The dotenv loading happens **before** any config files, allowing config files to reference environment variables. + +## Complete Data Flow + +```mermaid +sequenceDiagram + participant User + participant LoadConfig as loadConfig() + participant Dotenv as setupDotenv() + participant Resolve as resolveConfig() + participant Defu as defu (merger) + participant RC9 as rc9 + participant PkgTypes as pkg-types + + User->>LoadConfig: Call with options + LoadConfig->>LoadConfig: Normalize options + LoadConfig->>LoadConfig: Setup merger (defu or custom) + + opt dotenv enabled + LoadConfig->>Dotenv: setupDotenv(options) + Dotenv-->>LoadConfig: process.env populated + end + + LoadConfig->>Resolve: resolveConfig(".", options) + Resolve-->>LoadConfig: Main config object + + opt rcFile enabled + LoadConfig->>RC9: rc9.read({ cwd }) + opt globalRc enabled + LoadConfig->>PkgTypes: findWorkspaceDir() + PkgTypes-->>LoadConfig: workspace path + LoadConfig->>RC9: rc9.read({ workspace }) + LoadConfig->>RC9: rc9.readUser() + end + LoadConfig->>Defu: Merge all RC sources + Defu-->>LoadConfig: Merged RC config + end + + opt packageJson enabled + LoadConfig->>PkgTypes: readPackageJSON() + PkgTypes-->>LoadConfig: package.json object + LoadConfig->>Defu: Merge package.json values + Defu-->>LoadConfig: Merged pkg config + end + + LoadConfig->>LoadConfig: Resolve config functions + + LoadConfig->>Defu: Merge all sources + Note over Defu: overrides → main → rc →
packageJson → defaultConfig + Defu-->>LoadConfig: Merged config + + opt extends enabled + loop Each extend source + LoadConfig->>Resolve: resolveConfig(source) + Resolve-->>LoadConfig: Extended layer + end + LoadConfig->>Defu: Merge layers into config + Defu-->>LoadConfig: Extended config + end + + opt defaults provided + LoadConfig->>Defu: Merge defaults + Defu-->>LoadConfig: Final config + end + + opt omit$Keys enabled + LoadConfig->>LoadConfig: Remove $ prefixed keys + end + + LoadConfig-->>User: Resolved config + layers +``` + +## Key Files + +| File | Purpose | +|------|---------| +| `src/loader.ts` | Main `loadConfig()` function and `resolveConfig()` | +| `src/dotenv.ts` | Environment variable loading (`setupDotenv`, `loadDotenv`) | +| `src/types.ts` | TypeScript type definitions | +| `src/watch.ts` | Config watching with file system events | + +## Summary + +The configuration pipeline in c12 is: + +1. **Normalize options** - Set defaults and normalize paths +2. **Load environment variables** - Parse `.env` files and populate `process.env` +3. **Load main config** - Find and import the primary config file +4. **Load RC files** - Read from cwd, workspace, and home directories +5. **Load package.json** - Extract config values from package.json +6. **Merge all sources** - Use `defu` to merge in priority order +7. **Process extends** - Recursively load and merge extended configs +8. **Apply defaults** - Merge lowest-priority defaults +9. **Cleanup** - Remove internal `$` keys if requested + +`defu` is the core merging function used throughout the pipeline to ensure deep, predictable merging of configuration objects from all sources. diff --git a/doc/discovery/dynamic-configs.md b/doc/discovery/dynamic-configs.md new file mode 100644 index 0000000..dc2faf9 --- /dev/null +++ b/doc/discovery/dynamic-configs.md @@ -0,0 +1,538 @@ +# Dynamic Configuration Sources in c12 + +This document explores how c12 could support more dynamic configuration sources, such as drop-in config directories (systemd-style `.d` directories), and how configuration layering could be made more first-class. + +## Background + +Currently, c12 has a **fixed set of configuration sources** defined in `ConfigSource`: + +```typescript +export type ConfigSource = "overrides" | "main" | "rc" | "packageJson" | "defaultConfig"; +``` + +These sources are loaded in a **hardcoded order** within `loadConfig()` at `src/loader.ts:83-92`: + +```typescript +const rawConfigs: Record< + ConfigSource, + ResolvableConfig | null | undefined +> = { + overrides: options.overrides, + main: undefined, + rc: undefined, + packageJson: undefined, + defaultConfig: options.defaultConfig, +}; +``` + +This works well for the common case, but lacks flexibility for dynamic configuration discovery. + +## The Use Case: Drop-in Config Directories + +Inspired by systemd's drop-in configuration pattern, this feature would allow: + +1. A main config file: `myapp.config.ts` +2. A drop-in directory: `myapp.config.d/` +3. Individual override files in the directory: + - `myapp.config.d/10-admin-overrides.ts` + - `myapp.config.d/20-production.ts` + - `myapp.config.d/99-local.ts` + +Files in the `.d` directory are merged in **lexicographic order**, with later files overriding earlier ones. This allows: +- System administrators to layer configurations without modifying the base config +- Easy enable/disable by adding/removing files +- Clear provenance of where config values came from + +## Current Limitations + +### Fixed Source Types + +The `ConfigSource` type is a union literal, which means: +- New sources require type changes +- Cannot dynamically add sources at runtime +- Source ordering is fixed + +### Limited Source Metadata + +While `ConfigLayer` exists and has `meta` field, it's used differently than a comprehensive provenance system: + +```typescript +export interface ConfigLayer< + T extends UserInputConfig = UserInputConfig, + MT extends ConfigLayerMeta = ConfigLayerMeta, +> { + config: T | null; + source?: string; + sourceOptions?: SourceOptions; + meta?: MT; + cwd?: string; + configFile?: string; +} +``` + +The `meta` field is primarily for user-defined metadata, not automatic tracking of: +- Which provider provided the value +- Where in the merge order the value came from +- Priority/ranking of the source + +## Inspiration: Rust's Figment Crate + +Rust's [figment](https://docs.rs/figment/latest/figment/) takes a more flexible approach: + +### Provider Trait + +Any type can implement the `Provider` trait to become a configuration source: + +```rust +trait Provider { + fn metadata(&self) -> Metadata; + fn data(&self) -> Result, Error>; + fn profile(&self) -> Option; +} +``` + +### Metadata Tracking + +Every value is tagged with `Metadata`: + +```rust +pub struct Metadata { + pub name: Cow<'static, str>, // "TOML File" + pub source: Option, // Path, URL, etc. + pub provide_location: Option<&'static Location<'static>>, +} +``` + +This allows: +- Rich error messages showing exactly where a value came from +- "Magic" values like `RelativePathBuf` that know their config file location +- Debugging complex configurations + +### Third-Party Providers + +The ecosystem can provide custom providers: +- `figment-directory` - Config from directories +- `figment-file-provider-adapter` - Reads `_FILE` suffix variables +- Custom providers for any data source + +## Potential Design for c12 + +### Option 1: Provider Pattern + +Introduce a `ConfigProvider` interface: + +```typescript +export interface ConfigProvider { + /** Unique identifier for this provider */ + name: string; + + /** Metadata about this provider */ + metadata: ConfigProviderMetadata; + + /** Load configuration from this provider */ + load(context: ConfigProviderContext): Promise; + + /** Priority (lower = higher priority) */ + priority?: number; + + /** Should this provider be enabled? */ + enabled?(options: LoadConfigOptions): boolean; +} + +export interface ConfigProviderMetadata { + name: string; + source?: string; + description?: string; +} + +export interface ConfigProviderContext { + cwd: string; + envName: string | false; + [key: string]: any; +} +``` + +#### Drop-in Directory Provider Example + +```typescript +class DropInDirProvider implements ConfigProvider { + name = "drop-in-directory"; + + constructor( + private basePath: string, + private configName: string, + private pattern: string = "*.config.{ts,js,json,yaml,yml}", + ) {} + + metadata = { + name: "Drop-in Directory", + source: this.basePath, + }; + + priority = 50; // Between RC and package.json + + async load(context: ConfigProviderContext): Promise { + const dropInDir = path.join(context.cwd, `${this.configName}.d`); + + if (!fs.existsSync(dropInDir)) { + return null; + } + + const files = await glob(this.pattern, { cwd: dropInDir }); + const sorted = files.sort(); // Lexicographic order + + let merged: Partial = {}; + for (const file of sorted) { + const config = await loadConfigFile(path.join(dropInDir, file)); + merged = defu(merged, config); + } + + return merged as T; + } +} +``` + +### Option 2: Source Registry + +Add a source registration system to `LoadConfigOptions`: + +```typescript +export interface LoadConfigOptions { + // Existing options... + + /** + * Register additional config sources + * Sources are loaded in order of priority (lowest first) + */ + sources?: ConfigSourceEntry[]; +} + +export interface ConfigSourceEntry { + /** Unique identifier */ + id: string; + + /** Provider function returning config */ + provider: ResolvableConfig; + + /** Priority (lower = higher priority) */ + priority: number; + + /** Whether this source should be loaded */ + condition?: (options: LoadConfigOptions) => boolean; + + /** Metadata about this source */ + metadata?: Partial; +} +``` + +#### Usage Example + +```typescript +const config = await loadConfig({ + name: "myapp", + + sources: [ + { + id: "overrides", + priority: 10, + provider: { custom: "overrides" }, + }, + { + id: "drop-in-dir", + priority: 20, + condition: (opts) => opts.envName === "production", + provider: async (ctx) => { + const dropInDir = path.join(opts.cwd, "myapp.config.d"); + return await loadDropInConfigs(dropInDir); + }, + metadata: { name: "Drop-in Directory" }, + }, + { + id: "main", + priority: 30, + provider: { custom: "main" }, // Built-in loader + }, + ], +}); +``` + +### Option 3: Enhanced Built-in Sources + +Add a new `dropIn` option to the existing `LoadConfigOptions`: + +```typescript +export interface LoadConfigOptions { + // Existing options... + + /** + * Load config from a .d directory + * Files are merged in lexicographic order + */ + dropIn?: boolean | string | DropInOptions; +} + +export interface DropInOptions { + /** Directory name (default: .d) */ + dir?: string; + + /** File pattern to match */ + pattern?: string; + + /** Where in priority order to insert (default: after main, before defaults) */ + insertAfter?: "main" | "rc" | "packageJson"; + + /** Enable only for specific environments */ + env?: string[]; +} +``` + +#### Usage + +```typescript +// Simple: auto-detect myapp.config.d/ +await loadConfig({ name: "myapp", dropIn: true }); + +// Custom directory +await loadConfig({ + name: "myapp", + dropIn: { dir: "config.overrides.d" }, +}); + +// Environment-specific +await loadConfig({ + name: "myapp", + dropIn: { env: ["production", "staging"] }, +}); +``` + +## Layering Provenance + +To make layering "first class" as mentioned in issue #298, we need: + +### Enhanced Layer Metadata + +```typescript +export interface ConfigLayer { + config: T | null; + source?: string; + sourceOptions?: SourceOptions; + meta?: ConfigLayerMeta; // User metadata + cwd?: string; + configFile?: string; + + // New fields for provenance: + provider: string; // "main", "drop-in", "rc", etc. + priority: number; // Merge order + loadTime: Date; // When it was loaded + fingerprint?: string; // Content hash for change detection +} +``` + +### Layer Tree Visualization + +A resolved config could show its full provenance: + +```typescript +{ + config: { /* merged config */ }, + layers: [ + { provider: "overrides", priority: 10, configFile: undefined, ... }, + { provider: "drop-in", priority: 20, configFile: "./myapp.config.d/10-admin.ts", ... }, + { provider: "drop-in", priority: 20, configFile: "./myapp.config.d/20-production.ts", ... }, + { provider: "drop-in", priority: 20, configFile: "./myapp.config.d/99-local.ts", ... }, + { provider: "main", priority: 30, configFile: "./myapp.config.ts", ... }, + { provider: "rc", priority: 40, configFile: "/home/user/.myapprc", ... }, + ] +} +``` + +### Value Attribution + +For debugging, we could trace where each config value came from: + +```typescript +function traceValue( + config: ResolvedConfig, + keyPath: string[] +): ConfigLayer | undefined { + // Search layers in reverse priority order + for (const layer of [...config.layers].reverse()) { + let value = layer.config; + for (const key of keyPath) { + value = value?.[key]; + } + if (value !== undefined) { + return layer; + } + } + return undefined; +} + +// Usage: +const source = traceValue(config, ["server", "port"]); +console.log(`server.port came from: ${source.configFile}`); +``` + +## Implementation Considerations + +### Backward Compatibility + +Any changes should maintain backward compatibility: + +1. Default behavior unchanged - drop-ins disabled by default +2. Existing `ConfigSource` type could remain for internal use +3. New options as additions only + +### Performance + +Loading many small files has overhead: + +1. Consider caching resolved configs +2. Watch mode should efficiently track file additions/removals +3. Content fingerprinting for change detection + +### Error Handling + +With more sources, errors are more likely: + +```typescript +// Per-layer error tracking +export interface ConfigLayer { + // ... + error?: { + message: string; + source: string; + recoverable: boolean; + }; +} + +// Strict vs relaxed modes +await loadConfig({ + strict: true, // Fail on any source error + // or + strict: false, // Log warnings, continue +}); +``` + +### File Naming Conventions + +For `.d` directories, establish conventions: + +```typescript +// Numeric prefix for ordering: +// 00-base.conf +// 10-admin.conf +// 20-deployment.conf +// 99-local.conf + +// Or use alphanumeric sorting: +// admin.conf +// base.conf +// local.conf +// production.conf +``` + +## Related Enhancements + +### Watch Mode Integration + +For drop-in directories, watch mode should: +- Detect new files added +- Detect files removed +- Detect files renamed +- Re-merge in correct order when changes occur + +```typescript +watchConfig({ + name: "myapp", + dropIn: true, + onWatch: (event) => { + console.log(`Drop-in file ${event.type}: ${event.path}`); + }, +}); +``` + +### Configuration Validation + +With layered configs, validation should: + +1. Validate each layer independently +2. Validate the final merged result +3. Show which layer introduced validation errors + +```typescript +const config = await loadConfig({ + name: "myapp", + validate: (layer, merged) => { + // Check layer-specific constraints + // Check final merged constraints + }, +}); +``` + +## Existing Workarounds + +Before dynamic sources are implemented, you can: + +### Use `extends` for Multiple Files + +```typescript +// main.config.ts +export default { + extends: [ + "./configs/base.config", + "./configs/admin.config", + "./configs/production.config", + ], + // ... +}; +``` + +### Use Custom Resolver + +```typescript +const config = await loadConfig({ + name: "myapp", + async resolve(source, options) { + if (source.startsWith("drop-in:")) { + const dir = source.replace("drop-in:", ""); + return await loadDropInConfigs(dir); + } + return null; // Use default resolution + }, + + // Then use it via extends + overrides: (ctx) => ({ + extends: ["drop-in:./myapp.config.d"], + }), +}); +``` + +### Merge Multiple Loads + +```typescript +const [main, admin, local] = await Promise.all([ + loadConfig({ name: "myapp" }), + loadConfig({ name: "admin-overrides" }), + loadConfig({ name: "local-overrides" }), +]); + +const merged = defu(local.config, admin.config, main.config); +``` + +## Summary + +Dynamic configuration sources in c12 would enable: + +1. **Drop-in config directories** - Systemd-style `.d` directories +2. **Custom providers** - Third-party data sources +3. **Enhanced provenance** - Clear tracking of where values came from +4. **Flexible ordering** - Configurable source priorities +5. **Better debugging** - Layer-by-layer inspection + +The key design decision is between: +- **Provider pattern** - Maximum flexibility, ecosystem growth +- **Enhanced built-ins** - Simpler API, controlled feature set +- **Source registry** - Middle ground with explicit registration + +All approaches should maintain backward compatibility while enabling the dynamic configuration discovery that makes layering a first-class concept. From fc958abe6cce3caa4d72002ff3dd87e65e980d75 Mon Sep 17 00:00:00 2001 From: rektide Date: Mon, 9 Feb 2026 21:53:38 -0500 Subject: [PATCH 2/5] initial layering doc --- layering.md | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 layering.md diff --git a/layering.md b/layering.md new file mode 100644 index 0000000..f831d98 --- /dev/null +++ b/layering.md @@ -0,0 +1,7 @@ +# initial prompt + +we want to understand how configuration from various sources is layered together. supposedly `defu` is used to merge configs. where does this happen? when in the lifecycle of execution happens? what is the total execution pipeline for a normal run of c12? use mermaid diagrams . + +we want to try to understand more about how we might have more dynamic sources rather than a fixed path of sources. for instance we might want to make drop in config directories, such as talked about in https://github.com/unjs/c12/issues/298 . + +talking about how configuration becomes sourced. talk about how we can create a better registry of layers we are using. a registry of config layers, that can be built, and then ran, rather than a fixed pipeline. this project needs a more structured management of configuration, rather than being merely a fixed purpose tool. From c13861315d9877e6e44721095713ce386fe90f29 Mon Sep 17 00:00:00 2001 From: rektide Date: Mon, 9 Feb 2026 22:17:13 -0500 Subject: [PATCH 3/5] expand layering prompt --- layering.md | 132 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 128 insertions(+), 4 deletions(-) diff --git a/layering.md b/layering.md index f831d98..c271b24 100644 --- a/layering.md +++ b/layering.md @@ -1,7 +1,131 @@ -# initial prompt +# c12-layer: Understanding & Extending c12's Configuration Layering -we want to understand how configuration from various sources is layered together. supposedly `defu` is used to merge configs. where does this happen? when in the lifecycle of execution happens? what is the total execution pipeline for a normal run of c12? use mermaid diagrams . +## Goals -we want to try to understand more about how we might have more dynamic sources rather than a fixed path of sources. for instance we might want to make drop in config directories, such as talked about in https://github.com/unjs/c12/issues/298 . +This document guides analysis of c12's configuration layering system and proposes enhancements for more dynamic, introspectable configuration management. -talking about how configuration becomes sourced. talk about how we can create a better registry of layers we are using. a registry of config layers, that can be built, and then ran, rather than a fixed pipeline. this project needs a more structured management of configuration, rather than being merely a fixed purpose tool. +## Part 1: Understanding the Current System + +### Questions to Answer + +1. **Where does layering happen?** + - How does `defu` merge configurations? + - What is the call graph from `loadConfig()` to final merged config? + +2. **When in the lifecycle?** + - Map the complete execution pipeline from `loadConfig()` invocation to resolved config + - Identify when each source is loaded, when extends are resolved, when merging occurs + +3. **What is the layer resolution order?** + - Document the priority stack (per README: overrides → config file → RC → global RC → package.json → defaults → extended layers) + - How does environment-specific config (`$development`, `$production`, etc.) factor in? + +### Deliverables + +- **Mermaid sequence diagram**: Show the lifecycle from `loadConfig()` call through to final config +- **Mermaid flowchart**: Show decision points (does RC exist? does config extend? etc.) +- **Code citations**: Link to the actual source locations where merging happens + +--- + +## Part 2: Dynamic Configuration Sources + +### Problem Statement + +c12 currently has a fixed resolution pipeline. We want to explore: + +1. **Drop-in config directories** (à la systemd's `*.conf.d/`) + - Reference: https://github.com/unjs/c12/issues/298 + - Allow `.config.d/` directories where multiple configs can be dropped in + - Configs sorted alphabetically (or with numeric prefixes like `00-base.ts`, `10-overrides.ts`) + +2. **Pluggable source providers** + - Instead of hardcoded sources (file, RC, package.json), allow registering custom providers + - Examples: environment variables provider, remote config provider, vault/secrets provider + +### Design Questions + +- How would drop-in directories integrate with the existing layer system? +- Should drop-in configs be siblings to extended layers, or a separate concept? +- How do we handle ordering/priority for drop-in files? + +--- + +## Part 3: Layer Registry Architecture + +### Vision + +Transform c12 from a "fixed pipeline config loader" into a "structured config layer manager" with: + +1. **Explicit layer registry** + - Named, ordered collection of configuration sources + - Each layer has metadata: name, source type, priority, file path(s) + +2. **Two-phase execution** + - **Build phase**: Construct the layer registry, validate sources exist, resolve extends + - **Run phase**: Execute the registry to produce final merged config + +3. **Introspection capabilities** + - Query which layer provided a specific config key + - Trace config value provenance (like Rust's `figment` crate metadata) + - Debug mode showing layer-by-layer merge steps + +### Inspiration + +- **Rust's figment**: Providers with metadata, value provenance tracking +- **systemd**: Drop-in directories, clear override semantics +- **Kubernetes**: ConfigMap layering, strategic merge patches + +### Proposed API Sketch + +```typescript +// Build a layer registry explicitly +const registry = createLayerRegistry({ + name: "myapp" +}) + .addSource("defaults", { type: "static", config: { ... } }) + .addSource("base-file", { type: "file", path: "myapp.config.ts" }) + .addSource("drop-ins", { type: "directory", path: "myapp.config.d/" }) + .addSource("env-overrides", { type: "env", prefix: "MYAPP_" }) + .addSource("cli-overrides", { type: "static", config: cliArgs }); + +// Inspect before running +console.log(registry.layers); // See all registered sources +console.log(registry.resolve("database.host")); // Which layer provides this? + +// Execute to get final config +const { config, provenance } = await registry.load(); +``` + +--- + +## Part 4: Implementation Considerations + +### Backward Compatibility + +- `loadConfig()` should continue working unchanged +- New APIs are opt-in enhancements + +### Key Extension Points to Identify + +1. Where can we hook into layer resolution? +2. Can we intercept/wrap the merge function? +3. How do we inject additional sources into the pipeline? + +### Files to Analyze + +- `src/loader.ts` - Main loading logic +- `src/config.ts` - Config resolution +- Look for `defu` usage patterns +- Look for `extends` resolution logic + +--- + +## Success Criteria + +After this analysis, we should have: + +1. Clear understanding of c12's internals with diagrams +2. Feasibility assessment for drop-in directories +3. Draft design for layer registry architecture +4. Identified extension points or required changes to c12 From 43be190c438df2165675a5d1984e19da9efb1b14 Mon Sep 17 00:00:00 2001 From: rektide Date: Mon, 9 Feb 2026 22:46:09 -0500 Subject: [PATCH 4/5] feat: add pluggable ConfigProvider system Refactors c12's config loading to use a provider-based architecture: - Add ConfigProvider interface with name, priority, and load() method - Built-in providers: overrides, main, rc, packageJson, defaultConfig - Providers sorted by priority (lower = higher precedence) - New options.providers allows custom/replacement providers - Providers can access previously loaded configs via context - Backward compatible: loadConfig() works unchanged without providers - Exports: getDefaultProviders, sortProviders, create*Provider helpers This enables users to: - Inject custom config sources (env vars, remote config, vault) - Reorder or remove default sources - Build configs that depend on other loaded configs --- .oxfmtrc.json | 3 +- layering.md | 472 ++++++++++++++++++++++++++++++++++++-------- src/index.ts | 13 ++ src/loader.ts | 135 +++++-------- src/providers.ts | 274 +++++++++++++++++++++++++ src/types.ts | 24 +++ test/loader.test.ts | 129 +++++++++++- 7 files changed, 878 insertions(+), 172 deletions(-) create mode 100644 src/providers.ts diff --git a/.oxfmtrc.json b/.oxfmtrc.json index c46adf4..98c3902 100644 --- a/.oxfmtrc.json +++ b/.oxfmtrc.json @@ -1,3 +1,4 @@ { - "$schema": "https://unpkg.com/oxfmt/configuration_schema.json" + "$schema": "https://unpkg.com/oxfmt/configuration_schema.json", + "ignore": ["test/.tmp/**"] } diff --git a/layering.md b/layering.md index c271b24..6cbc5a9 100644 --- a/layering.md +++ b/layering.md @@ -1,131 +1,437 @@ -# c12-layer: Understanding & Extending c12's Configuration Layering +# c12-layer: Configuration Layering Analysis & Enhancement Proposal -## Goals +## Problem Statement -This document guides analysis of c12's configuration layering system and proposes enhancements for more dynamic, introspectable configuration management. +c12 is a powerful configuration loader, but it has limitations that prevent advanced use cases: -## Part 1: Understanding the Current System +1. **Fixed Pipeline**: Sources are hardcoded (overrides → config file → RC → package.json → defaults). Users cannot inject custom sources or reorder the pipeline. -### Questions to Answer +2. **No Drop-in Directories**: Unlike systemd's `*.conf.d/` pattern, there's no way to have a directory of config fragments merged automatically. See [unjs/c12#298](https://github.com/unjs/c12/issues/298). -1. **Where does layering happen?** - - How does `defu` merge configurations? - - What is the call graph from `loadConfig()` to final merged config? +3. **No Provenance Tracking**: After merge, it's impossible to know which layer contributed a specific key. Debugging configuration issues requires manual bisection. -2. **When in the lifecycle?** - - Map the complete execution pipeline from `loadConfig()` invocation to resolved config - - Identify when each source is loaded, when extends are resolved, when merging occurs +4. **Opaque Lifecycle**: The loading process is a black box—no hooks to observe or modify layers before final merge. -3. **What is the layer resolution order?** - - Document the priority stack (per README: overrides → config file → RC → global RC → package.json → defaults → extended layers) - - How does environment-specific config (`$development`, `$production`, etc.) factor in? +### Goals -### Deliverables - -- **Mermaid sequence diagram**: Show the lifecycle from `loadConfig()` call through to final config -- **Mermaid flowchart**: Show decision points (does RC exist? does config extend? etc.) -- **Code citations**: Link to the actual source locations where merging happens +1. Understand c12's current layering internals +2. Assess feasibility of drop-in directory support +3. Design a layer registry architecture with provenance tracking +4. Identify extension points for backward-compatible enhancements --- -## Part 2: Dynamic Configuration Sources - -### Problem Statement +## Findings: How c12 Works Today + +### Layer Resolution Order + +Priority from highest to lowest, based on [`loader.ts#L157-L164`](src/loader.ts#L157-L164): + +| Priority | Source | Code Location | +|----------|--------|---------------| +| 1 (highest) | `options.overrides` | Passed to loadConfig | +| 2 | Main config file (`.config.ts`) | [`L103-L108`](src/loader.ts#L103-L108) | +| 3 | RC files (cwd → workspace → home) | [`L115-L129`](src/loader.ts#L115-L129) | +| 4 | `package.json[name]` | [`L132-L141`](src/loader.ts#L132-L141) | +| 5 | `options.defaultConfig` | Passed to loadConfig | +| 6 | Extended layers (from `extends` key) | [`L167-L172`](src/loader.ts#L167-L172) | +| 7 (lowest) | `options.defaults` | [`L193-L195`](src/loader.ts#L193-L195) | + +### Environment-Specific Config + +Handled in [`resolveConfig` L411-L420](src/loader.ts#L411-L420): +- Checks for `$development`, `$production`, `$test` based on `options.envName` +- Also checks `$env.{envName}` object +- Merged with **highest priority** on top of the file's base config + +### Merge Points + +All merges use `defu` (or custom `options.merger`): + +| Operation | Location | Code | +|-----------|----------|------| +| RC sources merge | [`L128`](src/loader.ts#L128) | `_merger({}, ...rcSources)` | +| Package.json values | [`L140`](src/loader.ts#L140) | `_merger({}, ...values)` | +| Main 5-source merge | [`L158-L164`](src/loader.ts#L158-L164) | `_merger(overrides, main, rc, packageJson, defaultConfig)` | +| Extended layers | [`L171`](src/loader.ts#L171) | `_merger(config, ...layers.map(e => e.config))` | +| Env-specific | [`L418`](src/loader.ts#L418) | `_merger(envConfig, res.config)` | +| Final defaults | [`L194`](src/loader.ts#L194) | `_merger(config, defaults)` | + +### Lifecycle Sequence + +```mermaid +sequenceDiagram + participant User + participant loadConfig + participant resolveConfig + participant rc9 + participant pkg-types + participant extendConfig + participant defu + + User->>loadConfig: loadConfig(options) + loadConfig->>loadConfig: Normalize options + + opt options.dotenv + loadConfig->>loadConfig: setupDotenv() + end + + loadConfig->>resolveConfig: resolveConfig(".", options) + resolveConfig-->>loadConfig: {config, configFile} + + opt options.rcFile + loadConfig->>rc9: read(cwd) + opt options.globalRc + loadConfig->>rc9: read(workspace) + loadConfig->>rc9: readUser(home) + end + loadConfig->>defu: merge RC sources + end + + opt options.packageJson + loadConfig->>pkg-types: readPackageJSON() + end + + loadConfig->>defu: merge(overrides, main, rc, pkg, defaultConfig) + + opt options.extend + loadConfig->>extendConfig: extendConfig(config) + loop each extends source + extendConfig->>resolveConfig: resolveConfig(source) + extendConfig->>extendConfig: recursive + end + loadConfig->>defu: merge(config, ...layers) + end + + opt options.defaults + loadConfig->>defu: merge(config, defaults) + end + + loadConfig-->>User: ResolvedConfig +``` -c12 currently has a fixed resolution pipeline. We want to explore: +### Decision Flowchart + +```mermaid +flowchart TD + start([loadConfig]) --> normalize[Normalize options] + normalize --> dotenv{dotenv?} + dotenv -->|yes| loadDotenv[setupDotenv] + dotenv -->|no| mainFile + loadDotenv --> mainFile + + mainFile{Config file exists?} -->|yes| loadMain[resolveConfig: load file] + mainFile -->|no| rcCheck + loadMain --> rcCheck + + rcCheck{rcFile?} -->|yes| loadRC[rc9.read cwd] + rcCheck -->|no| pkgCheck + loadRC --> globalRC{globalRc?} + globalRC -->|yes| loadGlobal[rc9.read workspace + home] + globalRC -->|no| mergeRC + loadGlobal --> mergeRC[defu merge RC sources] + mergeRC --> pkgCheck + + pkgCheck{packageJson?} -->|yes| loadPkg[readPackageJSON] + pkgCheck -->|no| mainMerge + loadPkg --> mainMerge + + mainMerge[defu: overrides → main → rc → pkg → defaultConfig] + mainMerge --> extend{extends?} + + extend -->|yes| extendLoop[extendConfig recursively] + extendLoop --> mergeLayers[defu merge layers] + extend -->|no| defaults + mergeLayers --> defaults + + defaults{defaults?} -->|yes| applyDefaults[defu merge defaults] + defaults -->|no| cleanup + applyDefaults --> cleanup + + cleanup --> omitKeys{omit$Keys?} + omitKeys -->|yes| removeKeys[Remove $ prefixed keys] + omitKeys -->|no| done + removeKeys --> done([ResolvedConfig]) +``` -1. **Drop-in config directories** (à la systemd's `*.conf.d/`) - - Reference: https://github.com/unjs/c12/issues/298 - - Allow `.config.d/` directories where multiple configs can be dropped in - - Configs sorted alphabetically (or with numeric prefixes like `00-base.ts`, `10-overrides.ts`) +### Existing Extension Points -2. **Pluggable source providers** - - Instead of hardcoded sources (file, RC, package.json), allow registering custom providers - - Examples: environment variables provider, remote config provider, vault/secrets provider +1. **`options.resolve`** ([`L284-L288`](src/loader.ts#L284-L288)): Custom resolver can intercept any source before default resolution +2. **`options.merger`** ([`L71`](src/loader.ts#L71)): Replace `defu` with custom merge function +3. **`options.import`** ([`L385-L386`](src/loader.ts#L385-L386)): Custom module loader +4. **`extends` key**: Already supports arrays of sources with recursive resolution -### Design Questions +### Current Limitations -- How would drop-in directories integrate with the existing layer system? -- Should drop-in configs be siblings to extended layers, or a separate concept? -- How do we handle ordering/priority for drop-in files? +1. **No insertion points**: Can't add sources between existing ones (e.g., between RC and package.json) +2. **Post-hoc layers**: `ResolvedConfig.layers` preserves sources but only after merge—no pre-merge introspection +3. **No directory scanning**: `resolveConfig` handles single files only +4. **No key-level tracking**: `defu` merges destructively with no provenance --- -## Part 3: Layer Registry Architecture +## Proposed Solutions + +### Solution 1: Drop-in Directory Support + +**Problem**: Users want `myapp.config.d/` directories with sorted fragments like `00-base.ts`, `10-local.ts`. -### Vision +**Approach**: Add a `configDir` source type that scans a directory and sorts files. -Transform c12 from a "fixed pipeline config loader" into a "structured config layer manager" with: +```typescript +// New function in loader.ts +async function loadConfigDir( + dirPath: string, + options: LoadConfigOptions +): Promise[]> { + const dir = resolve(options.cwd!, dirPath); + if (!existsSync(dir)) return []; + + const entries = await readdir(dir); + const configFiles = entries + .filter(f => SUPPORTED_EXTENSIONS.some(ext => f.endsWith(ext))) + .sort(); // Alphabetical: 00-base.ts < 10-overrides.ts + + const layers: ConfigLayer[] = []; + for (const file of configFiles) { + const res = await resolveConfig(join(dir, file), options); + if (res.config) { + layers.push(res); + } + } + return layers; +} +``` -1. **Explicit layer registry** - - Named, ordered collection of configuration sources - - Each layer has metadata: name, source type, priority, file path(s) +**Integration Options**: -2. **Two-phase execution** - - **Build phase**: Construct the layer registry, validate sources exist, resolve extends - - **Run phase**: Execute the registry to produce final merged config +| Option | Where | Behavior | +|--------|-------|----------| +| A. New rawConfigs source | After `main` | `rawConfigs.configDir = loadConfigDir(...)` | +| B. Auto-extend | In `extendConfig` | Treat `.config.d/` as implicit extends | +| C. User-opt-in | Via `options.configDir` | Explicit enable with priority control | -3. **Introspection capabilities** - - Query which layer provided a specific config key - - Trace config value provenance (like Rust's `figment` crate metadata) - - Debug mode showing layer-by-layer merge steps +**Recommended**: Option C with priority parameter: +```typescript +loadConfig({ + name: 'myapp', + configDir: { + path: 'myapp.config.d/', + priority: 'after-main' // or 'before-rc', 'after-extends' + } +}) +``` -### Inspiration +--- -- **Rust's figment**: Providers with metadata, value provenance tracking -- **systemd**: Drop-in directories, clear override semantics -- **Kubernetes**: ConfigMap layering, strategic merge patches +### Solution 2: Pluggable Source Providers -### Proposed API Sketch +**Problem**: Users want custom sources (env vars, remote config, Vault) integrated into the pipeline. + +**Approach**: Define a `ConfigProvider` interface and allow registration. ```typescript -// Build a layer registry explicitly -const registry = createLayerRegistry({ - name: "myapp" +// New types +interface ConfigProvider { + name: string; + priority: number; // Lower = higher priority + load(options: LoadConfigOptions): Promise | null>; +} + +// Built-in providers +const builtinProviders: ConfigProvider[] = [ + { name: 'overrides', priority: 0, load: (o) => ({ config: o.overrides }) }, + { name: 'main', priority: 100, load: (o) => resolveConfig('.', o) }, + { name: 'rc', priority: 200, load: loadRcFiles }, + { name: 'packageJson', priority: 300, load: loadPackageJson }, + { name: 'defaultConfig', priority: 400, load: (o) => ({ config: o.defaultConfig }) }, +]; + +// User registration +loadConfig({ + providers: [ + ...defaultProviders, + { name: 'env', priority: 50, load: envProvider }, + { name: 'vault', priority: 150, load: vaultProvider }, + ] }) - .addSource("defaults", { type: "static", config: { ... } }) - .addSource("base-file", { type: "file", path: "myapp.config.ts" }) - .addSource("drop-ins", { type: "directory", path: "myapp.config.d/" }) - .addSource("env-overrides", { type: "env", prefix: "MYAPP_" }) - .addSource("cli-overrides", { type: "static", config: cliArgs }); +``` + +**Benefits**: +- Full control over source order +- Clean separation of concerns +- Easy to add/remove/reorder sources + +--- + +### Solution 3: Layer Registry with Two-Phase Execution -// Inspect before running -console.log(registry.layers); // See all registered sources -console.log(registry.resolve("database.host")); // Which layer provides this? +**Problem**: No way to inspect layers before merge or trace value provenance. -// Execute to get final config +**Approach**: Separate "build" and "load" phases. + +```typescript +// New API +interface LayerRegistry { + readonly layers: ReadonlyArray>; + + // Build phase + addSource(name: string, source: SourceDefinition): LayerRegistry; + validate(): Promise; + + // Introspection (pre-load) + getLayerByName(name: string): RegisteredLayer | undefined; + + // Execution + load(): Promise>; +} + +interface RegisteredLayer { + name: string; + priority: number; + source: SourceDefinition; + status: 'pending' | 'loaded' | 'not-found' | 'error'; + config?: T; + configFile?: string; +} + +interface ResolvedConfigWithProvenance { + config: T; + layers: RegisteredLayer[]; + provenance: Map; // key path → which layer +} + +interface LayerProvenance { + layerName: string; + configFile?: string; + keyPath: string; +} +``` + +**Usage**: +```typescript +const registry = createLayerRegistry({ name: 'myapp' }) + .addSource('defaults', { type: 'static', config: { port: 3000 } }) + .addSource('base', { type: 'file', path: 'myapp.config.ts' }) + .addSource('drop-ins', { type: 'directory', path: 'myapp.config.d/' }) + .addSource('env', { type: 'env', prefix: 'MYAPP_' }) + .addSource('cli', { type: 'static', config: cliArgs }); + +// Validate before loading +const validation = await registry.validate(); +if (!validation.ok) { + console.error('Missing sources:', validation.missing); +} + +// Load with provenance const { config, provenance } = await registry.load(); + +// Debug: where did database.host come from? +console.log(provenance.get('database.host')); +// → { layerName: 'drop-ins', configFile: 'myapp.config.d/20-database.ts', keyPath: 'database.host' } ``` --- -## Part 4: Implementation Considerations +### Solution 4: Provenance-Tracking Merger + +**Problem**: `defu` merges destructively—no way to know which layer contributed a key. + +**Approach**: Wrap merge with a tracking layer. + +```typescript +function createProvenanceMerger() { + const provenance = new Map(); + + function trackingMerger( + layerName: string, + configFile: string | undefined + ): (...sources: T[]) => T { + return (...sources) => { + // Use defu for actual merge + const result = defu(...sources); + + // Track which keys came from which layer + // (Simplified: real impl would deep-traverse) + for (const [key, value] of Object.entries(sources[0] || {})) { + if (value !== undefined && !provenance.has(key)) { + provenance.set(key, { layerName, configFile, keyPath: key }); + } + } + + return result; + }; + } + + return { trackingMerger, provenance }; +} +``` + +**Deep tracking** would require a recursive merge that records the path for every leaf value. + +--- -### Backward Compatibility +### Solution 5: Backward-Compatible Integration -- `loadConfig()` should continue working unchanged -- New APIs are opt-in enhancements +**Problem**: New features must not break existing `loadConfig()` users. -### Key Extension Points to Identify +**Approach**: Layer registry is opt-in; `loadConfig` continues unchanged. -1. Where can we hook into layer resolution? -2. Can we intercept/wrap the merge function? -3. How do we inject additional sources into the pipeline? +```typescript +// Existing API unchanged +const config = await loadConfig({ name: 'myapp' }); -### Files to Analyze +// New API for power users +const registry = await buildLayerRegistry({ name: 'myapp' }); +const { config, provenance } = await registry.load(); -- `src/loader.ts` - Main loading logic -- `src/config.ts` - Config resolution -- Look for `defu` usage patterns -- Look for `extends` resolution logic +// Or: loadConfig with provenance opt-in +const { config, provenance } = await loadConfig({ + name: 'myapp', + trackProvenance: true, // New option +}); +``` + +**Implementation**: Refactor `loadConfig` internals to use registry, but expose existing return type by default. --- -## Success Criteria +## Implementation Roadmap + +### Phase 1: Drop-in Directories +- Add `loadConfigDir()` function +- Add `options.configDir` to `LoadConfigOptions` +- Integrate into layer collection before extends resolution +- **Effort**: Small, self-contained change + +### Phase 2: Provider Interface +- Define `ConfigProvider` interface +- Refactor existing sources as built-in providers +- Add `options.providers` for custom sources +- **Effort**: Medium, requires restructuring loader.ts + +### Phase 3: Layer Registry +- Create `LayerRegistry` class +- Implement two-phase execution +- Add `buildLayerRegistry()` export +- **Effort**: Large, new module + +### Phase 4: Provenance Tracking +- Implement tracking merger +- Integrate with registry +- Add `trackProvenance` option to loadConfig +- **Effort**: Medium, requires careful deep-object traversal + +--- + +## Open Questions + +1. **Priority notation**: Should priorities be numeric (0-1000) or named slots (`before-main`, `after-rc`)? + +2. **Drop-in merge order**: Should drop-ins merge left-to-right (later files override) or right-to-left (earlier files have priority)? + +3. **Provenance granularity**: Track at key level only, or full path (`database.connection.host`)? -After this analysis, we should have: +4. **Async providers**: How to handle slow providers (Vault, remote config) gracefully? -1. Clear understanding of c12's internals with diagrams -2. Feasibility assessment for drop-in directories -3. Draft design for layer registry architecture -4. Identified extension points or required changes to c12 +5. **Caching**: Should registry cache loaded configs for repeated `.load()` calls? diff --git a/src/index.ts b/src/index.ts index 985e5e7..e50934e 100644 --- a/src/index.ts +++ b/src/index.ts @@ -4,4 +4,17 @@ export { SUPPORTED_EXTENSIONS, loadConfig } from "./loader.ts"; export * from "./types.ts"; +export { + type ConfigProvider, + type ProviderContext, + type ProviderResult, + getDefaultProviders, + sortProviders, + createOverridesProvider, + createMainProvider, + createRcProvider, + createPackageJsonProvider, + createDefaultConfigProvider, +} from "./providers.ts"; + export { type ConfigWatcher, type WatchConfigOptions, watchConfig } from "./watch.ts"; diff --git a/src/loader.ts b/src/loader.ts index 1ed3011..1f14c98 100644 --- a/src/loader.ts +++ b/src/loader.ts @@ -4,21 +4,24 @@ import { pathToFileURL } from "node:url"; import { homedir } from "node:os"; import { resolve, extname, dirname, basename, join, normalize } from "pathe"; import { resolveModulePath } from "exsolve"; -import * as rc9 from "rc9"; import { defu } from "defu"; -import { findWorkspaceDir, readPackageJSON } from "pkg-types"; import { setupDotenv } from "./dotenv.ts"; +import { + getDefaultProviders, + sortProviders, + type ConfigProvider, + type ProviderContext, + type ProviderResult, +} from "./providers.ts"; import type { UserInputConfig, ConfigLayerMeta, LoadConfigOptions, ResolvedConfig, - ResolvableConfig, ConfigLayer, SourceOptions, InputConfig, - ConfigSource, ConfigFunctionContext, } from "./types.ts"; @@ -70,7 +73,7 @@ export async function loadConfig< // Custom merger const _merger = options.merger || defu; - // Create context + // Create result context const r: ResolvedConfig = { config: {} as any, cwd: options.cwd, @@ -79,18 +82,6 @@ export async function loadConfig< _configFile: undefined, }; - // prettier-ignore - const rawConfigs: Record< - ConfigSource, - ResolvableConfig | null | undefined - > = { - overrides: options.overrides, - main: undefined, - rc: undefined, - packageJson: undefined, - defaultConfig: options.defaultConfig, - }; - // Load dotenv if (options.dotenv) { await setupDotenv({ @@ -99,69 +90,49 @@ export async function loadConfig< }); } - // Load main config file - const _mainConfig = await resolveConfig(".", options); - if (_mainConfig.configFile) { - rawConfigs.main = _mainConfig.config; - r.configFile = _mainConfig.configFile; - r._configFile = _mainConfig._configFile; - } + // Get providers (custom or default) + const providers = sortProviders(options.providers ?? getDefaultProviders()); - if (_mainConfig.meta) { - r.meta = _mainConfig.meta; - } + // Create provider context + const loadedConfigs = new Map(); + const providerCtx: ProviderContext = { + options, + merger: _merger, + resolveConfig: (source, opts) => resolveConfig(source, opts), + loadedConfigs, + }; - // Load rc files - if (options.rcFile) { - const rcSources: T[] = []; - // 1. cwd - rcSources.push(rc9.read({ name: options.rcFile, dir: options.cwd })); - if (options.globalRc) { - // 2. workspace - const workspaceDir = await findWorkspaceDir(options.cwd).catch(() => {}); - if (workspaceDir) { - rcSources.push(rc9.read({ name: options.rcFile, dir: workspaceDir })); + // Load all providers and collect results + const providerResults: Array<{ + provider: ConfigProvider; + result: ProviderResult; + }> = []; + + for (const provider of providers) { + const result = await provider.load(providerCtx); + if (result?.config !== undefined && result?.config !== null) { + loadedConfigs.set(provider.name, result.config); + providerResults.push({ provider, result }); + + // Capture main config metadata + if (provider.name === "main") { + if (result.configFile) r.configFile = result.configFile; + if (result._configFile) r._configFile = result._configFile; + if (result.meta) r.meta = result.meta; } - // 3. user home - rcSources.push(rc9.readUser({ name: options.rcFile, dir: options.cwd })); } - rawConfigs.rc = _merger({} as T, ...rcSources); - } - - // Load config from package.json - if (options.packageJson) { - const keys = ( - Array.isArray(options.packageJson) - ? options.packageJson - : [typeof options.packageJson === "string" ? options.packageJson : options.name] - ).filter((t) => t && typeof t === "string"); - const pkgJsonFile = await readPackageJSON(options.cwd).catch(() => {}); - const values = keys.map((key) => pkgJsonFile?.[key]); - rawConfigs.packageJson = _merger({} as T, ...values); } - // Resolve config sources - const configs = {} as Record; - // TODO: #253 change order from defaults to overrides in next major version - for (const key in rawConfigs) { - const value = rawConfigs[key as ConfigSource]; - configs[key as ConfigSource] = await (typeof value === "function" - ? value({ configs, rawConfigs }) - : value); - } + // Extract configs in priority order for merging + const configs = providerResults.map((pr) => pr.result.config) as Array; - if (Array.isArray(configs.main)) { - // If the main config exports an array, use it directly without merging or extending - r.config = configs.main; + // Check if main config is an array (special case: use directly without merging) + const mainResult = providerResults.find((pr) => pr.provider.name === "main"); + if (Array.isArray(mainResult?.result.config)) { + r.config = mainResult.result.config as T; } else { - // Combine sources - r.config = _merger( - configs.overrides, - configs.main, - configs.rc, - configs.packageJson, - configs.defaultConfig, - ) as T; + // Merge all provider configs in priority order + r.config = _merger(...(configs as [T, ...Array])) as T; // Allow extending if (options.extend) { @@ -173,19 +144,13 @@ export async function loadConfig< } // Preserve unmerged sources as layers - const baseLayers: ConfigLayer[] = [ - configs.overrides && { - config: configs.overrides, - configFile: undefined, - cwd: undefined, - }, - { config: configs.main, configFile: options.configFile, cwd: options.cwd }, - configs.rc && { config: configs.rc, configFile: options.rcFile }, - configs.packageJson && { - config: configs.packageJson, - configFile: "package.json", - }, - ].filter((l) => l && l.config) as ConfigLayer[]; + const baseLayers: ConfigLayer[] = providerResults + .filter((pr) => pr.result.layer) + .map((pr) => ({ + ...pr.result.layer, + config: pr.result.config, + })) + .filter((l) => l.config) as ConfigLayer[]; r.layers = [...baseLayers, ...r.layers!]; diff --git a/src/providers.ts b/src/providers.ts new file mode 100644 index 0000000..f985b42 --- /dev/null +++ b/src/providers.ts @@ -0,0 +1,274 @@ +import * as rc9 from "rc9"; +import { findWorkspaceDir, readPackageJSON } from "pkg-types"; + +import type { + UserInputConfig, + ConfigLayerMeta, + LoadConfigOptions, + ConfigLayer, + ResolvableConfig, +} from "./types.ts"; + +/** + * Context passed to config providers during loading + */ +export interface ProviderContext< + T extends UserInputConfig = UserInputConfig, + MT extends ConfigLayerMeta = ConfigLayerMeta, +> { + /** Normalized load options */ + options: LoadConfigOptions; + /** Merger function (defu or custom) */ + merger: (...sources: Array) => T; + /** Resolve a config file (used by main provider) */ + resolveConfig: ( + source: string, + options: LoadConfigOptions, + ) => Promise<{ + config?: T; + configFile?: string; + _configFile?: string; + cwd?: string; + meta?: MT; + }>; + /** Results from previously loaded providers (by name) */ + loadedConfigs: Map; +} + +/** + * Result returned by a config provider + */ +export interface ProviderResult< + T extends UserInputConfig = UserInputConfig, + MT extends ConfigLayerMeta = ConfigLayerMeta, +> { + /** The loaded configuration (or null/undefined if not found) */ + config: T | null | undefined; + /** Layer metadata for introspection */ + layer?: Partial>; + /** Additional metadata to merge into ResolvedConfig */ + meta?: MT; + /** Resolved config file path (for main provider) */ + configFile?: string; + /** Internal config file path */ + _configFile?: string; +} + +/** + * A pluggable configuration source provider + */ +export interface ConfigProvider< + T extends UserInputConfig = UserInputConfig, + MT extends ConfigLayerMeta = ConfigLayerMeta, +> { + /** Unique name for this provider */ + name: string; + /** + * Priority determines merge order. + * Lower numbers = higher priority (merged first, so they "win"). + * Built-in priorities: overrides=100, main=200, rc=300, packageJson=400, defaultConfig=500 + */ + priority: number; + /** + * Load configuration from this provider. + * Return null/undefined if this provider has no config to contribute. + */ + load(ctx: ProviderContext): Promise | null | undefined>; +} + +/** + * Built-in provider: overrides from options + */ +export function createOverridesProvider< + T extends UserInputConfig = UserInputConfig, + MT extends ConfigLayerMeta = ConfigLayerMeta, +>(): ConfigProvider { + return { + name: "overrides", + priority: 100, + async load(ctx) { + const config = await resolveResolvableConfig(ctx.options.overrides, ctx); + if (!config) return null; + return { + config, + layer: { + config, + configFile: undefined, + cwd: undefined, + }, + }; + }, + }; +} + +/** + * Built-in provider: main config file + */ +export function createMainProvider< + T extends UserInputConfig = UserInputConfig, + MT extends ConfigLayerMeta = ConfigLayerMeta, +>(): ConfigProvider { + return { + name: "main", + priority: 200, + async load(ctx) { + const result = await ctx.resolveConfig(".", ctx.options); + if (!result.configFile) return null; + return { + config: result.config, + configFile: result.configFile, + _configFile: result._configFile, + meta: result.meta, + layer: { + config: result.config, + configFile: ctx.options.configFile, + cwd: ctx.options.cwd, + }, + }; + }, + }; +} + +/** + * Built-in provider: RC files (.namerc) + */ +export function createRcProvider< + T extends UserInputConfig = UserInputConfig, + MT extends ConfigLayerMeta = ConfigLayerMeta, +>(): ConfigProvider { + return { + name: "rc", + priority: 300, + async load(ctx) { + const { options, merger } = ctx; + if (!options.rcFile) return null; + + const rcSources: T[] = []; + + // 1. cwd + rcSources.push(rc9.read({ name: options.rcFile, dir: options.cwd })); + + if (options.globalRc) { + // 2. workspace + const workspaceDir = await findWorkspaceDir(options.cwd!).catch(() => {}); + if (workspaceDir) { + rcSources.push(rc9.read({ name: options.rcFile, dir: workspaceDir })); + } + // 3. user home + rcSources.push(rc9.readUser({ name: options.rcFile, dir: options.cwd })); + } + + const config = merger({} as T, ...rcSources); + if (!config || Object.keys(config).length === 0) return null; + + return { + config, + layer: { + config, + configFile: options.rcFile, + }, + }; + }, + }; +} + +/** + * Built-in provider: package.json config + */ +export function createPackageJsonProvider< + T extends UserInputConfig = UserInputConfig, + MT extends ConfigLayerMeta = ConfigLayerMeta, +>(): ConfigProvider { + return { + name: "packageJson", + priority: 400, + async load(ctx) { + const { options, merger } = ctx; + if (!options.packageJson) return null; + + const keys = ( + Array.isArray(options.packageJson) + ? options.packageJson + : [typeof options.packageJson === "string" ? options.packageJson : options.name] + ).filter((t): t is string => typeof t === "string" && t.length > 0); + + const pkgJsonFile = await readPackageJSON(options.cwd!).catch(() => {}); + if (!pkgJsonFile) return null; + + const values = keys.map((key) => pkgJsonFile[key] as T | undefined); + const config = merger({} as T, ...values); + if (!config || Object.keys(config).length === 0) return null; + + return { + config, + layer: { + config, + configFile: "package.json", + }, + }; + }, + }; +} + +/** + * Built-in provider: defaultConfig from options + */ +export function createDefaultConfigProvider< + T extends UserInputConfig = UserInputConfig, + MT extends ConfigLayerMeta = ConfigLayerMeta, +>(): ConfigProvider { + return { + name: "defaultConfig", + priority: 500, + async load(ctx) { + const config = await resolveResolvableConfig(ctx.options.defaultConfig, ctx); + if (!config) return null; + return { config }; + }, + }; +} + +/** + * Get the default set of built-in providers in standard order + */ +export function getDefaultProviders< + T extends UserInputConfig = UserInputConfig, + MT extends ConfigLayerMeta = ConfigLayerMeta, +>(): ConfigProvider[] { + return [ + createOverridesProvider(), + createMainProvider(), + createRcProvider(), + createPackageJsonProvider(), + createDefaultConfigProvider(), + ]; +} + +/** + * Sort providers by priority (lower priority number = higher precedence) + */ +export function sortProviders( + providers: ConfigProvider[], +): ConfigProvider[] { + return [...providers].sort((a, b) => a.priority - b.priority); +} + +/** + * Helper to resolve a ResolvableConfig (handles functions) + */ +async function resolveResolvableConfig( + value: ResolvableConfig | null | undefined, + ctx: ProviderContext, +): Promise { + if (typeof value === "function") { + // Build legacy context from loaded configs + const configs: Record = {}; + const rawConfigs: Record | null | undefined> = {}; + for (const [name, config] of ctx.loadedConfigs) { + configs[name] = config; + rawConfigs[name] = config; + } + return value({ configs, rawConfigs } as any); + } + return value; +} diff --git a/src/types.ts b/src/types.ts index 4045a88..ba45a18 100644 --- a/src/types.ts +++ b/src/types.ts @@ -1,5 +1,6 @@ import type { DownloadTemplateOptions } from "giget"; import type { DotenvOptions } from "./dotenv.ts"; +import type { ConfigProvider } from "./providers.ts"; export interface ConfigLayerMeta { name?: string; @@ -146,6 +147,29 @@ export interface LoadConfigOptions< }; configFileRequired?: boolean; + + /** + * Custom configuration providers. + * + * When specified, these providers are used instead of the built-in sources. + * Use `getDefaultProviders()` and modify the array to customize while + * keeping default behavior. + * + * Providers are sorted by priority (lower = higher precedence) before loading. + * + * @example + * ```ts + * import { getDefaultProviders } from 'c12'; + * + * loadConfig({ + * providers: [ + * ...getDefaultProviders(), + * { name: 'env', priority: 150, load: myEnvProvider } + * ] + * }) + * ``` + */ + providers?: ConfigProvider[]; } export type DefineConfig< diff --git a/test/loader.test.ts b/test/loader.test.ts index 3dc5288..d2e9027 100644 --- a/test/loader.test.ts +++ b/test/loader.test.ts @@ -1,8 +1,13 @@ import { fileURLToPath } from "node:url"; import { expect, it, describe } from "vitest"; import { normalize } from "pathe"; -import type { ConfigLayer, ConfigLayerMeta, UserInputConfig } from "../src/index.ts"; -import { loadConfig } from "../src/index.ts"; +import type { + ConfigLayer, + ConfigLayerMeta, + ConfigProvider, + UserInputConfig, +} from "../src/index.ts"; +import { getDefaultProviders, loadConfig } from "../src/index.ts"; const r = (path: string) => normalize(fileURLToPath(new URL(path, import.meta.url))); const transformPaths = (object: object) => @@ -347,7 +352,7 @@ describe("loader", () => { configFile: "CUSTOM", configFileRequired: true, }), - ).rejects.toThrowError("Required config (CUSTOM) cannot be resolved."); + ).rejects.toThrowError(/Required config \(.*CUSTOM\) cannot be resolved\./); }); it("loads arrays exported from config without merging", async () => { @@ -377,4 +382,122 @@ describe("loader", () => { cwd: r("./fixture/jsx"), }); }); + + describe("providers", () => { + it("uses default providers when none specified", async () => { + const { config, layers } = await loadConfig({ + cwd: r("./fixture"), + name: "test", + overrides: { fromOverrides: true }, + }); + expect(config.fromOverrides).toBe(true); + expect(layers!.length).toBeGreaterThan(0); + }); + + it("allows custom providers to inject config", async () => { + const customProvider: ConfigProvider = { + name: "custom", + priority: 150, // Between overrides (100) and main (200) + async load() { + return { + config: { customValue: "injected", overridden: false }, + layer: { config: { customValue: "injected" }, configFile: "custom-provider" }, + }; + }, + }; + + const { config, layers } = await loadConfig({ + cwd: r("./fixture"), + name: "test", + providers: [...getDefaultProviders(), customProvider], + overrides: { overridden: true }, + }); + + // Custom value should be present + expect(config.customValue).toBe("injected"); + // Overrides should still win (priority 100 < 150) + expect(config.overridden).toBe(true); + // Custom layer should be in layers + const customLayer = layers!.find((l) => l.configFile === "custom-provider"); + expect(customLayer).toBeDefined(); + }); + + it("respects provider priority order", async () => { + const lowPriorityProvider: ConfigProvider = { + name: "low-priority", + priority: 1000, // Very low priority + async load() { + return { config: { testKey: "from-low" } }; + }, + }; + + const highPriorityProvider: ConfigProvider = { + name: "high-priority", + priority: 50, // Very high priority + async load() { + return { config: { testKey: "from-high" } }; + }, + }; + + const { config } = await loadConfig({ + cwd: r("./fixture"), + name: "test", + providers: [lowPriorityProvider, highPriorityProvider], + }); + + // High priority should win + expect(config.testKey).toBe("from-high"); + }); + + it("allows removing default providers", async () => { + // Only use a single static provider + const onlyProvider: ConfigProvider = { + name: "only", + priority: 100, + async load() { + return { config: { onlyThis: true } }; + }, + }; + + const { config, layers } = await loadConfig({ + cwd: r("./fixture"), + name: "test", + providers: [onlyProvider], + }); + + expect(config.onlyThis).toBe(true); + // Should only have the one layer (no main, rc, packageJson) + expect(layers!.length).toBe(0); // No layer metadata provided + }); + + it("provider can access previously loaded configs", async () => { + const firstProvider: ConfigProvider = { + name: "first", + priority: 100, + async load() { + return { config: { firstValue: 42 } }; + }, + }; + + const secondProvider: ConfigProvider = { + name: "second", + priority: 200, + async load(ctx) { + const firstConfig = ctx.loadedConfigs.get("first"); + return { + config: { sawFirst: firstConfig?.firstValue === 42 }, + }; + }, + }; + + const { config } = await loadConfig({ + cwd: r("./fixture"), + name: "test", + providers: [firstProvider, secondProvider], + }); + + expect(config.firstValue).toBe(42); + expect(config.sawFirst).toBe(true); + }); + }); }); From df94204fc92b77174c01341e02734280b5bf08ee Mon Sep 17 00:00:00 2001 From: rektide Date: Mon, 9 Feb 2026 23:02:22 -0500 Subject: [PATCH 5/5] docs: document ConfigProvider implementation in layering.md Add comprehensive documentation for the implemented provider system: - Architecture overview with mermaid diagram - Core types (ConfigProvider, ProviderContext, ProviderResult) - Built-in providers table - Usage examples (custom, replacing, reordering, dependencies, remote) - How the provider system works (step-by-step) - Impact on existing c12 users (zero breaking changes) - Minor differences noted (error message path resolution) - Future enhancements roadmap - Files changed summary --- layering.md | 392 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 392 insertions(+) diff --git a/layering.md b/layering.md index 6cbc5a9..e395d84 100644 --- a/layering.md +++ b/layering.md @@ -435,3 +435,395 @@ const { config, provenance } = await loadConfig({ 4. **Async providers**: How to handle slow providers (Vault, remote config) gracefully? 5. **Caching**: Should registry cache loaded configs for repeated `.load()` calls? + +--- + +## Implemented: ConfigProvider System + +Phase 2 of the roadmap has been implemented. The c12-layer fork now includes a fully functional pluggable provider system that allows users to customize the configuration loading pipeline while maintaining complete backward compatibility. + +### Architecture Overview + +The provider system introduces a `ConfigProvider` interface that abstracts each configuration source. Instead of hardcoded loading logic, `loadConfig()` now iterates through an ordered list of providers, each responsible for loading one source of configuration. + +```mermaid +flowchart LR + subgraph providers["Provider Pipeline"] + direction TB + p1[overrides
priority: 100] + p2[main
priority: 200] + p3[rc
priority: 300] + p4[packageJson
priority: 400] + p5[defaultConfig
priority: 500] + end + + loadConfig --> sort[Sort by priority] + sort --> providers + providers --> merge[defu merge all configs] + merge --> extends[Process extends] + extends --> result[ResolvedConfig] +``` + +### Core Types + +The implementation adds these types in [`src/providers.ts`](src/providers.ts): + +```typescript +/** + * Context passed to config providers during loading + */ +interface ProviderContext { + /** Normalized load options */ + options: LoadConfigOptions; + /** Merger function (defu or custom) */ + merger: (...sources: Array) => T; + /** Resolve a config file (used by main provider) */ + resolveConfig: (source: string, options: LoadConfigOptions) => Promise>; + /** Results from previously loaded providers (by name) */ + loadedConfigs: Map; +} + +/** + * Result returned by a config provider + */ +interface ProviderResult { + /** The loaded configuration (or null/undefined if not found) */ + config: T | null | undefined; + /** Layer metadata for introspection */ + layer?: Partial>; + /** Additional metadata to merge into ResolvedConfig */ + meta?: MT; + /** Resolved config file path (for main provider) */ + configFile?: string; + /** Internal config file path */ + _configFile?: string; +} + +/** + * A pluggable configuration source provider + */ +interface ConfigProvider { + /** Unique name for this provider */ + name: string; + /** + * Priority determines merge order. + * Lower numbers = higher priority (merged first, so they "win"). + */ + priority: number; + /** + * Load configuration from this provider. + * Return null/undefined if this provider has no config to contribute. + */ + load(ctx: ProviderContext): Promise | null | undefined>; +} +``` + +### Built-in Providers + +Five built-in providers replicate the original c12 behavior: + +| Provider | Priority | Factory Function | Description | +|----------|----------|------------------|-------------| +| overrides | 100 | `createOverridesProvider()` | Returns `options.overrides` | +| main | 200 | `createMainProvider()` | Loads `.config.ts` via `resolveConfig()` | +| rc | 300 | `createRcProvider()` | Loads `.namerc` files (cwd, workspace, home) | +| packageJson | 400 | `createPackageJsonProvider()` | Loads from `package.json[name]` | +| defaultConfig | 500 | `createDefaultConfigProvider()` | Returns `options.defaultConfig` | + +Each provider is a standalone factory function that can be individually imported and customized. + +### Exports + +The following are exported from the main `c12` module: + +```typescript +// Types +export type { ConfigProvider, ProviderContext, ProviderResult } from "./providers.ts"; + +// Factory functions for built-in providers +export { + getDefaultProviders, // Returns all 5 built-in providers + sortProviders, // Sort providers by priority + createOverridesProvider, + createMainProvider, + createRcProvider, + createPackageJsonProvider, + createDefaultConfigProvider, +} from "./providers.ts"; +``` + +### Usage Examples + +#### Basic: No Changes Required + +Existing code works without modification: + +```typescript +import { loadConfig } from 'c12'; + +// Works exactly as before +const { config } = await loadConfig({ name: 'myapp' }); +``` + +#### Adding a Custom Provider + +Inject a custom source between existing ones: + +```typescript +import { loadConfig, getDefaultProviders, ConfigProvider } from 'c12'; + +// Create a provider that loads from environment variables +const envProvider: ConfigProvider = { + name: 'env', + priority: 150, // Between overrides (100) and main (200) + async load(ctx) { + const config: Record = {}; + const prefix = `${ctx.options.name?.toUpperCase()}_`; + + for (const [key, value] of Object.entries(process.env)) { + if (key.startsWith(prefix)) { + const configKey = key.slice(prefix.length).toLowerCase(); + config[configKey] = value; + } + } + + if (Object.keys(config).length === 0) return null; + + return { + config, + layer: { + config, + configFile: 'environment', + }, + }; + }, +}; + +const { config, layers } = await loadConfig({ + name: 'myapp', + providers: [...getDefaultProviders(), envProvider], +}); + +// layers will include { configFile: 'environment', config: {...} } +``` + +#### Replacing Default Providers + +Use only specific sources: + +```typescript +import { loadConfig, createMainProvider, createOverridesProvider } from 'c12'; + +// Only load from overrides and main config file (no RC, no package.json) +const { config } = await loadConfig({ + name: 'myapp', + providers: [ + createOverridesProvider(), + createMainProvider(), + ], + overrides: { debug: true }, +}); +``` + +#### Reordering Providers + +Change the default priority order: + +```typescript +import { loadConfig, getDefaultProviders } from 'c12'; + +// Make package.json take precedence over RC files +const providers = getDefaultProviders().map(p => { + if (p.name === 'packageJson') return { ...p, priority: 250 }; + if (p.name === 'rc') return { ...p, priority: 350 }; + return p; +}); + +const { config } = await loadConfig({ + name: 'myapp', + providers, +}); +``` + +#### Provider Dependencies + +Access previously loaded configs in your provider: + +```typescript +const conditionalProvider: ConfigProvider = { + name: 'conditional', + priority: 250, + async load(ctx) { + // Check what the main config loaded + const mainConfig = ctx.loadedConfigs.get('main'); + + if (mainConfig?.featureFlags?.enableAdvanced) { + return { + config: { advancedSetting: 'enabled' }, + }; + } + + return null; // Don't contribute if feature not enabled + }, +}; +``` + +#### Remote Configuration Provider + +Load config from a remote source: + +```typescript +const remoteProvider: ConfigProvider = { + name: 'remote', + priority: 175, // After env, before main + async load(ctx) { + const endpoint = process.env.CONFIG_ENDPOINT; + if (!endpoint) return null; + + try { + const response = await fetch(`${endpoint}/${ctx.options.name}`); + if (!response.ok) return null; + + const config = await response.json(); + return { + config, + layer: { + config, + configFile: endpoint, + }, + }; + } catch { + // Network error - silently skip + return null; + } + }, +}; +``` + +### How the Provider System Works + +1. **Initialization**: `loadConfig()` normalizes options and sets up dotenv (unchanged) + +2. **Provider Selection**: + - If `options.providers` is specified, use those providers + - Otherwise, call `getDefaultProviders()` to get the built-in set + +3. **Sorting**: Providers are sorted by `priority` (ascending). Lower priority numbers are processed first and "win" in the merge. + +4. **Sequential Loading**: Each provider's `load()` method is called in order. The `ProviderContext` includes: + - The normalized `options` + - The `merger` function (defu or custom) + - A `resolveConfig` helper for loading files + - A `loadedConfigs` map with results from previous providers + +5. **Result Collection**: Non-null results are collected. Each result includes: + - `config`: The configuration object + - `layer`: Optional metadata for the `layers` array + - `configFile`, `_configFile`, `meta`: For the main provider + +6. **Merging**: All collected configs are merged using the merger function, in priority order + +7. **Extends Processing**: The `extends` mechanism works as before, after the main merge + +8. **Layer Preservation**: Provider results with `layer` data are added to `ResolvedConfig.layers` + +### Impact on Existing c12 Users + +#### Zero Breaking Changes + +The provider system is **fully backward compatible**. Existing code continues to work without any modifications: + +```typescript +// This code works identically before and after the change +const { config } = await loadConfig({ + name: 'myapp', + overrides: { debug: true }, + rcFile: '.myapprc', + packageJson: true, + defaults: { port: 3000 }, +}); +``` + +#### Behavioral Equivalence + +When `options.providers` is not specified, the system behaves identically to the original c12: + +1. Same source loading order (overrides → main → rc → packageJson → defaultConfig) +2. Same merge semantics (defu, or custom merger) +3. Same `extends` processing +4. Same `layers` array structure +5. Same handling of `$development`, `$production`, etc. + +#### Minor Differences + +There is one minor difference in error messages: + +| Scenario | Original | With Providers | +|----------|----------|----------------| +| `configFileRequired: true` with missing file | `Required config (CUSTOM) cannot be resolved.` | `Required config (/full/path/CUSTOM) cannot be resolved.` | + +The new message includes the full resolved path, which is more helpful for debugging. + +#### New Option + +One new option is added to `LoadConfigOptions`: + +```typescript +interface LoadConfigOptions { + // ... existing options ... + + /** + * Custom configuration providers. + * When specified, replaces the built-in source loading. + * Use getDefaultProviders() to include defaults. + */ + providers?: ConfigProvider[]; +} +``` + +### Testing the Provider System + +The implementation includes comprehensive tests in [`test/loader.test.ts`](test/loader.test.ts): + +```typescript +describe("providers", () => { + it("uses default providers when none specified"); + it("allows custom providers to inject config"); + it("respects provider priority order"); + it("allows removing default providers"); + it("provider can access previously loaded configs"); +}); +``` + +Run tests with: +```bash +pnpm test +``` + +### Future Enhancements + +The provider system creates a foundation for additional features: + +1. **Drop-in Directory Provider**: A `createConfigDirProvider()` that scans `*.config.d/` directories + +2. **Provenance Tracking**: Providers already return layer metadata; a tracking merger could record which provider contributed each key + +3. **Validation Phase**: A `validateProviders()` function that checks all sources exist before loading + +4. **Parallel Loading**: Independent providers could load concurrently for better performance + +5. **Provider Plugins**: A registry of community providers (Vault, AWS SSM, Consul, etc.) + +### Files Changed + +| File | Change | +|------|--------| +| [`src/providers.ts`](src/providers.ts) | New file: Provider types and built-in implementations | +| [`src/types.ts`](src/types.ts) | Added `providers` option to `LoadConfigOptions` | +| [`src/loader.ts`](src/loader.ts) | Refactored to use provider pipeline | +| [`src/index.ts`](src/index.ts) | Export provider types and functions | +| [`test/loader.test.ts`](test/loader.test.ts) | Added provider system tests | + +### Summary + +The ConfigProvider system transforms c12 from a fixed-pipeline loader into an extensible, customizable configuration platform while maintaining complete backward compatibility. Users who don't need customization see no changes; power users gain full control over the loading pipeline.