refactor: a tiny bit of everything #420

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft

luxass wants to merge 146 commits into main from this-is-a-lot-of-changes

.vscode/settings.json

-Original file line number
+Diff line change
@@ Expand Up / @@ -45,5 +45,7 @@ @@
       },
       "files.readonlyInclude": {
         "**/routeTree.gen.ts": true
-      }
+      },
+      "typescript.tsdk": "node_modules/typescript/lib",
+      "typescript.experimental.useTsgo": true
     }

AGENTS.md

-Original file line number
+Diff line change
@@ Expand Up @@
     ### Key Packages
     - **@ucdjs/ucd-store**: Store for managing Unicode Character Database files. Supports multiple file system bridges (Node.js, HTTP, in-memory). Core operations include mirror, analyze, and clean.
-    - **@ucdjs/ucd-store-v2**: Next-generation store with lockfile and snapshot support.
-      > NOTE: This is a temporary package name while the new store is being implemented. Once stable, it will replace `@ucdjs/ucd-store`.
-      > Core operations include mirror, analyze, and sync (different from v1).
     - **@ucdjs/lockfile**: Lockfile and snapshot management utilities for UCD stores. Provides file hashing, lockfile/snapshot validation, and test utilities.
     - **@ucdjs/schema-gen**: Uses AI (OpenAI) to generate TypeScript schemas from Unicode data files
     - **@ucdjs/cli**: Command-line interface for UCD operations (binary: `ucd`)
-      > Currently uses `@ucdjs/ucd-store-v2` and `@ucdjs/lockfile` under the hood.
+      > Currently uses `@ucdjs/ucd-store` and `@ucdjs/lockfile` under the hood.
     - **@ucdjs/client**: OpenAPI-based API client for the UCD API
     - **@ucdjs/fs-bridge**: File system abstraction layer that allows different storage backends
     - **@ucdjs/schemas**: Zod schemas for Unicode data files (includes lockfile and snapshot schemas)
@@ Expand Down Expand Up @@
 . Uses AI to infer field types and descriptions
 . Generates TypeScript interfaces using knitwork
-    ### UCD Store Migration
-    The project is currently migrating from `@ucdjs/ucd-store` to `@ucdjs/ucd-store-v2`. Key differences:
-    **Old Store (@ucdjs/ucd-store)**:
-    - Operations: mirror, analyze, clean
-    - No lockfile/snapshot support
-    **New Store (@ucdjs/ucd-store-v2)**:
-    - Operations: mirror, analyze, sync
-    - Integrated lockfile and snapshot support via `@ucdjs/lockfile`
-    - Currently used by CLI
-    - Will replace old store once stable
-    Both stores coexist during the migration period. When working with the CLI or testing new features, use ucd-store-v2.
     ### Internal Development Tools
     #### Moonbeam (@ucdjs/moonbeam)
@@ Expand Down @@

apps/api/README.md

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -2,18 +2,12 @@
  
    A RESTful API for accessing Unicode Character Database (UCD) data.

    ## 📦 Installation

    ## � Usage

    ```sh

    git clone https://github.com/ucdjs/api.ucdjs.dev.git

    cd api.ucdjs.dev

    pnpm install

    ```

    ## 🚀 Usage

    From the root directory:

    ```sh

    pnpm run dev

    pnpm run dev:apps

    ```

    ## 📖 API Documentation

apps/api/package.json

-Original file line number
+Diff line change
@@ Expand Up / @@ -2,7 +2,7 @@ @@
       "name": "@ucdjs/api",
       "type": "module",
       "private": true,
-      "packageManager": "pnpm@10.26.1",
+      "packageManager": "pnpm@10.26.2",
       "scripts": {
         "dev": "wrangler dev --port 8787 --inspector-port 9229",
         "build": "wrangler deploy --dry-run --outdir=dist --tsconfig=./tsconfig.build.json --env=production",
@@ Expand Down @@

apps/api/src/constants.ts

-Original file line number
+Diff line change
@@ -1,6 +1,10 @@
     export const V1_FILES_ROUTER_BASE_PATH = "/api/v1/files";
     export const V1_VERSIONS_ROUTER_BASE_PATH = "/api/v1/versions";
     export const V1_SCHEMAS_ROUTER_BASE_PATH = "/api/v1/schemas";
+    export const V1_CHARACTERS_ROUTER_BASE_PATH = "/api/v1/characters";
+    export const V1_PROPERTIES_ROUTER_BASE_PATH = "/api/v1/properties";
+    export const V1_BLOCKS_ROUTER_BASE_PATH = "/api/v1/blocks";
     export const WELL_KNOWN_ROUTER_BASE_PATH = "/.well-known";
     export const HTML_EXTENSIONS = [
@@ Expand Down @@

apps/api/src/lib/files.ts

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -1,5 +1,5 @@
  
    import type { Entry } from "apache-autoindex-parse";

    import { trimTrailingSlash } from "@luxass/utils";

    import { createGlobMatcher } from "@ucdjs-internal/shared";

    import { parse } from "apache-autoindex-parse";

    /**

    @@ -20,13 +20,102 @@ import { parse } from "apache-autoindex-parse";
  
     * console.log(entries); // [{ type: 'directory', name: 'UNIDATA', path: '/UNIDATA', ... }]

     * ```

     */

    export async function parseUnicodeDirectory(html: string): Promise<Entry[]> {

      const files = parse(html, "F2");

      return files.map(({ type, name, path, lastModified }) => ({

        type,

        name: trimTrailingSlash(name),

        path: trimTrailingSlash(path),

        lastModified,

      }));

    export async function parseUnicodeDirectory(html: string, basePath = ""): Promise<Entry[]> {

      const files = parse(html, {

        format: "F2",

        basePath,

      });

      return files;

    }

    export interface DirectoryFilterOptions {

      /**

       * A string to filter file/directory names that start with this query (case-insensitive).

       */

      query?: string;

      /**

       * A glob pattern to filter file/directory names.

       */

      pattern?: string;

      /**

       * Type of entries to include: "all" (default), "files", or "directories".

       */

      type?: string;

      /**

       * Field to sort by: "name" (default) or "lastModified".

       */

      sort?: string;

      /**

       * Sort order: "asc" (default) or "desc".

       */

      order?: string;

    }

    /**

     * Applies filtering and sorting to directory entries based on query parameters.

     *

     * @param {Entry[]} files - Array of directory entries to filter and sort

     * @param {DirectoryFilterOptions} options - Filter and sort options

     * @returns {Entry[]} Filtered and sorted array of entries

     */

    export function applyDirectoryFiltersAndSort(

      files: Entry[],

      options: DirectoryFilterOptions,

    ): Entry[] {

      let filtered = [...files];

      // Apply query filter (prefix search, case-insensitive)

      if (options.query) {

        // eslint-disable-next-line no-console

        console.info(`[v1_files]: applying query filter: ${options.query}`);

        const queryLower = options.query.toLowerCase();

        filtered = filtered.filter((entry) => entry.name.toLowerCase().startsWith(queryLower));

      }

      // Apply pattern filter if provided

      if (options.pattern) {

        // eslint-disable-next-line no-console

        console.info(`[v1_files]: applying glob pattern filter: ${options.pattern}`);

        const matcher = createGlobMatcher(options.pattern);

        filtered = filtered.filter((entry) => matcher(entry.name));

      }

      // Apply type filter

      const type = options.type || "all";

      if (type === "files") {

        filtered = filtered.filter((entry) => entry.type === "file");

      } else if (type === "directories") {

        filtered = filtered.filter((entry) => entry.type === "directory");

      }

      // Apply sorting (directories always first, like Windows File Explorer)

      const sort = options.sort || "name";

      const order = options.order || "asc";

      filtered = filtered.toSorted((a, b) => {

        // Directories always come first

        if (a.type !== b.type) {

          return a.type === "directory" ? -1 : 1;

        }

        // Within same type, apply the requested sort

        let comparison: number;

        if (sort === "lastModified") {

          // lastModified is always available from parseUnicodeDirectory

          comparison = (a.lastModified ?? 0) - (b.lastModified ?? 0);

        } else {

          // Natural name sorting (numeric aware) so 2.0.0 < 10.0.0

          comparison = a.name.localeCompare(b.name, undefined, { numeric: true, sensitivity: "base" });

        }

        return order === "desc" ? -comparison : comparison;

      });

      return filtered;

    }

apps/api/src/openapi.ts

-Original file line number
+Diff line change
@@ Expand Up @@
     export const OPENAPI_TAGS = {
       VERSIONS: "Versions",
       FILES: "Files",
+      CHARACTERS: "Characters",
+      PROPERTIES: "Properties",
+      BLOCKS: "Blocks",
       WELL_KNOWN: "Well-Known",
     } as const satisfies Record<string, string>;
@@ Expand Down @@

apps/api/src/routes/.well-known/ucd-config.json.ts

-Original file line number
+Diff line change
@@ Expand Up @@
           version: "0.1",
           endpoints: {
             files: V1_FILES_ROUTER_BASE_PATH,
-            manifest: `${WELL_KNOWN_ROUTER_BASE_PATH}/ucd-store.json`,
+            manifest: `${WELL_KNOWN_ROUTER_BASE_PATH}/ucd-store/{version}.json`,
             versions: V1_VERSIONS_ROUTER_BASE_PATH,
           },
           versions: versionStrings,
@@ Expand Down @@

apps/api/src/routes/v1_blocks/$block.ts

-Original file line number
+Diff line change
@@ -0,0 +1,115 @@
+    import type { OpenAPIHono } from "@hono/zod-openapi";
+    import type { HonoEnv } from "../../types";
+    import { createRoute } from "@hono/zod-openapi";
+    import { dedent } from "@luxass/utils";
+    import { UnicodeBlockSchema } from "@ucdjs/schemas";
+    import { cache } from "hono/cache";
+    import { MAX_AGE_ONE_DAY_SECONDS } from "../../constants";
+    import { badRequest, notFound } from "../../lib/errors";
+    import { createLogger } from "../../lib/logger";
+    import { VERSION_ROUTE_PARAM } from "../../lib/shared-parameters";
+    import { generateReferences, OPENAPI_TAGS } from "../../openapi";
+    const log = createLogger("ucd:api:v1_blocks");
+    const GET_BLOCK_ROUTE = createRoute({
+      method: "get",
+      path: "/{version}/{block}",
+      tags: [OPENAPI_TAGS.BLOCKS],
+      middleware: [
+        cache({
+          cacheName: "ucdjs:v1_blocks:block",
+          cacheControl: `max-age=${MAX_AGE_ONE_DAY_SECONDS}`,
+        }),
+      ],
+      parameters: [
+        VERSION_ROUTE_PARAM,
+        {
+          name: "block",
+          in: "path",
+          schema: { type: "string" },
+          required: true,
+          description: "Unicode block name or ID (e.g., Basic_Latin, CJK_Unified_Ideographs)",
+        },
+      ],
+      description: dedent`
+        ## Get Unicode Block Details
+        Retrieve detailed information about a specific Unicode block.
+        - Supports **block name or ID** (e.g., \`Basic_Latin\`, \`CJK_Unified_Ideographs\`)
+        - Returns **block information** including codepoint range and character count
+        - Optionally **includes character list** with minimal or detailed format
+        - Supports **pagination** via limit parameter
+        - Supports **caching** for performance optimization
+      `,
+      responses: {
+: {
+          content: {
+            "application/json": {
+              schema: UnicodeBlockSchema,
+              examples: {
+                default: {
+                  summary: "Basic Latin block",
+                  value: {
+                    name: "Basic Latin",
+                    aliases: ["ASCII"],
+                    range: { start: "U+0000", end: "U+007F" },
+                    count: 128,
+                    description: "ASCII characters and basic Latin",
+                    relatedScripts: ["Latin"],
+                  },
+                },
+                withCharacters: {
+                  summary: "Block with characters",
+                  value: {
+                    name: "Basic Latin",
+                    aliases: ["ASCII"],
+                    range: { start: "U+0000", end: "U+007F" },
+                    count: 128,
+                    description: "ASCII characters and basic Latin",
+                    relatedScripts: ["Latin"],
+                    characters: [
+                      { codepoint: "U+0041", character: "A", name: "LATIN CAPITAL LETTER A" },
+                      { codepoint: "U+0061", character: "a", name: "LATIN SMALL LETTER A" },
+                    ],
+                  },
+                },
+              },
+            },
+          },
+          description: "Block details",
+        },
+        ...(generateReferences([
+,
+,
+,
+,
+        ])),
+      },
+    });
+    export function registerBlockRoute(router: OpenAPIHono<HonoEnv>) {
+      router.openapi(GET_BLOCK_ROUTE, async (c) => {
+        const { block, version } = c.req.param();
+        if (!block || block.length === 0) {
+          log.warn("Empty block name");
+          return badRequest(c, {
+            message: "Block name cannot be empty",
+          });
+        }
+        // TODO: Fetch block data from Unicode database
+        // Include characters if requested, apply format and limit
+        log.info("Fetching block data", {
+          block,
+          version,
+        });
+        return notFound(c, {
+          message: `Block "${block}" not found. API implementation pending.`,
+        });
+      });
+    }

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refactor: a tiny bit of everything #420

Uh oh!

Diff view

Diff view

Uh oh!

There are no files selected for viewing

Uh oh!

Uh oh!

Uh oh!

refactor: a tiny bit of everything #420

Are you sure you want to change the base?

Uh oh!

refactor: a tiny bit of everything #420

Uh oh!

Diff view

Diff view

Uh oh!

There are no files selected for viewing

Uh oh!

Uh oh!