Misc Ideas for Enhancements

[thorwhalen]

To collect miscellaneous ideas for enhancement, new features, interfaces, etc.

[thorwhalen]

RAM-sensitive ZipReader

I'd like a ZipAccess(typing.Mapping) class in python that takes a zip file path.
It's __iter__ should give me an iterable of the filenames the zip contains, the __getitem__ should give me the bytes for that particular file, given the key.
Important though: I don't want to hold all the zips bytes in memory. I want the methods of my class to load only what they need to, and only temporarily.

import zipfile
from typing import Mapping

class ZipAccess:
    def __init__(self, zip_file_path):
        self.zip_file_path = zip_file_path

    def __iter__(self):
        with zipfile.ZipFile(self.zip_file_path, 'r') as zip_archive:
            return iter(zip_archive.namelist())

    def __getitem__(self, key):
        with zipfile.ZipFile(self.zip_file_path, 'r') as zip_archive:
            if key not in zip_archive.namelist():
                raise KeyError(f"'{key}' not found in the zip file.")
            with zip_archive.open(key) as file:
                return file.read()

# Example usage:
if __name__ == "__main__":
    zip_path = "example.zip"
    zip_access = ZipAccess(zip_path)

    # Iterating over filenames in the zip
    for filename in zip_access:
        print(filename)

    # Accessing the bytes of a specific file in the zip
    file_contents = zip_access['example.txt']
    print(file_contents.decode('utf-8'))  # Assuming it's a text file

[thorwhalen]

Clean up explicit tools

We have a module for "explicit" stores: dol/explicit.py.
By "explicit", I mean when the collection of keys (or values) are explicitly given.
It has always been a bit all over the place, though the theme that comes up a lot.

Note that a lot of the same functionalities can be taken care of, more or less, via wrap_kvs, so we may want to use these.

I'd propose to have a mini project where we analyze what's there, document and cover it with test it using AI, then try to replace with wrap_kvs-based code. Also, it would be nice to have some tutorial on this theme so that people (and AI) can better understand how to use it.

Things to check out

In the following discussion comment, I'm using wrap_kvs for some "explicit" mechanisms: Data acquisition stores

In config2py.base, there's a FuncBasedGettableContainer that wraps a Callable[[KT], VT] function so it has a (partial)
Mapping[KT, TT] interface

[thorwhalen]

Forwarding methods to extend mappings?

Given the "special methods" (__iter__, __len__, ...) get special treatment, which stops me from being able to simply change their definition "dynamically", which would make wrapping mappings a whole lot less verbose, I often thought that using non-special methods as the base, and simply forward the special methods to these might be a better idea.
I'm not yet sure if it's a good idea or not, but here are a few notes on that.

from collections.abc import Mapping

class MappingHooks(Mapping):
    def __iter__(self):
        return self.__iter()
    def __len__(self):
        return self.__len()
    def __contains__(self, key):
        return self.__contains(key)
    def __getitem__(self, key):
        return self.__getitem(key)
    
class ExampleMapping(MappingHooks, Mapping):
    """Just forwards all method calls to the wrapped dict"""
    def __init__(self, d: dict):
        self.d = d
    def __iter(self):
        return iter(self.d)
    def __len(self):
        return len(self.d)
    def __contains(self, key):
        return key in self.d
    def __getitem(self, key):
        return self.d[key]
    

m = ExampleMapping({'a': 1, 'b': 2})

assert list(m) == ['a', 'b']
assert len(m) == 2
assert 'a' in m
assert 'b' in m
assert 'c' not in m
assert m['a'] == 1
assert m['b'] == 2

Notes:

• Prefixing the names of the base methods with double-underscore seems to make it automatically treated as a special method. So for example, if you replace _iter with __iter (as I did origially) here, you'll get AttributeError: 'ExampleMapping' object has no attribute '_MappingHooks__iter'
• class ExampleMapping(Mapping, MappingHooks) leads to TypeError: Cannot create a consistent method resolution order (MRO) for bases Mapping, MappingHooks.
• Just writing __iter__ = _iter, __len__ = _len in a base class doesn't work since __iter__ is then bound to the particular _iter method (changing the _iter in a subclass doesn't have the effect expected.

[thorwhalen]

a KeyCodecs.mapped_keys and postelizing invertible_maps

The KeyCodecs.mapped_keys wrapper is useful to create a key codec that uses "explicit" mappings to encode and decode keys
(bringing functionality that used to be in the explicit.py module.

But the UX would even be more fantastic if the user could just specify an encoder (or decoder) function, and get the other one computed for them. As it is now, we have this functionality, but we need to feed an explicit mapping as the encoder (or decoder) that is missing. See below:

from dol import KeyCodecs, ZipFiles
from msword.tests.util import test_data_dir

zip_files = ZipFiles(os.path.join(test_data_dir, 'some_zip_file.zip'))
assert sorted(zip_files) == ['not_an_msword_doc.txt', 'simple.docx', 'with_doc_extension.doc']
remove_extensions = KeyCodecs.mapped_keys(zip_files, decoder=lambda x: os.path.splitext(x)[0])  # Note: Have to specify zip_files (keys) explicitly
s = remove_extensions(zip_files)
assert sorted(s) == ['not_an_msword_doc', 'simple', 'with_doc_extension']

This restriction comes from dol.util.invertible_maps itself which requires the explicit map, but at the level of KeyCodecs.mapped_keys, we could do something so that the user could just do this:

remove_extensions = KeyCodecs.mapped_keys(decoder=lambda x: os.path.splitext(x)[0])

and the keys to apply the decoder to would be dynamically fetched.

Or even more:

ZipFilesWithRemovedExtensions = KeyCodecs.mapped_keys(ZipFiles, decoder=lambda x: os.path.splitext(x)[0])

[thorwhalen]

A jsonbin store

jsonbins is a platform that allows users to store and retrieve JSON data over HTTP. The API provides endpoints for creating, updating, deleting, and retrieving JSON data. Users can also control the visibility and access permissions of their data.

See jsonbins api-reference.

There's no official python package for it. Found this package though.

Would like a MutableMapping interface for jsonbins.
The pypi name "bins" is available at the time of writing this.

We could start by using this this package, but I'd prefer to use ju and json2py packages to make the base python functions automatically from OpenAPI schema (swagger): But I didn't find an OpenAPI schema for jsonbins. We could probably make one semi-automatically from docs.

[thorwhalen]

Use the standard lib codecs module?

See this "codecs overview" conversation.

Key example:

import codecs

# Define encoder and decoder (both must return tuple (result, length))
def rot13_encode(input_str, errors='strict'):
    result = input_str.translate(str.maketrans(
        'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz',
        'NOPQRSTUVWXYZABCDEFGHIJKLMnopqrstuvwxyzabcdefghijklm'
    ))
    return result, len(input_str)

def rot13_decode(input_str, errors='strict'):
    # Same logic for ROT13 decoding
    return rot13_encode(input_str, errors)

# Define the CodecInfo object
rot13_codec_info = codecs.CodecInfo(
    name='rot13_custom',
    encode=rot13_encode,
    decode=rot13_decode
)

# Register codec
def rot13_search_function(encoding_name):
    if encoding_name == 'rot13_custom':
        return rot13_codec_info
    return None

# Register the codec globally
codecs.register(rot13_search_function)

Example use:

# Encoding
encoded = codecs.encode("Hello, World!", "rot13_custom")
print(encoded)  # Uryyb, Jbeyq!

# Decoding
decoded = codecs.decode(encoded, "rot13_custom")
print(decoded)  # Hello, World!

[thorwhalen]

Standard lib support for postget and preset

There are several standard library modules that can help use implement dynamic, context-aware encoding/decoding, especially when dispatching based on file extension, key patterns, or value types.

While none of them are built specifically for key/value or extension-based codecs, we can combine them with custom logic to build what we're after.

🧰 Helpful Standard Library Modules for Dynamic Serialization

mimetypes

Maps file extensions to MIME types, which you can use to select a codec.

import mimetypes

mime, _ = mimetypes.guess_type("file.json")  # 'application/json'

Use this to dispatch to codecs like 'json', 'gzip', 'text/plain', etc.

functools.singledispatch

Function overloading based on value type. Great for dispatching based on value type.

from functools import singledispatch

@singledispatch
def encode(value):
    raise NotImplementedError

@encode.register
def _(value: dict):
    return codecs.encode(json.dumps(value), 'utf-8')

@encode.register
def _(value: bytes):
    return value  # passthrough

Combine with key-based dispatch to dynamically choose both key and value logic.

pathlib

Gives you clean access to file extensions and path components, ideal for key-based logic.

from pathlib import Path

ext = Path("data/file.json.gz").suffix  # '.gz'

You can extract the full .suffixes for nested extensions (e.g. .json.gz).

types + inspect

Useful when value-based logic depends on inspecting callables, lambdas, or types.

from types import FunctionType
from inspect import isfunction

if isinstance(value, FunctionType) or isfunction(value):
    raise TypeError("Cannot serialize functions")

json, pickle, base64, gzip, bz2

While not “dispatch helpers,” these format-specific modules form your backend encoders/decoders. You can wrap them like codecs.

Example:

import json
import gzip

def gzip_json_encoder(value):
    return gzip.compress(json.dumps(value).encode())

def gzip_json_decoder(data):
    return json.loads(gzip.decompress(data).decode())

Register that with your own dispatch layer or codec wrapper.

🧩 Putting It All Together

You can build a dynamic encoder registry like this:

from pathlib import Path
from functools import singledispatch

def get_codec_from_key(key):
    suffix = Path(key).suffix
    if suffix == '.json':
        return 'json_codec'
    elif suffix == '.gz':
        return 'gzip_codec'
    return 'utf-8'

@singledispatch
def encode(value, key=None):
    raise NotImplementedError

@encode.register
def _(value: dict, key=None):
    return json.dumps(value).encode()

@encode.register
def _(value: bytes, key=None):
    return value  # already encoded

✅ Final Recommendation

Use a custom dispatch layer that:
• Leverages pathlib or mimetypes for key/extension logic.
• Uses singledispatch for value-type logic.
• Calls codecs or format-specific serializers as backends.

This gives you codec-like behavior far beyond what codecs supports, with full flexibility for key–value pairs.