basic knot merging

[oleksost]

• added Knot merging for LoRAs from https://arxiv.org/pdf/2410.19735
• currently Vs are merged with uniform Ties merging method without weights
• the Knot ingredients (SVD components) are stored in a local file, in the future they probably should be stored in the library. Now, aux data in the lib must be associated with expert, Knot components are not expert specific.

[Copilot]

Pull Request Overview

This PR introduces a new Knot merging transform for LoRA experts, computing SVD components, storing them locally, and merging expert weights uniformly via TIES.

• Added KnotMerge and KnotMergeConfig to perform SVD-based merges.
• Updated TiesMerge to factor out parameter-merging logic into merge_param.
• Added a test_knot_merge unit test to validate the new transform.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
tests/test_library_transforms.py	Added test_knot_merge to verify the KnotMerge flow.
mttl/models/library/library_transforms.py	Implemented KnotMerge transform and refactored TiesMerge.

Comments suppressed due to low confidence (3)

mttl/models/library/library_transforms.py:11

• Add import torch at the top of this file so that all references to torch.save, torch.load, and other torch APIs resolve correctly.

from typing import Dict, List, Union

mttl/models/library/library_transforms.py:361

• [nitpick] Rename the variable ties_mergert to ties_merger to fix the typo and clarify its purpose.

        ties_mergert = TiesMerge()

tests/test_library_transforms.py:96

• [nitpick] Consider adding assertions to verify that the merged weights themselves match expected values (e.g., compare against a manual U @ final_param calculation) to improve test coverage.

    assert len(merged_layers) == len(exp.expert_weights.keys()) == 1

[Copilot]

Correct the typo ezperts to experts in the docstring.

Suggested change

	Computes a weighted KnoT merge for LoRA ezperts as in https://arxiv.org/pdf/2410.19735
	Computes a weighted KnoT merge for LoRA experts as in https://arxiv.org/pdf/2410.19735

[Copilot]

The second assignment to sign_per_dim overrides the first; remove the redundant line or clarify which operation is intended.

Suggested change

sign_per_dim = expert_weights.sign().sum(0, keepdim=True).sign()