Internationalization test

.github/workflows/buildcheck-debug.yml

@@ -35,6 +35,11 @@ jobs:
     - name: Setup MSBuild.exe
       uses: microsoft/setup-msbuild@v2
 
+    # For reasons that are unclear, the build fails if the generated files do not exist at the beginning of the build.
+    # Explicitly build the generator to generate the files, then let the implicit build take care of everything else.
+    - name: Generate source
+      run: dotnet build ./Yafc.I18n.Generator/
+
     # Execute all unit tests in the solution
     - name: Execute unit tests
       run: dotnet test

.github/workflows/buildcheck-release.yml

@@ -34,6 +34,11 @@ jobs:
     - name: Setup MSBuild.exe
       uses: microsoft/setup-msbuild@v2
 
+    # For reasons that are unclear, the build fails if the generated files do not exist at the beginning of the build.
+    # Explicitly build the generator to generate the files, then let the implicit build take care of everything else.
+    - name: Generate source
+      run: dotnet build ./Yafc.I18n.Generator/
+
     # Execute all unit tests in the solution
     - name: Execute unit tests
       run: dotnet test

.github/workflows/crowdin-translation-action.yml

@@ -0,0 +1,35 @@
+name: Crowdin Action
+
+on:
+  push:
+    branches: [ internationalization-test ]
+
+jobs:
+  synchronize-with-crowdin:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: crowdin action
+        uses: crowdin/github-action@v2
+        with:
+          upload_sources: true
+          upload_translations: false
+          download_translations: true
+          localization_branch_name: l10n_crowdin_translations
+          create_pull_request: true
+          pull_request_title: 'New Crowdin Translations'
+          pull_request_body: 'New Crowdin translations by [Crowdin GH Action](https://github.com/crowdin/github-action)'
+          pull_request_base_branch_name: 'internationalization-test'
+          skip_untranslated_strings: true
+        env:
+          # A classic GitHub Personal Access Token with the 'repo' scope selected (the user should have write access to the repository).
+          GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}
+
+          # A numeric ID, found at https://crowdin.com/project/<projectName>/tools/api
+          CROWDIN_PROJECT_ID: ${{ secrets.CROWDIN_PROJECT_ID }}
+
+          # Visit https://crowdin.com/settings#api-key to create this token
+          CROWDIN_PERSONAL_TOKEN: ${{ secrets.CROWDIN_PERSONAL_TOKEN }}

.gitignore

@@ -1,6 +1,7 @@
 ## Custom ignores that were not the part of the conventional file
 
 Build/
+*.g.cs
 
 # Debug launch configuration
 Y[Aa][Ff][Cc]/Properties/launchSettings.json

Docs/Architecture/Serialization.md

@@ -0,0 +1,152 @@
+# Serialization system
+The project serialization system is somewhat fragile.
+It adapts properly to many changes, but can also fail silently.
+To reduce the chances of silent failures, it has some [guardrails in the tests](#guarding-tests), to ensure any relevant changes are intentional and functional.
+
+The serialization system only deals with public instance properties.
+Broadly, there are two kinds of public properties, which we'll call 'writable' and 'non-writable'.
+A writable property has a public setter, a constructor parameter with the same name and type, or both.
+Any other public property is non-writable.
+`[Obsolete]` writable properties may (and usually should) omit the getter.
+
+All constructor parameters, except for the first parameter when derived from `ModelObject<>`, must have the same name and type as a writable property.
+
+## Property types
+The primary data structures that get serialized and deserialized are types derived from `ModelOject` and concrete (non-abstract) classes with the `[Serializable]` attribute.
+These are serialized property-by-property, recursively.
+Properties are handled as described here:
+
+|Property type|Writable|Non-writable|
+|-|-|-|
+|[`ModelObject` and derived types](#modelobjects-and-serializable-types)|Supported if concrete|Supported|
+|[`[Serializable]` classes](#modelobjects-and-serializable-types)|Supported if concrete|Ignored|
+|[`ReadOnlyCollection<>` and types that implement `ICollection<>` or `IDictionary<,>`](#collections)|Error|Supported if content is supported|
+|[`FactorioOject`, all derived types, and `IObjectWithQuality<>`s](#factorioobjects-and-iobjectwithqualitys)|Supported, even when abstract|Ignored|
+|[Native and native-like types](#native-types)|Supported if listed|Ignored|
+|Any type, if the property has `[SkipSerialization]`|Ignored|Ignored|
+|Other types|Error|Ignored|
+
+Notes:
+* The constructor must initialize serialized collections to a non-`null` value.
+The property may be declared to return any type that matches the _Property type_ column above.
+* Value types are only supported if they appear in the list of [supported native types](#native-types).
+* `[Obsolete]` properties must follow all the same rules, except that they do not need a getter if they are writable.
+
+### `ModelObject`s and `[Serializable]` types
+Each class should have exactly one public constructor.
+If the class has multiple public constructors, the serialization system will use the first, for whatever definition of "first" the compiler happens to use.\
+**Exception**: If the class has the `[DeserializeWithNonPublicConstructor]` attribute, it should have exactly one non-public constructor instead.
+
+The constructor may have any number of parameters, subject to the following limitations:
+* The first constructor parameter for a type derived (directly or indirectly) from `ModelObject<T>` does not follow the normal rules.
+It must be present, must be of type `T`, and can have any name.
+* Each (other) parameter must have the same type and name as one of the class's writable properties.
+Parameters may match either directly owned properties or properties inherited from a base class.
+* If the parameter has a default value, that value must be `default`.
+(Or an equivalent: e.g. `null` for reference types and `Nullable<>`, `0` for numeric types, and/or `new()` for value types without a explicit 0-parameter constructor.)
+
+Writable properties that are not one of the supported types must have the `[SkipSerialization]` attribute.
+
+Collection properties (always non-writable) must be initialized by the constructor to a non-`null` value.
+The constructor is not required to initialize non-writable `ModelObject` properties.
+If it does not, the serialization system will discard non-`null` values encountered in the project file.
+
+### Collections
+Collection values must be stored in non-writable properties, must not be passed to the constructor, and must be initialized to an empty collection.
+Unsupported keys or values will cause the property to be silently ignored.
+Arrays and other fixed-size collections (of supported values) will cause an error when deserializing, even though they implement `ICollection<>`.
+
+Keys (for dictionaries) must be: `FactorioObject` or derived from it, `IObjectWithQuality<>`, `string`, `Guid`, or `Type`.
+
+Values may be of any type that can be stored in a writable property.
+Explicitly, values are allowed to contain collections, but may not themselves be collections.
+
+The serializer has special magic allowing it to modify a `ReadOnlyCollection<>`.
+Initialize the `ReadOnlyCollection<>` to `new([])`, and the serializer will populate it correctly.
+
+### `FactorioObject`s and `IObjectWithQuality<>`s
+When deserialized, `FactorioObject`s are set to the corresponding object from `Database.allObjects`.
+If that object cannot be found (for example, if the mod defining it was removed), it will be set to `null`.
+Properties may return `FactorioObject` or any derived type, even if the type is abstract.
+
+`IObjectWithQuality<>`s are the same, except that the object is fetched by calling `ObjectWithQuality.Get`.
+
+### Native types
+The supported native and native-like types are `int`, `float`, `bool`, `ulong`, `string`, `Type`, `Guid`, `PageReference`, and `enum`s (if backed by `int`).
+The `Nullable<>` versions of these types are also supported, where applicable.
+
+Any value types not listed above, including `Tuple`, `ValueTuple`, and all custom `struct`s, cannot be serialized/deserialized.
+
+## Guarding tests
+There are some "unit" tests guarding the types that are processed by the serialization system.
+These tests inspect things the serialization system is likely to encounter in the wild, and ensure that they comply with the rules given under [Property types](#property-types).
+They also check for changes to the serialized data, to ensure that any changes are intentional.
+
+The failure messages should tell you exactly why the test is unhappy.
+In general, the tests failures have the following meanings:
+* If [`ModelObjects_AreSerializable`](../../Yafc.Model.Tests/Serialization/SerializationTypeValidation.cs) fails, it thinks you violated one of [the rules](#modelobjects-and-serializable-types) in a type derived from `ModelObject`.
+* If [`Serializables_AreSerializable`](../../Yafc.Model.Tests/Serialization/SerializationTypeValidation.cs) fails, it thinks you violated one of [the rules](#modelobjects-and-serializable-types) in a `[Serializable]` type.
+* If [`TreeHasNotChanged`](../../Yafc.Model.Tests/Serialization/SerializationTreeChangeDetection.cs) fails, you have added, changed, or removed serialized types or properties.
+  * Ensure the change was intentional and does not break serialization (e.g. changing between `List<>` and `ReadOnlyCollection<>`) or that you have handled the necessary conversions (e.g. for changing from `FactorioObject` to `List<FactorioObject>`).
+Then update the dictionary initializer to match your changes.
+If you made significant changes, you should be able to run `BuildPropertyTree` in the debugger to produce the initializer that the test expects.
+* If you intended to add a new property to be serialized, but `TreeHasNotChanged` does not fail, the new property probably fell into one of the ["Ignored" categories](#property-types).
+* If you intended to add a new `[Serializable]` type to be serialized, but `TreeHasNotChanged` does not fail, make sure there is a writable property of that type.
+
+Test failures are usually related to writable properties.
+Non-writable properties that return array types will also cause test failures.
+
+## Handling property type changes
+The simplest solution is probably introducing a new property and applying `[Obsolete]` to the old property.
+The json deserializer will continue reading the corresponding  values from the project file, if present,
+but the undo system and the json serializer will ignore the old property.
+You may (should?) remove the getter from an obsolete writable property.
+
+Sometimes obsoleting a property is not reasonable, as was the case for the changes from `FactorioObject` to `ObjectWithQuality<>`.
+Depending on the requirements, you can either implement `ICustomJsonDeserializer<T>` (as used by `ObjectWithQuality<T>` in [081e9c0f](https://github.com/shpaass/yafc-ce/tree/081e9c0f6b47e155fbc82763590a70d90a64c83c/Yafc.Model/Data/DataClasses.cs#L819) and earlier),
+or create a new `ValueSerializer<T>` (as seen in the current `QualityObjectSerializer<T>` implementation).
+
+## Adding new supported types
+Consider workarounds such as using `List<NestedList<T>>` instead of `List<List<T>>`, where
+```c#
+[Serializable]
+public sealed class NestedList<T> : IEnumerable<T> /* do not implement ICollection<> or IList<> */ {
+    public List<T> List { get; } = [];
+    public IEnumerator<T> GetEnumerator() => List.GetEnumerator();
+}
+```
+
+### Writable properties and collection values
+If the type is already part of Yafc, you may be able to support it simply by adding `[Serializable]` to the type in question.
+If that isn't adequate, implement a new class derived from `ValueSerializer<T>`.
+The new type should be generic if the serialized type is generic or a type hierarchy.
+Add checks for your newly supported type(s) in `IsValueSerializerSupported` and `CreateValueSerializer`.
+
+The tests should automatically update themselves for newly supported types in writable properties and collections.
+
+### Dictionary keys
+Find the corresponding `ValueSerializer<T>` implementation, or create a new one as described in the previous section.
+Add an override for `GetJsonProperty`, to convert the value to a string.
+If `ReadFromJson` does not support reading from a `JsonTokenType.PropertyName`, update it to do so, or override `ReadFromJsonProperty`.
+In either case, return the desired object by reading the property name.
+Add a check for the newly supported type in `IsKeySerializerSupported`.
+
+The tests should automatically update themselves for newly supported types in dictionary keys.
+
+### Non-writable properties
+To add support for a new type in a non-writable property, implement a new
+```c#
+internal sealed class NewReadOnlyPropertySerializer<TOwner, TPropertyType[, TOtherTypes]>(PropertyInfo property)
+    where TPropertyType : NewPropertyType[<TOtherTypes>] where TOwner : class
+    : PropertySerializer<TOwner, TPropertyType>(property, PropertyType.Normal, false)
+```
+Include `TOtherTypes` if the newly supported  type is generic.
+Serialize nested objects using `ValueSerializer<TOtherType>.Default`, to ensure that newly introduced `ValueSerializer<T>` implementations are supported by your collection.
+
+If your newly supported type is an interface, add a check for the interface type in `GetInterfaceSerializer`.
+
+If your newly supported type is a class, find the two calls to `GetInterfaceSerializer`.
+In the outer else block, add another check for your newly supported type.
+
+Changes to non-writable property support may require changes to `SerializationTypeValidation` in the vicinity of the calls to `AssertNoArraysOfSerializableValues`,
+and will require changes to `TreeHasNotChanged` in the vicinity of the `typeof(ReadOnlyCollection<>)`, `typeof(IDictionary<,>)` and/or `typeof(ICollection<>)` tests.

Docs/CodeStyle.md

@@ -74,22 +74,3 @@ Most of the operators like `.` `+` `&&` go to the next line.
 The notable operators that stay on the same line are `=>`, `=> {`, and `,`.
 
 The wrapping of arguments in constructors and method-definitions is up to you.
-
-# Quality Implementation
-Despite several attempts, the implementation of quality levels is unpleasantly twisted.
-Here are some guidelines to keep handling of quality levels from causing additional problems:
-* **Serialization**
-  * All serialized values (public properties of `ModelObject`s and `[Serializable]` classes) must be a suitable concrete type.
-  (That is, `ObjectWithQuality<T>`, not `IObjectWithQuality<T>`)
-* **Equality**
-  * Where `==` operators are expected/used, prefer the concrete type.
-  The `==` operator will silently revert to reference equality if both sides of are the interface type.
-  * On the other hand, the `==` operator will fail to compile if the two sides are distinct concrete types.
-  Use `.As<type>()` to convert one side to the interface type.
-  * Types that call `Object.Equals`, such as `Dictionary` and `HashSet`, will behave correctly on both the interface and concrete types.
-  The interface type may be more convenient for things like dictionary keys or hashset values.
-* **Conversion**
-  * There is a conversion from `(T?, Quality)` to `ObjectWithQuality<T>?`, where a null input will return a null result.
-  If the input is definitely not null, use the constructor instead.
-  C# prohibits conversions to or from interface types.
-  * The interface types are covariant; an `IObjectWithQuality<Item>` may be used as an `IObjectWithQuality<Goods>` in the same way that an `Item` may be used as a `Goods`.

Docs/LinuxOsxInstall.md

@@ -1,20 +1,20 @@
 ### Arch 
 There is an AUR package for yafc-ce: [`factorio-yafc-ce-git`](https://aur.archlinux.org/packages/factorio-yafc-ce-git) 
-Once the package is installed, it can be run with `factorio-yafc`. Note that at least dotnet 6 or later is required.
+Once the package is installed, it can be run with `factorio-yafc`. Note that dotnet runtime v8 is required.
 
 ### Debian-based
 - Download the latest Yafc-ce release.
-- [Install dotnet core (v8.0 or later)](https://learn.microsoft.com/en-us/dotnet/core/install/linux-debian)
+- [Install dotnet core (runtime version: v8)](https://learn.microsoft.com/en-us/dotnet/core/install/linux-debian)
 - Install SDL2:
   - `sudo apt-get install libsdl2-2.0-0`
   - `sudo apt-get install libsdl2-image-2.0-0`
   - `sudo apt-get install libsdl2-ttf-2.0-0`
   - For reference, have following libraries: SDL2-2.0.so.0, SDL2_ttf-2.0.so.0, SDL2_image-2.0.so.0
-- Make sure you have OpenGL available
-- Use the `Yafc` executable to run.
+- Make sure you have OpenGL available.
+- Use the `Yafc` executable to run (you might need to give it executable permissions: `chmod +x Yafc`).
 
 ### OSX
-- [Install dotnet core (v8.0 or later)](https://dotnet.microsoft.com/download)
+- [Install dotnet core (runtime version: v8)](https://dotnet.microsoft.com/download)
 - For Arm64 Macs, that's it. You can skip to the final step of launching Yafc.
 - For Intel Macs, you can skip to the step of getting SDL libraries with `brew`.
 - If you want to build Lua from source, here's how you can do that:

Docs/MoreLanguagesSupport.md

@@ -1,17 +1,21 @@
 # YAFC support for more languages
 
-YAFC language support is experimental. If your language is missing, that is probably because of one of two reasons:
+You can ask Yafc to display non-English text from the Welcome screen:
+- On the Welcome screen, click the language name (probably "English") next to "In-game objects language:"
+- Select your language from the drop-down that appears.
+- If your language uses non-European glyphs, it may appear at the bottom of the list.
+  - To use these languages, Yafc may need to do a one-time download of a suitable font.
+Click "Confirm" if Yafc asks permission to download a font.
+  - If you do not wish to have Yafc automatically download a suitable font, click "Select font" in the drop-down, and select a font file that supports your language.
 
-- It has less than 90% support in official Factorio translation
-- It uses non-European glyphs (such as Chinese or Japanese languages)
-
-You can enable support for your language using this method:
-- Navigate to `yafc.config` file located at `%localappdata%\YAFC` (`C:\Users\username\AppData\Local\YAFC`). Open it with the text editor.
-- Find `language` section and replace the value with your language code. Here are examples of language codes:
-    - Chinese (Simplified): `zh-CN`
+If your language is supported by Factorio but does not appear in the Welcome screen, you can manually force YAFC to use the strings for your language:
+- Navigate to `yafc2.config` file located at `%localappdata%\YAFC` (`C:\Users\username\AppData\Local\YAFC`). Open it with a text editor.
+- Find the `language` section and replace the value with your language code. Here are examples of language codes:
+	- Chinese (Simplified): `zh-CN`
 	- Chinese (Traditional): `zh-TW`
 	- Korean: `ko`
 	- Japanese: `ja`
 	- Hebrew: `he`
-	- Else: Look into `Factorio/data/base/locale` folder and find folder with your language.
-- If your language have non-European glyphs, you also need to replace fonts: `Yafc/Data/Roboto-Light.ttf` and `Roboto-Regular.ttf` with any fonts that support your language glyphs.
+	- Else: Look into `Factorio/data/base/locale` folder and find the folder with your language.
+- If your language uses non-European glyphs, you also need to replace the fonts `Yafc/Data/Roboto-Light.ttf` and `Roboto-Regular.ttf` with fonts that support your language.
+You may also use the "Select font" button in the language dropdown on the Welcome screen to change the font.