Fix catalog importer by LoloZarro · Pull Request #119 · lostb1t/Gelato

LoloZarro · 2026-02-28T02:33:12Z

Fix catalog import:

Collections not getting created or updated
Collection getting recreated instead of updated
Collection overwriting themself
No safeguards for catalogs having less item than configured and constantly looping over the same items
No checks for existing items in the collection, leading to the same items getting imported and the collection not progressing
Import will now only stop after having imported reaching the configured amount for actual new items added to it
Every catalog in the settings will get it's own collection (e.g. Netflix -> Netflix movies & Netflix series)
Minor changes

LoloZarro · 2026-02-28T02:35:02Z

Do not merge yet, still concluding my tests.

LoloZarro · 2026-02-28T19:27:23Z

@lostb1t
Test OK!

Test Results

Catalogs are correctly imported
Catalogs are correctly grouped into Collections
Catalog import will create a collection for each catalog
Catalog import will now always update the collection if the setting is enabled
Catalog import will now correctly expand the according collection instead of recreating it each run
Catalog import will now run until the configured amount of new items are saved
Catalog import will now detect if a catalog delivers the same items each time and abort the catalog import after 3 tries

j4ckgrey · 2026-02-28T19:43:22Z

Do not merge yet got me wondering 😂 is he gonna read your PR?

lostb1t · 2026-03-01T07:31:32Z

There's a reason collections items are cleared before inserting. To preserve the ordering of the catalog.

LoloZarro · 2026-03-01T12:52:28Z

@lostb1t
Changed, it will now always empty the collection and populate the items in order received from the catalog.

lostb1t · 2026-03-01T13:03:28Z

Any AI used?

LoloZarro · 2026-03-01T13:10:55Z

Only for research and boilerplate, every line was written by me.

LoloZarro · 2026-03-01T14:03:53Z

@lostb1t
I will now trigger a library scan after importing the catalogs. This should mitigate the issue, where the Collections Library will appear empty after modifying the collections.

I only did limited testing, but it seems to work so far.
If you want I can remove it and make a separate PR after merging this.

lostb1t · 2026-03-01T14:10:48Z

im a bit confused on what this Pr actually does different then the current code. Im aware if the empty collection folder but what other issues did you encounter that made you make this PR?

LoloZarro · 2026-03-01T14:30:40Z

There are several things mentioned in the first comment.

Collections not getting created or updated:
The previous code never updated the catalogs if the items exceeded max items for that catalog.
Example:
Catalog limit is 100 -> logic will fetch items -> gets 99 -> adds items to the library -> then fetches the next page -> gets 99 -> adds 1 item to the library then aborts without updating the catalog

Collection getting recreated instead of updated
The previous code never actually updated the collections, instead it recreated the collection with the set limit.

Example:
I set the limit to 10 -> Collection will not always contain 10 items (the 10 newest)

The collection never progressed which makes the global limit useless. Now it will only stop after importing the set amount of actual new items configured in the setting.

Example:
I set the limit to 10 -> Collection already contains 10 items -> it will now loop until it encounters 10 items not yet in the collection -> library will contain 20 items after the import.

No safeguards for catalogs having less item than configured and constantly looping over the same items
Previous code unnecessarily looped over the same items over and over again if it got less than the limit set in the setting.

Example:
Netflix Top 10 will always deliver 10 items, if I set the limit to 1000 it will loop 100 times over the same 10 items. Now it will detect stale results and abort.

Every catalog in the settings will get it's own collection (e.g. Netflix -> Netflix movies & Netflix series)
The previous code had a bug, where it would create the collections for the catalog, unfortunately it would overwrite itself if a catalog had the same name.

Example:
Netflix catalog has one for movies and series.
Before it would create a collection Netflix with the provider for series, import it then go to the next catalog which would be Netflix movies and would then update the collection Netflix and overwrite the provider with the movies.

Collection providers can only contain one value for the same key.

Now every catalog you see in the settings will get it's own collection.

There were some other things, but this should give you an overview of the key changes.

lostb1t · 2026-03-01T13:34:54Z

Plugin.cs

-                    cfg.MovieFolder = _manager.TryGetMovieFolder(cfg);
-                    cfg.SeriesFolder = _manager.TryGetSeriesFolder(cfg);
+
+                    cfg.MovieFolder = _manager.TryGetConfigFolder(cfg.MoviePath);


Whats the reason this has been moved?

Passing the whole Plugin config just to extract a string property seems excessive.

lostb1t · 2026-03-01T13:35:12Z

Services/CatalogService.cs

        GelatoPlugin.Instance.SaveConfiguration();
    }
-
-    public CatalogConfig? GetCatalogConfig(string id, string type)


Whats the reason this has been moved?

This just fetches something from the config and does not really belong in the service in my opinion. It can also be static and does not have any dependencies.

The helper can be called from anywhere without having to inject the CatalogService just for this.

lostb1t · 2026-03-01T13:35:59Z

Services/CatalogImportService.cs

 {
+    private const string ProviderKey = "Stremio";
+
+    // TODO: Add property for "FullName" on CatalogConfig Object and use that instead of manually creating it here.


Why not do this todo?

lostb1t · 2026-03-01T14:52:31Z

Services/CatalogImportService.cs

+            existingCollectionIds = currentCollectionItems.ToHashSet();
+        }
+
+        if (existingCollectionIds?.Count >= cfg.MaxCollectionItems)


Catalogs are dynamic, this prevents new items to shown up in collections.

If a collection reaches the max items configured in the settings, it should not be expanded further, that seems fine to me.

Granted, the check in the beginning should be removed or at least tied to the above block as it would currently block new items from even getting imported in the library.

Proposal:
I will remove the check in the beginning.
When updating the collection I will check if the max size has been reached and then only update the collection with the newest items up to the max configured collection size.

Would that work for you?

but that results in the exact same logic as there is now? Currently it purges all items in a collection and reinsterts.

Are you maybe talking about MaxItems instead of MaxCollectionItems?

The MaxItems setting on the Catalogs themself are now being used to set the amount of new items (not currently present in the collection), while the MaxCollectionItems are used to limit how many items the collection will include in total (it's the upper cap).

The current logic does not use the MaxCollection items.

The MaxItems currently limits the size of the Collection instead of using the MaxCollectionItems.

If you set the MaxItems setting on the Catalog in the settings the resulting size will always be that value. If you set it to 50 and run the import twice the collection will contain 50 items.

With this change if you set it to 50 and run it twice it will contain 100.

I changed it as proposed. The check is still there and will generate a warning, but it will continue with adding new items to the library and will update the collection with the newest amount of items configured in MaxCollectionItems.

So even if the max size has been reached the collection will be updated and contain newest items up to the configured cap.

lostb1t · 2026-03-01T15:05:06Z

Services/CatalogImportService.cs

+
+                        allSeenLibraryCatalogItems.Add(item);
+
+                        if (existingCollectionIds?.Contains(item.Id) ?? true)


existing checks arent needed as its cleared

See comment above

lostb1t · 2026-03-01T15:05:41Z

Services/CatalogImportService.cs

+            var currentChildren = collection.GetLinkedChildren().Select(i => i.Id).ToArray();
+            var currentChildrenSet = new HashSet<Guid>(currentChildren);
+
+            var newItems = allRetrievedCatalogItems


existing checks arent needed as its cleared

See comment above

- Collections not getting created or updated - Collection getting recreated instead of updated - Collection overwriting themself - No safeguards for catalogs having less item than configured and constantly looping over the same items - No checks for existing items in the collection, leading to the same items getting imported and the collection not progressing - Import will now only stop after having imported reaching the configured amount for actual new items added to it - Every catalog in the settings will get it's own collection (e.g. Netflix -> Netflix movies & Netflix series) - Minor changes

LoloZarro force-pushed the catalog_importer_fix branch from e18bcb6 to 6df8eb4 Compare February 28, 2026 22:42

LoloZarro force-pushed the catalog_importer_fix branch from 6df8eb4 to ca36a68 Compare March 1, 2026 12:51

LoloZarro force-pushed the catalog_importer_fix branch from ca36a68 to df77f62 Compare March 1, 2026 14:01

lostb1t requested changes Mar 1, 2026

View reviewed changes

LoloZarro force-pushed the catalog_importer_fix branch from df77f62 to 609c424 Compare March 1, 2026 20:18


		allSeenLibraryCatalogItems.Add(item);

		if (existingCollectionIds?.Contains(item.Id) ?? true)

Uh oh!

Conversation

LoloZarro commented Feb 28, 2026

Uh oh!

LoloZarro commented Feb 28, 2026

Uh oh!

LoloZarro commented Feb 28, 2026

Uh oh!

j4ckgrey commented Feb 28, 2026

Uh oh!

lostb1t commented Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LoloZarro commented Mar 1, 2026

Uh oh!

lostb1t commented Mar 1, 2026

Uh oh!

LoloZarro commented Mar 1, 2026

Uh oh!

LoloZarro commented Mar 1, 2026

Uh oh!

lostb1t commented Mar 1, 2026

Uh oh!

LoloZarro commented Mar 1, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LoloZarro Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lostb1t commented Mar 1, 2026 •

edited

Loading

LoloZarro Mar 1, 2026 •

edited

Loading