Memory leak on file import.

[globalshrug]

Hydrus version

v652

Qt major version

Qt 6

Operating system

Other (aka none of the above, specify in the comments)

Install method

Running from source

Install and OS comments

Tahoe 26.2

Bug description and reproduction

I'm not sure if this is new, but:

I am importing a large batch of files (which I don't usually do - that's why I don't know if this is new): on the order of ~38k or so. I opted for advanced import options so I could apply tags upon import.

First thing I noticed (and this is something I have noticed in previous versions actually), is that when applying a tag based on the "last directory" option, it starts "filling in" as you type which introduces lag. I have to type quickly to get the entire word/tag in before it tries to apply it, if there is a large import. At that point I have to wait with a spinning wheel until it applies the tag to all files - this takes a weirdly long time, on the order of > 2 min in some cases on a modern CPU. I had an OS-level dialog come up asking me to close apps as not everything could be held in memory.

Once my system was responsive again, I opened Activity Monitor and applied a tag in "tags for all" and watched it happen again (I wanted to make sure I could reproduce it).

At this point, with my 35k files example, I watched memory usage spike up to 82GB. I don't know how the 82GB number is being calculated - compressed or uncompressed, how much is shared etc. At the very least I guess it's being swapped as this machine only has 16GB built in (I know, I know..)

Once it hits its peak, which in this case was 82GB, after sitting there for ~30s or so, it went back down to 50GB and then deflated from there back to ~6.7GB which seems normal/sane.

Happy to reproduce and provide logs or debug output if need be, just let me know what I need to do on my end.

[hydrusnetwork]

Thank you for this report. I am very sorry for the trouble--82GB is obviously nuts!

Unfortunately, I am really not sure what caused this. We've seen some memory explosions in some clients often related to PTR processing, duplicate work, or file import proper (i.e. the actual stage where file imports happen and thumbs are streaming in) but it can be tricky to pin down. I recently fixed a big long-term one with the asynchronouse job system not cleaning up after itself sometimes. I can't figure out what would cause it from this file tagging dialog.

I don't have 38k files to hand, but my dev machine here has 8k example files. If I throw them all in at once and do what you did, I'm afraid I do not get any lag as it updates. Even if I scroll through my whole 8k items with some filename and directory and 'tags for all/selected' stuff set up, it is nice and smooth, and I don't see the memory use bounce around more than a couple of MB. There was a time when any update to the tags you put in would cause the whole list to be updated, but these days it only needs to refresh the like 30 rows that are currently in view. There's no 'global' refresh where it, say, would need to do 38k hard drive hits or something. Sidecars need a hard drive hit, but I'm confident they do the same efficient "only for rows in view" regen tech, so it isn't too bad, and it doesn't sound like you have complex sidecars set up. The 'metadata' sort data is faked to be the same as the '#' column so we don't need a tag/sidecar hit for sort, either. The final 'apply' can take a while, when it freezes everything for everything, but usually not more than a second or two.

In the dialog before that, where hydrus does an early parse to figure out filetype, that can be CPU/memory heavy if you hit a ton of difficult videos or something and there's 15k+ calls to the ffmpeg exe, but if it was that hitting you, you'd have seen the memory spike before the tagging dialog and then drift down. You are seeing it happen in the lag period.

So I can't explain your symptoms except to wonder if there was another thing going on in your client at the same time and it was that, or the combination of that and the filename parsing, that was blowing you up. Maybe your 16GB was already just about full up and hydrus was suddenly doing work on a really slow swap disk and it spiralled out of control? 6.7GB feels like a pretty chunky client already with a heavy session.

So, some ideas:

• Do you sync with the PTR?
• If you do use the PTR, or if you otherwise have many siblings/parents, what is your sync status under tags->sibling/parent sync->review? Everything all good, or does something have lots of work still to do?
• When you did this, were there any large import queues happening in the background? 12kx14x png files can be a memory killer on import.
• What's your 'session weight' under the pages menu?
• Do you regularly get multi-second lag in hydrus, or was this event quite unusual?

I have a memory profiler in the hydrus debug menu, but it is truly awful atm. I'm hoping to put some time aside in 2026 to overhaul it, along with the CPU profiler, so we can capture better data when these things happen.

Although it sucks as an answer for now, since you are a little ram limited, it might be healthy to slim down your session if possible, and if you have more big imports to do, try cutting them into batches of 10k files, or set up a test run with 100 files and then create an 'import folder' with the same rules to handle the rest. We've had a couple of users who wanted to import ~250k files at once, and an import folder worked well rather than pushing it all through the UI and a single giganto import page.

[globalshrug]

There was a time when any update to the tags you put in would cause the whole list to be updated, but these days it only needs to refresh the like 30 rows that are currently in view. [...] The final 'apply' can take a while, when it freezes everything for everything, but usually not more than a second or two.

Hmm, this is exactly what it 'feels' like is happening.. the entire list being updated one at a time. The final apply definitely takes a while, IIRC it was easily > 10s if not 30s or more for the entire 38k batch.

If we had a feature to 'delete file and record' (not on disk but in db), I could test this out more thoroughly. I've actually wanted that feature before but never asked for it -- but especially in a case like this where you want to test a specific circumstance and importing a batch of files multiple times as though it's the first time may be helpful.

So I can't explain your symptoms except to wonder if there was another thing going on in your client at the same time and it was that, or the combination of that and the filename parsing, that was blowing you up. Maybe your 16GB was already just about full up and hydrus was suddenly doing work on a really slow swap disk and it spiralled out of control? 6.7GB feels like a pretty chunky client already with a heavy session.

Right.. I don't think much else was going on though.

• Do you sync with the PTR?

Nope.

• When you did this, were there any large import queues happening in the background? 12kx14x png files can be a memory killer on import.

No, I just had the one. I have subscriptions but they are very light and IIRC none were running at the same time.

• What's your 'session weight' under the pages menu?

~ 1.2 million.

• Do you regularly get multi-second lag in hydrus, or was this event quite unusual?

It's pretty responsive overall, there are some lag spikes here and there but this was exceptional.

I have a memory profiler in the hydrus debug menu, but it is truly awful atm. I'm hoping to put some time aside in 2026 to overhaul it, along with the CPU profiler, so we can capture better data when these things happen.

Looking forward to that!

Although it sucks as an answer for now, since you are a little ram limited, it might be healthy to slim down your session if possible, and if you have more big imports to do, try cutting them into batches of 10k files, or set up a test run with 100 files and then create an 'import folder' with the same rules to handle the rest. We've had a couple of users who wanted to import ~250k files at once, and an import folder worked well rather than pushing it all through the UI and a single giganto import page.

Yeah. I don't mind that it takes a while, I've just come to expect that with very large batches. I have noticed that it's very much better under about 16-18k files, then performance degrades rapidly. But that RAM spike affected other things that were running and seemed excessive and unusual so I thought I'd report it.

[globalshrug]

I accidentally re-created this so I was able to make better notes of the conditions at play.

It happened while importing about 3k files. I actually paused that import to see if having the import active or paused would change anything, and it didn't. I left it paused while doing the following. (Edit: and, I'm not sure that doing this while already importing has any bearing on this, I'd be willing to bet not.)

As a baseline, python/hydrus was sitting at about 2.6GB, with a page weight of 1.2M and with minimal activity going on. Some watchers, and the paused import, and very little/almost nothing else active at the time I did this.

I dragged 4 folders into Hydrus totalling 21,740 files, checked the 'delete originals' button, clicked on add tags/urls, typed into the "second last directory" field and watched as python/hydrus locked up. Memory spiked up to 27.6GB, plateaued, and eventually came back down. Once responsive again I entered another tag and watched almost exactly the same thing happen again. Each time it locks up for ~50s while nothing changes at all, then memory usage comes back down and the tag is reflected as being applied to all of the files, and memory usage returns back to high (~7GB) and then within 5-10s, back down to baseline.