Prevent Recursion Errors with django_q2 Tasks

[jacobfelknor]

It appears that using the inventree python library with functions that require pickling - like django_q2 background jobs - can fail in some cases. I'll admit that it's unclear to me exactly what conditions lead to it failing, because thusfar I've only noticed it on this ProjectCode case but have definitely used it elsewhere with no issues.

The following is a simplified version of some code I have in a different Django site of mine.

from inventree.project_code import ProjectCode
from my_site.models import MyOtherSiteProjectCode

@receiver(post_save, sender=MyOtherSiteProjectCode)
def queue_create_inventree_project_code(sender, instance: MyOtherSiteProjectCode, created: bool, **kwargs):
    if created:
        # push off to background worker
        async_task(create_inventree_project_code, instance.display, instance.description)

def create_inventree_project_code(code: str, description: str) -> ProjectCode:
    # NOTE: simplified for demonstration purposes
    inventree_api = InvenTreeAPI(INVENTREE_BASE_URL, username=USERNAME, password=PASSWORD)
    project_code = ProjectCode.create(inventree_api, {"code": code, "description": description})

Traceback is provided below

16:36:07 [Q] INFO Process-ae64a4e9c9174446868cea1ddc1c51a9 processing queen-utah-video-neptune 'tasks.create_inventree_project_code'
Process Process-0281948da1c2493c9ff0002eb56580fd:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/workdir/.venv/lib/python3.11/site-packages/django_q/monitor.py", line 44, in monitor
    for task in iter(result_queue.get, "STOP"):
  File "/home/workdir/.venv/lib/python3.11/site-packages/django_q/queues.py", line 73, in get
    x = super(Queue, self).get(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/multiprocessing/queues.py", line 122, in get
    return _ForkingPickler.loads(res)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/workdir/.venv/lib/python3.11/site-packages/inventree/base.py", line 350, in __getattr__
    if name in self._data.keys():
               ^^^^^^^^^^
  File "/home/workdir/.venv/lib/python3.11/site-packages/inventree/base.py", line 350, in __getattr__
    if name in self._data.keys():
               ^^^^^^^^^^
  File "/home/workdir/.venv/lib/python3.11/site-packages/inventree/base.py", line 350, in __getattr__
    if name in self._data.keys():
               ^^^^^^^^^^
  [Previous line repeated 963 more times]
RecursionError: maximum recursion depth exceeded

Which points to this block in base.py

class InventreeObject(object):
    """ Base class for an InvenTree object """
    
    ...
    
    def __getattr__(self, name):

        if name in self._data.keys():
            return self._data[name]
        else:
            return super().__getattribute__(name)
    ...

Specifically, the call to self._data appears to be problematic during pickling (according to the _ForkingPickler.loads(res) line in traceback). Pickling is not a process I fully understand, but it seems like there is an intermediate state where the object is not fully "initialized." Asking for the ._data property, which doesn't exist yet in this "uninitialized state", attempts a lookup of name in self._data.keys(), which self references and causes the infinite recursion.

Additionally, I believe the call to super().__getattribute__(name) is unnecessary. I see the intent here.... it's to fall back on the normal attributes when the attribute we ask for is not in the _data dictionary. However, python's behavior means that we never even reach the __getattr__ function call if the "normal" attribute already exists.

I therefore suggest we update this __getattr__ function to instead be

def __getattr__(self, name):
    try:
        data = object.__getattribute__(self, "_data")
    except AttributeError:
        # Appears to happen during pickling. Raise immediately to prevent recursion errors
        raise AttributeError(name)

    if name in data:
        return data[name]

    # if we're in this block, there already wasn't a "normal" attribute with this name. Raise
    raise AttributeError(name)

To prove this works, here is a minimal example:

class Demo:
    def __init__(self):
        self._data = {"x": 33}
        self.y = "normal attribute"

    def __getattr__(self, name):
        try:
            data = object.__getattribute__(self, "_data")
        except AttributeError:
            # Appears to happen during pickling. Raise immediately to prevent recursion errors
            raise AttributeError(name)

        if name in data:
            return data[name]

        # if we're in this block, there already wasn't a "normal" attribute with this name. Raise
        raise AttributeError(name)
    
d = Demo()

print(d.x)   # comes from _data
print(d.y)   # normal attribute

[SchrodingersGat]

@jacobfelknor thanks for the fix :)