How to control object destruction order with Micropython GC?

[xyzzy42]

I created an new micropython object, which wraps a resource provided by a library. This resource must be allocated and freed. I use m_new_obj_with_finaliser() to cause the delete method to be called when the object is GCed, to free this resource.

Suppose I create a new object of this type, called A, and then another object is created, B, and object B uses object A. The only reference to A is contained by object B.

Now B becomes unreferenced.

My understanding is the GC will now discover that neither A nor B can be found by following any referenced objects. And both can be garbage collected, in any order. The problem is that B still uses A. Garbage collecting A first causes a problem when B still exists and is still using it.

[peterhinch]

I'm puzzled over why you should care about an unreferenced object. How would you even know it was unhappy?

In any event, my understanding of the GC is that while A is referenced by B, it won't be collected. B will be collected first. This causes A to be unreferenced and it will be collected.

[dlech]

I think it would be more accurate to say "unreachable" instead of "unreferenced". If A has a pointer to B and nothing else has a pointer to A or B, then both are "unreachable" and will be garbage collected.

I think a way to solve the ordering problem is to call a B.close() method in A's finalizer before freeing A that tears down whatever is needed so that B is no longer using A. This way if A's finalizer runs first, then B still won't be using it by the time the finalizer finishes.

[trentp-igor]

This requires that A know about B, which unfortunately isn't the case.

[peterhinch]

What type of resource is at issue here? If it's a memory region such as an array, the GC should collect it once it becomes unreachable.

If it's an object like a socket it must explicitly be instantiated and taken down. If the resource is set up by an instance of A, and is bound to that instance, the usual way of ensuring it is taken down is to design A so that it supports use in a context manager. When class B wants to use class A, it does something like

with A() as a_instance:
    # Use the instance. A.__exit__() takes down the resource

If the resource is owned by the A class rather than by the A instance, the approach is similar with A maintaining a class variable instance counter (taking the shared resource down when the counter reaches zero).

[trentp-igor]

I've made an object for external flash chips. But the object for the flash chip is garbage collected when it has a mounted filesystem still on it, if the filesystem and the flash object both become unreferenced.

I'm familiar with context managers, but that doesn't work here since the use is done by the vfs functions.

There's also plenty of times a context manager can't be used, such as an object that should be created in a class method and persist in the objects' attributes across method calls.

[zcattacz]

Sometimes I have a class's __init__ register the instance to a attribute associated with the class and removed on __del__, to keep track and spot lingering instances for resource statistics. If the top level class is not GCed, the instance shouldn't be GCed either.

[peterhinch]

@trentp-igor The filesystem won't become unreferenced. Consider this:

def setup():
    flash = get_device()  # Return a reference to a Flash object
    os.mount(flash, "/fl_ext")

On exit from the function, the local variable flash goes out of scope. But it won't be gc'd because there is still a reference to it in os.mount(). The reference will only be lost when os.umount() is called. Only then can the Flash object be gc'd.

Re objects owned by the class:

class Foo:
    @classmethod
    def initiate(cls):
        cls.resource = get_device()  # Return a reference to some object

Instances can access the resource via Foo.resource. This class variable won't go out of scope as it's owned by the class, not by any instance. The object will remain reachable and will not be gc'd.

If you want to deallocate the device you will need to take control. As I suggested above, one way is to create instances in a context manager,
designing the Foo class to maintain an instance count in a class variable. If the count goes to zero, set cls.resource=None. At this point the device will become unreachable and can be gc'd.

As a general point it is quite difficult to make MicroPython leak memory. I can think of three techniques, only one of which is remotely common. None relate to object destruction order.

I am wary of the use of del. In CPython there is no guarantee that it will be called before the script returns to the REPL. Given that most firmware runs forever I have my doubts that its use is a sound programming technique. I have never used it. However this is not gospel and I'm willing to be proved wrong.

[trentp-igor]

The flash object becomes unreachable when Micropython is soft-rebooted. At this point the flash device and any open files will be GCed. mpy doesn't just reset, it GCs existing objects. However, since all are equally unreachable, there is no defined order in which the GC is done. This can result in the flash device being deleted before any open files on the file system, which then try to write to it in their close functions.

There's no memory leak. The issue is garbage collected objects being used after being garbage collected.

[peterhinch]

A soft reboot is a potential disaster if any filesystem is open. It is equivalent to a hardware reset: all RAM is initialised, all context is lost. Python files are re-compiled on import.