garbage collector/memory management possibly flawed

[GitHubsSilverBullet]

Recent discussions about memory allocation led to a small test-program I'd like to offer for discussion. It seems as if the garbage collector behaves a bit strange and unpredictable at times. In particular, bytearrays not freed after ›del‹. Sometimes only freed after leaving a function. Sometimes not freed at all. (Only when really forced)

May it is behaving as intended, that's why I'd like to open it for discussion.

Here's the code:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import gc

def show_mem(n=0):
    gc.collect()
    print(f'({n:02}) {round(gc.mem_alloc()/1000, 1)} kB')

def works_as_expected1():
    show_mem(10)
    ba = bytearray(ALLOC_SIZE)
    show_mem(11)
    del ba
    show_mem(12)

def works_as_expected2():
    show_mem(20)
    ba = bytearray(ALLOC_SIZE)
    show_mem(21)
    ll = len(ba)
    print(ll)
    del ba
    show_mem(22)

def does_not_works_as_expected():
    show_mem(30)
    ba = bytearray(ALLOC_SIZE)
    show_mem(31)
    print(len(ba))  # somehow holds the ›ba‹ object
    del ba          # can't delete
    show_mem(32)    # ›ba‹ object still locked

ALLOC_SIZE = 150_000

works_as_expected1()
show_mem(13)
print()

works_as_expected2()
show_mem(23)
print()

# this is bad
does_not_works_as_expected()
show_mem(33) # after leaving the function, ›ba‹ is freed
print()

# this is OK
show_mem(40)
ba = bytearray(ALLOC_SIZE)
show_mem(41)
del ba
show_mem(42)
print()

# this is truly ugly
show_mem(50)
ba = bytearray(ALLOC_SIZE)
show_mem(51)
print(len(ba))  # now ba is REALLY locked
del ba          # and one can't free the object
show_mem(52)    # 100kB lost forever…
print()

try:  # catch exception, because this WILL throw an error
    ba2 = bytearray(ALLOC_SIZE) # should work
except MemoryError as me:
    print(me) # but doesn't

n = gc.threshold()
gc.threshold(0) # … unless we REALLY force a full gc
show_mem(60)
gc.threshold(n)

ba2 = bytearray(ALLOC_SIZE) # now it will work

# picorun "memory.py"
# (10) 5.0 kB
# (11) 105.0 kB
# (12) 5.0 kB
# (13) 5.0 kB
#
# (20) 5.0 kB
# (21) 105.0 kB
# 100000
# (22) 5.0 kB
# (23) 5.0 kB
#
# (30) 5.0 kB
# (31) 105.0 kB
# 100000
# (32) 105.0 kB
# (33) 5.0 kB
#
# (40) 5.0 kB
# (41) 105.1 kB
# (42) 5.0 kB
#
# (50) 5.0 kB
# (51) 105.1 kB
# 100000
# (52) 105.1 kB
#
# (60) 5.1 kB

[kjm1102]

In my week in the wilderness I found manipulating 20k files was really all about contiguous memory. In this regard import micropython; micropython.mem_info(1) helped me greatly compared to the garbage collector.

Jimmo insists the gc does eventually get around to releasing blocks https://github.com/orgs/micropython/discussions/14090 but, like you, I've found it to be a bit unpredictable when it comes to deletions.

[stinos]

Just a note but some of the comments are not actually correct.

now ba is REALLY locked

That's not how it works: that Python statement does not actually keep any reference to ba. As to why it affects GC, I'd say coincidence i.e. you might even be able to come up with another statement not even using ba which has the same effect. Would have to debug the GC (can enable this in modgc.c) to see what is actually going on, I'd suspect ba is still in the C stack somewhere.

… unless we REALLY force a full gc

That's not what gc.threshold does: it sets the threshold for when to call gc.collect when there is an allocation. It does not affect manual calls to gc.collect directly. But again: it might seem to do something in your case, for instance because sprinkling extra statements in between pushes something of the C stack after which GC will see it as free and ok to collect.

[GitHubsSilverBullet]

@stinos Please run the code and verify what's happening.
It is doing EXACTLY what is written in the comments. And a print statement shouldn't hold/lock ba, so that it's not possible to del/free ba within the function.
One can work around all this, of course, but that's not really the point.
And a simple

gc.threshold(0)
gc.threshold(-1)

does free ›ba‹ up. Even without calling ›gc.collect()‹

Edit: Here's a simpler one:

#!/usr/bin/python3
# -*- coding: UTF-8 -*-
# vim:fileencoding=UTF-8:ts=4

import gc

def show_mem(n=0, force=False):
    if force:
        gc.threshold(0) # force a full gc
        gc.threshold(-1)
    else:
        gc.collect()
    print(f'({n:02}) {round(gc.mem_alloc()/1000, 1)} kB')

ALLOC_SIZE = 150_000
FORCE_ERROR = False
# FORCE_ERROR = True  # Will throw an exception

show_mem(10)
ba = bytearray(ALLOC_SIZE)
show_mem(11)
ba[-50000:] = b''
show_mem(12, True)

if FORCE_ERROR:
    print(len(ba))

show_mem(13)
del ba
show_mem(14)
try:
    ba = bytearray(ALLOC_SIZE)
except MemoryError as me:
    print(me)
show_mem(15)

[stinos]

It is doing EXACTLY what is written in the comments.

It's doing something, my point is that the explanation you give to it is only a description of what you think it does, not what actually happens. For example:

And a simple
gc.threshold(0)
gc.threshold(-1)
does free ›ba‹ up. Even without calling ›gc.collect()‹

Here's the C code of the threshold function:

mp_obj_t gc_threshold(size_t n_args, const mp_obj_t *args) {
    if (n_args == 0) {
        if (MP_STATE_MEM(gc_alloc_threshold) == (size_t)-1) {
            return MP_OBJ_NEW_SMALL_INT(-1);
        }
        return mp_obj_new_int(MP_STATE_MEM(gc_alloc_threshold) * MICROPY_BYTES_PER_GC_BLOCK);
    }
    mp_int_t val = mp_obj_get_int(args[0]);
    if (val < 0) {
        MP_STATE_MEM(gc_alloc_threshold) = (size_t)-1;
    } else {
        MP_STATE_MEM(gc_alloc_threshold) = val / MICROPY_BYTES_PER_GC_BLOCK;
    }
    return mp_const_none;
}

it does not call collect(), it only sets the gc_alloc_threshold variable. However every single allocation might cause collect() to be called so in your case I'm guessing one of the other statements leads to an allocation, in turn leading to collect.

[GitHubsSilverBullet]

The explanation I'm giving is, what a Python programmer sees when writing and running a Python program. To a regular Python programmer everything else (underlying C++, C or Assembler code) is completely opaque. Not more, not less.
To a regular Python programmer it looks and feels and behaves like an object isn't freed.
And it looks and feels and behaves as if the two threshold function calls solve a problem that shouldn't be there in the first place.
The underlying intricacies are for the description of a problem of no meaning whatsoever.

[stinos]

I'm well aware, but as far as a Python programmer is concerned, del only removes the name of an object and it is wrong to equate that to 'can be garbage collected', because as far as Python is concerned I don't think there's a hard rule in the standard which says it should. Nor is there in MicroPython.

So the GC is probably correct. Maybe not what you expected, but to explain why such expectation is wrong, well, you'd need to be aware of those underlying intricacies so that's full circle again :)

[dhalbert]

Python (the language) makes no guarantees about when garbage collection happens. CPython uses reference counting for most garbage collection. MicroPython uses mark and sweep. So CPython can free an object immediately after its reference count to zero. MicroPython will only free it when the mark and sweep runs. To get it to run, use gc.collect().

In addition MicroPython's gc is conservative, because it is not necessarily always able to tell whether something on the stack is a pointer or binary data. So some objects may go uncollected if there happens to be a binary data that looks like an object pointer.

[GitHubsSilverBullet]

It's a result of the creator's intention to create a mark&sweep style garbage collector which traces C stack and registers to decide when something is free, yes.

So your point is: No matter what one does, be it correct or not, that memory is gone and can't be reclaimed with ›gc.collect()‹ until a hard reset occurs and one starts all over again?

Except when one toys with the threshold() back and forth, then all of a sudden it does works?

Sorry, that's not convincing at all.

[stinos]

So your point is: No matter what one does, be it correct or not, that memory is gone and can't be reclaimed with ›gc.collect()‹ until a hard reset occurs and one starts all over again?

No, not at all. The point is it will be reclaimed when MicroPython sees no references to the memory it anymore. Which can be right after del. Or a couple of statements later. Depends.

Except when one toys with the threshold() back and forth, then all of a sudden it does works?

Again: depends. As mentioned earlier: could be conincidence. Could be that toying around with other statements (as in: I'm pretty sure that if you start adding random other statements instead of gc.threshold, you'll also find that that is enough tfor MicroPython to suddenly allow collecting the memory) .

[dhalbert]

The object pointer to the bytearray is probably lying around on the stack and hasn't been overwritten yet. As @stinos mentions, eventually it will get overwritten and the stale reference will disappear.

[GitHubsSilverBullet]

I'm pretty sure that if you start adding random other statements instead of gc.threshold, you'll also find that that is enough tfor MicroPython to suddenly allow collecting the memory .

Quite possible. Unfortunately one doesn't write programs to just include some ›random statements‹ in order to get the real program to do its real business. But I get your drift, it's rather a journey, an adventure, a funny trial and error thing, until a Micropython program runs as intended. ;-)

[Josverl]

Such is the charm of not needing to do your own memory management. One only retains the illusion of control

[GitHubsSilverBullet]

One more attempt to make the point:

#!/usr/bin/python3
# -*- coding: UTF-8 -*-
# vim:fileencoding=UTF-8:ts=4
import gc

# comment out the two lines and it'll never work!
# leaving the dummy_func() in, but not even calling it,
# it'll throws an exception during first attempt to allocate the bytearray,
# but then works on the next attempt.
def dummy_func():
    return

ALLOC_SIZE = 150_000
ba = bytearray(ALLOC_SIZE)
print(len(ba))
del ba
attempt = 0
while True:
    try:
        attempt = int(attempt) + 1
        gc.collect()
        ba = bytearray(ALLOC_SIZE)
        break
    except MemoryError as me:
        print(attempt, me)
print('success')

# With the dummy_func() commented out:
# ====================================
# ~/tmp$ pyboard.py memory2.py 
# 150000
# 1 memory allocation failed, allocating 150000 bytes
# 2 memory allocation failed, allocating 150000 bytes
# 3 memory allocation failed, allocating 150000 bytes
# …
# …
# 999 memory allocation failed, allocating 150000 bytes

# With the dummy_func() still in:
# ===============================
# ~/tmp$ pyboard.py memory2.py 
# 150000
# 1 memory allocation failed, allocating 150000 bytes
# success

[projectgus]

@GitHubsSilverBullet That is indeed a bit odd to see behaviour like this one. I can't immediately explain it, but memory fragmentation can have odd consequences when you're allocating large blocks of contiguous RAM relative to the total RAM.

Can you please let us know:

• Micropython banner output (can get this by opening a REPL and typing Ctrl-D if necessary). This has the exact version and the board you're using.
• As someone hinted above, if you put micropython.mem_info(1) near the top and each time around the loop, what's the full output? This lets you see the exact layout of the heap.

[GitHubsSilverBullet]

@projectgus I really thought that I'd shown the version. Sorry for that, must have been another thread.
It's: MicroPython v1.22.1 on 2024-01-05; Raspberry Pi Pico with RP2040

Here's a new piece of code that's even stranger (observe the magic value).
Using : pyboard.py memory3.py | uniq -f1 to filter duplicate lines.
Albeit, at this time the entire (memory) system is so non-deterministic and unpredictable, I'll simply stop this exercise.
What I learned so far is, one has to knead and mangle the code in many and often funny ways until its execution finally leans towards what the developer had in mind. Probably every time a new firmware arises.
Thanks for the help anyways and without further ado:

#!/usr/bin/python3
# -*- coding: UTF-8 -*-
# vim:fileencoding=UTF-8:ts=4
#
# ~/tmp$ pyboard.py memory3.py | uniq -f1
#
# MicroPython v1.22.1 on 2024-01-05; Raspberry Pi Pico with RP2040

from micropython import mem_info as MI
import gc

# change the magic value to 336 and it crashes at line 23
# change the magic value to 337 and it works
# needless to say that there's plenty of memory (>223KiB) available.

magic = bytearray(336)
ba = bytearray(200_000)
del ba
MI(1)
gc.threshold(0)
gc.threshold(-1)
MI(1)
ba = bytearray(200_000)

#   ~/tmp$ pyboard.py memory3.py | uniq -f1
#
#   stack: 500 out of 7936
#   GC: total: 233472, used: 205200, free: 28272
#    No. of 1-blocks: 50, 2-blocks: 11, max blk sz: 12500, max free sz: 1755
#   GC memory layout; from 20007000:
#   00000000: h=MLhhhBDhhBTTBDhTBDBDBh===BTB=hh====B=BBBBBTB=BTB=BBBTB=TBTB=Bh
#   00000400: ===DB=h===========h===================BBB...h=Bh========h=======
#   00000800: ==========h=====================================================
#   00001000: ==========h=hTAAhh===...h====......h=====h=h====================
#   00001400: h=======h========h==============================================
#   00001800: ================================================================
#   00032000: =====================================...........................
#          (27 lines all free)
#   stack: 500 out of 7936
#   GC: total: 233472, used: 205200, free: 28272
#    No. of 1-blocks: 50, 2-blocks: 11, max blk sz: 12500, max free sz: 1755
#   GC memory layout; from 20007000:
#   00000000: h=MLhhhBDhhBTTBDhTBDBDBh===BTB=hh====B=BBBBBTB=BTB=BBBTB=TBTB=Bh
#   00000400: ===DB=h===========h===================BBB...h=Bh========h=======
#   00000800: ==========h=====================================================
#   00001000: ==========h=hTAAhh===...h====......h=====h=h====================
#   00001400: h=======h========h==============================================
#   00001800: ================================================================
#   00032000: =====================================...........................
#          (27 lines all free)
#   Traceback (most recent call last):
#     File "<stdin>", line 23, in <module>
#   MemoryError: memory allocation failed, allocating 200000 bytes
#   ~/tmp$

[projectgus]

@GitHubsSilverBullet Thanks, you found a weird memory leak bug! Due to a dangling pointer in the compile stage, there was a memory address which would never be garbage collected when compiling and running Python code on the fly.

This is why the weird 336/337 thing happens, and also why the bug only triggers with random quantities of code in between the two steps (calling a particular gc function didn't make a difference, I was able to reproduce calling print!)

If the first ab buffer ends up at this particular address then this will happen, and that address depended on what else is allocated in the Python heap when the code ran (hence changing buffer size helped, and also having different compiled Python code size moved things around in memory and made the bug go away.)

Have submitted a fix at #14136

Albeit, at this time the entire (memory) system is so non-deterministic and unpredictable, I'll simply stop this exercise.
What I learned so far is, one has to knead and mangle the code in many and often funny ways until its execution finally leans towards what the developer had in mind. Probably every time a new firmware arises.

The extreme non-determinism here is because of this bug, I'm fairly certain. You should have a much more predictable experience after it's fixed.

That said, in general allocating and then re-allocating a single large memory buffer (say, a decent proportion of the total memory) multiple times in MicroPython is often going to lead to problems, due to the non-bug limitation of memory fragmentation. You'll run into less roadblocks if you can structure code to allocate that buffer once and then reuse it, or refactor it into multiple smaller buffers.

[GitHubsSilverBullet]

@projectgus
Angus, first of all, thanks a lot for your work and help!

allocating and then re-allocating a single large memory buffer (say, a decent proportion of the total memory) multiple times in MicroPython is often going to lead to problems,

How true. The same memory fragmentation can happen using C or a number of other languages as well. Usually not a problem on larger computers, on a fairly limited µC it's worth keeping in mind.