Crash accessing or writing huge values (regression from 1.3.8) #931
Description
We've recently upgraded Bolt from 1.3.8 to 1.4.0 (yeah 1.3.8 was an old one I know), and found a curious problem: starting with 1.3.9, Bolt started crashing when trying to store, access or enumerate values larger than 256 MB.
Yes, I know, it's a crazy size for a value. Yes, we accidentally have one in our database. Bolt 1.3.8 didn't have a problem with it, and now Bolt simply crashes.
Reproduction is trivial:
package main
import (
"log"
bolt "go.etcd.io/bbolt"
)
func main() {
db, err := bolt.Open("/tmp/bbolt-crash.db", 0600, nil)
if err != nil {
log.Fatal(err)
}
defer db.Close()
data := make([]byte, 0xFFFFFFF+1)
err = db.Update(func(tx *bolt.Tx) error {
b, err := tx.CreateBucketIfNotExists([]byte("test"))
if err != nil {
return err
}
err = b.Put([]byte("key"), data)
if err != nil {
log.Fatal(err)
}
return nil
})
if err != nil {
log.Fatal(err)
}
}The crash is:
panic: runtime error: slice bounds out of range [::268435458] with length 268435455
goroutine 1 [running]:
go.etcd.io/bbolt/internal/common.UnsafeByteSlice(...)
/Users/andreyvit/Developer/prairie/bbolt/internal/common/unsafe.go:26
go.etcd.io/bbolt/internal/common.WriteInodeToPage({0x1400002e0c0?, 0x1, 0x4?}, 0x1401018e000)
/Users/andreyvit/Developer/prairie/bbolt/internal/common/inode.go:81 +0x288
go.etcd.io/bbolt.(*node).write(0x14000072070?, 0x1401018e000?)
/Users/andreyvit/Developer/prairie/bbolt/node.go:199 +0xa0
go.etcd.io/bbolt.(*node).spill(0x14000072070)
/Users/andreyvit/Developer/prairie/bbolt/node.go:334 +0x1dc
go.etcd.io/bbolt.(*Bucket).spill(0x1400002e080)
/Users/andreyvit/Developer/prairie/bbolt/bucket.go:786 +0x278
go.etcd.io/bbolt.(*Bucket).spill(0x14000158018)
/Users/andreyvit/Developer/prairie/bbolt/bucket.go:753 +0xc0
go.etcd.io/bbolt.(*Tx).Commit(0x14000158000)
/Users/andreyvit/Developer/prairie/bbolt/tx.go:204 +0x260
go.etcd.io/bbolt.(*DB).Update(0x100b35db8?, 0x14000104f00)
/Users/andreyvit/Developer/prairie/bbolt/db.go:915 +0xc4
main.main()
/Users/andreyvit/Developer/prairie/bbolt/cmd/bbolt-crash-repro/bboltcrash.go:19 +0x120
exit status 2
The crash has been introduced in this commit:
commit ea511567eb216de0ef8539eacbd56bed8d1aa2a7
Author: Benjamin Wang <[email protected]>
Date: Sat Jan 28 14:37:24 2023 +0800
refactor both bolt and guts_cli based on the common package
Signed-off-by: Benjamin Wang <[email protected]>
because it changed from
func unsafeByteSlice(base unsafe.Pointer, offset uintptr, i, j int) []byte {
// [...]
return (*[maxAllocSize]byte)(unsafeAdd(base, offset))[i:j:j]
}to
func UnsafeByteSlice(base unsafe.Pointer, offset uintptr, i, j int) []byte {
// [...]
return (*[pageMaxAllocSize]byte)(UnsafeAdd(base, offset))[i:j:j]
}moving from maxAllocSize (which is platform-dependent but generally very large on modern systems) to pageMaxAllocSize (which is always 0xFFFFFFF.
Also, that's the only code that uses pageMaxAllocSize in Bolt, and the comment on that constant isn't super helpful:
// DO NOT EDIT. Copied from the "bolt" package.
const pageMaxAllocSize = 0xFFFFFFFSo I cannot judge if this was an intentional limitation or an oversight. If it's intentional, I propose panicking earlier and with a more helpful message, or, better yet, returning an error.
Note that bbolt check also panics when trying to check such a file.