Provide a guarantee on unmapped memory region (null region)

[ishitatsuyuki]

Most OS implement safeguards on null arithmetic, by reserving the first memory pages as unmapped (reference).

It's good to guarantee some amount of spaces, to enable the use of raw pointers with small enum. (By the way, you have 16 bits to play with on amd64.)

For example, this one is currently UB. At least one page of width would be good to implement those enum-like structures.

[strega-nil]

I'm not sure what you're talking about. I feel examples are in order.

[seanmonstar]

@ubsan the line linked to shows that the futures crate is assuming that a pointer address of 1 could not possibly be anything, and @ishitatsuyuki is saying that the current Rust implementation doesn't guarantee that.

[strega-nil]

@seanmonstar ah, okay.

@ishitatsuyuki I would like to make certain that you understand that that example is not undefined behavior. It simply may not do what you'd like it to.

[strega-nil]

I am personally very much against any of these guarantees. 0 and 1 may be valid pointer values, for example, if you're writing an operating system.

[seanmonstar]

Well, rust does currently believe that 0 is indeed null: https://github.com/rust-lang/rust/blob/1.14.0/src/libcore/ptr.rs#L55

p.is_null() checks for 0 as well.

[strega-nil]

@seanmonstar The standard (or core) library can make assumptions that other code should not make.

[ishitatsuyuki]

It's quite okay to implement such things, since it's implemented in almost any operating system as a security measure, and we have already guaranteed that 1 byte = 8 bit (which doesn't apply to some minor architecture).

[nagisa]

It's quite okay to implement such things, since it's implemented in almost any operating system as a security measure, and we have already guaranteed that 1 byte = 8 bit

There’s an important distinction between OS and Architecture. rustc does not (and probably never will) target any architecture where 1B≠8b, but nothing is preventing one from writing an OS (possibly in Rust, even!), where 0th page is map-able by the kernel or even the user.

[ishitatsuyuki]

@nagisa:
NULL has been 0 for a long time, both defined in POSIX and Windows. I don't really see the point in reinventing this. Redox follows the de facto standard too.
Hence, convenient enum in pointer support is important, mostly the atomic one in the futures crate example I mentioned. We should have flexibility in 3rd-party crates as well; unlike Go, Rust's std is quite minimal, lacking async I/O or coroutine things.

[ticki]

Note: pointer 1 is used internally in both libcore and libstd as a non-null placeholding pointers (in e.g. empty Vec<T>).

[ticki]

@nagisa You're confusing virtual and physical memory.

[joshtriplett]

It seems fine to assume that 0 means "NULL" in the vast majority of libraries, though some low-level systems programming code may need to do otherwise. (Such code would have to handle that assumption carefully, though it does seem reasonable to require that the language itself outside of any library should not break if you attempt to write a given value to a 0 pointer.)

But expecting the availability of any particular non-zero invalid pointer seems like a non-portable assumption, albeit a relatively safe one on a large subset of environments. A more portable (if slightly slower) approach would declare a dummy object, obtain its pointer, and use that value as the placeholder.

I have an alternative proposal, which could provide the same effect with more readable code:

Teach rustc for any given target platform about the range of invalid pointer values; for any hosted environment, this would typically mean "any pointer in the zeroth page of memory". Then, extend the "forbidden value" optimization work in rust-lang/rust#36237 (which generalized the original null-pointer optimization) to optimize enum types containing pointers and a small number of values. That would provide a much more general and transparent memory size optimization, allowing many different enums to shrink to the size of a pointer on most platforms.

With that optimization, the following enums would each take exactly the size of a pointer on most platforms:

enum E1 {
    Invalid,
    Pointer(*SomeType),
}

enum E2 {
    Invalid1,
    Invalid2,
    Pointer(*SomeType),
}

enum E3 {
    Boolean(bool),
    Pointer(*SomeType),
}

enum E4 {
    Byte(u8),
    Pointer(*SomeType),
}

enum E5 { // Existing null-pointer optimization
    Invalid,
    Reference(&SomeType),
}

enum E6 {
    Invalid1,
    Invalid2,
    Reference(&SomeType),
}

enum E7 {
    Byte(u8),
    Reference(&SomeType),
}

Consider how much code would use substantially less memory with this optimization, without requiring any manual special-cases.

[ishitatsuyuki]

The optimization is also awesome, however the example I mentioned use this trick on atomic values. Without a guarantee this cannot be made safe.

[joshtriplett]

@ishitatsuyuki If you're willing to make exactly the same non-portable assumption, you could have that same guarantee. Given the optimization above, it doesn't seem too hard to add a way to assert that at compile time. The same compile-time assertion would also allow you to build your own unsafe code relying on the same assumption. (And, as a bonus, if someone attempts to compile your code on a platform that doesn't support that assumption, they'll get a compile-time error rather than runtime breakage.)