Allow instance and metaobject members in one declaration?

[munificent]

I've been really interested in metaclasses/metaobjects for a long time. The metaobjects proposal is excellent. However, I'm worried about the usability of specifying the static interface of a type separate from its instance interface.

There was a chat thread a while back about this and @kevmoo brought up how weird it would be to have separate interfaces for, say, serializing and deserializing. What users probably want is a single conceptual "interface" that incorporates both instance and static members.

For example, here's a simple API for serializing and deserializing JSON using the metaobjects proposal:

interface class JsonDecodable<T> {
  T fromJson(Map<String, Object?> json);
}

interface class JsonEncodable {
  Map<String, Object?> toJson();
}

And here's how a user class might implement it:

class Person implements JsonEncodable static implements JsonDecodable<Person> {
  final String name;
  final int age;

  Person(this.name, this.age);

  factory Person.fromJson(Map<String, Object?> json) =>
      Person(json['name'] as String, json['age'] as int);

  Map<String, Object?> toJson() => {'name': name, 'age': age};
}

And here's some example code to round-trip an object:

T roundTrip<T extends JsonEncodable static extends JsonDecodable<T>>(T value) {
  var json = value.toJson();
  return T.fromJson(json);
}

main() {
  var clone = roundTrip(Person('Diana', 23));
  print(clone.name); // "Diana".
}

If I understand the proposal right, this all works, and it's definitely a capability we don't have now, so this looks good. The fact that roundTrip() can be called with any type that implements the serialization API is exactly what users have been asking for.

User experience

But I find what they have to write to wire this all up to be kind of verbose and unintuitive. By that I mean:

Declaring a static API

interface class JsonDecodable<T> {
  T fromJson(Map<String, Object?> json);
}

Here, there's nothing to indicate that expected use case is that this is a static interface and that an implementation of fromJson() is expected to be a static method or a constructor.

Further, it's strange that I have to make two separate interfaces, one for the static members in the API and one for the instance members:

interface class JsonEncodable {
  Map<String, Object?> toJson();
}

In this trivial example, it kind of works out logically because it so happens that the instance member and static member represent different capabilities (encoding and decoding, respectively). But I imagine that in other use cases the API will be a mixture of instance and static behavior that isn't so neatly separated and then it would feel strange to have to split the members into two separate interfaces. Then in the static half, you have to remember to declare them as instance members on the metainterface.

Implementing an API

On the implementation side, there is also the same bifurcation and repetition:

class Person implements JsonEncodable static implements JsonDecodable<Person> {
  // ...
}

You have two separate implements clauses and they must be paired for the API to work correctly, but the user still has to write them twice. It would be neater if they could just do:

class Person implements JsonCodable<Person> {
  // ...
}

(It would be even nicer if we had self types so that we didn't have to do the CRTP here, but that's a separate issue.)

Using an API

Likewise, in roundTrip():

T roundTrip<T extends JsonEncodable static extends JsonDecodable<T>>(T value) {
  ...
}

The bound has a pair of constraints that must always be paired to fully access the API. It would be simpler as:

T roundTrip<T extends JsonCodable<T>>(T value) {
  ...
}

Generality

Of course, this verbosity does provide expressiveness in return. One of the cool things about the metaobject proposal is that it lets you mix objects and metaobjects together. You can do things like:

interface class Fooable {
  void foo();
}

class StaticallyFooable static implements Fooable {
  static void foo() { print('static method'); }
}

class InstantlyFooable implements Fooable {
  void foo() { print('instance method'); }
}

main() {
  var fooables = [
    StaticallyFooable, // Use type literal to get metaobject.
    InstantlyFooable(), // Consruct to get instance.
  ];

  for (var fooable in fooables) {
    fooable.foo();
  }
}

The metaobjects really are objects and can play along with other objects and do all the other things you might want an object to do.

I'm just not sure if that flexibility is that useful. The cost of it is verbosity when an API wants to offer a mixture of instance and static abstractions, and I think some cognitive confusion because members are no longer clearly stratified into instance and static ones.

Alternative semantics

An alternative approach is that a single interface can define both the instance and metaobject's member sets, similar to how a single concrete class can define both instance static members (unlike, say Scala and Kotlin where the static members are off to the side in separate companion object declarations).

Then when you implement a type, you sign up to implement both its instance and metaobject members. In the example here, it would look something like:

interface class JsonCodable<T> {
  static T fromJson(Map<String, Object?> json);
  Map<String, Object?> toJson();
}

And then you'd use it like:

class Person implements JsonCodable<Person> {
  ...

  factory Person.fromJson(Map<String, Object?> json) =>
      Person(json['name'] as String, json['age'] as int);

  Map<String, Object?> toJson() => {'name': name, 'age': age};
}

T roundTrip<T extends JsonCodable<T>>(T value) {
  var json = value.toJson();
  return T.fromJson(json);
}

Of course, the problem is that this is massively breaking. We already allow implemented classes to have static members and we don't treat them as part of the implemented interface. Likewise, when you extend a class, you only inherit the instance members, not the static ones or the constructors. In other words, the metaclass inheritance chain does not parallel the class inheritance chain.

If we made this change, every interface with a static method would break every class implementing it.

Note that the current behavious isn't a design flaw. I believe that 90% of the time, you don't want static members to be inherited by subclasses. I don't think there's much useful value in being able to call bool.hash(123), or Set.iterableToFullString(someListNotEvenASet). The fact that static members aren't inherited has not really been a problem in Dart for its entire life.

(Constructors are a sort of different story. Pretty often, I do find myself wishing I could just inherit a constructor instead of having to define my own and forward everything. But more often than that, I don't want to inherit all the constructors.)

This suggests that maybe there could be three flavors of member:

• Instance members which are called on instances of the type and inherited by subclasses.
• Static members which are called directly on the type but only accessible on that type and its corresponding metaobject and are not inherited by subclasses or their metaobjects.
• "Meta" members that are also called directly on the type but are inherited by subclasses.

Does this mean we really need three kinds of dispatch (not even counting extensions)? I don't think so. I think, just as metaobjects model static method calls using instance dispatch, we can model the whole thing as instance dispatch using a carefully organized class hierarchy.

Let's say you have:

class Parent {
  a() {}
  static b() {}
  meta c() {}
}

class Child extends Parent {
  d() {}
  static e() {}
  meta f() {}
}

Here, I'm using meta as a not-great temporary keyword to define these "static but inherited" things. Given that class hierarchy, here's how it behaves:

main() {
  var x = Child();
  x.a(); // OK, inherited from Parent instance method.
  //x.b(); // Error, can't call statics on instance.
  //x.c(); // Error, can't call statics on instance.
  x.d(); // OK.
  //x.e(); // Error, can't call statics on instance.
  //x.f(); // Error, can't call statics on instance.

  var y = Child;
  //y.a(); // Error, can't call instance methods on metaobject.
  //y.b(); // Error, statics aren't inherited.
  y.c(); // OK, inherited from Parent meta methods.
  //y.d(); // Error, can't call instance methods on metaobject.
  y.e(); // OK.
  y.f(); // OK.
}

To make that go, the class (and metaclass) hierarchy looks something like:

┌──────┐  ┌────────────┐  ┌──────────┐
│Parent│  │ParentStatic│  │ParentMeta│
│------│  │------------│═▶│----------│
│a()   │  │b()         │  │c()       │
└──────┘  └────────────┘  └──────────┘
    ▲                          ▲
    ║                          ║
┌──────┐  ┌────────────┐  ┌──────────┐
│Child │  │ChildStatic │  │ChildMeta │
│------│  │------------│═▶│----------│
│d()   │  │e()         │  │f()       │
└──────┘  └────────────┘  └──────────┘
    ▲           ▲
    ┊           ┊
 ╭─────╮     ╭─────╮
 │  x  │     │  y  │
 ╰─────╯     ╰─────╯

Here, square boxes are classes, rounded boxes are objects, ═▶ is "inherits from" and ┈▶ is "is an instance of". So there are three classes for each class declaration:

• The "normal" class where instance members go.
• The "static" metaclass where non-heritable static members go.
• The "meta" metaclass where heritable "meta" members go.

When you construct an instance of a class, the instance points to its normal class. That's how normal instance dispatch works. That class inherits from the superclass.

When you use a type literal to get your hands on a metaobject for some class, the object points to its static metaclass. That has all of the static members that are only available directly on that class. The static class inherits from the "meta" class, so it also includes all of the "heritable static" members. The static class does not inherit from the superclass's static class. That way, superclass static methods like Parent.b() are not inherited by Child. However, the metaclass does inherit from the superclass's metaclass. That way heritable static methods like Parent.c() are inherited.

It's all instance dispatch and single inheritance. Just with a, umm, somewhat complex compiler-generated hierarchy.

Syntax and prior art

I don't like meta as a keyword. Perhaps a better one is class:

class Parent {
  a() {}
  static b() {}
  class c() {}
}

class Child extends Parent {
  d() {}
  static e() {}
  class f() {}
}

So "class" methods are accessed through the class but unlike statics are also inherited. I don't love it either but... it turns out that Swift actually has this exact feature.

In Swift, you can declare static methods or class methods. The former are not inherited but the latter are. As far as I can tell, it's pretty much exactly the distinction I'm talking about here.

Should we make the metaobject proposal for Dart work like that too?

[kevmoo]

Love it, @munificent ! Points for updating your initial post with a tl;dr section outlining your prefered proposal w/ Codable as an example.

[Wdestroier]

Very cool!

Can't these abstract static methods either not have a body or (if they have a body) have an abstract modifier? Then it would not be breaking, right? Examples:

1:

abstract class JsonCodable<T> {
  static T fromJson(Map<String, dynamic> json); // <-- Here it's implicitly abstract. Currently an error.
  Map<String, dynamic> toJson();
}

class Person implements JsonCodable<Person> {
  ...

  @override
  static Person fromJson(Map<String, dynamic> json) => /* <-- The compiler knows it's abstract,
because all types are known at compile-time. */
      Person(json['name'] as String, json['age'] as int);

  Map<String, dynamic> toJson() => {'name': name, 'age': age};
}

2:

class Person2 {
  ...

  abstract static fromJson(Map<String, dynamic> json) => // <-- Here it's explicitly abstract.
      Person(json['name'] as String, json['age'] as int);
}

class Student implements Person2 {
  ...

  // @override
  static fromJson(Map<String, dynamic> json) => /* <-- The compiler knows it's an abstract static method,
because it's declared in the parent class. */
      Person(json['name'] as String, json['age'] as int, json['school'] as String);
}

[munificent]

Can't these abstract static methods either not have a body or (if they have a body) have an abstract modifier? Then it would not be breaking, right?

Yes, we definitely need to come up with a non-breaking syntax. You're right that for abstract static methods, it would be a non-breaking addition to just not have a body because currently that's a syntax error.

But I think we also do want to support concrete inherited static (or "class") members, so we still need to figure out a syntax for how to express those too. Using class as a modifier like Swift does would probably work, though I'm somewhat worried that that would collide with allowing actual class declarations inside a class (#336, #3021).