Runtime regression from Scala 2: A non-closure lambda literal is no longer a constant value and is not equal to itself

[neko-kai]

Compiler version

3.3.1

Minimized code

object app extends App {
  def x(): Int => String = (i: Int) => i.toString

  locally {
    println(x() == x()) // true on Scala 2, false on Scala 3...
    println(x().hashCode -> x().hashCode) // same on Scala 2, different on Scala 3
  }
}

Output

% coursier launch scala:2.13.12 -- example.scala
true
(1240730624,1240730624)
% coursier launch scala:3.3.1 -- example.scala
false
(580212331,390994841)

Expectation

Expected repeated invocations of x() to produce the exact same lambda instance, as before in Scala 2 and currently in Java:

class app {
    static java.util.function.Function<Integer, String> x() {
        return ((Integer i) -> i.toString());
    }

    public static void main(String[] args) {
        System.out.println(x() == x());
        System.out.println(x().equals(x()));
        System.out.println(x().hashCode());
        System.out.println(x().hashCode());
    }
}

Outputs:

true
true
707610042
707610042

We rely on this property very heavily in https://github.com/7mind/izumi (since Java behaves the same way, we assumed this property couldn't be broken in the future)

[mbovel]

I could reproduce:

➜  ~/scala-snippets-4 scala-cli run -S 2.13 closuresEquality.scala     
true
(729864207,729864207)
➜  ~/scala-snippets-4 scala-cli run -S 3.nightly closuresEquality.scala
false
(128526626,1265210847)

Seems like Scala 2 hoist closures that don't depend on their environment while Dotty doesn't.

Compare ASTs

➜  ~/scala-snippets-4 scala-cli run --server=false -S 2.13 -Xprint:delambdafy closuresEquality.scala
[[syntax trees at end of                delambdafy]] // closuresEquality.scala
package <empty> {
  object app extends Object with App {
    final protected def args(): Array[String] = app.super.args();
    @deprecated(message = "the delayedInit mechanism will disappear", since = "2.11.0") override def delayedInit(body: Function0): Unit = app.super.delayedInit(body);
    final def main(args: Array[String]): Unit = app.super.main(args);
    final override <stable> <accessor> def executionStart(): Long = (app.this.executionStart: Long);
    private[this] var executionStart: Long = _;
    override <accessor> def _args(): Array[String] = (app.this._args: Array[String]);
    private[this] var _args: Array[String] = _;
    override <accessor> def _args_=(x$1: Array[String]): Unit = app.this._args = (x$1: Array[String]);
    override <stable> <accessor> def initCode(): scala.collection.mutable.ListBuffer = (app.this.initCode: scala.collection.mutable.ListBuffer);
    private[this] var initCode: scala.collection.mutable.ListBuffer = _;
    final override <accessor> protected[this] def scala$App$_setter_$executionStart_=(x$1: Long): Unit = app.this.executionStart = (x$1: Long);
    final override <accessor> protected[this] def initCode_=(x$1: scala.collection.mutable.ListBuffer): Unit = app.this.initCode = (x$1: scala.collection.mutable.ListBuffer);
    def x(): Function1 = $anonfun();
    final <static> <artifact> def $anonfun$x$1(i: Int): String = java.lang.Integer.toString(i);
    final <synthetic> def delayedEndpoint$app$1: Unit = {
      scala.Predef.locally({
        scala.Predef.println(scala.Boolean.box(app.this.x().==(app.this.x())));
        {
          scala.Predef.println(scala.Predef$ArrowAssoc.->$extension(scala.Predef.ArrowAssoc(scala.Int.box(app.this.x().hashCode())), scala.Int.box(app.this.x().hashCode())));
          scala.runtime.BoxedUnit.UNIT
        }
      });
      ()
    };
    def <init>(): app.type = {
      app.super.<init>();
      app.super./*App*/$init$();
      app.this.delayedInit(new app$delayedInit$body(app.this));
      ()
    };
    final <static> <artifact> def $anonfun$x$1$adapted(i: Object): String = app.this.$anonfun$x$1(unbox(i))
  };
  final <synthetic> class app$delayedInit$body extends runtime.AbstractFunction0 {
    <paramaccessor> private[this] val $outer: app.type = _;
    final def apply(): Object = {
      app$delayedInit$body.this.$outer.delayedEndpoint$app$1();
      scala.runtime.BoxedUnit.UNIT
    };
    def <init>($outer: app.type): app$delayedInit$body = {
      if ($outer.eq(null))
        throw null
      else
        app$delayedInit$body.this.$outer = $outer;
      app$delayedInit$body.super.<init>();
      ()
    }
  }
}

➜ scala-cli compile -S 3.nightly -Xprint:genBCode closuresEquality.scala 
Compiling project (Scala 3.4.0-RC1-bin-20231209-fdd06ce-NIGHTLY, JVM)
[[syntax trees at end of                  genBCode]] // /Users/mbovel/scala-snippets-4/closuresEquality.scala
package <empty> {
  @SourceFile("closuresEquality.scala") final module class app extends Object, 
    App {
    def <init>(): Unit =
      {
        super()
        super[App].$init()
        locally(
          {
            println(Boolean.box(app.x().==(app.x())))
            {
              println(
                {
                  val ev$1: Integer =
                    ArrowAssoc(Int.box(app.x().hashCode())).asInstanceOf[Integer
                      ]
                  ArrowAssoc.->$extension(ev$1, Int.box(app.x().hashCode()))
                }
              )
              scala.runtime.BoxedUnit.UNIT
            }
          }
        )
        scala.runtime.Statics.releaseFence()
        ()
      }
    private <static> var executionStart: Long
    final def executionStart(): Long = app.executionStart
    private <static> var scala$App$$_args: String[]
    def scala$App$$_args(): String[] = app.scala$App$$_args
    private <static> var scala$App$$initCode:
      scala.collection.mutable.ListBuffer
    def scala$App$$initCode(): scala.collection.mutable.ListBuffer =
      app.scala$App$$initCode
    def scala$App$$_args_=(x$1: String[]): Unit = app.scala$App$$_args = x$1
    def scala$App$_setter_$executionStart_$eq(x$0: Long): Unit =
      app.executionStart = x$0
    def scala$App$_setter_$scala$App$$initCode_$eq(
      x$0: scala.collection.mutable.ListBuffer): Unit =
      app.scala$App$$initCode = x$0
    protected final def args(): String[] = super[App].args()
    @deprecated(message = "the delayedInit mechanism will disappear",
      since = "2.11.0") override def delayedInit(body: Function0): Unit =
      super[App].delayedInit(body)
    final def main(args: String[]): Unit = super[App].main(args)
    private def writeReplace(): Object =
      new scala.runtime.ModuleSerializationProxy(classOf[app])
    def x(): Function1 =
      {
        {
          closure(this.x$$anonfun$adapted$1)
        }
      }
    private final def x$$anonfun$1(i: Int): String = Int.box(i).toString()
    private final def x$$anonfun$adapted$1(i: Object): String =
      this.x$$anonfun$1(Int.unbox(i))
  }
  final lazy module val app: app = new app()
}

[mbovel]

@sjrd does the Spec say anything about that?

[sjrd]

The spec doesn't say anything about this. I don't think the Java spec says anything about it either. Relying on this behavior is just playing with chance.

[neko-kai]

The spec doesn't say anything about this. I don't think the Java spec says anything about it either. Relying on this behavior is just playing with chance.

@sjrd Technically yes, of course. But, in this answer by Brian Goetz he does say that you can 'conclude' this property at least for Oracle's Java implementation.

From an implementation perspective, you can conclude a little more. ... if you evaluate the same lambda at the same capture site, and that lambda is non-capturing, you get the same instance, which can be compared with reference equality.

[odersky]

I verified that the following does work (i.e. Scala 2 and Scala 3 both return true):

class App {
  def x(): Int => String = (i: Int) => i.toString

  locally {
    println(x() == x()) // true on Scala 2, false on Scala 3...
    println(x().hashCode -> x().hashCode) // same on Scala 2, different on Scala 3
  }
}

@main def Test = App()

This could be a simple fix in LambdaLift. I'll investigate.

[pshirshov]

@sjrd : even if this is not specified, it might be a very good idea to explicitly specify this. Otherwise we consider lambdas to be second-class citizens and we suddenly get unsoundness where it can be easily avoided.

Also I think that we should also define sound equality rules for for the lambdas with non-empty closures, I think we should create a separate issue for that. I don't see any obvious performance penalties in that. If I understand correctly, both equals and hashCode rely on default implementations right now and, effectively, are useless for bound lambdas.

[SethTisue]

the main Scala 2 PR is #19224 scala/scala#5099 by @retronym — perhaps it could be raided for implementation ideas and/or test cases. not sure if there were any sequels/followups

as the PR description indicates, the primary motivation at the time was serialization (see also scala/scala#4652):

allows lambdas that call the local methods to be serializable, regardless of whether or not the enclosing class is serializable

but also performance, and compatibility with Scala 2 does seem like a reasonable additional motivation for forward-porting

[odersky]

@SethTisue #19224 is this issue. What is the Scala 2 PR that addressed it?