Include parallel collections in Toolkit?

[SethTisue]

I was a bit surprised recently to realize that we didn't include scala-parallel-collections in the Toolkit.

Just now I looked at the old spreadsheet of candidate libraries, assuming I'd see it was considered and rejected or postponed, but I don't even see a spreadsheet entry for it? Was it really never considered?

I think we should include it, or at least discuss including it. It has a lot of things going for it:

• It's in the scala. namespace
• It's applicable in many/all problem domains
• It doesn't really have any direct competition. (Only heavyweight competition, such as parallel streaming in fs2.)
• It's mature, good quality (few known bugs), and has needed little maintenance
• It's cross-published for Scala 2.13 and 3
• Although it's community-maintained, the Scala org has a strong interest in keeping it alive, as it's mentioned in Scala books and such. It's already a Scala "module" which is the next thing down from being in the standard library
• We have a strong team of volunteers at @scala/collections who are interested in helping with collections stuff

What about the library's usefulness/importance?

For some tasks, it's extremely convenient and can give a large speed benefit. I think it didn't end up being quite as widely used as we'd originally envisioned, but when the unit of work that you need to happen in parallel is large enough, it's super easy to use parallel collections to get a big speedup.

Here's an example. I have a Scala-CLI script that does several hundred independent GitHub API queries. With only this small diff:

+//> using dep org.scala-lang.modules::scala-parallel-collections:1.0.4
...
+import scala.collection.parallel.CollectionConverters.* // for .par
...
-for file <- files
+for file <- files.par

I sped up the script from taking 5 minutes to taking less than 30 seconds.

[iusildra]

Would be nice to have indeed

For bigger projects that would require things like fs2, is the toolkit used over the alternatives from the same stack ?

[SethTisue]

If you depend on Toolkit and other libraries too, it all just ends up piled into your classpath and your application code is free to use or ignore or whatever it wants.

[armanbilge]

For bigger projects that would require things like fs2, is the toolkit used over the alternatives from the same stack ?

@iusildra see also the typelevel toolkit.

https://typelevel.org/toolkit/

[lihaoyi]

I feel like something like parallel collections would be nice, but probably not parallel collections themselves.

IIRC the codebase was pretty old, had a lot of weird non-idiomatic/no-longer-idiomatic APIs (e.g. configuring things via mutability?), wasn't super well maintained, and didn't get widely adopted even after 10+ years.

AFAIK the recommendation for years about using scala-parallel-collections was "Don't", so unless that has changed for some reason it feels very weird to bundle it with the toolkit

[OndrejSpanel]

@lihaoyi I am not using toolkit and I am not likely to use it in a near future, but I am using parallel collections in my application to parallelize a few inner loops processing thousands of items. If I am not to use parallel collections, what should I use instead?

recommendation for years about using scala-parallel-collections was "Don't"

Could you provide some link? I do not see any kind of warning in https://docs.scala-lang.org/overviews/parallel-collections/overview.html. It says:

The hope was, and still is, that implicit parallelism behind a collections abstraction will bring reliable parallel execution one step closer to the workflow of mainstream developers.

[SethTisue]

I'll answer this point by point.

IIRC the codebase was pretty old

I would describe it as mature and well-established. No one has needed to write a competitor, because this one does the job. I don't see why it matters how old the code is unless there is actually something wrong with the code.

had a lot of weird non-idiomatic/no-longer-idiomatic APIs (e.g. configuring things via mutability?)

I acknowledge that the configuring the thread pool usage is a little weird, but it's rare that anyone doing Toolkit-y, script-y type things would need to configure it at all. Most of the time you just want to do some stuff in parallel and it does that. Easy things easy, harder things possible.

wasn't super well maintained

It doesn't need any maintenance. There aren't any open bugs to speak of.

didn't get widely adopted even after 10+ years.

No? There are 3500 hits at https://github.com/search?q=scala-parallel-collections&type=code .

You might have this impression if people aren't talking about it, but sometimes there isn't a lot to say about a workhorse library like this.

AFAIK the recommendation for years about using scala-parallel-collections was "Don't"

I agree with Ondrej. There is no such recommendation.

One aspect of the parallel collections that did have a negative reputation was how much they deepened and complicated the collections hierarchy when they were introduced. But we fixed that in Scala 2.13, when the parallel collections were re-engineered, became a separate library, and stopped being intertwined with serial collections.

[JD557]

From the toolkit introduction:

The Toolkit supports:

• Scala 3 and Scala 2
• JVM, Scala.js, and Scala Native

I was under the impression that only libraries supporting Scala.js and Scala Native would be included, which I don't think is the case for the parallel collections.

Although I guess one could publish them for Scala.js and Scala Native 0.4.x with dummy implementations that just wrap the sequential collections.

(As an aside, I do use the parallel collections a lot on scala-cli scripts, so I'm excited to see this landing)

[lihaoyi]

One aspect of the parallel collections that did have a negative reputation was how much they deepened and complicated the collections hierarchy when they were introduced. But we fixed that in Scala 2.13, when the parallel collections were re-engineered, became a separate library, and stopped being intertwined with serial collections.

This may be where my impression came from. I must admit, I don't have any concrete objections here, other than a vague feeling. Perhaps it's no longer a problem since it's been modularized.

How do parallel collections play with scala.concurrent.ExecutionContext? My understanding is that implicit ec: ExecutionContext is the de-facto standard in way to handle thread pools and executors in Scala, being in the standard library and all.

If parallel collections are now a module, does that mean they are open to modifications? If so, then we should definitely consider sanding off some of the rough edges as part of including it in the toolkit, e.g. replacing the mutable threadpool thing with an implicit ec: ExecutionContext, and going over the rest of the doc/API/architecture to see if there's anything else we could modernize. This would be similar to what we're doing with uPickle and OS-Lib and others. If we do that, then I have no objection to including it in the toolkit

If I am not to use parallel collections, what should I use instead?

I've generally used scala.concurrent.Future here:

val foo = items.map(x => Future{doThing(x)}).map(Await.result(_, Duration.Inf))

I admit it's a bit more clunky to use than .par, but the API can definitely be simplified if necessary, and having a consistent ExecutionContext interface that all your asynchronous operations use is definitely nice.

[He-Pin]

Nowadays, there are fs2, zio , pekko/akka streams out there, -1 for including parallel collection

[Ichoran]

There isn't anything I'm aware of that matches the simplicity of the parallel collections, so I think it's worth including for that reason.

[som-snytt]

Just to echo Mr Li's impression, but maybe it's just that (as happens) the real-world effectiveness did not match heightened expectations.

As usual, Mr Ichoran's summation is especially succinct and persuasive.

The reply to He-Pin's objection, and one is liable to mix up one's kerrs, is that the toolkit is about simplicity and scope. So an easy-to-use solution of limited scope is desirable.

[He-Pin]

What about sending a MR to scala/dotty compiler and let the compiler use it first？
@som-snytt

@SethTisue submit a poll on reddit and see？

[som-snytt]

I forgot to say that I checked
https://mvnrepository.com/artifact/org.scala-lang.modules/scala-parallel-collections/1.0.4/usages?p=1
but I wasn't sure what it implies.

Released for Scala 3 nullifies any suggestion the project is moribund.

Actually, #14 in this list is impressive:
https://mvnrepository.com/open-source/collections

Worth adding that lack of tickets does not directly imply quality, but may imply usability in proportion to the use cases people use it for.

@He-Pin also worth adding that suitability for a limited use case does not imply suitability for others. But one may ask if "inclusion in the toolkit" constitutes an endorsement, and should the user receive further guidance.

I'm sorry this topic missed the recent survey that folks were complaining was already too long. "And before we let you hit submit, please tell us your thoughts on the parallel collections. Yes, that one. Good for noobs who have no opinions about concurrency and/or parallelism? or is par short for paradigmatic?"

[He-Pin]

@som-snytt Hum, interesting , Zio is on the list too, IIRC, zio has zero dependency.

@SethTisue What's the status of this scala/scala-parallel-collections#22