Thursday 17 October 2019
Alexandre Archambault, Sébastien Doeraene
Early in 2019, the Scala Center advisory board accepted a proposal from Spotify, whose goal is to improve the handling of conflicts in dependency management in sbt.
The work we’ve been doing may interest you if:
- You’ve run into trouble with version conflicts between dependencies and didn’t know what to do or where to look for advice.
- You want more control on how version conflicts are resolved in your build.
- You want automated assistance with deciding whether a version upgrade might break something in your project.
At the time we started this project, Ivy was still used by default to resolve libraries in sbt, but efforts were in progress to use coursier instead, so we focused on the latter as a vehicle for improvements. Since then, coursier support in sbt became official, and ships by default in sbt 1.3.0. Besides having a simpler model than the one used before in sbt or in Ivy, coursier has the advantage of being usable from the command line, and can benefit other users of the coursier API.
For the third point about static analysis, we found the missinglink tool for Maven, originating from Spotify themselves. This tool can analyze the classpath of a Maven project, and detect ahead of time (at “link” time) whether binary incompatibilities can be encountered at run-time. The only issue was that it was not usable from sbt, rendering it useless for a large subset of Scala users. We built and released an sbt plugin for missinglink to address this hole.
Version ordering and reconciliation
While most people have a simple intuition about library resolution (a newer version of a library gets picked over an older version), the details of how versions are compared and reconciled are quite complicated. We contributed a dedicated documentation page on the coursier website to help users understand how resolution works in the not-so-trivial cases. In order to improve the experience of upgrading from sbt 1.2.x and Ivy-based resolution, we tweaked resolution in coursier 2.0.0-RC3-4 with the help of Eugene Yokota to be more in line with Ivy, the former sbt behavior, and the semantic versioning specification. The documentation page should be updated accordingly soon.
On top of that, Eugene Yokota recently wrote a blog post comparing how versions are compared and reconciled in both Ivy and coursier.
Strict conflict manager
By default, both Ivy and coursier tolerate that the dependency graph contains two different versions of a library, in which case they pick a winner. Sometimes, we want more control, and do not want any conflict resolution to happen. Instead, we would like an error to be reported if there are two different versions of a library. A strict conflict manager does precisely that, and we added one to coursier. Instructions follow for enabling the strict conflict manager 1) in an sbt build, and 2) from the coursier command line.
From sbt, enable the strict conflict manager either through the original
conflictManager key, like
conflictManager := ConflictManager.strict
or enable it in a possibly more fine-grained way if sbt-coursier is around with
versionReconciliation += "*" % "*" % "strict"
(Note that to use sbt-coursier from an sbt 1.3.x project, the coursier-based sbt launcher is required.
Get it via its custom sbt-extras runner or generate one with
coursier bootstrap sbt-launcher && ./sbt.)
update task will then succeed only if the strict checks pass.
If they don’t, either force the versions of faulty dependencies with
dependencyOverrides += "org.typelevel" % "cats-core_2.12" % "1.5.0"
or adjust their version reconciliation, like
versionReconciliation ++= Seq( "org.typelevel" %% "cats-core" % "relaxed", // "semver" reconciliation is also available "*" % "*" % "strict" )
A number of version reconciliation types are available:
"strict"requires all dependees to depend on the exact selected version (if they depend on an interval, the selected version only needs to be contained in it)
"semver"requires all dependees to depend on the same major version as the selected version (if they depend on an interval, the selected version only needs to be contained in it too)
"default"is the default version reconciliation in coursier, enabled in the coursier CLI or with the sbt-coursier plugin. It is described in more detail in this page.
"relaxed"does not trigger any conflict. It is the default in the coursier support of sbt 1.3.x, which allows for better compatibility with former sbt versions. It ignores the lowest versions and version intervals, until they can be reconciled with the same algorithm as
Since reconciliation strategies can be set per artifact, or per organization, it becomes much easier to declaratively tell sbt, for example, that some project adheres to SemVer, and that it is fine to resolve dependencies on that specific project according to SemVer, but not others.
From the command line, we can use the new strict conflict manager of coursier with:
$ coursier resolve --strict \ org.typelevel:cats-effect_2.12:2.0.0 \ org.typelevel:cats-core_2.12:1.5.0 Resolution error: Unsatisfied rule Strict(*:*): Found evicted dependencies: org.typelevel:cats-core_2.12:2.0.0 (1.5.0 wanted) └─ org.typelevel:cats-core_2.12:1.5.0
To ignore some strict checks, one can exclude the faulty dependency with
$ coursier resolve \ --strict-exclude 'org.typelevel:cats-core*' \ org.typelevel:cats-effect_2.12:2.0.0 \ org.typelevel:cats-core_2.12:1.5.0 org.scala-lang:scala-library:2.12.9:default org.typelevel:cats-core_2.12:2.0.0:default org.typelevel:cats-effect_2.12:2.0.0:default org.typelevel:cats-kernel_2.12:2.0.0:default org.typelevel:cats-macros_2.12:2.0.0:default
--strict-exclude automatically enables
--strict), or only include a subset of the dependencies in the strict checks, like
$ coursier resolve \ --strict-include 'org.scala-lang:*' \ org.typelevel:cats-effect_2.12:2.0.0 \ org.typelevel:cats-core_2.12:1.5.0 org.scala-lang:scala-library:2.12.9:default org.typelevel:cats-core_2.12:2.0.0:default org.typelevel:cats-effect_2.12:2.0.0:default org.typelevel:cats-kernel_2.12:2.0.0:default org.typelevel:cats-macros_2.12:2.0.0:default
Alternatively, forcing the version of the faulty dependency makes the strict checks ignore it:
$ coursier resolve \ --strict \ --strict-exclude 'org.scala-lang:*' \ --force-version org.typelevel:cats-core_2.12:1.5.0 \ org.typelevel:cats-effect_2.12:2.0.0 \ org.typelevel:cats-core_2.12:1.5.0 org.scala-lang:scala-library:2.12.9:default org.scala-lang:scala-reflect:2.12.6:default org.typelevel:cats-core_2.12:1.5.0:default org.typelevel:cats-effect_2.12:2.0.0:default org.typelevel:cats-kernel_2.12:1.5.0:default org.typelevel:cats-macros_2.12:1.5.0:default org.typelevel:machinist_2.12:0.6.6:default
Static analysis checks
First, add the following line in
addSbtPlugin("ch.epfl.scala" % "sbt-missinglink" % "0.1.0")
then simply run the following task for the project you want to test:
This will check that the transitive dependencies of your project do not exhibit any binary compatibility conflict, assuming that the methods of your
Compile configuration (in
src/main/) are all called.
You can add that task to your CI script, for example.
If the checks succeed, you have a guarantee that no
LinkageError can happen at run-time, unless run-time reflection is involved, making static analysis impossible.
The plugin is very simple at the moment, and will be extended to support more features of
missinglink, in particular the ability to add exclusions to the checks performed.
missinglink itself proves insufficient for the needs of Scala developers, we may also contribute improvements to it in the future.
We have seen a few new ways you can improve the management of your transitive dependencies. In particular, we have introduced:
- reconciliation strategies, which let you declaratively specify the compatibility guarantees of individual libraries, allowing coursier to do a better, safer job at conflict resolution, and
sbt-missinglink, an sbt plugin to easily check ahead of time that your resolved transitive dependencies are actually binary compatible.