Skip to content

Advise library authors on how best to depend on child dependencies #14080

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 19, 2024
Merged

Advise library authors on how best to depend on child dependencies #14080

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions lib/elixir/pages/references/library-guidelines.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,3 +42,11 @@ Keep in mind your library's [lockfile](`Mix.Project#module-configuration`) (usua
On the other hand, contributors of your library, need a deterministic build, which implies the presence of `mix.lock` in your Version Control System (VCS), such as `git`.

If you want to validate both scenarios, you should check the `mix.lock` into version control and run two different Continuous Integration (CI) workflows: one that relies on the `mix.lock` for deterministic builds, and another one, that starts with `mix deps.unlock --all` and always compiles your library and runs tests against latest versions of dependencies. The latter one might be even run nightly or otherwise recurrently to stay notified about any possible issue in regard to dependencies updates.

### Dependency Version Requirements

When depending on other libraries, the dependency version requirements are ultimately up to you. However, you should consider the effects that an overly strict dependency requirement can have on users of your library. Most dependencies adopt [Semantic Versioning](https://semver.org/), and therefore provide reasonable guarantees about what each release contains. For instance, if you use `{:some_dep, “== 0.2.3”}`, this prevents users from using any other version but the one that you specified, which means that they cannot receive bug fix upgrades to that package. When in doubt, use a dependency in the format of `"~> x.y"`. This prevents the user from using a higher major version of the library, but allows them to upgrade to newer minor and patch versions, which should only include bug fixes and non-breaking improvements.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't see the library guidelines docs, GTK these exist!

I want to write on the Elixir Forum thread too, but writing just a bit here 😄

The >= operator is only used once in this whole section. And I get your explanation, but in my mind >= should be the first (but not the last!) thing you should think about in a library. Of course it depends on the dependency, but if you're using the main, most stable parts of the dependency's API, I think you should be using >= (and then maybe later add a and < x.y.z if needed because of some incompatibility)

I think that absolutely sometimes library developers should be using ~> x.y(.x). I think that this documentation could use some examples as to how to make the decision in addition to the abstract description. The biggest example that I often see is just using Jason.encode / Jason.decode. I know that since JSON is moving into core that's going to become an out-of-date example, but I'm sure we could find better ones (like if you're only using Ecto.Repo.get, and a handful of really stable functions for example).

Application developers should (😅) have tests which run whenever there are updates (including in the mix.lock) and/or they should be verifying updates somehow. So I don't think it's up to the library developer to protect the app developer from future updates to sub-dependencies unless a known issue comes up (and if the library maintainer is AWOL, the app developer can limit the version of the sub-dependency without even needing to use override 😉 )

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but in my mind >= should be the first (but not the last!) thing you should think about in a library.

I agree to disagree! I believe ~> is a good default for application and library authors (RE that forum thread) as long as people follow semver which they tend to do in my experience. When hex.pm displays example version requirement it would show ~> major.minor (post 1.0) and ~> major.minor.patch (pre 1.0) exactly because of where the breakage is allowed to happen per semver.

FWIW I sometimes use >= without a corresponding < but it is very rare, I'd say exclusively for :ex_doc and :postgrex where the former is a dev-only dependency and the latter has extremely stable public API and I only ever use it in apps anyway.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Libraries should not use >= for their child dependencies. Users might end up upgrading child_dep and taking care of breaking changes, and main_dep would allow that while not necessarily handling those breaking changes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wholeheartedly agree that libraries should not use >= for dependencies 👍

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, well... 🤷 😅

I guess the main thing I've been frustrated by a number of times in when libraries authors go AWOL, and I guess that's really the problem I'm trying to solve. And really I'm specifically trying to solve the problem of maintainers going AWOL using a dependency when they're using the most stable parts of a dependency... Yes, semver is great when people use it, but updating the major version generally doesn't mean that the whole API of the library changes.

The options as I see them are a personal fork (which can go out of date), making your own fork (requiring the library name in hex to be something like better_<library> 😅), or override. Maybe there are better options?

But I take the point that an application developer isn't usually going to be able to easily understand what went wrong if a sub-dependency is updated 🤔

I also don't know that I would say libraries should never use >=. All tools are good for different things, so I try never to say never 😉

But maybe the best, main option is tooling to help library maintainers know when there are new versions of their dependencies which aren't covered yet.

Also, this is maybe harder, recommending that people have more than one maintainer...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess the main thing I've been frustrated by a number of times in when libraries authors go AWOL, and I guess that's really the problem I'm trying to solve. And really I'm specifically trying to solve the problem of maintainers going AWOL using a dependency when they're using the most stable parts of a dependency... Yes, semver is great when people use it, but updating the major version generally doesn't mean that the whole API of the library changes.

Is incorrect version requirements really the biggest or most common problem you will face when trying to use a library from 5 years ago that is not being maintained more. I think it's more likely that there will be actual incompatibilities to fix and a bunch of other things to fix like deprecations. Are we not optimizing for something rare that the tools will tell you about and is fixed by adding override: true.

On the other hand if there were no version requirements then you would have to figure out the incompatibilities yourself and hope that you have test coverage for it all, including any transitive dependencies.

The options as I see them are a personal fork (which can go out of date), making your own fork (requiring the library name in hex to be something like better_ 😅), or override. Maybe there are better options?

What is the problem with override?

Copy link
Contributor Author

@zachdaniel zachdaniel Dec 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copying my comment from the forum here:

I’m suggesting that this is built into the mix dependency resolver. Right now the following happens all the time:

  • I want to add foo to my app. I already have bar and baz.
  • foo depends on a newer version of bar.
  • I can’t update bar because baz depends on it.
  • I check how baz and foo use bar, and confirm that its fine to just override.
  • So I add {:bar, "~> ...", override: true}
  • Someone else adds buzz to the app, which also depends on an old version of bar.
  • Problem 1: We never find out that we just overrode bar for the sake of buzz as well.
  • Next, foo releases an update that depends on more stuff from the old version of bar.
  • Problem 2: mix tells us we can update foo, so we update it and have bugs.

If instead of override: true, I could say:

{:bar, "~> x.x", override: [foo: "x.x.x"]}

which would say “this override only overrides the dependency that foo at exactly version x.x.x has on bar”, then we are protected from any of those accidental changes.

Adding buzz would produce an appropriate dependency conflict warning, solving Problem 1. We can then go look at the code/docs and decide if we want to override the bar dependency for that version of buzz as well.

foo won’t appear to be automatically upgradeable, solving Problem 2.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I (and Parker for that matter) said "should not", not "must not", exactly because of this.

Yeah... in my mind "should not" can mean "never" or "basically never" 🤷

It is very easy to say what was stable in hindsight, but very difficult to say what will be stable.

I would disagree 😅 I really should spend some time to come up with examples, so maybe it's not worth continuing here, but I can imagine often being confident enough that a specific library's usage specific usage of a specific depedency, especially when that dependency is large enough, can be pretty much relied upon. An example off the top of my head would be using Phoenix.PubSub.subscribe / Phoenix.PubSub.subscribe (especially if you're not using any opts)

Anyway, the PR might not be the best place. I have lots of other thoughts 😉 I'll get back to the forum thread in a bit. Need to finish up the workday!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In those rare exceptions, it is of course up to the library author. However, as the maintainer of many small to large Elixir packages, I reserve the right to break whatever I feel like breaking in a major release, trusting that others have depended on a major version 😅 But I do see where you are coming from. I think it is better to make the guidelines fit the 99% case, even if there can be valid cases for deviation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the issue you described @zachdaniel. My suggestion is simply to say: {:bar, "~> x.x", override: [:foo, :baz]}, then we can:

  1. warn if we need to override anyone else
  2. warn if foo or baz are updated and no longer need the override


The exception to this is pre 1.0 libraries using [Semantic Versioning](https://semver.org/), which provide [no guarantees](https://semver.org/spec/v2.0.0.html#spec-item-4) about what might change from one version to the next. In this scenario, depending on the full patch version, i.e `"~> 0.1.2"` is a better default.

A common mistake is to use a dependency in the format of `"~> x.y.z"` to express "a version greater than `x.y.z`". For example, if you are depending on `"~> 1.2"`, and the dependency publishes a fix in version `1.2.1` that you need for the next version of your library. If you use `"~> 1.2.1"` to express that dependency, you are preventing users from upgrading to `"1.3.0"` or higher! Instead of `"~> 1.2.1"`, you should use `"~> 1.2 and >= 1.2.1"` as the version requirement. This allows users to use any version less than `2.0`, and greater than `1.2.1`.