New, easier ember-try config, for most addons

For those who don't know, ember-try provides the ability to run a command against various sets of dependencies. It is commonly used by Ember addons to run their tests against every version of Ember with which they want to maintain compatibility.

Until now, ember-try has required a configuration file with a "scenario" specified for each set of dependencies that the user wanted to run against. That file can quickly become unwieldy and also become out-of-date as new versions of Ember are released.

For addons that are only varying the Ember version between scenarios, a new feature has been added to ember-try that solves these problems. It allows addons to use a semver range to declare ember compatibility in package.json under "versionCompatibility.ember" under the "ember-addon" key.

For example, in package.json

"ember-addon": {
  "configPath": "tests/dummy/config",
  "versionCompatibility": {
    "ember": ">=1.12.1"
  }
}

ember-try uses this statement to determine which versions of Ember to run against. It looks up possible Ember versions from Github's API, so if a new version is released and if that version satisfies the semver range that is set in versionCompatibility, it will be added to the addon's test runs. ember-try will also add runs for beta and canary versions of Ember, as well as a default run which is the dependency as it is set in bower.json.

To keep the number of test runs sane, ember-try limits the versions that will run to the latest point release in a minor version. For example, with a range of 1.13.0 - 2.0.0, it will run with 1.13.13, 2.0.0, skipping 1.13.0 - 1.13.12.

Advanced Configuration

When using versionCompatibility, configuration can still be set in config/ember-try.js or in another file that is provided at runtime.

If a configuration file containing scenarios is used, it will "win" over anything in versionCompatibility, unless useVersionCompatibility: true is set in that configuration file, in which case any scenarios in the file will be merged with those generated by the semver range. In this way scenarios defined by the semver range can be overridden or altered.

The configuration that ember-try will run with can be seen by running ember try:config.

As an example, running ember try:config with a semver range of 1.13.0, would yield the following:

{
  scenarios: [
    {
      name: "default",
      bower: {
        dependencies: {}
      }
    },
    {
      name: "ember-1.13.0",
      bower: {
        dependencies: {
          ember: "1.13.0"
        }
      }
    },
    {
      name: "ember-beta",
      allowedToFail: true,
      bower: {
        dependencies: {
          ember: "beta"
        }
      }
    },
    {
      name: "ember-canary",
      allowedToFail: true,
      bower: {
        dependencies: {
          ember: "canary"
        }
      }
    }
  ]
}

If an addon didn't want to allow Ember beta to fail, all that is needed is the following configuration file:

module.exports = {
  useVersionCompatibility: true,
  scenarios: [
    {
      name: 'ember-beta',
      allowedToFail: false
    }
  ]
}

Try before you buy

ember try:ember <semver-range> allows trying out the behavior of setting versionCompatibility in package.json without doing so.

Limitations

When using the versionCompatibility feature, addons will want to run ember try:each in CI. This will sequentially run tests with each version of Ember that applies, as described above.

Currently most addons make use of Travis CI's build matrix to parallelize running tests for each Ember version. Using Travis CI's build matrix requires defining each scenario in two places, the ember-try config and in .travis.yml. With versionCompatibility, ember-try is dynamically determining the versions to run and so using the build matrix is not possible.

This unfortunately means slower builds. Most addons will not consider this an issue because their builds will be fast enough. For those addons for which it is an issue, there are a couple of options: Either continuing to specify scenarios like today or using ember try:ember with a few semver ranges. It should also be possible to use the output of ember try:config and manual parallel testing setup on CircleCI to parallelize running of ember try while still using versionCompatibility.

This is an example of the output of a build using versionCompatibility with Ember > 1.12.0 on Travis CI.

Set versionCompatibility even if not using it for ember-try runs

The values set in versionCompatibility will soon be used on EmberObserver.com to allow filtering for compatible versions.

Considerations

Why not use versions from package.json or bower.json dependencies?

Due to the lack of a reliable lock file mechanism, most addons are have the versions in their dependencies locked down, rather than floating. Not using the dependency values also means that addon developers need to take an intentional step of deciding what to support, rather than having support inferred from whatever a package manager or blueprint automatically filled out.