Skip to content

Commit

Permalink
IMPROVE: rewrite schemas in YAML (#123)
Browse files Browse the repository at this point in the history
# General
* VERSION: v0.21.0
* IMPROVE: rewrite schemas in YAML (#123)
* CHANGE: minor changes to `description` fields of schemas
* DOCS: update CONTRIBUTING.md (see #127)
* DEV: add build script (`npm run build`)
* DEP: azure-storage@2.10.1
* DEP: jasmine@3.2.0

# Location
* CHANGE: reference GeoJSON schema from DLx directly, rather than schemastore.org

# Orthography
* FIX: type for "name" field incorrectly set to "string" instead of "object"
  • Loading branch information
dwhieb authored Sep 29, 2018
1 parent 14930c9 commit 65be065
Show file tree
Hide file tree
Showing 106 changed files with 5,450 additions and 1,898 deletions.
1 change: 1 addition & 0 deletions .eslintignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
schemas/json/index.js
155 changes: 123 additions & 32 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,82 +2,173 @@

:star2: Thank you for contributing to DLx! :star2:

Below is some information on how you can contribute to the DLx data format, and how to get started.
Below is some information on how you can contribute to the DLx data format project, whatever your level of technical expertise.

* [General Guidelines for DLx Projects][1]
## Quick Links

[Open an Issue][8] | [User Documentation][3] | [Pull Requests](#contributing-code--changes-to-the-schemas) | [DLx Contributing Guidelines][1] | [Code of Conduct][2]

## Contents

* [General Guidelines for DLx Projects](#general-guidelines-for-dlx-projects)

* [Questions?](#questions)

* [Suggesting Features](#suggesting-features)

* [Reporting Bugs & Other Issues](#reporting-bugs--other-issues)

* [Contributing Code / Changes to Schemas](#contributing-code--changes-to-the-schemas)

## General Guidelines for DLx Projects

Here are some general guidelines that apply to all DLx projects:

* [Code of Conduct][2]

* [Contributing Guidelines][1]

## Questions?

Just have a question you need answered? Consider [joining the DLx Slack channel][3], where you can chat with other users and developers about various DLx projects.
Check out the [user documentation][3].

You can also ask a question by [opening an issue in this repository][4].
You can also ask a question by [opening an issue in this repository][8].

## Reporting Bugs & Other Issues

Found a problem in the specification? [Open an issue on GitHub][8] describing the problem and its severity. Does the issue affect just a single schema, or several schemas? Does the issue have the potential to cause errors in applications using data in this format? Include as much detail as possible.

## Suggesting Features

Have a feature you'd like to suggest? The project would especially benefit from the following types of suggestions (other suggestions are fine too):

- Is there some part of the specification that could be structured more simply, without losing any information?
- Is there some part of the specification that could be structured more simply, without losing any information?

- Is there a use case, property, or other piece of information that linguists commonly use that hasn't been included in the specification?
- Is there a use case, property, or other piece of information that linguists commonly use that hasn't been included in the specification?

- Are there any parts of the specification that are unclear or could be made clearer?
- Are there any parts of the specification that are unclear or could be made clearer?

- Is there a type of linguistic object or data that isn't included in the specification?
- Is there a type of linguistic object or data that isn't included in the specification?

- Can the documentation be improved?
- Can the documentation be improved?

To suggest a feature, simply [open an issue in the GitHub repository for this project][4], and explain your suggestion. You should provide an example in JSON of how your suggestion would work, and if possible a valid [JSON Schema][5] describing the new format.
To suggest a feature, simply [open an issue in the GitHub repository for this project][8], and explain your suggestion. If you are able, consider providing an example in JSON of how your suggestion would work, and if possible a valid [JSON Schema][5] describing the new format.

If you're suggesting an improvement to the documentation, please include the original wording and how you would change it, or at least a short description of what needs to be changed.

When suggesting a feature, think about whether this would require a major, minor, or patch update to the specification (following [semantic versioning principles][6]), and include that in the comments for your issue.

## Reporting Bugs & Other Issues
## Contributing Code / Changes to the Schemas

If you'd like to contribute changes to the code or the schemas in the repository, follow these steps:

1. **[Open an issue][8]** with information about the feature, fix, or change you'd like to make. Follow the guidelines in the [Suggesting Features](#suggesting-features) section above.

**Note:** It is a good idea to wait until your suggested change is approved by a maintainer before writing any code.

1. **Fork the repository** and clone it to your computer.

1. **Set up your development environment** by [installing Node.js][4].

1. **Install project dependencies** by running `npm install` in the project folder from the command line.

1. **Update the documentation** (`README.md`, `CONTRIBUTING.md`) to reflect the changes you plan to make.

1. **Write or update tests** for the changes you are going to make.

- The tests are located in the `/test` folder. Each schema has one test file where all its tests are located. For example, the tests for the Language schema are located in `test/Language.test.js`.

- Tests are run using [Jasmine][9] for Node.js. See the [Jasmine documentation for Node.js][10] for more information on how to write tests in Jasmine.

- The tests use the [`ajv` library][7] to validate the schemas against the JSON Schema format, and validate the sample data against the DLx schema. See the [`ajv` documentation][7] for more information on how to use this library to validate schemas.

- The file `schemas.test.js` checks that each schema is a valid JSON Schema. You should not need to change this file.

- Each schema should be validated against *both* valid and invalid sample data. Each test file contains some valid sample data in the `data` variable near the top of the file. Update this sample data to match your proposed changes.

- The first test in each file validates the sample data against its schema using `ajv`. You should not need to change this test.

- The remaining tests in each file check that the sample data does *not* validate against the schema if the data is formatted incorrectly. Each test generally checks one attribute in the schema. Update these tests as needed.

1. **Make changes to the schema(s)**.

- The schemas are located in `schemas/yaml`. Do not update the schemas in `schemas/json`, or your changes will be overwritten.

- The schemas use the [JSON Schema format][5] to describe how data should be structured. See the [latest version of the JSON Schema][11] for more details.

- Use the file `schemas/template.yml` as a guideline / model for other schemas.

- Update the `"description"` field of any schemas that you change, as needed.

- Increment the version number of any schemas that you update, following [semantic versioning][6].

- `major`: When the schema is changed in a way that makes it incompatible with previous versions

- `minor`: When new properties are added, or optional properties removed, or other changes are made to the schema that are backwards-compatible

- `patch`: Typos, changes to descriptions, minor bugs and hotfixes, or other changes that do not address the functionality of the schema

1. **Run the tests** using `npm test` in the command line. Fix any issues that arise.

1. **Double-check the documentation** (`README.md`, `CONTRIBUTING.md`) to make sure it still reflects the changes you made.

1. **Get the latest changes from the `master` branch** and resolve any conflicts that arise. Run the tests again.

Found a problem in the specification? [Open an issue on GitHub][4] describing the problem and its severity. Does the issue affect just a single schema, or several schemas? Does the issue have the potential to cause errors in applications using data in this format? Include as much detail as possible.
1. **Run the build script** by running `npm run build` from the command line. This does the following steps for you automatically:

## Making Pull Requests
1. **Generate the JSON versions of the schemas** (`npm run convert`)

If you'd like to make a pull request and contribute to the code for the DLx specification, first open an issue with information about the feature or fix you'd like to make, following the guidelines in the [suggesting features](#suggesting-features) section above. It is a good idea to have your feature request or bug fix request approved by a maintainer before writing any code.
1. **Regenerate the documentation website** (`npm run docs`)

Once an issue is made and has been assigned to you, you can follow the steps in the [maintainer's guidelines][7] to prepare your pull request (not all of the steps may be applicable).
1. **Run the tests again** (`npm run test`)

For this project in particular, you will want to include the following steps:
1. **Open a pull request**. See GitHub's advice on [How to write the perfect pull request][12].

1. Fork the repository and clone it to your computer.
1. **Address any changes requested by the reviewer**.

1. Install project dependencies by running `npm install` in the project folder from the command line.
1. **Wait for a maintainer to merge your changes!**

1. Increment the version number in the `version` field of `package.json`.
_These steps are for maintainers only:_

1. Increment the version number(s) of the particular schemas you'll be updating, if any.
1. **Increment the version number** of the branch by running `npm version {major|minor|patch} -m "Your commit message"` from the command line.

1. Update any existing documentation (e.g. the readme, or the `description` property of a schema) so that it reflects the changes you'll be making.
- Include a note closing the associated issue (e.g. `closes #167`).

1. In the `/test` folder, find (or create) the test file for the schema(s) you're making changes to, and update the tests with valid (and invalid) sample data to test against your schema. The tests use the [`ajv` library][7] to validate data against schemas, and are run in Node.
1. **Run upload script**: `npm run upload`

1. Make any necessary changes to the schemas. If adding a new schema, place it in the `/schemas` folder, and name the file with the name of the object, in SnakeCase. Make sure that object schemas have a `type` property.
1. **Open a pull request** into the `master` branch, and include the release notes in the comments.

1. Run `npm test` from the command line to check that the schemas are valid and that your new / updated tests pass.
1. **Squash & merge** the PR into the `master` branch.

1. Update any documentation again, if needed.
1. **Create a release**

1. Regenerate the documentation by running `npm run docs` from the command line.
- Title: vX.X.X

1. Add commit message that closes the related issue (e.g. `closes #167`).
- List the changes for each schema separately under its own heading. General project changes get their own heading as well.

1. Commit and push
- Each change should have one of the following labels:

1. Open a pull request into the `master` branch, and include the release notes in the comments.
- CHANGE: (breaking) changes
- DEP: changes to dependencies
- DEV: changes to development environment
- DOCS: changes to docs
- FIX: bug fixes
- IMPROVE: code enhancements
- NEW: new features
- REMOVE: removed features
- TEST: changes to tests

1. Address any changes requested by the code reviewer in your pull request.
1. **Publish to npm**: `npm publish`

[1]: https://github.com/digitallinguistics/digitallinguistics.github.io/blob/master/CONTRIBUTING.md
[2]: https://github.com/digitallinguistics/digitallinguistics.github.io/blob/master/CODE_OF_CONDUCT.md
[3]: https://slack.digitallinguistics.io/
[4]: https://github.com/digitallinguistics/spec/issues/
[3]: https://digitallinguistics.github.io/spec/
[4]: https://nodejs.org/en/
[5]: http://json-schema.org/
[6]: http://semver.org/
[7]: https://www.npmjs.com/package/ajv
[8]: https://github.com/digitallinguistics/spec/issues/new
[9]: https://jasmine.github.io/
[10]: https://jasmine.github.io/setup/nodejs.html
[11]: http://json-schema.org/latest/json-schema-validation.html
[12]: https://blog.github.com/2015-01-21-how-to-write-the-perfect-pull-request/
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,8 @@ Following the recommended data format in your own project is as easy as making s
Note that most schemas have a strongly-recommended (but optional) `type` property indicating the schema that that object adheres to.

## Want to Contribute?
Check DLx's [general contributing guidelines][18].

Check out the [Contributing Guidelines][18] for this project.

## Maintainers
This repo is maintained by:
Expand All @@ -215,7 +216,7 @@ This repo is maintained by:
[15]: http://developer.digitallinguistics.io/spec/schemas/Reference.html
[16]: http://developer.digitallinguistics.io/spec/schemas/URL.html
[17]: http://developer.digitallinguistics.io/spec/schemas/Location.html
[18]: https://github.com/digitallinguistics/digitallinguistics.github.io/blob/master/CONTRIBUTING.md
[18]: https://github.com/digitallinguistics/spec/blob/master/CONTRIBUTING.md
[19]: https://github.com/dwhieb/
[20]: https://slack.digitallinguistics.io/
[21]: https://github.com/digitallinguistics/spec
Expand Down
80 changes: 80 additions & 0 deletions build/convert.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
/**
* Converts all the YAML versions of the DLx schemas to JSON,
* and saves them to /schemas/json
*/

/* eslint-disable
no-shadow,
*/

// IMPORTS
const path = require(`path`);
const yamljs = require(`yamljs`);

const {
readdir,
readFile,
writeFile,
} = require(`fs`).promises;

// VARIABLES
const schemasPath = path.join(__dirname, `../schemas`);

// METHODS

/**
* Reads a schema from a filename and returns an Object containing "name" and "schema" attributes
* @param {String} filename The filename of the schema to read
* @return {Object}
*/
async function readSchema(filename) {

const schema = await readFile(path.join(schemasPath, `yaml`, filename), `utf8`);

return {
name: filename.replace(`.yml`, ``),
schema,
};

}

/**
* Writes a converted schema object to a JSON file in the /schemas/json folder
* @param {String} name The name of the schema
* @param {Object} schema The schema as a JavaScript Object
* @return {Promise} Returns a Promise that resolves when complete
*/
async function writeSchema({ name, schema }) {
const schemaPath = path.join(schemasPath, `json/${name}.json`);
const json = JSON.stringify(schema, null, 2);
await writeFile(schemaPath, json, `utf8`);
}

// TOP-LEVEL SCRIPT
void async function convert() {

// Read all the schemas from their YAML files
const filenames = await readdir(path.join(schemasPath, `yaml`));
const schemas = await Promise.all(filenames.map(readSchema));

// Parse each schema into JavaScript
schemas.forEach(schema => {
// eslint-disable-next-line no-param-reassign
schema.schema = yamljs.parse(schema.schema);
});

// Write the schemas to /schemas/json
await Promise.all(schemas.map(writeSchema));

// Update schemas/json/index.js
const lines = schemas
.map(({ name }) => name)
.map(name => ` ${name}: require('./${name}'),`)
.join(`\n`);

const indexText = `module.exports = {\n${lines}\n};`;
const indexPath = path.join(schemasPath, `json/index.js`);

await writeFile(indexPath, indexText, `utf8`);

}();
Loading

0 comments on commit 65be065

Please sign in to comment.