From FIWARE Forge Wiki
FIWARE Platform (GEi) Development Guidelines
This guide is intended to describe the best practices for FIWARE Platform Development, particularly for those projects which deal with the development of a GEi or accompanying module. There are three kinds of guidelines included:
- MUST Guidelines. They are mandatory and your GEi project must conform to that.
- SHOULD Guidelines. They are not mandatory but highly recommended if you want to have a mature development process.
- MAY Guidelines. They are nice to have.
The Guide assumes that the reader knows the basics of Github, Travis, test driven development and continous integration. If that is not the case please read one of the many tutorials available on the Internet.
Summary of MUST Guidelines
- The SCM Tool MUST be Github.com.
- Github MUST be used during the whole development lifecycle. See #Development_Lifecycle which describes a set of best practices for collaborative open source development.
- The repository's README.md file MUST include:
- the following sentence: 'This project is part of FIWARE'.
- a link to fiware.org.
- a link to catalogue.fiware.org (to the section corresponding to the particular GEi). (if the GEi has already an entry in the catalogue).
- A Tracking system MUST be used in order to manage the development work. Such tracking system MUST include at least all the bugs/known issues of your component.
- When a commit is actually solving a bug/issue there MUST be a cross-reference between the commit message and the corresponding bug/issue in the Tracking System.
- Developers oriented documentation MUST be included as part of the Github content.
- Code and documentation MUST be synced. A recommended practice (SHOULD) is that every Pull Request with a direct impact on documentation include documentation changes as well.
- To guarantee that documentation is of high quality, development related documents MUST be peer-reviewed and QA verified. See #Documentation for the best documentation practices.
- Should you want to benefit from FIWARE automatic documentation generation systems you MUST use markdown for documentation and Apiary Blue Print for API specification.
- Files describing the Licensing and Contribution Policy MUST be included in the repository. They MUST be consistent with those reflected in the FIWARE Catalogue.
- Support responsiveness is a MUST. If a feature is not implemented / not available / buggy, an alternative workaround MUST be provided.
- NEW!!! Project repositories MUST include information badges as per the following Guidelines.
- The SCM Tool MUST be Github.com. Accompanying tools or plugins that integrate well with Github (ex. Gerrit) MAY be used.
- Github MUST be used during the whole development lifecycle. What that actually means is that you MUST use Github in your everyday development tasks and not as a mere code snapshot repository. See below.
- The name of the Github Repository MAY be 'fiware' + '-' + "Code Name of your GEi". Example:
fiware-orion. Nonetheless,the repository's README.md file MUST include:
- the following sentence: 'This project is part of FIWARE'.
- a hyperlink to fiware.org.
- a hyperlink to catalogue.fiware.org (to the section corresponding to the particular GEi). (if the GEi has already an entry in the catalogue).
When the GEi code is spread over various repositories, the FIWARE hyperlinks can be included on the README.md pertaining to the main repository or at the GH user/organization level.
- Every repository in Github SHOULD have at least one branch corresponding to the main development line, usually named 'master', and a certain number of additional branches/tags corresponding to releases. Please note that if you need to do bug fixing over previous releases you will implement such a fix over the corresponding branch and later you will uplift the code to the master branch, using
The development lifecycle of a GEi SHOULD be as follows:
1. When developing a new feature / bug try to divide the work into smaller parts. For instance, dividing by front-end and back-end implementation. There can be other criteria that might depend on each feature / bug. The aim of this division is to simplify the review process. Big commits are usually hard to review, hard to change later and prone to conflicts.
2. Once you have divided the work in parts, the implementation of each part should be done in a different branch that you will use later to Pull Request to the GEi Repository. One recommended practice is that you fork the original repository to your GH account and develop the feature in your own branch (which will belong to your own repository). Later, you will make a Pull Request from that branch to the GEi repository. See .
3. The Pull Request should include not only the feature's source code but unit or integration tests together with relevant documentation.
4. At the time of landing (merging), a Pull Request should include only one commit. While developing you may have more than one commit but ultimately only one commit should remain. For doing so, use the git squash functionality. Nonetheless, there can be situations for which a Pull Request might be better explained with more than one commit. In that particular case you should also try to squash your commits into functionally coherent blocks. Nonetheless, every commit message must be descriptive of what feature or bug the code is implementing, for instance:
ISSUE 345. POST Operation /entities. Part I Parsing Requests.
ISSUE 345. POST Operation /entities. Part II DB Persistence.
ISSUE 345. POST Operation /entities. Part III Rendering Responses.
5. Once a Pull Request is done, a code review will be performed. Find a competent reviewer that can perform a code review. There can be more than one reviewer but one of them should be a main developer or technical owner/architect of the project. Typically the reviewer will test the feature, review the code (style, robustness, structuration, performance) and suggest some changes. The code review must be made using the code review facilities provided by Github.
6. Try to address all the comments suggested by reviewers. If a comment is not going to be addressed a rationale should be provided. Remember to squash your changes and to push them to your branch (
git push -f).
7. Wait for a final positive review. This may imply one or more iterations over steps 5 and 6.
8. Land (merge) the new code. But before landing ensure that all tests are passing. Remember to squash commits as needed. Under certain conditions your Pull Request could not be directly landed (merged). In that case you would need to rebase your branch with the master branch, resolve all conflicts, push them and finally merge the new code in the master branch.
9. If later you discover that new code is breaking something do not hesitate to back out the culprit code. That's the advantage of landing only one commit. Backing out code is as easy as
git revert <commit_hash> .
GEi projects developed as open source SHOULD have a public continuous integration system.
The continuous integration system SHOULD be running every time a new Pull Request (PR) is done. As a result the PR owner can know in advance if her code is breaking something.
Before landing new code the continuous integration system MAY be run in a temporary branch that merge the Pull Request and the destination branch. Nonetheless, in the event of continous integration errors the (offending) code MUST be backed out or a fix should be provided urgently. Use git revert for the former.
Travis-Ci MAY be your continuous integration system as it is super-easy to integrate it with Github. Jenkins is another popular tool which has the advantage of being able to run on different OS. Drone.IO is another option which MAY be considered as well, as it has a nice integration with Docker.
Regardless the continuous integration system you MAY use a continuous deployment system as well. Puppet (preferred) and Chef (allowed) are two popular recipe systems to be considered. They can be integrated with Travis.
You MUST include Travis badges (or other badges) in your README.md file. Travis badges can be used to report the current status of your build, (it should be green!).
A Tracking system MUST be used in order to manage the development work. Such tracking system MUST include at least all the bugs/known issues of your component.
The Tracking system SHOULD be public. You SHOULD use Jira as a public tracking tool. For the time being, as the Jira instance of FIWARE is essentially private, you MAY use GH Issues (or a similar tool) as an alternative.
When a commit is actually solving a bug/issue there MUST be a cross-reference between the commit message and the corresponding bug/issue in the Tracking System.
For new features, the GH issue SHOULD include a link to the corresponding feature page on the Wiki (if public). Please DO NOT link Jira entries as they are private. For improving management processes, private Jira entries MAY cross-reference GH issues.
When users report a new bug, it SHOULD be given an estimation of when the bug will be solved. If the bug is consider minor or a very edge case and it will not be resolved, it SHOULD be marked as 'WontFix' and closed. This decision MUST be communicated to the reporter together with a suggested workaround.
The roadmap of your component SHOULD be public.
Be prepared for external contributions. When someone makes a Pull Request be responsive and consider carefully the proposal made. If you want to create community around your component you should be open minded. On the other hand code authored by external contributors should not break the basic design principles of your component. Be prepared for trade-offs.
To avoid documentation inconsistencies, development related documents MUST be handled in the same way than Source Code, this implies:
- Developers oriented documentation MUST be included as part of the Github content.
- MarkDown (.md) is the recommended format for document files. Restructured Text (reST) might be used as well.
- Additional developer oriented documentation (advanced topics) MUST be present in a '/doc' folder at the root of your repository (or in an specific documentation repository associated to the GEi). It is noteworthy that you MUST use markdown format if you want to benefit from the automatic documentation generation system that will be provided by FIWARE.
- Code and documentation MUST be synced. To this aim, every Pull Request with any impact in existing documentation SHOULD include any related documentation changes.
- Inconsistencies or lacks of documentation SHOULD be detected in Code Reviews and QA phases, opening bugs when necessary.
- A README.md MUST be always present in the root folder of any repository associated to the GEi. The purpose of such a document associated to a GEi is to document:
- GEi overall description.
- How to Deploy the GEi (basic/default installation procedure)
- How to run tests
- A walkthrough guide on using the main APIs
- How to get access to the advanced API and Documentation topics
More specifically, the referred README SHOULD include:
- A Table of Contents to make navigation through the document easier
- Simple GE/Service Description and purpose
- Include information about testbed environments if available
- How to Build & Install
- Make your simplest full stack deployment as easy as possible
- Include System requirement info: SO, CPU/Storage Capacity...
- Include installation support for your dependencies
- Provide a “Hello World” example using curl (basic acceptance test)
- Include troubleshooting information for the whole process
- API Overview of the main data flow
- It is a tutorial, not a reference. It does not need to be exhaustive, it needs to guide the user
- Provide always curl examples for this section
- More API Examples
- Just the important bits of your API used in examples
- A link to the API Reference Documentation. Apiary Blueprints SHOULD be used. We recommend the usage of Apiary if you want to benefit from tools (to be released soon) that will allow to automatically generate FIWARE specifications.
- How to run tests
- End to End tests (MUST). This will be part of the sanity checks included in the "Installation & Administration Guide", thus a link will be needed.
- Unit tests (SHOULD)
- Performance tests (MAY)
- List of links to Advanced topics (/doc folder)
- User & Programmers Manual (MUST)
- Installation & Administration Guide (MUST)
- Other documents, for instance: HA deployment, detailed architecture, Advanced configuration topics, advanced functionalities, ...
GEis SHOULD be configurable through a config file.
If the GEi can be configured through a config file a template/example configuration file MUST be present and fully aligned with every branch.
The config file must be seen as part of the documentation (for integrators and deployers). So it is mandatory to document every config param, its available options and operation advices. Redis configuration is an illustrative example.
Software Releases (source code)
Each code release SHOULD be properly tagged in your Github repository.
Each component should always show a consistent view in the “Releases” and “Tags” tabs at GitHub.com. Examples:
The release notes SHOULD include links for downloading the source code.
The release notes associated to each Release SHOULD include the changelog for that release, i.e. the list of changes regarding the previous version, including (if possible) links to the GitHub issues related to each change.
GEis MUST provide Docker containers according to Docker Guidelines.
Other Binary release systems are to be defined.
Additional Repository Content
Files describing the Licensing and Contribution Policy MUST be included in the repository. They MUST be consistent with those reflected in the FIWARE Catalogue.
Stackoverflow SHOULD be used for thorough technical questions. Educate your users to tag their questions properly in Stackoverflow.
Each FIWARE GEri MUST be associated to an Stackoverflow tag. Such tag could be specific for a GEri or default to a chapter generic tag, for instance <fiware-iot>.
Stackoverflow tags SHOULD be of the form [fiware-<GEi codename>]. Example fiware-orion. Creating new tags requires Stackoverflow Karma(this has nothing to do with the karma we use internally in FIWARE). You MAY ask someone from the FIWARE team with enough Karma to create tags for your GEi. Otherwise you can use the FIWARE Q&A platform.
- Exception to the rule: In the case the GEri is associated to an existing tag (e.g. the tag was created before that GEri joins FIWARE), it could be used instead of the usual fiware-XXXX as an exception, as long as the two following conditions occur: 1) the tag clearly reflect the name of the GEri, 2) the tag is already widely used in StackOverflow (i.e. more than 30 questions using it). Anyway, this is a workaround valid only in the case a tag synonym cannot be created due to reputational constraints.
New releases of your components SHOULD be announced through different channels: FIWARE Blog, dedicated Twitter or similar channels (TBD), your own Blog ...
Support responsiveness is a MUST. If a feature is not implemented / not available, an alternative workaround MUST be provided. Afterwards it can be decided whether it is necessary / worth adding such a new feature to your backlog.
These Guidelines enable specifically external contributions coming from the community (i.e. people who are not part of the regular staff assigned to each GEri project). They have been inspired on real experience gained while working in the Mozilla Open Source Community. See  and .
A public backlog including potential work items (bug, features, documentation, test cases, ...) to be externally contributed (welcome contributions) MUST be available.
A work item is a good candidate to become a welcome contribution if:
- Should be fixed
- It is not time-sensitive or blocking
- It has clear steps to reproduce
- It is well understood
Welcome contribution work items MUST have a mentor assigned (who will typically be a member of the regular GEri staff). The Mentor's role includes:
- She will ensure that the right contributor is assigned.
- She understands how it can be done and will provide guidance on the best way to approach the problem.
- She will answer questions, review changes, and make the process as easy as possible by reacting promptly.
- She will make sure that a task is progressing. If nothing is heard from a contributor for a reasonable period of time the mentor might unassign the task.
- After a successful contribution, she recognizes the contributor and propose new work items to be accomplished, as now a mentor should have a good sense of the contributor's capabilities.
The public backlog MUST provide a mechanism to allow assignment of work items to external contributors. Contributors who already come with a proposed patch MUST be given preference. Assignment will be subject to approval by GERi owners, particularly mentors. Github users activity profile MAY be used in order to assess the skills of the potential contributor.
For those projects using Github as a public backlog, work items will be 'Github issues' tagged as 'Welcome Contributions'. The assignment process will consist of a comment made by the contributor on the corresponding Github issue, indicating the intent to fix the issue. A mentor can decide to acknowledge the request by actually assigning the issue. Nonetheless, the Mentor will have the final assignment decision depending on the proved skills of the requester.
Spontaneous contributions (i.e. contributions not associated with any welcome contributions published by the GEri owners) MAY be accepted if the GEri development staff consider them valuable.
Contributions MUST follow the standard FIWARE "Development Lifecycle" and the advices given by mentors. The contributor's role includes:
- Regular communication (at least once a week) with the mentor; the mentor needs confidence that the task is moving in a good direction.
- Respect the mentor's guidance, particularly with regards to patch submission and review.
Credits MUST be given to external contributors. A CREDITS file MUST be present at the root directory of the GEri code repository, including one line per contributor to the project. An example can be found here.
The public backlog organization for Welcome Contributions SHOULD be as follows:
- All the work items which are suitable for external contributions (Welcome Contributions) will be tagged as 'mentored'.
- A good Welcome Contribution requires:
- A clear description of the problem, and steps to reproduce (if applicable)
- A broad description of what an acceptable solution to the problem would look like
- Clear steps that should be taken for the fix - including links to the lines of code to be tweaked.
- Identification of any relevant automated tests, as well as instructions for adding any new tests
- Links to relevant documentation (eg. if running automated tests, link to the page and even give the full command).
- Every Welcome Contribution will have a category which will denote its difficulty and importance level:
- Entry. Entry level work items are typically low complexity tasks such as those which have to do with simple (one liner) bug fixing. It includes an extremely narrow scope, clear hardware and platform requirements, and prompt reviewer followup. They are aimed at making a new contributor familiar with the development environment and workflow.
- Intermediate. These are less trivial tasks and are focused on actually providing a solution to an issue identified. Few if any of these will be good starting points for new contributors. Actually, they are aimed at those contributors who have completed successfully at least one or two entry level work items.
- Advanced. These are work items which difficulty is high or that have been brought up as important issues by the priority-triage processes. Previous experience in the project will be a must in order to be commissioned to the task.
Apart from the presence on the credits file, different recognitions can be given to external contributors. Some ideas, coming from the Mozilla Community can be found at . The greater recognition that can be made to a contributor is promotion to the 'active contributor' role (see []). These decision MUST be made by the GEri owners taking into account the number, quality and sustainability in time of the contributions provided. The FIWARE Bounty Programme [] will be an additional instrument to award contributors.
Should you have any doubt please contact José Manuel Cantera or Miguel Carrillo