FIWARE Agile Development Methodology
From FIWARE Forge Wiki
by Juanjo Hierro, Chief Architect and Technical Manager of FIWARE
This page elaborates on how Agile principles are being applied in FIWARE. The approach being adopted has been very much inspired in the work carried out by Dean Leffingwell with Juha‐Markus Aalto on how to apply Agile in development of large systems.
We first advice you should read Dean Leffingwell's and Juha-Markus Aalto's white paper on a lean and scalable requirements information model for agile enterprises ]. Much of the methodology we are following comes from this source. We have further refined some of the definitions, and elaborated some of the concepts, described there in order to manage such a complex and large project like FIWARE.
We assume that you already know the basis about Agile. Information in this page doesn't intend to be a stand-alone and complete tutorial on Agile. Please take this into account.
About Epics, Features, User Stories and Work Items
Entries in Backlogs used in FIWARE can be of any of the following types:
- User Story
- Work Item
- Reported Bug
The Backlog in a product development project is fundamentally about "functionality that has to be developed" in the product. Seeking for clarity, and because I have found it helpful when explaining Agile concepts, I have explained that entries in the backlog indeed map into "work to be done". Actually, a lot of the work you have to do when developing a software product has to do with developing the software that implements the functionality the users expect from your product and it is the description of functionality to be supported in your product what we intend to capture through Themes, Epics, Features or User-Stories. The rest of the work may be categorized as Work Items (creating specific parts the documentation, solving a support ticket, running an analysis or defining part of the API spec before we actually implement it, etc).
Epics, Features or User Stories describe functionality at different level of granularity. Epics describe functionality at a very high level. Epics are further refined into Epics, Features and User stories. When should I label a given backlog entry as an Epic, a Feature or a User-Story ?
User Stories comply with "INVEST" properties which means they should be "Independent, Negotiable, Valuable, Estimatable, Small and Testable". Some resources that may be helpful in finding out what is meant by INVEST can be found at ,  and even  but in general:
- A User Story has to be something sufficiently limited in scope as to be affordable in one Sprint, unitary tests included. ("Small" property)
- A User Story should be detailed enough as to be able to define a test for it. ("Testable" property)
- A User stories should include enough information enabling a developer to make a rough estimation of the resources needed for designing, developing, and testing the functionality within one Sprint ("Estimatable" property).
- There is no need to cover all details nor to have everything finalized before starting actual development. In other words, you shouldn't enter into that level of detail at which you are probably wasting time and unnecessarily delaying development. Just have the details that make you confident that development may fit in the duration of the Sprint. There should be details that may be worked out while developing. ("Negotiable" property)
"Independent" and "Valuable" are also relevant characteristics but I guess there may be many Epics and Features that would fulfill those properties. That's why I would make emphasis on the previous ones.
Therefore, based on the above, you should ask your development teams the following questions to determine whether a new entry you want to insert in the backlog should qualify as an User-Story or not:
- Can you commit to get it developed by the end of the next Sprint ?
- Would you be able to define a test for it ?
If their answer to these questions is "No" or you see the doubt in their eyes ... it's not a User-Story.
Note that labeling an entry as User-Story in the backlog depends very much on a fundamental parameter of your project: the duration of a Sprint. In this respect, Sprints are defined to last one month in FIWARE. What could be labeled as a User-Story would vary a lot if we had defined a different duration for Sprints.
You can consider a Feature just as a special kind of Epic. What Epics can be labeled as Features ? Those describing functionality that you feel confident you will be able to develop in the course of a Product Release. Therefore, labeling an entry as Feature in the backlog depends very much on another fundamental parameter of your project: the duration of a Release. Duration of a Release should be a multiple of the duration of a Sprint and use to be 60 to 120 working days long. In this respect, Releases (also named as minor Releases) in FIWARE last three months. We have also introduced the concept of Major Releases which last a determined number of minor Releases. Therefore, we use a two digit notation to identify a Release (e.g., Release 1.3)
Note that being able to state that you expect a given functionality described as an Epic to be implemented in your product by the duration of a Release simply means that you have enough information about the functionality as to feel confident that you will be able to refine it to the level of User-Stories which, in turn, can be addressed in development sprints, all during the course of the Release.
Why are we dealing with Features in addition to Epics ? Because it gives hints about what functionality we expect to address within a certain period of time and this is important for the overall management of a large product development project like FIWARE and helps to share valuable information with customers about our product roadmap. Users need to have some hints on what functionality is going to be available when, so that they can plan their developments better. Other than this, you may think on Features as just Epics you expect to develop before some given date. Of course, Agile is about re-planning the work to be done whenever you feel necessary and priorities change. That means that a Feature you initially planned for Release 2.1 of your product may get delayed and postponed for a later Release.
Recursive Epics allow to refine abstract conceptual items as much as needed, but is not that critical as the frontier between Epics and Features or User Stories. Deriving a Feature allows us to map the corresponding functionality into a product Release. Deriving a User-Story means you can stop refining and start actual development of a concrete functionality in your product. Whenever you derive a new backlog entry, either as result of refining previously existing Epics or Features or just because it captures new functionality, you should ask yourself: "Is the description of this functionality detailed enough as to make it affordable in the course of a Sprint ? Does it own the desired INVEST properties ?". If the answer is "Yes", then you should label it as an User-Story. If not, then you should ask yourself: "Is the description of this functionality detailed enough as to make it affordable in the course of a Release ?". If the answer is "Yes", then you should label it as a Feature. Otherwise, it is an Epic. When you derive a User-Story directly from an Epic, we recommend to create a Feature with the same description. You may think in a first approach that this is unnecessary duplication of entries, but this will allow to keep a complete description of planned features for all Releases, which is important in large products/systems.
Note than a given Epic may be refined into several entries, each linked to a finer-grained description of the target functionality, but still to be considered as an Epic. Last but not least, an Epic may also be refined into a mixture of several Epics, Features and User Stories. This is natural consequence of first deriving the finer-grained entries and then categorize them properly.
A Work item refers to work that needs to be done in the course of a Sprint, trying to meet a concrete goal. It may relate to work to be done in order to refine an existing Epic (or a feature) and derive actual Features (or User-stories) derived from it. In general, any work you need to address to progress development but may be difficult to describe in terms of "functionality supported by the product". Work Items help to report activities performed by your development team you want to report because it consumes a significant amount of resources and you want to leave a trace of that. They are also extremely helpful in monitoring progress at a higher-level, which is something that is always needed in development of a large product/system.
You may derive a mixture of finer-grained Epics plus a list of Features, User-Stories and Work Items when you refine an existing Epic. Similarly, you may obtain finer-grained Features, plus a list of User Stories and Work Items when you refine an existing Feature.
A Reported bug refers to work needed to fix malfunction of functionality developed. It allows to have them identified and be addressed at a specific sprint or release.
To illustrate this, let's consider a concrete example. You may find out that your system needs to provide some sort of Directory functionality. In a first approach, you may have identified an Epic linked to this functionality. Here you are the description of the Goal linked to that Epic:
- As a user I can create/delete/modify entries in a Directory and then retrieve entries using different methods
You may afterwards turn this Epic into three Features A, B and C:
- A = As a user I can create/delete/modify entries in the Directory.
- B = As a user I can discover entries that match some given criteria, expressed in a query language similar to SQL.
- C = As a user I can subscribe to events linked to creation/deletion/modification of entries in the Directory. Those events are received spontaneously, through a callback service I can register linked to my subscription. Subscriptions may have a lifetime, be paused, resumed and canceled.
(note: you should, of course, provide a more detailed description for each of these features in real life, but let's keep it simplified here for convenience)
You may decide that such Features will be exposed through a well defined API. Therefore, all the three Features may get refined into:
- a WorkItem M = Specifying a well-defined REST interface 'I' through which operations linked to features A and B are exposed
- another WorkItem N = Extending specifications of interface 'I' to include operations supporting feature C
- three UserStories O, P and Q corresponding to implementation of the operations in the REST interface 'I' linked to features A, B and C respectively.
When dealing with realization of the backlog through subsequent Sprints, it may happen that you decided to address WorkItem M in Sprint X of the Release 1.1 and then in Sprint X+1, also part of the same Release, you decided to address the implementation of UserStories O and P, leaving WorkItem N and UserStory Q to the Release 1.2. This illustrates a major concept in how Agile works differently than the traditional waterfall models: we are not waiting for finalizing the complete specification of interface 'I' to start implementing part of it.
Creating the first version of the FIWARE backlogs
In defining how we were going to use Agile in FIWARE, we have had to deal with a fundamental characteristic of the project: it is not about developing from the scratch but from a set of selected products (assets) resulted from previous projects, many of which hadn't been developed using Agile. If we had started development of every FIWARE Generic Enabler (that's the way we name components in FIWARE) from the scratch, the FIWARE GE backlogs would contain Themes/Epics/Features/User-stories that, all together, would summarize the whole functionality of FIWARE. But this is not the case.
When creating the backlog for a given FIWARE GE, we have considered that it should contain the Themes/Epics/Features/User-stories that, at the start of the FIWARE project, map to "functionality to be developed" in the reference implementation of the GE we are building based on a number of baseline products. We won't capture all the functionality that was already implemented in the baseline products. That would mean trying to carry out a kind of "reverse engineering" of baseline products, deriving User-Stories from what those baseline products already support. This, apart from meaning a huge effort would not be be that useful to development teams and would simply delay start of our development. Something which would precisely go against a cornerstone axiom in Agile: trying to do things that are useful for the development teams. Thus, a potential user who wants to have a detailed picture of all target functionality to be supported by a given FIWARE GE should study:
- the FIWARE Product Vision, in order to understand what is overall expected for the GE
- the documentation about the products that have been adopted as baseline for the reference implementation of the GE (available on Materializing the FIWARE Vision): this should give the user a clear picture of what functionality has already been covered and therefore will be supported in FIWARE
- the backlog linked to the GE (also available on Materializing the FIWARE Vision: this will allow the user to understand what's going on and is planned on the roadmap
In summary: the FIWARE backlogs were created to drive developments to be carried out after the FIWARE project started. They are not about documenting what he have done during many years in our respective labs.
Management of the FIWARE Backlog
The FIWARE project uses a Product Backlog to drive the development of the reference implementations of Generic Enablers (GE) in FIWARE. You can find a full description of all the Themes, Epics, Features and User-Stories linked to each FIWARE Chapter and GE visiting the FIWARE Wiki section on Materializing the FIWARE Vision.
FIWARE deals with management of the lifecycle of Backlog entries using FusionForge trackers. Each FIWARE chapter owns a Backlog Management tracker (access only allowed to FIWARE Chapter members):
- Cloud Hosting Backlog Management tracker
- Data/Context Management Backlog Management tracker
- IoT Services Enablement Backlog Management tracker
- Apps/Services Ecosystem and Delivery Backlog Management tracker
- Security Backlog Management tracker
- Interface to Networks and Devices Backlog Management tracker
Any ticket created in a given Chapter tracker is linked to a Theme/Epic/Feature/User-story described on the Wiki. Work Items to be carried out also have a ticket on the proper Chapter tracker. Note that Work Items are not documented on the Wiki. We have taken this decision in FIWARE because Themes/Epics/Features/User-stories are the only Backlog entries that are relevant to FIWARE users.
Sprints and Releases are numbered according to the schema defined for Releases and Sprints numbering
There is a comprehensive set of tutorials explaining FIWARE project members:
- How to assign identifiers to FIWARE Backlog entries (convention to follow)
- How to upload the full description of backlog entries to the Wiki
- How to create and configure trackers in FusionForge
- How to create entries in the "Backlog Management" Tracker of a FIWARE Chapter
- How to follow-up a tracker in FusionForge
Requesting the addition of new entries to the FIWARE Backlog
FIWARE users can issue request for the addition of new Epics or Features in FIWARE, linked to existing Generic Enablers or proposals for new ones. They do so by creating tickets on the FIWARE Epic/Feature Requests tracker
In order to get access to the FIWARE backlog one needs to:
- Register an account at the FIWARE FusionForge
- Join the FIWARE project in FusionForge
- Provide a link (in the ticket) to a complete backlog entry draft in the Uncategorized Enabler backlog on this wiki
- Issue a ticket on the FIWARE Epic/Feature Requests tracker
In the first place, and once it is formally verified (authorized user, formally correct, in the Uncategorized Enabler backlog, etc), the ticket will be handled by the FIWARE Chief Architect, Deputy Chief Architect or any of the FIWARE Chapter leads who may assign a ticket to any member of the FIWARE team. Handling a ticket may involve several interactions between the ticket Issuer, and the FIWARE team. FIWARE team members handle tickets following the workflow provided on How to handle FIWARE Theme/Epic/Feature requests issued by UC projects. As a result of this workflow, a new Theme, Epic or Feature may be added to the FIWARE Backlog. Since tickets and associated tasks are visible to the Issuers and even to external Observers, status and progress is completely transparent.