This is the odd idea to first publish an unfinished flow of consciousness to try to get some feedback before investing the time into a full blown medium article. Offer criticism, improvement, ideas and experience as you like. Or already use what ever you find useful now. But higher level stack planning is really falling under the table lately. So I update this post consecutively.
Title: 1.3 App Structure with Screens & Components: Get ready for A/B/Multivariate testing and continuous improvement
Technology: Meteor 1.3, React
Keywords: Meteor, Continuous Improvement, Testing, Variance Testing, Application Structure, Workflow
// Working Paper, v0.1 - 03-17-2016 (thrown together out of internal communication)
This is a longer article about an appropriate folder and file structure, incl. conventions, for successfull continious improvment and A/B/Multivariate testing in Meteor 1.3 React websites & apps. A mathematical / data driven way to find out if a nav-bar should be on the right or left side, or if the background should be green or red to improve a certain goal. Here is what I got so far.
I often see startups struggle to prepare to validate design decisions - or at least - proof that they could do design validations to VCs and other sources of capital. Preparation in this case means to allow the tech stack to
- isolate visible (screen) variations
- dynamically map users and allow them “to be beta”
- have a workflow where developers can work on variations
- without having merge conflicts all the time
In the most extreme way, you could create a zillion app variations and have them delivered by a sticky load balancer. However, this is incredible intransparent and a real challenge for multivariate testing. Especially if there is no dev ops guy doing some automation with a statistical background.
I have now advised - for the first time - not to use webpack and rely on Meteors new module system. So I needed to define a Project Structure that allows continuous improvement on multiple levels
- server & services
- api
- ui
- ui elements
- build targets
- +Variations of all of those
while beeing testable and easily enforced. Not very easy.
I have done those kinds of “best practice” boilerplates a lot to counteract or reduce the fatigue of a fresh dev team. Give them recipes and don’t allow them to introduce unnecessary complexity.
With react, there is a notion that everything is big tree of smart or dumb components. That sounds so much simpler than helpers, templates and weakly defined controllers. While that is true, it does not tell you anything about how to weigh the structure for different kinds of components.
And there are so many different kinds: function, relation, feature, user story, load order, coupling, section, complexity…
I did a lot of research, especially within the react community, to find an approach that proofs solid for parallel development. But before talking about that, let’s take a look at the current structure of the official Meteor 1.3 react todos example (react branch): https://github.com/meteor/todos/tree/react
This repo is as much in progress as this document. It may change a bit to the following illustrations.
Official Meteor 1.3 todos App Structure
The folder folder current folder structure looks like this
apache
.
├── client/
│ ├── main.html
│ ├── main.js <--------------------------------+
│ └── main.less+---------+ Single |
├── i18n/ | | import to |
│ └── en.i18n.json v v main.js |
├── imports/ Styles in client |
│ ├── api/ in UI/ |
│ │ ├── lists/... |
│ │ └── todos/... |
│ ├── startup/ |
│ │ ├── client/ |
│ │ │ └── routes.jsx +--------------------+
│ │ └── server/
│ │ ├── fixtures.js +--------------------+
│ │ ├── register-api js +----------------|
│ │ ├── reset-passwo d-email.js|---------|
│ │ └── security.js +--------------------+
│ └── ui/ |
│ ├── components/... |
│ ├── containers/... |
│ ├── helpers/... |
│ ├── layouts/... |
│ ├── pages/... |
│ └── stylesheets/... |
├── node_modules/... |
├── packages/... |
├── public/... 4 imports to |
├── resources/... main.js in |
├── server/ serVer/ |
│ └── main.js <--------------------------------+
├── tests/
├── README.md
├── circle.yml
├── mobile-config.js
└── package.json
- everything lives under the
imports/folder - it has three main parts: api, ui, startup
- files are imported to the server or client by referencing them in a
main.jsfile in the client and server - in this structure, the
server/main.jshas various imports - the
ui/folder splits between types of react components: components, containers, layout and pages
It is probably quite obvious that running tests (variations of containers, components and pages) would be quite a headache. Also, since there is no scoping of files into “features”, the list of a chat window and the list of a blog element could collide.
The NEW “Screens & Components” 1.3 React App Structure
The idea of the following structure is to facilitate the addition of aspects of the app. Since aspects of an app can be defined as different sections (or routes) within an application, the _screens/ folder follows the apps route in its directory structure.
I have forked and adapted the official react todos example to the following illustration. Check it out:
https://github.com/D1no/todos (default Branch: react-screen-component)
The app structure looks now like this:
.
├── client/
│ ├── main.html
│ ├── main.js #import "./imports/client.entry";
│ └── main.less #@import "{}/imports/style.entry.less";
├── i18n/
│ └── en.i18n.json
│ +---------------------- Main Dev Folder -----------------+
│
├── imports/
│ ├── #components/ # UI Components & Extensions
│ ├── _screens/ # Compositions of UI Components
│ ├── api/ # Collections, Methods and Pub/Sub
│ ├── lib/ # Helpers & former Meteor Packages
│ ├── server/ # Strict security related files
│ ├── stylesheets/ # Global styles (if used)
│ ├── client.entry.jsx +--->
│ ├── server.entry.js |---> Loader agnostic / entries
│ └── style.entry.less +--->
│
│ +--------------------------------------------------------+
├── node_modules/...
├── packages/...
├── public/...
├── resources/...
├── server/
│ └── main.js #import "./imports/server.entry";
├── tests/
├── README.md
├── circle.yml
├── mobile-config.js
└── package.json
Mainly, the imports/ is now the full center of attention and provides only one single file per build target/entity for reference. This facilitates clutter to the outside directories and allows to just switch out the meteor build tool for example the webpack:webpack alternative (referencing the client, server, etc entries).
Also, the imports folder hierarchy is now much flatter.
Screens follow Routes
The _screens/ directory follows the apps router.jsx, which leads to natural scoping of files and an easy way for multiple devs to work on different features without facing pre-mature merge conflicts.
...
imports/
├── #components/...
├── _screens/ <---------------+ Directory structure
│ ├── _shared/ follow routes. Routes
│ │ ├── _auth.jsx follow features.
│ │ ├── _auth.less
│ │ ├── _notFound.jsx
│ │ └── _notFound.less
│ ├── join/
│ │ └── Join.jsx
│ ├── lists/ +-----+ Feature Lists (Specific)
│ │ ├── ListsContainer.jsx |
│ │ ├── _Header.jsx | lists are a Feature
│ │ ├── _Header.less | of the App and are
│ │ ├── _Lists.jsx | made available under
│ │ ├── _Lists.less | the app.com/lists route.
│ │ ├── _TodoItem.jsx |
│ │ └── _TodoItem.less | Acceptance Test that
│ ├── signin/ +-----+ lists do work as expected
│ │ └── SignIn.jsx
│ ├── _App.jsx +-----+ Feature App (Global)
│ ├── _App.less |
│ ├── _ListList.jsx | feature of using
│ ├── _ListList.less | routes for navigation
│ ├── _Menu.jsx | welcome messages,
│ ├── _Menu.less | connectivity status
│ ├── AppContainer.jsx | etc.
│ └── routes.jsx +-----+
├── api/...
├── lib/...
├── server/...
├── stylesheets/...
├── client.entry.jsx
├── server.entry.js
└── style.entry.less
...
Neglecting the awesome React chrome extension, it’s now pretty straight forward for a new developer - tasked to “change the lists background to blue” - to find the relevant sceen and ui components without knowing the full application. The only thing he has to know, is that he needs to follow the directory structure according to the _screens/routes.jsx.
import React from 'react';
import { Router, Route, browserHistory } from 'react-router';
// route components
import AppContainer from './AppContainer.jsx';
import ListContainer from './lists/ListsContainer.jsx';
import SignIn from './signin/SignIn.jsx';
import Join from './join/Join.jsx';
import _notFound from './_shared/_notFound.jsx';
export const renderRoutes = () => (
<Router history={browserHistory}>
// Dictating folder structure
<Route path="/" component={AppContainer}>
<Route path="lists/:id" component={ListContainer}/>
<Route path="signin" component={SignIn}/>
<Route path="join" component={Join}/>
<Route path="*" component={_notFound}/>
</Route>
</Router>
);
The routes.jsx is a key file within the project and is the Nr. 1 source of merge conflicts for new features and parallel programming.
Variable & Filenaming Conventions
for now copyed from the readme.md of the https://github.com/D1no/todos
- Each react component file is
-
UpperCase if it exports a
Class -
lowerCase if it exports a
constor afunction - A
screencomponent is prefixed with an_underscore, if it is a partial screen element that either belongs to the next non-_upper-level screen component or is shared among various, down level screens (like a _layout variant) inside a_shared/folder - Variables have to be declared, default exported and imported as the file name.
For example, you now easily understand the role and location of various components
<div className="page lists-show">
<_Header list={list}/>
<div className="content-scrollable list-items">
{loading ? <message title="Loading tasks..."/> : Todos}
</div>
</div>
Note: This structure paradigm resulted out of a recent, large, assisted Meteor 1.3 restructure. This example illustrates the main points for discussion and further adaption. The _ and # proof helpful to quickly spot relevant components in long import trees and enforce a semantic order.
Multi-Dev Workflow
Assuming an agile, user story driven workflow, it is common to split tickets (user stories) among mutliple developers to get going quickly. A user story can be
- an improvement to an existing feature
- or the addition of a new one
Going back to the original project structure with a ui folder, it is clear that multiple devs will add and change files in every folder. Always. With the “Screens & Components” structure, a new dev could create a feature - for example a dashboard - by creating a new dashboard folder under the _screens/ folder and recycling existing components available to the app. Probably adding (rather than changing) new ones as well.
The “dashboard developer” adds the dashboard to the routes.jsx as a new route:
(...)
<Route path="dashboard" component={Dashboard} />
(...)
This is probably going to be the only merge conflict for the integration of the feature and a good hook-moment for QA.
Unit & Acceptance Testing
ToDo: Unit tests live next to components. Feature / acceptance tests live within the _screens folder next to its feature domain tested with chimp. Discussed here: [Solved]: Meteor acceptance test
Value of Variance Testing (A/B/Multivariate)
Variance testing is a big part of proper software development and the interface of the business guys and the techs. And usually the part where arguments end and start between an CEO and CTO or a Product Owner and a Tech Lead.
Therefore, having a transparent variance testing set-up that is well understood from sales to junior dev is the corner stone of a great product development. Because the user feedback of the variance needs to flow back from the customer, through the people and mapped successfully to the test cases and resulted data.
One of the most obvious ways you know it works is this:
Sales: My customers hate the beta and want to go back to the old system!
It’s clear to which variation the feedback belongs to: Some beta! This is a very rough A/B test - you know now that B sucks but without some more qualitative information from the sales guy, you will need an intern sorting and aggregating all the praise and complaints to get some insight. And still, you may just listen to the loudest voice.
So it makes sense to use 4 different “Betas”. Beta1, Beta2, Beta3, Beta4… with lesser variations so you know what probably sucks, if all the sales people of “Beta3 Customers” hear “they hate the beta and want to go back!”. Further splitting up those groups and variations leads you at some point to testing id’s or testing objects. Google runs hundreds of testing objects in parallel to validate variation models.
But they can do that because they don’t have to worry about
- enough traffic per-variation to make meaningful, statistical comparisons (significance)
- having benchmark conversion rates out of past experience
- about a large number of goals (in fact a relative low number of goals - talking about the search engine)
However, beeing able to easily do variation testing for a start up means
- less technical considerations and debates (just say whatever and test it)
- on average, the risk profile of users / pages views is limited and weighted
- it forces to state a desired improvement (formulate a goal - hint: “easy to use” is a pretty hard to measure goal)
- learning quickly what engages users
Variations on the Application Level with the Screens & Component Structure
Its simple. In order to do variance testing you have to create variations. Now think about your application, stack or thought out app. How do you test if a green login button is better than a blue one?
Well, you maybe got some ideas already. Its React. So we could create a button with a property color and have our test variation randomly decide to render with a 50:50 chance either green or blue.
(...)
<Button color={ Math.floor(Math.random() * 2) == 0) ? 'green' : 'blue' } />
(...)
A user will now randomly see a green or blue button. You could track this variation with some self written code, Google Analytics or other third parties (more on that later).
Now I want to also test a lighter background color of the text field, round corners of the dialog and a different font with the green option. And I want to add a pink option now. And now it is getting tricky: We have to inject in all those places (components) variation generators - that stay consistent - and track those outcomes. Very hairy.
What we can do instead: We track variations of the user experience. And the user experience is strongly defined by our routes (the beauty of dynamic applications) and reflected by our _screens/ directory tree.
So in the case of the todos example, we simply copy our login screen. Three times. And create our variations by changing
...
├── _screens/
│ ├── _shared/... Starting point:
│ │
│ +-------------------- Our Join Feature --------+
│ | │ |
│ | ├── join/ Lets run 3 variations to |
│ | │ └── Join.jsx validate the login flow. |
│ | │ |
│ +----------------------------------------------+
│ │
│ ├── lists/...
│ ├── signin/...
│ ├── ...
│ └── routes.jsx
...
To this:
...
├── _screens/
│ ├── _shared/... With 3 Screen Versions
│ │
│ +-------------------- Our Join Feature --------+
│ | │ |
│ | ├── join/ # 3 Variations are tested |
│ | │ ├── 01_green/ |
│ | │ │ └───── Join.jsx # Variation 1 |
│ | │ ├── 02_blue/ |
│ | │ │ └───── Join.jsx # Variation 2 |
│ | │ └── 03_pink/ |
│ | │ └───── Join.jsx # Variation 3 |
│ | │ |
│ +----------------------------------------------+
│ │
│ ├── lists/...
│ ├── signin/...
│ ├── ...
│ └── routes.jsx
...
We simply copy pasted the Join.jsx and apply the variations discussed above. This has a few advantages to the method before:
- a variation in itself is nothing special and could come from another repo with branches etc
- a variation can be changed at will without worrying about killing atomic variations
- a front-end developer has to know nothing about variation testing to make changes to a component and certiantly wont trip over noise such as
Math.floor(...
Running Variations with the Router
What? What? Yes -> Running variations over a router sounds awful at first. After all, a router should be a URI, right?
A Uniform Resource Identifier - something that by definition does not change. Well, starting with the invention of cookies which changed the user’s experience on hitting the server, the immutable characteristic of a URI already faded.
Today we have dynamic routers where we ensure that a route delivers rather a constant experience (meeting an expectation) - like www.myapp.com/register - but is not linked to specific content no more. On apps we often don’t even see routes.
The routes.jsx file becomes our key place to run variations and is the flipboard of continuous improvement. We simply vary the content underneath an expected experience to gather data about user behavior. Like so
import React from 'react';
import { Router, Route, browserHistory } from 'react-router';
// route components
import AppContainer from './AppContainer.jsx';
import ListContainer from './lists/ListsContainer.jsx';
import SignIn from './signin/SignIn.jsx';
import _notFound from './_shared/_notFound.jsx';
/* Join Experiment (3 Variations)
Google Docs: (some URL to a Google Docs document explaining the reasoning)
Owner: Person XYZ / email
Primary Goals
1) maximise the login speed of our users and
2) prefer facebook login
3) reduce bounce rate
*/
import Join01_green from './join/01_green/Join.jsx';
import Join02_blue from './join/02_blue/Join.jsx';
import Join03_pink from './join/03_pink/Join.jsx';
const joinVariations = [Join01_green, Join02_blue, Join03_pink];
// We could add a more complex variation picker at will
const Join = joinVariations[Math.floor(Math.random() * joinVariations.length)];
/*
Router
*/
export const renderRoutes = () => (
<Router history={browserHistory}>
<Route path="/" component={AppContainer}>
<Route path="lists/:id" component={ListContainer}/>
<Route path="signin" component={SignIn}/>
// We could pass a testing id down the route for better tracking
<Route path="join" component={Join}/>
<Route path="*" component={_notFound}/>
</Route>
</Router>
);
Tests are now run “automatically” and can be easily removed or lowered by only touching the router.jsx. This should be done together with a guy from the business side, the tech side and/or the experiment owner.
Performance
Right now I expect to hear something like this:
Are you crazy?!? If you copy & past Screens like that you will have a hell of a lot of duplicate code served to the client! That will certainly kill load times! Especially with many experiments of large screens such as the
ListsScreen! We need webpack for dynamic requires!
And you are right and wrong at the same time. Let’s talk about wepback: Webpack exists to lazy load new code. That means, if you have a complex app with many features, you are able to lazy load additional aspects of your application to not bombard the client on its first landing. And that is awesome. And that is what I hope comes soon to meteor.
But here we talk about duplicate code.
Yes! That’s the problem! It’s not DRY!
Exactly. It’s a problem but for someone else, who already solved this perfectly well: gzip compression
Meteor, also all load balancers, are able to compress data before it is send over the wire. For that gzip is used. A compression algorithm which is supported natively in browsers since Internet Explorer 6. The sole role of this algorithm is to tokenize duplicate strings to expand them at native speed again on the other side.
This is what gzib is made for and the right tool for variations: Mostly little changes in overall similar code (=strings). The overhead of having a few KB - expanded - on the client can be neglected if you load just one picture.
In an ad’hock experiment: Duplicating the “lists” feature of the todos app 50 times while changing three props randomly (color, sorting, inline css) via a pre-run-script resulted in a 0,74% larger transfer volume after gzib.
Since I often catch premature load optimisations, read: Only the gzip Size Matters by TJ VanToll
https://www.tjvantoll.com/2014/01/27/only-the-gzip-size-matters/
The network layer is much better suited for quick wins in load times. Also check your http headers etc. rather than trying to solve this in a reflex over the application layer with added complexity.
How to track variations
This is still a ToDo
- Basic in App
- Google Analytics (A/B Testing)
- Optimizely (Multivariate Testing)
Okay, this is the status quo. I wanted to roll it out - at least to the community - before Meteor 1.3 lands to give people the chance to review their stack and maybe find some inspiration in this approach.
If you do testing, continuous improvement, variations validation or anything related to this post, please share your experience.
If you happen to struggle with a stack strategy or need help finding a proper implementation / workflow - feel free to contact me.
