OSS Summer Refresh

on Wednesday, 22 July 2015

One would not believe it given the weather here in Denmark, but it is summer and what happens with summer? Amongst other interesting things here comes an update of some OSS projects.

NMoneys

The amendment 160 from ISO has been applied, containing only minor changes in currency denomination.

But majority of changes happen behind the visible artifacts:

  • All tools and libraries have been updated (NuGet, NUnit and psake)
  • The whole build process has been refactored heavily to support...
  • ...continuous integration and remote deployments via AppVeyor

Not So Continuous Deployment

Now it is easier for me to push a new version locally (with the new \build\Deploy.ps1 script) but it is also possible to deploy using AppVeyor. It is a nice to have but it opens new scenarios for quick fixes in which there is no need for a development computer:

  1. Make a small fix using GitHub's text editor
  2. Make a commit
  3. Mend documentation in the Wiki
  4. Code is built, tests are run and all artifacts created
  5. Packages are pushed to NuGet
  6. GitHub Release is created

And all of that can be done away from my development machine! A nice to have, but definitely pretty sweet.

Let the bits flow downward

As usual you can get the packages from NuGet or the binaries from the Release.

Testing.Commons

Believe it (or not) it has been almost 3 years from the last release of Testing.Commons, more than one year from Testing.Commons.NUnit and Testing.Commons.ServiceStack last ones.

One can see those projects as "stable" or "nearly abandoned", depending on how one can see a non-empty bottle. It is pretty obvious that this project is not as popular as NMoneys, but they are still used in all my professional projects.
But being quiet (dormant) is not a reason for not getting a facelift and new features.

  • Testing.Commons offers a new way to organize your builders.
  • Testing.Commons.NUnit got an update on its dependencies: ExpectedObjects, NUnit and Testing.Commons itself
  • Testing.Commons.ServiceStack got a new .UriFor() method to calculate the URL to a request DTO decorated (or not) with RouteAttribute and got it namespace changed to Testing.Commons.Service_Stack to prevent silly namespace conflicts.
  • All tools and libraries have been updated (NuGet, NUnit and psake)

Time for an update

Get latest version of all three packages frrom NuGet.

NUnit.Runner.lite

The benjamin of the family (but used in all my OSS projects) got a new version for the latest stable release of NUnit.

No fanciness in the build here, just the NuGet package.

Some Order in (Not) Chaos

on Wednesday, 15 July 2015

It has been a long while (2013) since I promised writing about Psake and using Powershell for build scripts.

But sometimes, something unexpected has to happen for old things to become current again. That something was my curiosity in using AppVeyor as a automated deployment platform.

Hosted CI++

It is not now that I “discovered” AppVeyor (I created a working barebones NMoneys build long time ago, when Google Code was still alive) but it has been lately that I was interested in its potential for automating deployments.
I am not going to be analyzing AppVeyor: go ahead, browse their magnificent documentation and don’t take my word on how easy it is to get going. Do try it.

AppVeyor is a hosted Continuous Integration/Delivery (CI/CD) for the Windows platform so you don’t have to have your own server and go through the process of setting up a server.
Best of all it is free for OSS projects.

CI/CD 101

The basic idea of CI is “listening” to your source code repository and when new commits are made, perform some actions (usually compiling the source code, running unit tests, perform some sort of static analysis and some more stuff).
The basic idea of CD is build upon CI in order to automatically (or with minimal human intervention) push your software to the environments that needs to be deployed for people to use.

It can get more deep and complicated than this, but it is basically automation performed somewhere else than your development machine.

NMoneys Build History

In the beginning of time NMoneys was built using nothing but MSBuild scripts. I hated it, so whenever I had to extend it (automating API documentation) I migrated and enhanced the script using Psake as the task framework and Powershell as the language instead of XML. The main purpose for this build was (and is) to generate the artifacts (the various Nuget packages) whenever a new release was due.

A new release was a matter of executing the build script to get the freshly built NuGet packages after making sure everything was ok (tests). This is, but not for every build there is a corresponding deployment, a bunch of manual steps existed like: pushing the packages to NuGet and creating a binary release with unsigned and singed assemblies.

nmoneys_build

  • Clean: clean solution and recreate the /release folder, that will contain everything we are likely to deploy
  • Compile: invoke msbuild.exe to compile the solution
  • Sign: use IL trickery to sign an assembly once built
  • Document: use ImmDoc.Net and the MS HTML Help Compiler to generate a .chm with the API documentation
  • Test: invoke nunit-console.exe to run all tests of the solution and create visual reports on their execution
  • Copy Artifacts: copy binaries, source files, test reports, documentation and package manifest to the /release folder
  • Build Artifacts: invoke nuget.exe to pack the different packages and zip.exe to generate the bin and signed artifacts to be released

    Could we do better? I would not be writing if we couldn’t Winking smile

    CI with AppVeyor

    Translating the process to AppVeyor was not difficult at all, but extracting a common set of script functionality that can be invoked both by the local Psake build and the remote engine took a bit longer.
    All I can say that the scripts are so much cleaner and easy to follow than before and because of that only, the process was worth it.

    Appveyor has a rich and complex build pipeline and their default build and test stages make redundant the local Compile and Test tasks, as well was the Clean, because, for every build that is started, a brand new machine is provisioned, so there is nothing to clean. Of their complete pipeline we use the following steps:

    appveyor_build

    The equivalency between the two is roughly like this:

    • Init + BuildCompile + Sign + Document
    • TestTest + CopyArtifacts + BuildArtifacts

    CI Build?

    Isn’t it weird to have a Deploy stage in a CI pipeline?
    Only if the deploy has the meaning of “pushing components to another environment for their execution”.

    Appveyor has two types of deployment: Inline (the last stage of the pipeline) and Environment. Whereas the Inline deployment is executed for each build execution, the Environment deployment is asynchronously triggered by “human intervention”.
    The Deploy stage we refer to in our graphic is, thus, an inline deployment and in our case the “only” thing we do is “promote” artifacts from the /release folder to AppVeyor artifacts for them to be deployed later on in en Environment deployment.
    The manual promotion (as opposed of the artifact definition as configuration of the build) is provoked by the similarity in package naming which would make impossible to deploy only one package to a given environment.

    Details, please

    All these changes have turned the old NMoneys build process into a very powerful, lean and streamlined deployment pipeline.

    Details matter. Details are fun. I have learned a ton of Powershell in the refactoring process, so… why not sharing it? But this post is already long, so there will be another one with those beloved details.

  • Switching Fun

    on Wednesday, 8 July 2015

    In my last post about Feature Toggles I mentioned that, for my simple scenario, I chose FeatureSwitcher amongst the plethora of packages (254 packages for the feature toggle search, never went beyond page 1).

    Some Whys

    I needed configuration support and it is supported by the means of another package. Check for solving my problem.

    I also liked their object model to deal with configuration in code. Likeability check.

    I dislike magic strings. Check for being based on types.

    How did I use it?

    I cannot show the real code but I can show bits of a sample project.
    The project is an OWIN web site that uses Nancy and Razor to render an application with an optional feature.

    As Martin Fowler suggests (and we do not always agree):

    Toggle tests should only appear at the minimum amount of points to ensure the new feature is properly hidden.

    That is, disable the menu that gives access to the feature to the bare minimum to disable the feature.

    toggling_both

    What is it that is actually needed to accomplish this bare minimum?

    1. Install the Nuget Package FeatureSwitcher.Configuration. That will reference two assemblies to our solution.
    2. Create a feature. That is a type that implements the IFeature marker interface
    3. Set-up configuration by adding the configuration section group. In this case, have the actual features are located in separate file:
    4. Tell FeatureSwitcher that configuration is to be used and that features are named after the type that implements the interface.
    5. Check whether the feature is enabled and render the markup if it is.

    Pushing through the last mile

    When I mentioned I liked to have types instead of just strings is not only for type safety (aka intelli-sense dependency), but because we can easily attach behavior to types. So now, a feature is not only able to allow telling whether it is enabled or not, but methods can me added to perform the activation steps, for instance.

    If we didn’t do anything else, the feature could be “guessed” and accessed though the URL (no, it is not as crazy as it sounds, I have had clients that demanded to scramble routes to avoid guessing from users and/or competitors. True Story). Now we can have the activation code right inside the feature and interrogate itself for its status:

    So if someone gets a hold on the route, he will be greeted by the hilarious 404 status code:

    toggling_disabled_optional

    Moderate Fun with Feature Toggles

    A Feature Toggle is just a way to enable of disable certain software features based on several criteria (configuration flag, time, user, some complex context,…).

    The problem feature toggles try to solve is: how does one release code that is not totally finished without any apparent  impact on users. Let’s say your team is working on two features simultaneously, one large and one small, and we want to get the small one “out of the door” (that is, deployed to production) as soon as possible while disallowing users to operate the large, unfinished one.

    This simplistic concept has made into the hotspot recently as the suggested alternative for feature branches in environments in which Continuous Integration/Delivery is dogmatically advocated.
    And I say dogmatically because I have not been in the position in which a feature branch (in SCM) did not solve that problem while still being confident in the integrity of my build (safe-guarded by my CI server). It should be really easy to “clone” an existing build definition in the CI server into another one and listen to changes in a different branch. If it isn’t I think it might be time to change your CI Server instead of discarding features branches altogether.

    As you can imagine I am pretty much against feature toggles as I believe they are the doorstep to happy feature branching in multiple places, with the corresponding readability/maintainability nightmare.
    Having said so, feature toggles are just another tool that one needs to pull from the tool-belt when almost every other tool failed.

    The Challenge in Hand

    Big feature: almost a complete sprint.
    Small surface of activation: it all starts with a scheduled tasks that perform a state transition on certain entities under certain conditions and everything else goes on from there. No task ⇒ whole feature is invisible.
    No feature branch upfront: code is in trunk and it would be risky to fiddle with SCM to pick commits out.

    Whoever validates the big feature as correct is on holiday and some other smaller features need to get to production.

    A simple solution

    Given the small activation surface, it makes sense to have a feature toggle that prevents the scheduled tasks from running and hence, the feature is virtually not there.

    When the feature can be validated by the business owners, it will be enabled in configuration and eventually the toggle will disappear.

    Some Implementation Details

    Instead of rolling my own, I spent a couple of hours looking at what was already out there and I finally chose FeatureSwitcher. Why? I think we’ll wait for that until next post on the subject.

    Automating PhoneGap Builds with Gulp. Part IV

    on Friday, 26 June 2015

    In part III we got PhoneGap Build (PGB) “cooking” our artifacts and generating the binaries stew.

    D-Ploy

    That compilation takes some time so we can:

    1. sit and wait
    2. be proactive and check the build status

    We chose the first options as deploying is not done every time the app is built (we install the application on the devices from PGB itself while developing). But if we wanted to check whether a build is ready to be deployed we could poll the PGB API and check the status for the platform that we want to deploy.

    The idea is (after some sanity check), using the aforementioned phonegap-build-api package, download the resource for a given platform (using the right file extension) and calling the HockeyApp method that uploads the application version for the chosen platform and environment.

    Wrapping it up

    Package, queuing a build, waiting and pushing a version.

    That is all it takes to have a new version of the application for a given platform ready for the testers.

    Two commands. And the peace of mind that there is very little room to screw up. I’ll type them once again, just for the heck of it.

    Automating PhoneGap Builds with Gulp. Part III

    on Monday, 22 June 2015

    In Part II we got to create a .zip file with the artifacts to be uploaded to PhoneGap Build (PGB)

    Compile and forget.

    The point of automating is not create a .zip file and point and click your way through PGB for it to compile. That is why PGB offers a “read” and a “write” APIs.

    Being a PGB “private app” we need to call the /apps/:id endpoint and attach the .zip containing the artifacts, which raises the question: is there an easier way than crafting the HTTP calls ourselves? And there is: use the phonegap-build-api package to handle communication, authentication (using the slightly less secure token authentication) and form manipulation and learn from their documentation.

    According to phonegap-build-api’s documentation on the method, attaching the file is as simple as indicating the path to the file, but since globs are used in Gulp’s src, we need to know the path to the single file matched by the glob. For that introspection we use gulp-tap.
    Of course, the build task depends on the compress task but there is also this unlock task. What would it be?

    Unlock it

    Apps in Android may or may not be signed with a key, but iOS apps absolutely need to. For signing purposes there are manual steps to be carried out the resulting of which is some signing keys uploaded to PGB. From then on, those keys can be used to sign the application while building. We chose to have a key per build environment, but you can do differently. In any case, those keys are unlocked (suitable for build) for a period of time, after which, they lock again, resulting in failed builds. To prevent those failed builds keys need to be unlocked and, fortunately enough, the unlocking process can be automated via the API.

    Watch it! There is no such thing as updating an existing signing key. If you are referring to keys by id (like we do), changing the iOS provisioning profile requires creating a new key (that has a new id), which forces us to change the keys in our config.js. We chose to cope with it (changing config.js is part of the manual process of adding test devices to the provisioning profile and uploading it again) but one can overcome this small annoyance by getting the keys by name via the API and extracting the key before unlocking or building.

    Watch it! I found little documentation about the format of the data to be sent when unlocking the key (the unlockData object from our example). If you do not follow this structure: { form: { data: { password: 'your_password' } } } (or { form: { data: { key_pw: 'kP4zzw0rd', keystore_pw: 'p4zzw0rd' } } } for Android) the call to unlock will say yay! while the build will say nay! and then your frustration levels soar with every try.

    Automating PhoneGap Builds with Gulp. Part II

    on Tuesday, 16 June 2015

    So far, in Part I we read about the motivations and the targets we want to hit with our automation.

    Packing up the goods

    Being a PhoneGap Build (PGB) private application, we need to upload a .zip file that contains everything we need to build the app for a given environment.
    So, creating a .zip file with the app and assets it is.

    We follow the default ionic source tree structure:

    which means having the config.xml outside the /www folder (as PGB expects). Besides, both the config.xml and the angular services have to be “configured” for the environment they are to be deployed to, so we took the approach of copying everything we needed to a temporary location (tmp/) and create the .zip file for that location (previous cleaning it).

    First challenge, harvest command line arguments. Gulp does not have a built-in, standard way, so we chose the yargs package that exposes them from the argv variable.
    config is a module we built that exposes variables and utility functions that deal with configuration of the process.
    Second challenge is the parallelize-everything mindset of Gulp, that offers no guarantee that a dependent task has finished before the executed task begins. In our case we definitely wanted to finish up “mirroring” the artifacts into tmp/ before performing file transformations or start packing. To achieve such simple requirement we used gulp-sync that offers the gulpsync.sync(['t1', 't2', …]) syntax.
    We use gulp-zip to compress the files in tmp/, but the dependencies are interesting in themselves.

    mirror copies just the files we need to build (minified versions) and makes heavy usage of inclusion and exclusion of globs. Word of advice, copy just what is needed and optimize your images. We ended up from a 60MB app to a still heavy 10MB (thanks you, high-res displays) and I cannot tell you how much faster that is.
    Another interesting feature is, since we are using TFS, we have those dreaded read-only flags on files that would prevent manipulating the content of the files, so gulp-chmod allows to remove the flag when copied.

    config transforms the config.xml file with the variables known at build time, such as version and application namespace. To “poke” the content of the files we use gulp-replace. services does the same for for our services.js file for replacing the tokens such as API endpoints and the like from the source code itself.

    The end result is a .zip file that can be uploaded to PGB and it will compile.