NMoneys and MongoDB

on Thursday, 3 September 2015

MongoDB is a public favorite of mine. It was the first document database I worked with and despite their quirks (less and less as time passes) I can strongly recommend to have a look to it when you are searching for a server-based, solid document database.

Half-broken by Default

Unfortunately, (and we can blame NMoneys for it) MongoDB does not get on well, by default, with Money instances.
Yes, they can be stored (not “pretty” { "CurrencyCode" : 208, "Amount" : "123.45" } alas), but what it can’t be done by default is retrieving them and that seriously compromises my definition of working “out-of-the-box”.

Fortunately enough, MongoDB offers a pretty extensive customization over the serialization/deserialization process.

Looking Back

There is a big gap from version 1 to version 2 of the MongoDB driver, meaning a single source file cannot target both versions.

Initially, NMoneys will support custom serialization for the version 1 and when time comes a second package will be created to support v2.

Picking up the package

NMoneys_MongoDBLegacy_Nuget

Documentation is available in here.

Compacting JSON

on Wednesday, 26 August 2015

JSON has become one of the most widely accepted data interchange/representation formats. Not only as a Web data interchange, but also as data representation of multiple document databases.

Justified dumbness

One might question why does one have to write JSON at all since there are libraries for that, right?
The only justified scenario I can find (and the one that lead me to write this entry) is testing.

Writing JSON literals using string formatting (or concatenation) would be a top 5 of “What on Earth are you doing there?” in a code review. But it makes sense if one is verifying that the JSON is properly generated.

Pain in The Neck

Thing is, names in JSON need to be written between double quotes, and so are string values. Since I write tons of C# I have to put up with escaping double quotes in string literals (\" for normal literals or "" for verbatim ones). In any case, it is annoying. And when I get annoyed too much, I do something about it.

Working Inspiration

I was working on some MongoDB support for NMoneys (more on this to come in future posts) when, while looking at some of their unit tests, I saw a test like this:

But of course!!!! That is genius! Since I did not come up with it, at least I had to do something. And so I abstracted away the logic to do the replacements, guess a couple of clever names and add some syntactic sugar. And so, JsonString became a new member of the Testing.Commons family.

Triple Clarification: I am not claiming ownership of the idea, I saw it from the MongoDB guys, which they may or may not have invented “the trick”

Compact JSON Vs. Expansion

I love names, despite being one of the two hardest things in Computer Science.

So how would I name a JSON literal that uses ' instead of escaped "? I would name it Compact JSON.
And the proper JSON that results from Compact JSON? Expanded JSON.

Using Compact JSON

There are several ways in which this technique can be used.

Instantiating the class

One can pass the Compact JSON literal to the JsonString constructor. From then one, one can get the expanded JSON by either invoking .ToString() or by passing the instance whenever a string is expected via implicit conversion.

Extension method

Invoke the .Jsonify() extension method on the Compact JSON literal to get the expanded JSON as a string.

Asserting Compact JSON

I mentioned one should avoid writing JSON literals (except when testing). In that case one is more likely to be using those literals while asserting the result of one’s code.
Then it makes sense that Testing.Commons.NUnit contains a JsonEqualConstraint.

Using the Constraint Directly

Using the Must Extensibility Model

Modifying EqualConstraint

Our last option would be continue using the EqualConstraint, but changing how it behaves via a custom IEqualityComparer<> encapsulated in the extension method .AsJson()

OSS Summer Refresh

on Wednesday, 22 July 2015

One would not believe it given the weather here in Denmark, but it is summer and what happens with summer? Amongst other interesting things here comes an update of some OSS projects.

NMoneys

The amendment 160 from ISO has been applied, containing only minor changes in currency denomination.

But majority of changes happen behind the visible artifacts:

  • All tools and libraries have been updated (NuGet, NUnit and psake)
  • The whole build process has been refactored heavily to support...
  • ...continuous integration and remote deployments via AppVeyor

Not So Continuous Deployment

Now it is easier for me to push a new version locally (with the new \build\Deploy.ps1 script) but it is also possible to deploy using AppVeyor. It is a nice to have but it opens new scenarios for quick fixes in which there is no need for a development computer:

  1. Make a small fix using GitHub's text editor
  2. Make a commit
  3. Mend documentation in the Wiki
  4. Code is built, tests are run and all artifacts created
  5. Packages are pushed to NuGet
  6. GitHub Release is created

And all of that can be done away from my development machine! A nice to have, but definitely pretty sweet.

Let the bits flow downward

As usual you can get the packages from NuGet or the binaries from the Release.

Testing.Commons

Believe it (or not) it has been almost 3 years from the last release of Testing.Commons, more than one year from Testing.Commons.NUnit and Testing.Commons.ServiceStack last ones.

One can see those projects as "stable" or "nearly abandoned", depending on how one can see a non-empty bottle. It is pretty obvious that this project is not as popular as NMoneys, but they are still used in all my professional projects.
But being quiet (dormant) is not a reason for not getting a facelift and new features.

  • Testing.Commons offers a new way to organize your builders.
  • Testing.Commons.NUnit got an update on its dependencies: ExpectedObjects, NUnit and Testing.Commons itself
  • Testing.Commons.ServiceStack got a new .UriFor() method to calculate the URL to a request DTO decorated (or not) with RouteAttribute and got it namespace changed to Testing.Commons.Service_Stack to prevent silly namespace conflicts.
  • All tools and libraries have been updated (NuGet, NUnit and psake)

Time for an update

Get latest version of all three packages frrom NuGet.

NUnit.Runner.lite

The benjamin of the family (but used in all my OSS projects) got a new version for the latest stable release of NUnit.

No fanciness in the build here, just the NuGet package.

Some Order in (Not) Chaos

on Wednesday, 15 July 2015

It has been a long while (2013) since I promised writing about Psake and using Powershell for build scripts.

But sometimes, something unexpected has to happen for old things to become current again. That something was my curiosity in using AppVeyor as a automated deployment platform.

Hosted CI++

It is not now that I “discovered” AppVeyor (I created a working barebones NMoneys build long time ago, when Google Code was still alive) but it has been lately that I was interested in its potential for automating deployments.
I am not going to be analyzing AppVeyor: go ahead, browse their magnificent documentation and don’t take my word on how easy it is to get going. Do try it.

AppVeyor is a hosted Continuous Integration/Delivery (CI/CD) for the Windows platform so you don’t have to have your own server and go through the process of setting up a server.
Best of all it is free for OSS projects.

CI/CD 101

The basic idea of CI is “listening” to your source code repository and when new commits are made, perform some actions (usually compiling the source code, running unit tests, perform some sort of static analysis and some more stuff).
The basic idea of CD is build upon CI in order to automatically (or with minimal human intervention) push your software to the environments that needs to be deployed for people to use.

It can get more deep and complicated than this, but it is basically automation performed somewhere else than your development machine.

NMoneys Build History

In the beginning of time NMoneys was built using nothing but MSBuild scripts. I hated it, so whenever I had to extend it (automating API documentation) I migrated and enhanced the script using Psake as the task framework and Powershell as the language instead of XML. The main purpose for this build was (and is) to generate the artifacts (the various Nuget packages) whenever a new release was due.

A new release was a matter of executing the build script to get the freshly built NuGet packages after making sure everything was ok (tests). This is, but not for every build there is a corresponding deployment, a bunch of manual steps existed like: pushing the packages to NuGet and creating a binary release with unsigned and singed assemblies.

nmoneys_build

  • Clean: clean solution and recreate the /release folder, that will contain everything we are likely to deploy
  • Compile: invoke msbuild.exe to compile the solution
  • Sign: use IL trickery to sign an assembly once built
  • Document: use ImmDoc.Net and the MS HTML Help Compiler to generate a .chm with the API documentation
  • Test: invoke nunit-console.exe to run all tests of the solution and create visual reports on their execution
  • Copy Artifacts: copy binaries, source files, test reports, documentation and package manifest to the /release folder
  • Build Artifacts: invoke nuget.exe to pack the different packages and zip.exe to generate the bin and signed artifacts to be released

    Could we do better? I would not be writing if we couldn’t Winking smile

    CI with AppVeyor

    Translating the process to AppVeyor was not difficult at all, but extracting a common set of script functionality that can be invoked both by the local Psake build and the remote engine took a bit longer.
    All I can say that the scripts are so much cleaner and easy to follow than before and because of that only, the process was worth it.

    Appveyor has a rich and complex build pipeline and their default build and test stages make redundant the local Compile and Test tasks, as well was the Clean, because, for every build that is started, a brand new machine is provisioned, so there is nothing to clean. Of their complete pipeline we use the following steps:

    appveyor_build

    The equivalency between the two is roughly like this:

    • Init + BuildCompile + Sign + Document
    • TestTest + CopyArtifacts + BuildArtifacts

    CI Build?

    Isn’t it weird to have a Deploy stage in a CI pipeline?
    Only if the deploy has the meaning of “pushing components to another environment for their execution”.

    Appveyor has two types of deployment: Inline (the last stage of the pipeline) and Environment. Whereas the Inline deployment is executed for each build execution, the Environment deployment is asynchronously triggered by “human intervention”.
    The Deploy stage we refer to in our graphic is, thus, an inline deployment and in our case the “only” thing we do is “promote” artifacts from the /release folder to AppVeyor artifacts for them to be deployed later on in en Environment deployment.
    The manual promotion (as opposed of the artifact definition as configuration of the build) is provoked by the similarity in package naming which would make impossible to deploy only one package to a given environment.

    Details, please

    All these changes have turned the old NMoneys build process into a very powerful, lean and streamlined deployment pipeline.

    Details matter. Details are fun. I have learned a ton of Powershell in the refactoring process, so… why not sharing it? But this post is already long, so there will be another one with those beloved details.

  • Switching Fun

    on Wednesday, 8 July 2015

    In my last post about Feature Toggles I mentioned that, for my simple scenario, I chose FeatureSwitcher amongst the plethora of packages (254 packages for the feature toggle search, never went beyond page 1).

    Some Whys

    I needed configuration support and it is supported by the means of another package. Check for solving my problem.

    I also liked their object model to deal with configuration in code. Likeability check.

    I dislike magic strings. Check for being based on types.

    How did I use it?

    I cannot show the real code but I can show bits of a sample project.
    The project is an OWIN web site that uses Nancy and Razor to render an application with an optional feature.

    As Martin Fowler suggests (and we do not always agree):

    Toggle tests should only appear at the minimum amount of points to ensure the new feature is properly hidden.

    That is, disable the menu that gives access to the feature to the bare minimum to disable the feature.

    toggling_both

    What is it that is actually needed to accomplish this bare minimum?

    1. Install the Nuget Package FeatureSwitcher.Configuration. That will reference two assemblies to our solution.
    2. Create a feature. That is a type that implements the IFeature marker interface
    3. Set-up configuration by adding the configuration section group. In this case, have the actual features are located in separate file:
    4. Tell FeatureSwitcher that configuration is to be used and that features are named after the type that implements the interface.
    5. Check whether the feature is enabled and render the markup if it is.

    Pushing through the last mile

    When I mentioned I liked to have types instead of just strings is not only for type safety (aka intelli-sense dependency), but because we can easily attach behavior to types. So now, a feature is not only able to allow telling whether it is enabled or not, but methods can me added to perform the activation steps, for instance.

    If we didn’t do anything else, the feature could be “guessed” and accessed though the URL (no, it is not as crazy as it sounds, I have had clients that demanded to scramble routes to avoid guessing from users and/or competitors. True Story). Now we can have the activation code right inside the feature and interrogate itself for its status:

    So if someone gets a hold on the route, he will be greeted by the hilarious 404 status code:

    toggling_disabled_optional

    Moderate Fun with Feature Toggles

    A Feature Toggle is just a way to enable of disable certain software features based on several criteria (configuration flag, time, user, some complex context,…).

    The problem feature toggles try to solve is: how does one release code that is not totally finished without any apparent  impact on users. Let’s say your team is working on two features simultaneously, one large and one small, and we want to get the small one “out of the door” (that is, deployed to production) as soon as possible while disallowing users to operate the large, unfinished one.

    This simplistic concept has made into the hotspot recently as the suggested alternative for feature branches in environments in which Continuous Integration/Delivery is dogmatically advocated.
    And I say dogmatically because I have not been in the position in which a feature branch (in SCM) did not solve that problem while still being confident in the integrity of my build (safe-guarded by my CI server). It should be really easy to “clone” an existing build definition in the CI server into another one and listen to changes in a different branch. If it isn’t I think it might be time to change your CI Server instead of discarding features branches altogether.

    As you can imagine I am pretty much against feature toggles as I believe they are the doorstep to happy feature branching in multiple places, with the corresponding readability/maintainability nightmare.
    Having said so, feature toggles are just another tool that one needs to pull from the tool-belt when almost every other tool failed.

    The Challenge in Hand

    Big feature: almost a complete sprint.
    Small surface of activation: it all starts with a scheduled tasks that perform a state transition on certain entities under certain conditions and everything else goes on from there. No task ⇒ whole feature is invisible.
    No feature branch upfront: code is in trunk and it would be risky to fiddle with SCM to pick commits out.

    Whoever validates the big feature as correct is on holiday and some other smaller features need to get to production.

    A simple solution

    Given the small activation surface, it makes sense to have a feature toggle that prevents the scheduled tasks from running and hence, the feature is virtually not there.

    When the feature can be validated by the business owners, it will be enabled in configuration and eventually the toggle will disappear.

    Some Implementation Details

    Instead of rolling my own, I spent a couple of hours looking at what was already out there and I finally chose FeatureSwitcher. Why? I think we’ll wait for that until next post on the subject.

    Automating PhoneGap Builds with Gulp. Part IV

    on Friday, 26 June 2015

    In part III we got PhoneGap Build (PGB) “cooking” our artifacts and generating the binaries stew.

    D-Ploy

    That compilation takes some time so we can:

    1. sit and wait
    2. be proactive and check the build status

    We chose the first options as deploying is not done every time the app is built (we install the application on the devices from PGB itself while developing). But if we wanted to check whether a build is ready to be deployed we could poll the PGB API and check the status for the platform that we want to deploy.

    The idea is (after some sanity check), using the aforementioned phonegap-build-api package, download the resource for a given platform (using the right file extension) and calling the HockeyApp method that uploads the application version for the chosen platform and environment.

    Wrapping it up

    Package, queuing a build, waiting and pushing a version.

    That is all it takes to have a new version of the application for a given platform ready for the testers.

    Two commands. And the peace of mind that there is very little room to screw up. I’ll type them once again, just for the heck of it.