RavenDB Indexes While Developing

on Friday, 27 February 2015

At work, my team and I have been working with RavenDB for quite some time.

Despite some people objections, the team usually shares a single development database. There is still the occasional “I made this breaking change… Guys! Get latest!!” but happens quite rarely.

Besides documents, there is another data artifact that is very necessary for RavenDB to work: indexes (and transformers). Favoring explicit indexes (instead of automatically created ones) means we end up with bunch of indexes defined in code.

From Code to Database

RavenDB offers the IndexCreator high-level api to easily create indexes. Question is: when will that code that creates data indexes from their code definitions executed?

When application starts

An strategy we have followed in the past was placing index creation code right after the DocumentStore initialization code. Since that initialization is recommended to occur once per application, we ended up re-creating the indexes every time the app starts.

An app in development re-starts many times and the problem that I pointed out in the first paragraph (developers that do not have the latest version of the code running) worsens seriously: an application with the outdated index definition will override the just-updated index.

So, all in all, that approach was not such a good idea at all.

On demand

Currently we use a small console program (or LINQPad script) that updates the indexes when needed.
Such approach can even be used to automatically deploy index definitions from environment to environment (given we have access to that environment database, of course).

The multiple database challenge

What happens when multiple databases are accessed in the application? Out-of-the box, the simpler to use IndexCreator.CreateIndexes() takes an assembly, which means that if all indexes for all databases live in the same assembly (and we tend to favor single –or few– assembly solutions), all the databases will contain all assembly definitions.

Not all databases contain the same collections, so some indexes are empty (consuming little to no resources); and transformers won’t be used in queries against that database. But having artifacts that should not be there is still a small annoyance.

OSS to the Rescue

To solve this, and some other problems, rvn-izer has been created.

Rvn.Izr_Nuget

Installing the Nuget package in the project that contains the indexes, gives access to the CreateInAttribute that can be used to decorate indexes and transformers with the name of the database they correspond.

Filter by database name

Using the Indexes fluent interface, we can get the ExportProvider instance that contains the artifacts corresponding to the specified database and invoke the corresponding IndexCreator.CreateIndexes() overload to stop creating artifacts where they do not belong.

Safety first

Another interesting feature Indexes offers is the ability to return artifacts not decorated with the attribute. That opens the chance to create an automated test that verifies, for instance, that we are not forgetting to decorate a given artifact and, thus, not creating it.

Useful?

Play with it, use and abuse it and, if you fancy more features, PR your way through.

Order of Things used to be Wrong

on Tuesday, 17 February 2015

Well, not everywhere. Only in my project. When I was querying RavenDB. Here how I solved it.

The catalog Model

Our catalog, as many other product catalogs in the world is made up of products and categories. A product must belong to at least a given standard category and can belong to zero or more non-standard categories. Although the most usual case is a product “living inside” one standard and one non-standard categories.

image

 

Each category object contains its globally unique identifier and a value that indicates the relative order of the product inside that category.

Querying the Catalog

When displaying the products that belong to a given category (standard or otherwise) the products collection is queried, filtering by the desired category id and sorting by the relative order of the product inside that category.

Raven-izing the query

RavenDB goes as far as requiring an index for each query you do to your data (or an ad-hoc index will be created the first time the query is performed). In that index, we will include the fields that are part of the filter (the category ids) as well as the fields that are used for sorting.
But… we do not have an specialized sorting field for each category in our products. No we don’t. But we can ask Raven to create one while indexing:

New entries in the index will contain a dynamic field that is named after the category and valued after the relative order. With those fields in the index we can query our catalog with the following query:

It seemed to work. The feature was demoed several times and approved by several people: change the order of a category –> products are displayed in that order. End of story? Hardly.
It did not pass the test that every feature has to pass in order to be considered a success: the reality test.

Real-World problems

We started t get bug reports in production:

— customer: “it does not work. Products are not in the order specified in the category”.
— developer: Head scratching. ”documents have the right information”. More scratching. “it works in staging, you saw it with your own eyes, look”
— “not in production, fix it”
— “what is what you are doing exactly?”
— “I change category A, then category B, then…”
— Vicious scratching. “let me look into it… once more”

And indeed, as usual, customer was right. When changing more than one category at a time, the order of products was less than predictable.

Why? Oh why?

After asking around, a colleague pointed me the solution: Do not use .OrderBy() extension. Instead, use .AddOrder() which allows the type of the field that we are using.

And the customer has not complained since.

NMoneys 3.6.0.0

on Thursday, 8 January 2015

With a new year, comes a new release of NMoneys.

Inside

The amendment 159 from ISO has been applied. Amongst the changes are:

  • LTL has been deprecated in favor of EUR
  • Changes to CVE information

Fixed

Issue #28 has been fixed and now, one has full control over what Money.Parse() does. It is still a pretty weak method (and that is hardly fixable, but ideas and contributions are welcome) and its usage is still not recommended.

Download

Use your favorite package manager (or Nuget Smile with tongue out) to reference it: 3.6.0.0

If you get your kicks from downloading random zip files and manually referencing assemblies, follow the badge:

Dynamic Documents

on Saturday, 22 November 2014
More things coming out from my recent session about MongoDB.

Documents and Types

MongoDB is a document database. Such documents are stored as JSON (well, kind of). JSON has a very challenged simple type system.
.NET is an object-oriented environment that promotes types. Such type system is pretty rich.
Are MongoDB’s documents and .NET Framework types doomed to miscommunicate? They would be if not for the existence of the serialization bridge the driver provides.

Types Everywhere

Since .NET is type-happy, majority of the times we are working in our happy strongly-typed world.
Believe it or not, there developers out there that are very happy (and productive) in a less strongly-typed world.
I will measure my words carefully, trying to prevent them understood as heresy:
There are certain types of applications (or parts of all applications) where strong typing bring little benefits.
Wow. There I said it.
Imagine this scenario: data is carefully crafted and validated and taken care of at the time of writing it to persistence storage and whenever needs to be presented to the user, there is little (if any) manipulation of it, because those details were taken care of beforehand, at a time when computing cost is not so important as write operations tend to be fewer than read operations. For what is worth, we could be storing textual representations of the data and no one would care if that was the case.
In those scenarios we do not need types. The structured data only travels through the wire to be presented on a screen.

Type-wannabe: the Document

The serialization part of the MongoDB driver for C# offers a straightforward type to solve the problem: BsonDocument. It is a nicely designed and useful type that allows reading information from MongoDB without much fuss about the underlying type.
The “downside”? C# is not helping to have a clean syntax.
Let’s take this simple document:

And how to access when I started snowboarding:

From 33 to 42 characters. It does not look like much, but it's a whooping 27% increase in typing.
I do not know about you, but I would gladly accept a 27% increase in my paycheck, thank you very much.

Lipstick on a Type

In .NET 4, dynamics were introduced as a way to marry two different worlds: strongly and loosely typed with a more palatable syntax. I suspect it was just a trick to prevent developers doing Interop from quitting in mass.
That sounds like almost useful and applicable to our small problem in hand.
Unfortunately, the driver still lives in a pre-.NET 4 era, so they cannot provide out-of-the-box support for dynamics. There are some attempts out there, but I was not happy with taking a dependency on another (arguably more capable) JSON parsing library (JSON.Net) to just do that.
Since it was for educational purposes you are allowed to simply waste some of your scarce free time in search for a solution that no one cares about Smile

Dynamic document

It turns out it is really easy to provide support for accessing instances of BsonDocument as dynamic object, taking benefit from the usual dot notation. Thanks to DynamicObject.
By simply wrapping an instance of BsonDocument inside an inheritor of DynamicObject and overriding .TryGetMember() to wrap nested documents we are able to happily use dot notation (with a slight overhead):

To use it, just invoke the .AsDynamic() over an instance of BsonDocument

With this minimal baggage, you are able to, for example, write razor views that display data coming from an instance of such DynamicBsonDocument.
As for getting dynamic objects into BsonDocument instances… Well, that is a whole different story. That I am not ready or willing to tell… for the moment Winking smile

Easier projections in MongoDB

I have already written about the talk I gave in the local user group about MongoDB.
I always try to push the limits of what I want to show by adding a couple of more advanced (yet useful) scenarios. The one I want to describe here is document projections using the .NET official driver.

What is a projection anyway?

The term refers to a result of a query that is a subset of the source of information.
In relational databases, it is a subset of the complete list of columns being queried.
For MongoDB, it is a subset of the fields returned for all matched documents.

mDb4net

Last week I was able to give a talk in the local .Net user group, AAnug.

The topic in question was MongoDB and, more specifically, how it can be used by .NET developers (using the official driver).

Code, code, code

It was a mostly code-oriented session but I did not want to type everything myself (horrible typist, worse in front of people if possible), but being helped by a series of code snippets that I could use from LINQPad (no need to fire off heavy Visual Studio instances for simple code snippets).

Snippets

Crafting snippets was not the most fun task in the world (for the lack of a less offensive term) as I could not find a good tool that can edit .snippet files comfortably (leaving out Visual Studio plugins, it runs slow enough already). The one I used (Snippet Editor) does the job, but just that.

Git

I used branches for each topic of the session, so changing to a brand new workspace for the next demo was a matter of checking out a branch. For topics that used LINQPad (majority), I created a /snippets subfolder and configured LINQPad to load custom snippets from it:

linqpad

That way, I only had the snippets I needed for the current topic in Intellisense and I had way less to type, minimizing the amount of embarrassment.

The Result

I feel the session went pretty smooth and I could deliver all the important bits in time (I only overflowed for the “Bonus Tracks”) without rushing. So it is definitely a technique that I will use again for code-heavy topics.
My only fear is that by not having to type it and even though I tried to emphasize what’s important from the code, some people might have felt overwhelmed by everything that happened on the screen. But I did not get negative comments from it, so it might be only an unfounded fear of mine.

You can head to its Git repository to see the contents and even browse the slides in its GitHub Pages site.

BDD Mini Match-up

on Thursday, 18 September 2014

I cannot say I am a big BDD-er. I have used it for commercial projects, mostly to define complex scenarios for complex pieces of code (a custom discount engine comes to mind) but it comes with a price tag that I am not willing to pay for the majority of tests I write (unit tests).

I am not going to describe what BDD is, philosophy, syntaxes, flavors… I am just going to mention my experience with a couple of tools one can use if developing for the .NET Framework.

A little experiment

Quite long time ago I started SharpRomans as a learning experience with some interns here at my company. The idea was to create an implementation of roman numerals using some BDD techniques in order to get rich executing documentation.
The experiment was never completed until some time ago I decided to re-launch it, this time in Github.

The interesting thing of the experiment in regards of BDD was the tool I used: StoryQ.

StoryQ


I chose this veteran framework for a veteran project and I have been quite a happy user. In fact, I generated the whole Specification section of the project wiki by transforming the resulting XML report into Markdown (not too shabby for not having touched XSLT in years).

The way the tool was used in SharpRomans was simply using the library to write tests that are run by NUnit. An example of such test could be:

As one can see in the code, it is a quite pleasant fluent interface that leverages delegates in quite a clever manner to later infer the names of the steps and the arguments and their position via the usage of underscore placeholders.

The execution of such test produces several reports. A console output:

And a “pretty” HTML report (resulting from transforming an XML into HTML thanks to XLST):

storyq_html

It is easy to get the code syntax right, but if one prefers using the included “Text-To-Code converter”, it is a simple WPF application that allows converting a Cucumber like syntax into .NET code that uses their fluent interface. It is a great addition.

While look any further?

As I have said, I am pretty happy with the tool. What could have taken me to evaluate alternatives?
Well, besides the sheer desire to improve and do more with less, I have had my share of complaints for which I take some responsibility for not notifying the author or submitting a patch (although, in my defense, I tried, but was too complex for the time I allocated for it).

  • The code generated by the tool cannot be customized Sad smile
  • If one wants a different reports, one has to put with transforming XML, which is not my cup of tea.
  • Stability is a double-edge sword. People might look at the activity of the project and be put off by the fact that is has not shown any activity for years

The contender. BDDfy

If one can say only a good thing about the project is that they have raised the level of documentation for other OSS projects. It is pretty impressive.

Another thing that can be said is that it covers a lot more ground, in the sense that it is more ambitious in the styles and workflows that can support. To compare apples to other similar fruits I am going to stick to its FluentAPI.

And it generates (by default) a couple of nice reports. A console output:

And the mandatory “pretty” HTML report to show off our stakeholders what their money can buy:

bddfy_html

The API feels quite the same, with fluent .Give(), .When(), .Then() but there are a couple of key differences:

story “context” is provided as a set of attribute properties

one can have (as with StoryQ) all-scenarios-in-one-test. When everything goes alright (all scenarios pass), everything is good; but, when one scenario breaks, all subsequent scenarios are not run (unlike StoryQ). To solve this issue (and that is my opinion about it) we can easily move to a test-per-scenario (which StoryQ can’t handle) at the cost of syntactic noise in the code

And if one looks at the console output, one can see that the story part is repeated for each of the scenarios, which I, personally, don’t fancy.

So…?

They are both quite nice frameworks and both can support my way of doing BDD. StoryQ is lean and BDDfy is more powerful and its flexibility can accommodate other workflows different than mine.

A couple of nice extras that BDDfy brings to the table are:

  • a nice looking metro report (that has to be configured without any sweat)

bddfy_metro

  • an even nicer markdown report.

That might have come very handy to generate the spec for SharpRomans, which I could have copied and pasted to the wiki. But then, I would not have written this puppy (and that might have been a good thing Smile with tongue out).