How to integration test ASP.NET Core services, including Startup and dependency injection

After 3 years, I feel I have something valuable to share.

I have recently started to work with ASP.NET Core and even though it's a thousand times easier to test than ASP.NET MVC, thanks to its built-in Dependency Injection (IServiceCollection, IServiceProvider) and the very great TestServer, I could not find a well rounded pattern to easily test all aspects of an application, mocking at different levels and using the powerful fact that everything comes from the IServiceProvider by design.

This is very much work in progress so please, if you happen to come across this page, get in touch with any alternatives you have found or any improvements you think can be applied.

Maintaining my style, let's get on with some code. It's worth more than a thousand words.

This is what I consider the ideal integration/acceptance/end-to-end test:

public async Task WhenXThenY(
	IRepository<SomeEntity> repository, // For this to work, a TestServer must have been started and
	// this IRepository must be the same instance that the Controller is going to use when processing the request
	WebApiClient client, // This must be wrapping an HttpClient connected to the TestServer
	ISomeExternalServiceGateway gateway // Must also be the same instance to be used by the Controller)
	{
	var entityInRequiredState = CreateEntityInStateX();
	repository.GetById(entityInRequiredState.Id).Return(entityInRequiredState);
	await client.Post($"controller/action/{entityInRequiredState.Id}");
	entityInRequiredState.ShouldBeInStateY();
	}

https://gist.github.com/rodolfograve/16e603a86565536063932e03fa4afc10#file-aspnetcoretest-cs

There are a few things to notice here:

1. Somehow, a TestServer must be started using as much of the production configuration as possible. Obviously, we still need to be able to mock what we want.
2. The WebApliClient must send its requests to the TestServer.
3. The IRepository must be the same that is going to be used by the Controller when processing the request, otherwise our setup is useless.
4. We need to use a framework that lets you decorate your tests to customize the way the parameters are injected. I'm currently using AutoFixture.

The key to make this work is to use the IServiceProvider used by the server as the source for all the instances you want to configure to setup your test. Turns out this is not trivial to achieve without doing some research and understanding some ASP.NET Core secrets, which is why I think this is a valuable thing to share.

The typical code to start an ASP.NET Core web server is:

https://gist.github.com/rodolfograve/16e603a86565536063932e03fa4afc10#file-defaultaspnetcoreprogram-cs

The trick to get access to the IServiceProvider created by the WebHostBuilder, and the reason I'm sharing all this, is to provide an instance of IStartup instead of the type:

https://gist.github.com/rodolfograve/16e603a86565536063932e03fa4afc10#file-modifiedaspnetcoreprogram-cs-L13

With the above in place, we can now use AutoFixture to create an instance of TestServer, keep a reference to the IServiceProvider and use it to obtain instances to be injected into the test:

1. The attribute, to instruct AutoFixture how to resolve all parameters of the test method.
2. The AutoFixture ICustomization that creates a fixture for the TestServer and configures a SpecimenBuilder to delegate all requests to the IServiceProvider.

Apologies for not embedding the code here but Blogger refuses to accept any of the solutions I have found to embed code. I do dislike Blogger a lot but I haven't had the time and energy required to find something else.

I will probably turn all the above into a little NuGet package. In the meantime, the gist contains all you need to get it working:

https://gist.github.com/rodolfograve/16e603a86565536063932e03fa4afc10

Risk, acceptance tests, why unit tests are not enough, and what can you do about it

It's time to release again (hopefully after only 1-2 weeks after your previous release) and you must answer the question: how confident are you that the software is going to do what it's supposed to do?

It's ultimately the Product Owner's decision if you should release or not, but in order to make the decision she must answer that very same question. Based on what? What information can we give her to help her support her decision?

Most teams go about it like this:

Hey, all unit tests and integration tests are passing and we have a 98% test coverage! We've manually checked that the software satisfies the acceptance criteria for the new stories we played, and we also did some manual regression testing. Everything seems to be OK. We're happy to release.

For a team to state they're happy to release when this is all the information they can provide is an act of faith in their skills. They're assuming that because unit tests are passing then the software is going to work as expected.

Unfortunately we've all been trained to fully believe in our unit tests and how good we are: surely if we test the individual methods and then the interactions by faking dependencies, we can imply that everything works, right?

Well, I'd argue there are a lot of assumptions and untested implications in the proposition "the system works because we have good unit tests coverage and those tests are passing". See my previous post about the value of unit tests, and by all means, do your Googling... the subject's been heating up recently.

Long story short, you may have all the unit tests in the world, 100% code coverage (whatever that means), and still have no certainty that the software is going to work as the developers intented. More importantly, because unit tests verify that software does what the developers wanted, you'll have even less certainty that the software is going to do what the Product Owner wanted!

Here's a good exercise you can do to get an idea of the implications you do in this process: take a couple of user stories you have already delivered and released, and try to make a map of the unit tests that guarantee that user story works as expected.

Take 1 minute and think about how would you go about it right now...

The first sign of alert should be that you had never thought about this, which is usually the case.

In most cases only imagining the exercise should be enough, but if you go ahead and are actually able to do it then you should end up with either the realization that you have blind spots (particularly around interactions) or in the best case, a big mess of tests scathered all around the code with no clear relation to a particular acceptance test.

If your Product Owner is not very technical then deciding to release after been given the former assessment on the quality of the release is an act of faith in the team. One most of us has been part of. If the Product Owner happens to be a techy then she's probably into the unit tests "religion" and is as convinced as everyone else that you can release with very high (and false) certainty things are to work as expected.

OK, you got me thinking. What can I do?

What if you could provide the following assessment on the quality:

Hey, all unit tests are passing, all the end-to-end automated acceptance tests which run in a production-like environment are passing. We've done some manual exploratory testing and didn't find any issues.

The automated acceptance tests cover 83% of all the acceptance tests, 98% of those considered critical and 95% of the new acceptance tests we've just implemented.

We're happy to release.

If you've followed your process correctly then acceptance tests represent all your Product Owner cares about. If you can tell the Product Owner 83% of the acceptance tests have been verified in a production-like environment you're basically telling her that at least 83% of the features are known to work (the only risk being how different from production the test environment is). And she can now make an informed decision of the type "no, I really need X and Y to be verified" or "great, let's go ahead".

But isn't the end-to-end automation of acceptance tests expensive?

Yes, it is. It will consume time you could spend doing something else, like implementing new features. But assuming your software is of a reasonable complexity it should very quickly be impossible to manually verify each acceptance test which is why I included risk in the title.

It's all about managing risk. If the Product Owner and the team are happy to live with the risk of regressions of bad behaviour then it makes sense to cut down on this automation, or apply it only to critical areas. In most cases this is not the case though, and the only reason releases are approved is because there's no clear picture of what the certainty is.

Here's another exercise (more of a challenge really): next time you have to release, create a list of all the acceptance tests in the stories you have already delivered and released. Then try to explain to your Product Owner how many of those have been validated for this release. Feel free to do all the magic and hand waving you want, just stay honest.

You can say "this acceptance test is covered because we have this and this and this unit tests, you see...", and "this other was tested manually and because it uses most of the same components as acceptance test X then we consider X has also been validated". This other might also work: "we haven't touched this component for a long time and because users haven't complained so far we can safely assume it's working properly".

See how much confidence can you get from your Product Owner for that release.

If your application is in any way critical to your company by this point you'll probably have a very pallid face in front of you. Clear exceptions might be some start-ups, experiments, prototypes, etc.

After perhaps a bit of hatred and arguing, and after pointing out that it's unfeasible to verify every single acceptance test before each release, this would be the time to propose that you invest in the automation of your acceptance tests, in a production-like environment.

Note that all this is very related to BDD, which has been around for a long time, but most people associate BDD with a framework and it also implies that you write your tests first, which is not the point I'm trying to make here.

Any hints on automating the acceptance tests?

Actually, yes, I have a couple of things we've been trying and I haven't seen anywhere else like having a unique set of tests that can be run fast, without any I/O, and also in a full-deployed environment.

Being able to run those tests fast give the developers a very quick feedback on their changes, much like unit tests do, especially if you use a tool like NCrunch.

The full-deployed version will be much slower so it should probably run as part of your continuos integration, but will give you much more confidence in return.

I'll try to post more details about this approach soon, but whatever you do, please start being objective about the quality of the software you're releasing. If you decide to take the step and start to automate your acceptance tests systematically then make sure you invest in quality: you'll need to maintain this code together with your production code..

Are unit tests waste?

[Edit]
If you're interested in how to get real confidence out of your investment in testing, see Risk, acceptance tests, why unit tests are not enough, and what can you do about it

A co-worker sent me this article from James O Coplien a few days ago: http://www.rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf.
If you get past the title (which you should), and don’t take everything it says as “the truth”, there are a lot of interesting things we could take from it. This is my personal resume from the article but I encourage everyone to read it.
You can apply many of these personally, other would require team agreement and others would even require changes to wider policies:

People confuse automated tests with unit tests: so much so that when I criticise unit testing, people rebuke me for critising automation.

Keep this in mind whilst reading the article (or this resume). This is not about removing automated tests, is about the value of testing one method in isolation (unit) versus testing at the system or sub-system level.

(...) don’t forget the Product Owner perspective in Scrum or the business analyst or Program Manager: risk management is squarely in the center of their job, which may be why Jeff Sutherland says that the PO should conceive (and at best design) the system tests as an input to, or during, Sprint Planning (...) Software engineering research has shown that the most cost-effective places to remove bugs are during the transition from analysis and design, in design itself, and in the disciplines of coding. It’s much easier to avoid putting bugs in that to take them out (…) one of my favourite cynical quotes is, “I find that weeks of coding and testing can save me hours of planning.” (...) “There’s something really sloppy about this ‘fail fast’ culture in that it encourages throwing a bunch of pasta at the wall without thinking much… in part due to an over-confidence in the level of risk mitigation that unit tests are achieving”.

This is said in many other places but you can never say it enough. Translated into process: take your time in the design meetings, don’t rush it and think it through! Do your best to get it right the first time. Don’t leave it to “the QA process” to find your errors later.

(...) you can model any program as a Turing tape, and what the program can do is somehow related to the number of bits on that tape at the start of execution. If you want to thoroughly test that program, you need a test with at least the same amount of information: i.e., another Turing tape of at least the same number of bits (…) to do complete testing, the number of lines of code in unit tests would have to be orders of magnitude larger than those in the unit under test (...) Few developers admit that they do only random or partial testing and many will tell you that they do complete testing for some assumed vision of complete. Such visions include notions such as: "Every line of code has been reached," which, from the perspective of theory of computation, is pure nonsense in terms of knowing whether the code does what it should. Unit tests are unlikely to test more than one trillionth of the functionality of any given method in a reasonable testing cycle. Get over it. (Trillion is not used rhetorically here, but is based on the different possible states given that the average object size is four words, and the conservative estimate that you are using 16-bit words).

These are some of the reasons why we absolutely need automated Acceptance Tests. It also links to the fact that we shouldn’t care about “impossible” scenarios: if something can’t happen in production then we don’t need to test it, which in turn links back to minimizing configuration options because each option increases the testing effort exponentially.

The purpose of testing is to create information about your program. (Testing does not increase quality; programming and design do. Testing just provides the insights that the team lacked to do a correct design and implementation.)

Never, ever, lose sight of this.

The third tests to throw away are the tautological ones. I see more of these than you can imagine — particularly in shops following what they call test-driven development (...) However, as with most unit tests, it’s better to make this an assertion than to pepper your test framework with such checks (...) When I look at most unit tests (...) they are assertions in disguise. When I write a great piece of software I sprinkle it with assertions that describe promises that I expect the callers of my functions to live up to, as well as promises that function makes to its clients. Those assertions evolve in the same artefact as the rest of my code (...) Turn unit tests into assertions. Use them to feed your fault-tolerance architecture on high-availability systems. This solves the problem of maintaining a lot of extra software modules that assess execution and check for correct behavior; that’s one half of a unit test. The other half is the driver that executes the code: count on your stress tests, integration tests, and system tests to do that.

This is what Code Contracts should be doing for us. Unfortunately Code Contracts seems to be taking too long to become a finished product but if you choose to get rid of it then you should at least replace the contracts with Debug.Assert.

(...) one question to ask about every test is: If this test fails, what business requirement is compromised? Most of the time, the answer is, "I don't know." If you don't know the value of the test, then the test theoretically could have zero business value. The test does have a cost: maintenance, computing time, administration, and so forth. That means the test could have net negative value.

Make sure you link your automated acceptance tests back to the original acceptance test which in turn should be linked to a requirement (User Story, Use Case, etc). If you're using some BDD framework then you might have the option to consider the automated tests as your actual acceptance tests.

If you cannot tell how a unit test failure contributes to product risk, you should evaluate whether to throw the test away. There are better techniques to attack quality lapses in the absence of formal correctness criteria, such as exploratory testing and Monte Carlo techniques. (Those are great and I view them as being in a category separate from what I am addressing here.) Don’t use unit tests for such validation.

Again, Acceptance vs Unit Tests. They aren’t the same and Acceptance Tests that link directly to requirements provide much more value.

Most programmers believe that source line coverage, or at least branch coverage, is enough. No. From the perspective of computing theory, worst-case coverage means investigating every possible combination of machine language sequences, ensuring that each instruction is reached, and proving that you have reproduced every possible configuration of bits of data in the program at every value of the program counter. (It is insufficient to reproduce the state space for just the module or class containing the function or method under test: generally, any change anywhere can show up anywhere else in a program and requires that the entire program can be retested. Long fragment but very interesting. This is why we need to minimize the number of interactions in a system.

Even if “in general” his explanation seems correct, a good design can (and should) mitigate this by applying encapsulation, SRP, cohesion, preferring immutable state when possible, etc. Make each module in the system be a black-box API to other modules, and define the interaction with well-defined contracts.

(...) The classes he was testing are code. The tests are code. Developers write code. When developers write code they insert about three system-affecting bugs per thousand lines of code. If we randomly seed my client’s code base — which includes the tests — with such bugs, we find that the tests will hold the code to an incorrect result more often than a genuine bug will cause the code to fail! Interesting point.

The numbers are clear but I believe he’s not considering that tests should be much simpler than “production code”, so the bug ratio must be much lower (the number of bugs is not a linear function of the complexity but exponential or similar).

Create system tests with good feature coverage (not code coverage) — remembering that proper response to bad inputs or other unanticipated conditions is part of your feature set.

No comments. Just do it.
In summary (from the article itself, not my words. Bold means I consider it controversial, the rest I consider we should be doing it):

1. Keep regression tests around for up to a year — but most of those will be system-level tests rather than unit tests.
2. Keep unit tests that test key algorithms for which there is a broad, formal, independent oracle of correctness, and for which there is ascribable business value.
3. Except for the preceding case, if X has business value and you can test X with either a system test or a unit test, use a system test — context is everything. I understand he’s using “unit tests” referring to the scope (1 method). Speed is a different concern here.
4. Design a test with more care than you design the code.
5. Turn most unit tests into assertions.
6. Throw away tests that haven’t failed in a year.
7. Testing can’t replace good development: a high test failure rate suggests you should shorten development intervals, perhaps radically, and make sure your architecture and design regimens have teeth.
8. If you find that individual functions being tested are trivial, double-check the way you incentivize developers’ performance. Rewarding coverage or other meaningless metrics can lead to rapid architecture decay.
9. Be humble about what tests can achieve. Tests don’t improve quality: developers do.

NuGet.Server + IIS 7.5: nuget push returns 404 error with large packages

After installing a NuGet server using the NuGet.Server package I was getting 404 error when trying to push packages:

F:\test>NuGet.exe push MyPackage.1.0.0.0.nupkg -Source http://nugetrepo/ -ApiKey myApiKey

Pushing MyPackage 1.0.0.0 to 'http://nugetrepo:81/'...

Failed to process request. 'Not Found'.
The remote server returned an error: (404) Not Found..

Short version

The cause in my case was that the package was too big. If your package is moderately large you need to set the maxAllowedContentLength setting to a larger number:

<system.webserver>
    <security>
        <requestFiltering> 
                <requestLimits maxAllowedContentLength="32212254720"> </requestLimits>
        </requestFiltering>
</system.webserver>

This setting controls the biggest file size accepted by IIS. Notice this setting is specified in bytes.

You also need to set ASP.NET's maxRequestLength setting to a corresponding large number (this time in Kb):

<system.web>
    <httpRuntime maxRequestLength="31457280" targetFramework="4.5"></httpRuntime>
</system.web>

Full version

Obvious things discarded (wrong api key, wrong URL, server not available, etc) I then turned to Fiddler. 2 requests are made by the NuGet.exe client:

• The first one a GET to the site root (http://nugetrepo). This is sucessful.
• A subsequent PUT to http://nugetrepo/api/v2/package which fails with a 404 error.

I went on and enabled IIS Failed Request Tracing (see here) which gave me the key piece of information: the error was a 404.13. For some reason I didn't see that 404.13 on Fiddler.

Once I knew the cause it was fairly straight forward to find the solution.

Hope this helps! It will certainly help myself in the future.

TEAM.BabyHost: leave your baby playing in the PC

TEAM.BabyHost

If you ever wanted to leave your baby or toddler playing some game in the computer without fearing he will inadvertently (sometimes I doubt it!) do nasty things, like closing all your other windows, open every possible  application and discovering all kind of keyboard shorcuts you had never imagined existed:

TEAM.BabyHost is your friend

TEAM.BabyHost lets you open any web page in a controlled sandbox. With so many flash games already available you just need to navigate to the web page or point it to a local flash file.

Get your copy here in bitbucket.

How does it work?

TEAM.BabyHost is a full screen application which intercepts most of the special keyboard shortcuts available in Windows applications. In fact, it can intercept all shortcuts except for Ctrl+Alt+del and Windows+L. Those might be coming in the future though so if you like the application stay tuned.

As a full-screen application it prevents babies from using the mouse outside of the window too!

How does it look?

Feedback

Your feedback is very welcome. Get in touch by sending an email to rodolfograve@gmail.com, by leaving a comment here or by creating an issue in bitbucket.

Single Action Controller: more improvements

It's been more than a year since the last entry and I've done more improvements to the implementation of Single Action Controller in TEAM.Commons.Web. Most of the improvements have been around removing magic strings and using types instead.

The following gist shows some examples of what you get:


If you like type safety and finding errors at compile time rather than build time, you're going to love this approach, together with enabling view compilation.

On top of that I've also added some methods to help with testing of applications built with this library. This is how you verify that an action returns a View:


And this is how you check that an action returns a redirect to another Single Action Controller:

Single Action Controller - Take 2

After almost a year of using the first version of Single Action Controllers over the ASP.NET MVC (http://rodolfograve.blogspot.com/2011/05/teamcommons-mvc-single-responsibility.html), I have applied a few improvements on it. I particularly didn't like the requirement to inherit from a custom class (SingleActionController), and the boilerplate code required in your Global.asax.

So, the current implementation (also available as a NuGet package at http://nuget.org/packages/TEAM.Commons.Web) doesn't require your controller class to inherit from SingleActionController but from the usual Controller class, and provides a helper method for Autofac users:

Global.asax:

protected void Application_Start()
{
  var builder = new ContainerBuilder();

  // Register your dependencies

  builder.RegisterTypesForSingleActionControllers(Assembly.GetExecutingAssembly(), "Base namespace of your controllers");
}

A controller:

using MyCompany.MyProject.ViewModels.Author;

// Name your namespace after your controller.
// This works very nicely with the directory structure you create in your project.
namespace MyCompany.MyProject.Controllers.Author
{
  // Name your class after your Action and make it inherit from the standard Controller
  public class Index : Controller
  {
    // One single action named Execute
    public ActionResult Execute(FilterDataModel filter) { ... }
  }
}

TEAM.Commons: MVC + Single Responsibility Principle (SRP) = Single Action Controller

Since a long time ago I've been concerned about the increasing size and complexity of our ASP.NET MVC controllers: we have been clearly going against the Single Responsibility Principle.

UPDATE:

an improved version of this implementation is available here.

Common controllers

This is some sample code to illustrate how one of our typical controllers looked like:

using MyCompany.MyProject.Models.Author;

namespace TEAM.MvcSACDemo.WebUI.Controllers
{

  public class AuthorsController : Controller
  {

    // Dependencies (smell: too many dependencies)
    protected readonly IDbSessionFactory DbSessionFactory;
    protected readonly IIndexQueries IndexQueries;
    protected readonly IEditQueries EditQueries;

    // Actions
    // Smells: too many methods; too many qualifications in the classes' names)
    public ActionResult Index(AuthorsFilterDataModel filter) { ... }

    public ActionResult Create() { ... }

    public ActionResult Insert(AuthorInsertDataModel newAuthor) { ... }

    public ActionResult Edit(long id) { ... }

    public ActionResult Update(AuthorUpdateDataModel) { ... }
  }
}

The violation of the Single Responsibility Principle should be obvious.

Our new approach

My initial thought was to change the rule we have been using to create a controller, which is one controller per entity, and find some other rule that produces more controllers with less actions while keeping or increasing the organization level.

A few days ago I stumbled on this post by Derek Greer, and immediately asked my friend Yaniel Díaz help me to develop this idea. After a couple of iterations we had a couple of classes and extension methods that will let us (and also you if you like) implement Single Action Controllers keeping must of the nice ASP.NET MVC conventions.

First, let's see how do Single Action Controllers look like:

using MyCompany.MyProject.Models.Author.Index;

// Name your namespace after your controller.
// This works very nicely with the directory structure you create in your project.
namespace TEAM.MvcSACDemo.Controllers.Author
{
  // Name your class after your Action and inherit from SingleActionController
  // You could choose to name it IndexAction instead
  public class Index : SingleActionController
  {
    // Only one dependency
    protected readonly IIndexQueries IndexQueries;

    // One single action (notice the use of namespace to remove extra qualification in the classes' names)
    public ActionResult Execute(FilterDataModel filter) { ... }
  }
}

And this is how it all looks in the Solution Explorer:

Except for the "Author" and "OtherEntity" nested folders, it's all like "traditional" MVC.

Using it in your application

All the required classes are available in the TEAM.Commons package under the TEAM.Commons.Web.MVC.SingleActionControllers namespace. TEAM.Commons is also available as a NuGet package, which is the preferred way to share code these days.

After you install TEAM.Commons in your solution (or copy the code you need), you must register your routes using the custom SingleActionControllerRoute:

public static void RegisterRoutes(RouteCollection routes)
{
  routes.IgnoreRoute("{resource}.axd/{*pathInfo}");
  // Best practice to avoid MVC handling the favicon request
  routes.IgnoreRoute("{*favicon}", new { favicon = @"(.*/)?favicon.ico(/.*)?" });

  // Only change the original MapRoute by our MapSingleActionControllersRoute
  routes.MapSingleActionControllersRoute(
    "Default",
    "{controller}/{action}/{id}",
    new { controller = "Author", action = "Index", id = UrlParameter.Optional },
    new { }
  );
}

and then register SingleActionControllersFactory as the ControllerFactory:

// You must pass the fully qualified name of the assembly
// and the namespace of the controllers. 
// In future versions we'll simplify this step.
ControllerBuilder.Current.SetControllerFactory(new SingleActionControllerFactory(Assembly.GetExecutingAssembly().FullName, "TEAM.MvcSACDemo.Controllers"));

The internals

Thanks to ASP.NET MVC design we just had to implement a custom ControllerFactory, a custom Route and a custom base class for controllers.

Check the source code at the TEAM.Commons repository. It should be self-explanatory and could give you ideas about how to do it yourself in case you don't want to use TEAM.Commons.

Conclusions

The implemented solution is not perfect and I'm sure it could be improved, but we needed to go back to SRP in our controllers as soon as possible. I'll update this post when we find a new solution.

I hope you find this as useful as we do :-)

Unobtrusive and useful Google Chrome plugins for developers

There are zillions of plugins for Google Chrome, and several of them are developer-oriented. However, some of these plugins interrupt my usual workflow, or make Chrome slow and unresponsive, or are simply not well tested and crash.

These are the plugins I've found to be more useful:

• FireBug Light (as good as in FireFox).
• Screen Capture - by Google (there are many screen capture plugins, I just picked this one)
• Resolution Test (another classic, very useful)
• jsshell (Amazing for testing javascript and jQuery)
• Magic Inputs Filler (saves a lot of time of filling forms for testing).
• CSSViewer
• Simple REST client (Build custom HTTP request to test your services)

TEAM.Commons published as a NuGet Package

NuGet is a nice tool to share code (source or compiled).

I've published TEAM.Commons as a NuGet package in http://nuget.org/List/Packages/TEAM.Commons. With this step it's much easier to use this set of classes: just type Install-Package TEAM.Commons in your NuGet console.

Unobtrusive jQuery UI (jquery.ui.unobtrusive)

I am very enthusiastic about unobtrusive JavaScript. It's a design pattern and usual warnings apply:

• It doesn't apply to every scenario.
• It will hurt if you use it wrong.
• Long etc.

Very short version of this post

I hope someone creates a complete jquery.ui.unobtrusive plugin. Since I'm a developer I have already started one at:
https://bitbucket.org/rodolfograve/jquery.ui.unobtrusive

The bad news is that I'm not an expert JavaScript nor jQuery developer, so I'm sure the quality of what I'm doing can be greatly improved. It will be great if someone with more expertise could continue this work or start his own and make it public.

Long version

I think unobtrusive JavaScript solves several old and current problems:

• How do we make advanced HTML + JavaScript stuff from server-side code?
• How do we reuse JavaScript code that interacts with the elements of a page?

Until now I've seen incomplete and ugly approaches (the latter is obviously a personal opinion):

• Don't do advanced HTML + JavaScript from server-side code at all and create custom JavaScript code for each page you have.
• Generate small pieces of script from server-side code, intermingled with the final HTML code (outside the head tag), and handle elements' IDs and classes.

With unobtrusive JavaScript we have a new great approach: generate HTML attributes in your tags and let your unobtrusive scripts do their science (I appreciate science more than magic).

This new approach is:

• Flexible: you can use whatever you like to generate the attributes (ASP.NET MVC HtmlHelpers, your own framework, write attributes by hand in your HTML, etc).
• Testable: unobtrusive scripts can (or even must) be developed as reusable components, with their own tests (JavaScript as a Framework).
• Readable: your HTML code will declaratively express what it does (no need to look IDs nor classes in JavaScript files).

No more obscure separation between the appearance and the behavior. This separation is good and encouraged, but it's much better without the "obscure" feature added by current implementations (you must look your IDs or CSS classes in your JavaScript code to know what's going on with an HTML element).

<html>
  <head>
<script type="text/javascript">
  $(document).ready(function() {
    $("myDatePicker").datepicker({
      showOn: "button",
      buttonImage: "../themes/mytheme/calendar.gif"
    });
  });Notice this piece of script usually is in another file (even worst).
</script>
  </head>
  
  Lots of stuff, and then hidden some where in the page:

  <input id="myDatePicker" name="myDatePicker" type="text" >

With unobtrusive JavaScript you keep the separation at the implementation level, and avoid the separation at the code level.

<input type="text" ui-datepicker="true" ui-datepicker-showOn="button"
                                             ui-datepicker-buttonImage="../themes/mytheme/calendar.gif" >

This last code looks much better to me and I wish we soon have the unobtrusive "mapper" of the most popular JavaScript frameworks out there.

What do you think?

TEAM.Commons: Messaging

This is the 6th of several posts about TEAM.Common, a set of functionality I use in every project and that I'd like to share. The index is here: http://rodolfograve.blogspot.com/2010/09/teamcommons-introduction.html

Before talking about the details of TEAM.Commons.Threading I need to explain TEAM.Commons.Messaging, a set of classes about communication between processes based on the concept of transport.

ISendingTransport

public interface ISendingTransport
{
    void Send(object message);
    void Send(object[] messages);
}

IReceivingTransport

public interface IReceivingTransport
{
    // Blocks the current thread until a message is received.
    IList<object> WaitForMessageWithoutTimeout();

    /// Blocks the current thread until a message is received or timeout ellapses.
    IList<object> WaitForMessage(TimeSpan timeout);
}

There are two available implementations of these transports:

InProcessTransport

public class InProcessTransport : ISendingTransport, IReceivingTransport
{
...
}

This class implements both interfaces so that it can be used for communications between threads, handling all the related complexity. It's a very important piece used in the implementation of the advanced threading features in TEAM.Commons.Threading.

MsmqSendingTransport
Send messages through an MSMQ Queue.

MsmqReceivingTransport
Receives messages from an MSMQ Queue.

Remember, you can get all this code for free at bitbucket: http://bitbucket.org/rodolfograve/team.commons/overview

Check it out to get ideas, or simply use it as it is. It's working out for me and my team.

TEAM.Commons: Threading - part 2

This is the 5th of several posts about TEAM.Common, a set of functionality I use in every project and that I'd like to share. The index is here: http://rodolfograve.blogspot.com/2010/09/teamcommons-introduction.html

In TEAM.Common.Threading you can find these abstractions:

WorkerThread
A thread running a task in an "infinite" loop, until some other thread asks it to stop.
This is a very basic piece I use it all the time for implementing services and it's also the foundation for the "advanced" features.

public abstract class WorkerThread
{
 Exception Error { get; }
 void ForceStop();
 bool IsStopped { get; }
 void Start();
 void Stop();
 event EventHandler<System.ComponentModel.RunWorkerCompletedEventArgs> Stopped;
}

ProducerConsumersProcess
My approach to parallelization was to use the one producer -> several consumers technique:
You must create an instance of this class passing an instance of IProducer and a list of IConsumerWorker.

public class ProducerConsumersProcess<T>
{
    int ConsumedItemsCount { get; }
    int ProducedItemsCount { get; }
    void Start();
}

With a very simple interface, ProducerConsumersProcess<T> coordinates the producer and the consumers using a buffer (an InProcessTransport, see next post) and handling a lot of edge cases and exceptions.

IProducer

public interface IProducer<T>
{
    T ProduceOne();
    bool HasFinished();
}

IConsumerWorker<T>

public interface IConsumerWorker<T>
{
    event EventHandler<RunWorkerCompletedEventArgs> Stopped;
    void Start(IReceivingTransport receivingTransport);
    void StopWhenStarved();
    void InterruptExecution();
    bool IsStopped { get; }
    int ProcessedItemsCount { get; }
    Exception Error { get; }
    string Id { get; }
}

These are the foundations for all the good stuff. The rest are classes that help in the most common cases:

DelegatedProducer<T>
Just pass two delegates:

public DelegatedProducer(Func<T> producerDelegate, Func<bool> hasFinishedDelegate)

Combine it with lambda expressions and you have a very flexible and expressive syntax.

EnumeratorProducer<T>
Just pass an IEnumerable:

public EnumeratorProducer(IEnumerable<T> enumerableSource)

Combine it with the yield operator and/or lambda expressions and again, you have a very flexible and expressive syntax.

EnumeratorProducer<T>
Just pass an IEnumerable:

public DelegatedConsumerWorker(string id, TimeSpan starvationTimeOut, int processedItemsCountMonitorizationStep, Action<T> consumerDelegate)

You know what comes now: mix it with lambda expressions and...

Remember, you can get all this code for free at bitbucket: http://bitbucket.org/rodolfograve/team.commons/overview

Check it out to get ideas, or simply use it as it is. It's working out for me and my team.

TEAM.Commons: Threading - part 1

This is the 4th of several posts about TEAM.Common, a set of functionality I use in every project and that I'd like to share. The index is here: http://rodolfograve.blogspot.com/2010/09/teamcommons-introduction.html

(Multi)Threading is a complex, error prone task. You must take a lot of details into account everytime you're going to do it. On the other side, it's a very powerful tool and refusing to use it is not a valid solution. At least not if you want to do some large processings or interactive UIs.

AJAX (web), BackgroundWorker and Dispatcher (desktop), have solved the interactive UIs part (at least I feel comfortable enough with those these days), but I didn't feel the same about large processings that could be parallelized.

At some point I got tired of avoiding multithreading until there was no other option and got my self to implement some abstractions that would allow me and my team to implement parallel processes really, really quickly, with industry level robustness.

The result is TEAM.Commons.Threading. Next thing is an example so that you can decide if you like it and want to go on reading this post, or stop right here.

// Create the list of consumers... we are creating 5 consumers for this example
var consumers = Enumerable.Range(1, 5).Select(x =>
        new DelegatedConsumerWorker<string>(
            "Consumer" + x.ToString(),
            TimeSpan.FromSeconds(1),
            2,
            item => Console.WriteLine("Processed: " + item)
        )).ToArray();

// Create a producer based on an IEnumerable of strings from "Item-01" to "Item-20".
var producer = new EnumeratorProducer<string>(Enumerable.Range(1, 20).Select(x => "Item-" + x.ToString("00")));

// Create the MainProcess... all the internal logic for parallelization is here.
ProducerConsumersProcess<string> p = 
    new ProducerConsumersProcess<string>(
        producer,
        TimeSpan.FromMilliseconds(800),
        consumers,
        3);

// Start the process. This call blocks until the parallel process fails or ends.
p.Start();

Console.WriteLine("Process finished!");

Output is:

Processed: Item-01
Processed: Item-02
Processed: Item-03
Processed: Item-04
Processed: Item-05
Processed: Item-06
Processed: Item-08
Processed: Item-09
Processed: Item-07
Processed: Item-11
Processed: Item-12
Processed: Item-10
Processed: Item-14
Processed: Item-16
Processed: Item-17
Processed: Item-13
Processed: Item-15
Processed: Item-18
Processed: Item-19
Processed: Item-20
Process finished!

If you run the code (TEAM.Commons.Threading.SampleConsole) you'll see how the execution pauses at some points. That's because the buffer gets emptied and the consumers are waiting for the producer to produce more items. In the real world you should adjust the parameters to avoid these pauses: buffer size, producer pause time when buffer is full, consumers pause time when buffer is empty.

Next post will be about the details of TEAM.Commons.Threading.

Remember, you can get all this code for free at bitbucket: http://bitbucket.org/rodolfograve/team.commons/overview

Check it out to get ideas, or simply use it as it is. It's working out for me and my team.

TEAM.Commons: Formatting objects for human reading

This is the 3rd of several posts about TEAM.Common, a set of functionality I use in every project and that I'd like to share. The index is here:http://rodolfograve.blogspot.com/2010/09/teamcommons-introduction.html

Converting an arbitrary object into a human-readable string is something I need all the time, specially for logging.

TEAM.Commons has the PrettyFormat() extension method for the object class that performs this task. For an instance of a class like this:

public class CyclicClass
{
    public int IntProperty { get; set; }
    public SimpleClass SimpleProperty { get; set; }
    public CyclicClass CyclicProperty { get; set; }
}

The output of .PrettyFormat() would be:

CyclicClass {
IntProperty='7',
SimpleProperty='SimpleClass {
 StringProperty='My string',
 IntProperty='5',
 DateProperty='3/2/2010 12:00:00 AM',
}',
CyclicProperty='CyclicClass {
 nIntProperty='7',
 SimpleProperty='SimpleClass {
  -- EXCLUDED (Too deep) --
 }',
  CyclicProperty='CyclicClass {
   -- EXCLUDED (Too deep) --
  }',
 }',
}

As you can see, by default it will go down only 1 level . Properties beyond that level will be rendered as "-- EXCLUDED (Too deep) --". There is an overload to specify an arbitrary deepness: .PrettyFormat(10).

The PrettyFormat method handles a lot of special cases: enum, IEnumerable, DateTimes, string, int, etc, providing an appropiate format for these types. You don't want to see an string as:

String {
 Length=14,
 Empty=...
}

but rather as:

The value of my string

Thanks to the "magic" of extension methods, you can even call it on null references, saving a lot of "if" instructions:

object o = null;
o.PrettyFormat();

The output of the former code is: <NULL>

Remember, you can get all this code for free at bitbucket: http://bitbucket.org/rodolfograve/team.commons/overview

Check it out to get ideas, or simply use it as it is. It's working out for me and my team.

TEAM.Commons: Infrastructure for the Query part of CQRS (or just for querying)

This is the 2nd of several posts about TEAM.Common, a set of functionality I use in every project and that I'd like to share. The index is here: http://rodolfograve.blogspot.com/2010/09/teamcommons-introduction.html

It all started with some CQRS reading (Command and Query Responsibility Segregation).

The idea of having a separated model only for querying seemed very valuable even outside a "classic" CQRS implementation, so we started applying it everywhere using an architecture like:

At this point, the implementation of the SqlSomeModelQueries was always something like this:

public IEnumerable<somemodel> GetAllWithCondition(string condition1, int condition2)
{
  using (var connection = new SqlConnection(ConnectionString))
  {
    var cmd = connection.CreateCommand("select A, B, C, ... from T1 inner join T2 ... where ...");
    using (var reader = cmd.ExecuteReader())
    {
      while (reader.Read())
      {
        SomeModel result = new SomeModel()
        {
          // Missing null checkings, safe conversions, etc
          A = Convert.ToInt32(reader["A"].GetValue()), 
          B = Convert.ToString(reader["B"].GetValue())
          // ... More properties
        };
        yield return result; // We don't want to keep all the items in memory.
      }
    }
  }
}

Obviously, after implementing this 3 times I decided I needed some infrastructure that will save us from repeating all this code over and over, with all the benefits that come with it.

You can find the complete implementation in https://bitbucket.org/rodolfograve/team.commons. Check the MapperExtensions and DbConnectionExtensions classes. Using this infrastructure the code above is:

public IEnumerable<SomeModel> GetAllWithCondition(string condition1, int condition2)
{
  using (var connection = new SqlConnection(ConnectionString))
  {
    foreach (var item in connection.GetAllWithStreaming<SomeModel>("select A, B, C, ... from T1 inner join T2 ... where ..."))
    // There is an overload to pass an SqlCommand in case you need to use an SqlCommand with parameters.
    {
      yield return item;
    }
  }
}

There are many details you must take into account if you want to do this by yourself:

• Reflection is always very tricky to get right.
• Reflection is slow, so use something like Fasterflect and its cache features.
• There are a lot of special cases.
• You should add as much information as possible in the exception messages.

Remember, you can get all this code for free at bitbucket: http://bitbucket.org/rodolfograve/team.commons/overview

Check it out to get ideas, or simply use it as it is. It's working out for me and my team.

TEAM.Commons: Introduction

This is the 1st of several posts about TEAM.Common, a set of functionality I use in every project and that I'd like to share.

1. Querying.
2. Formatting objects into readable strings.
3. Threading - part 1 (introduction)
4. Threading - part 2 (parallelization)
5. Messaging (in-process and inter-process)

For some time I've been trying to organize, stabilize and share some common code I use in all my projects and that I find myself copying over and over.

This code matures with me and it will be nice to keep some part of my evolution as a developer in this code's history.

Last time I published what I called Romialyo SDK. Since then I have already made some improvements and stopped using git in favor of Mercurial, so here is the new TEAM.Commons.

I'll try to blog about it because I feel there are many things in there that could be used by many people and more importantly, I'd love to get some feedback on this code because it contains what I feel are the best things I've coded.

Any improvement on this code will be a direct improvement on my abilities.

Remember, you can get all this code for free at bitbucket: http://bitbucket.org/rodolfograve/team.commons/overview

Check it out to get ideas, or simply use it as it is. It's working out for me and my team.

Another award for TEAM's TimeTracker

This has happened many times in the last few months, but every time is important:

TEAM's TimeTracker has been awarded the "Famous Software Award" by "Download.FamousWhy.com":


These later awards have inspired me to spend some time on my old applications trying to make them better, and I have recently released version 2.4.0 of TEAM's TimeTracker and version 0.2.0 of TEAM's Db4o Management Studio.

I'd like to ask everyone who is using or have used these application to leave their comments and suggestions at bitbucket.org:

• TEAM's TimeTracker: http://bitbucket.org/rodolfograve/team.timetracker/issues/new
• TEAM's Db4o Management Studio: http://bitbucket.org/rodolfograve/team.db4omanagementstudio/issues?status=new

I'm a db4o MVP

Well, this came as a great surprise to me. Today I received an email from db4o.com explaining I had been selected as db4o MVP thanks to my TEAM's db4o Management Studio application. (Blog post here. Download from here.

I'm the 11th in the list of MVPs: dVPDirectory2010

I've been using db4o for a while. About a year ago I created the db4o Management Studio because I found no suitable tool to inspect a db4o database. This is an issue when you don't have the definition of the classes you have stored in that database, or even when you just don't want to spend time writing an application to inspect the database.

Finally, here is an extract from the email:

Let me be the first to congratulate you! You have been selected as a db4o Most Valued Professional (dVP) awardee for the year 2010.
The dVP is our community program first launched at the db4o User Conference (dUC) in July 2006 in London. The purpose of the initiative is to honor individuals like yourself who have shown a large commitment towards the db4o Community - by providing valuable peer support in the forums or outstanding contributions to the db4o software ecosystem (such as db4o related projects, articles, books, translations, etc) or by consistently spreading the word about the db4o technology (through blogs, webcasts, etc). The status is awarded for a given year (in this case 2010) and has to be re-earned every year.

Nice :-)

Handling configuration in Enterprise environments

Have you ever faced the problem of handling a complicated configuration in an Enterprise environment where there are dozens of components interacting (databases, web services, FTP sites, queues, etc)?

Mix this with automated deployment and some deployment environments (integration, QA, staging, production) and you have a complex problem to solve. A wrong solution could undermine your whole architecture, making deployment chaotic and untestable.

We've faced this situation and we've come up with what I believe is a nice solution. Let's state our requirements, which is always a good exercise when solving a problem:

R1- Automatic deployment
We need our solution not to interfere with automatic deployment.

To me, this means to keep code and configuration separated. I don't want a super-complex deployment script which must handle deployment in every environment. I also don't want to include all my environment's passwords in this deployment script.

R2- Ease of use and maintenance for system administrators
Our solution must be easy to use and maintain for system administrators. It must be very easy for administrators to create and maintain configuration for each environment and to track changes done to the configuration.

R3- Robustness
I want all the applications to share the same configuration, specially in our environment, where several applications interact with the same databases/web services/etc. It must be impossible for application A to point to a database Db_staging while application B points to Db_integration. They must always point to the same database.

R4- Flexibility
At least in our case, the staging/QA/integration environments are far from being identical to our production environment, mainly due to infrastructure expenses: we have 10+ servers in production and we only have 2 servers per environment. We can not afford to have 10+ servers per environment.

Besides, you never know what a developer will need in order to test the application in their boxes.

So, our solution must allow a great deal of flexibility, assuming nothing about the different environments.

R5- Ease of use for developers

It must be very easy for developers to get the configuration values and to tamper with them in their development environments.

R6- Security
Finally, for us, it was very important to have one single database user per application. But this could be a very particular requirement.

The (simplified) solution
S1- Global configuration file: create a file (or files) that contain all your configuration resources (connection strings to databases, ftp connection details, urls to web services, etc).

We choose an XML file with key/values:

<add name="MyDB1" value="Data Source=stagingSQL;Initial Catalog=MyDB;User ID=user;Password=pass"/>

S2- Create a copy of this file per environment, and put it under version control so that you can check for modifications.

S3- Create a project in your CI server that deploys this configuration file(s) to the right servers when a change is detected.

So far we have handled requirements:
R1- We have automatic deployment of the configuration because it is a simple plain-text file under version control that is automatically deployed by our CI server (or any other mechanism you prefer).

R2- System administrators only have to commit changes to the configuration files in the right branch in order to modify an envinroment's configuration.

R3- There isn't a configuration file per application but a single "source of truth": the global.config file. There is no way you can get two different values for the same key.

R4- There are no limits to what you can place in a text file. The only requirement is that the keys are the same for all the files, otherwise the applications won't find the resources they are asking for. That should be flexible enough and some naming conventions can make it simple.

R5- Build an API to access the configuration values. Make it very transparent to users. Our client calls look like this: Environment.Current.GetDbConnectionString("MyKey"), which is very similar to AppSettings.ConnectionStrings["MyKey"].

R6- Using a very simple template for the values you can make your API to automatically handle this kind of security:

• Have a users.config file where you have ApplicationIds -> User/Password.
• Make your API request the running application to provide its ApplicationId before the application can access any configuration values: Environment.Initialize("MyAppId"); Any call to Environment.Current should fail if no ApplicationId has been provided.
• Make your connection strings look like this:
  <add name="MyDB1" value="Data Source=stagingSQL;Initial Catalog=MyDB;User ID={user};Password={password}"/>
• Finally, make your GetDbConnectionString load the ConnectionString, the ApplicationId -> User/Password values, and replace the placeholders with the actual values.

We have found this approach to be a very nice solution. Hope it helps you find your own, or you can use this one as it is.