Castle Windsor, Service Locator and a possible memory leak !!!

In recent days, in one of my projects, we found an innocently looking bug that was a little too overweight – sometimes weighing more than 6GB in size in a 8GB web server. Well, a memory leak that caused the application pool to crash by exhausting the system memory. So I thought that, why not share it here so that if anyone else, facing the same problem, can get some insight. Here goes the plot –

The main goal of the project was to build a modular framework to be used to build robust web applications easily. For that reason the most obvious choice was to use Inversion of Control Pattern.

If you are already here then it is highly normal that you are familiar with the IOC( Inversion of Control) pattern already, but if you are not, then here is a good article for you.

As you all know, castle windsor is a widely used service locator library we used it for our IOC implementation. Therefore, we registered our services like this –

_container.AddComponentLifeStyle<IDocumentService, DocumentService>(LifestyleType.Transient);

If you are new to castle windsor and the life cycle of the items it resolves, grab its details from here.

In short, LifestyleType.Transient means each time you ask castle windsor to resolve or give you an instance of an interface it will create a new one each time. But the default implementation of castle windsor with transient life cycle does the following –

  • It creates a new instance each time ask for
  • Keeps a reference of that instance created, so that it can refer it later
  • Assumes that you will release the instance manually, again remember it, manually.

The first two options is not a problem, but the third one is a possible leak. If in any case you do not release the instance manually its a never-ending memory leak. Because the default garbage collector of CLR will never clear it since castle windsor is holding a reference and castle will never clear it because it thinks you will clear it yourself.  That’s what we exactly did and well it took only 2 hours to consume all the server memory (6GB +) and crash it.

If you are interested about the possible memory management scenarios, see this article here, I realized our problem reading this one.

Well, now comes the big question – Whats the easiest solution?

Well you can release all the instance manually that you resolved using castle windsor or you can grab the solution mention here.

I will save your time. Just in the castle service locator where you are creating the kernel, add the following lines –

_container.Kernel.ReleasePolicy = new NoTrackingReleasePolicy();

Basically what it does is that, it prevents castle windsor from keeping any references of resolved instances and thus when you are done with your code and the object needs releasing the default GC collects it and the memory is freed. Which removed the memory leak problem. And you know what now the memory consumption never goes over 600MB. 🙂

It’s highly usual that you are using NHibernate with castle windsor and you think NHibernate is causing the leak?.. well don’t be harsh on NHibernate ; its castle and your unreleased instances, who is causing the leak. 😀

Managed Extensibility Framework –

Introduction

Before I begin lets discuss about a common development scenario and how we might deal with it –

Suppose we have a requirement as follows (This was a real-time scenario once for me )-

  1. The system will collect table data from flat files,  like – plain text (txt).
  2. The task is to collect column information and show the list of columns in a Grid.

Now, lets consider we have started our development process and also completed it and delivered. Your client is happy and so are you.

Now, comes the tricky part. A few days later your happy client asks –

 could you please add an option so that we can also parse Excel WorkSheets, such as xls, xlsx, xlsm, xlsb files or may be delimited CSV files?

If you never anticipated this would happen, I assume you didn’t keep an option to add a parser later. Now what should you do? You will open the source again and modify it and them again build and deliver it. For a small project like this it is still troublesome, there are other projects with millions of lines of codes. What will you do with them? Open the source again and build?

Oh Boy, ain’t that tiresome!!!

Lets, talk about another approach –

Instead of hard coding the parser in the application could we could have done the following –

  1. Create a Plugin directory and keep a parser as a portable assembly (.dll) which will have a generic interface to expose methods.
  2. When executed the program will search for available plugins in that folder and will see that it has Text (.txt) file parser.
  3. Then the program will issue commands or all methods based on exposed interface of the parser.

Now, in this situation even if my client wants more parser, we can just build a new one implementing the base interface and build it as a class library and finally just put it in the Plugin folder and the parse will work out of the box.

Now, I think you have got a rough idea of what I am talking about. So lets, just mention it in words –

Using Managed Extensibility Framework(a.k.a MEF) we can do the above mentioned thing in minutes. In other words, we can inject business logic right into the application and will not even need rebuild it.

Before we start coding, you might wanna know where to get it. If you are using .net framework 4.0 you already have it and for those who are using 3.5, you can get the source from http://mef.codeplex.com/ and add a reference to the sources to get the feature sets. I haven’t tried with other versions of .net framework but if you still need to know, post me a comment and I will dig a little more deeper to get you the info.

 

The Solution

We should create and organize out solution first.

Here is the basic process that I followed for this session –

  • Open MS Visual Studio ( I am using 2010 ultimate and will use C# + .net framework 4.0 for this project) and create a Class Library project. I would name this project MEFIntro.Lib.
      I use “.” extended project names to keep the names meaningful and  easy to mange. Its not a restriction
  • Remove the default class file “Class1.cs” from the project as we will not be needing this and we will create our classes ourselves.
  • So far we have created the library project. We also need an executable project that we can use to show the output of the library. So lets add another project named MEFIntro.Desktop and this would be a Windows Forms Application. I also renamed the form to “MainForm“.
  • Add a reference from MEFIntro.Lib to MEFIntro.Desktop.
  • This is how your screen should look like –
  • Now we will need one more project, that we will use to create Extensions. Lets name it MEFIntro.Lib.Ex.
  • Add reference for the MEF library (System.ComponentModel.Composition) to both of your projects – MEFIntrol.Lib and MEFIntro.Lib.Ex.
  • Our final solution will look like this. We have all the projects that we need for this tutorial. If more is needed we will add later

 

The Extensible Core

Lets create the core section of our MEF Project.

By the by, I have downloaded and installed MS Visual Studio 11 Beta and loving it very much. So, from now on all other tutorials and blogs will be based on MS Visual Studio 11 Beta. Hope you will like the IDE too, its loaded with cool staffs….. 🙂

The core engine of our project will be responsible for the following tasks –

  1. Load all plugins and extensions those we will create and keep as plug & play components.
  2. Expose methods that our library will support.
  3. Provide Options and Interfaces for future plugin and extension development.

Okay, enough high level blabbering, lets jump into more low-level description. To support all the features mentions above we need to expose some interfaces, so that our extensions can inherit them and build their own logic in it, more like a strategic pattern. And at run-time our core engine will collect them using MEF and execute as needed.

When I said “interface” in above paragraph, I literally meant Interface of C# programming language. So don’t be afraid, we are inside the codes. Literally!!!

next, we need to expose some methods to be implemented in the implemented class. Our requirement is to collect column list i.e. metadata, from the files. So lets call this method “GetMetaData“. So our core engine interface will look exactly like the following –

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace MEFIntro.Lib.Interface
{
	public interface ICoreEngine
	{
		String[] GetMetaData(String filename);
	}
}

So far, we have expose the interface that our extension methods and classes will use to integrate themselves with the engine. But we haven’t told your engine about the extensions. In other words, our engine still does not know that there could be extensions to use. So lets add 2 interfaces for that-

  1. The top-level interface of the extensions, all extensions targeting our engine will have to implement that interface, otherwise we will not consider as a plugin. I named it – IFileExtension. Our plan here is to use one extension class for one file of files. Like one class for .txt files and another extension class for .xls files. These classes will implement this interface.
  2. The second interface is for holding information about the extension classes. At run-time MEF will search with these data and give us the correct class to use. This idea will be much clear when you see the code in action. For now, lets just put a pin in it. I named this interface – IFileData
Now, we summarize what we have with a screenshot –

I prefer using a single file for all the interfaces that are under one namespace for all my projects, this helps me finding them later on. This is not a requirement. You are free to implement it your way as long as they all are available when needed.

So far we only have the abstract structure of our engine. But we need some concrete ones. Lets, look at our concrete implementation of our concrete core engine. I will describe the codes shortly –

Lets see what we have here –

  1. We have a concrete implementation of our core interface named CoreEngine.
  2. We have overridden the GetMetaData method. For now it is kept blank, we will add implementation logic later.
  3. The class also has some interesting fields, namely –
    IEnumerable<lazy<IFileExtension, IFileData>> operations;
    private CompositionContainer_container;
     This is main container of all the plugins. At run-time the framework will automatically collect our plugins and put them in the operations field. All we have to do is call it in the appropriate manner.
  4. And one other interesting method ComposeParts.
    This method does composition an puts together all the parts. you can understand the codes very easily by looking at it.

So, far we have nothing done with our implementation. Lets complete our core engine and call some extension methods. First we need to add at least one extractor class  for this example. Lets add one for text ( .txt) files. The task is simple just open the files first line, split by the delimiter character and return the string array –

Now since we have one extension that we can call to test our implementation, lets finish the core. First we need to extract the file extension, then check whether we have any extractor defined for this extensions and if we do we will call it and return the value. Our final code for the GetMetaData will look like this –

That’s it, we have the core section complete.

Chapter 1: Introduction to FluentNHibernate

After working over 2 years with FluentNHibernate, I have now decided I should write a step by step learning schedule for Fluent NHibernate. The reason? … Well lets just say it’s better to wait for it. Once you learn FluentNHibernate, I guess, you will never go back to your old SQL query based development, trust me, you won’t.

When I learned FluentNH, it took me a while to find solutions, as help was not that much available all the time.

But I must be glad to those extraordinary people at stackoverflow (http://stackoverflow.com/ ) who helped me find solutions to problems and also helped me learn FluentNH.

This is an approach of mine, to bring all those thoughts and solutions in one place, so that you can find all without going under the hassle that I went through.

Before beginning with FluentNHibernate, I would like to introduce you with it a bit.

For those, who have worked with FluentNH earlier, you can skip the next section.

FluentNH is an ORM in long Object Relation(al) Mapper. Now, the big question is, What is a ORM? well keep reading….

  1. What is a ORM?
    In simple words, I would say ORM is an advanced technique which is actually originated from OOP ( Object Oriented Programming) concept and is created on the sole purpose to provide developers like us, the capability to perform persistent store management, such as managing a database, with a complete OOP approach.Confused? Okay lets see an example of what I meant. Consider  a very simple case -” Suppose you have 1 table in database name “User” with 2 fields in it username and password.Now, lets say we have to create a single user to the database, what will be a our approach? Okay, lets say we are creating a huge ERP solution, and thus we will use the 3-tier architecture. So we have Presentation Layer, Business Logic and Data Access Components. No matter what we use, somewhere in the DataAccess layer we will need to create a SQL connection and generate a query of some thing like –Select * From <Table> Where <Condition> Order By <field>This is just for one single table and running a query from it. Think about the whole database with hundreds of tables. Well, its better not to think about it, you know what I am talking about. This is where a ORM kicks in.A ORM is the abstraction layer, that gives you the opportunity to run queries just like normal codes in C#. For example in one of your codes you might use something like this –Person p = Repository.Get<Person>(“Id”, <someid>);

    Thats right, no more sql queries and no more table adapters and no more data sets. That singe line will do the rest.

    If you use a ORM you don’t have to type all the sql codes by hand. You create a nice simple repository class and let the ORM handle the test. And frankly speaking, until I knew about ADO.Entity Framework (v3), I never thought any library or ORM will be able to compete with FluentNH.

I think, thats enough for you to just give you a very rough idea of what is a ORM. But until you use it in your projects, I guess it won’t be of much help.

Stay tuned for the next articles on FluentNH.