Having fun with code

Building a Web API in SharePoint 2010 with ServiceStack

May 4, 2012 – 11:00 pm

Background

I do lots of SharePoint work/architecture/consulting. Recently, while building out a custom service application, I was exploring various ways of creating a web API, hosted within SharePoint. Obviously, using WCF is the standard in SharePoint, but it certainly is not the only way.

So, that’s when the thought of trying ServiceStack came into existence.

Why ServiceStack? Because it gives you XML, JSON, JSV, CSV, SOAP 1.1 and SOAP 1.2 response types out-of-the-box! That’s pretty cool, but just a start. It also provides a consistent way of developing services and great performance (check out these benchmarks). Check out this interesting read by Phillip Haydon also as he takes ServiceStack for a spin. Most of all, ServiceStack is super fun.

The End-Goal

Ok, so the end-goal is to create the barebone essentials for building out a web service API, hosted within SharePoint 2010, using ServiceStack.

Getting Started

Let’s flesh out the SharePoint project. I’ll create 1 class library project, and 1 empty SharePoint project in my solution. The solution MUST be a Farm Solution, because we’re going to have to do a web.config mod, BUT, only to deploy the infrastructure. This will play really nicely with sandboxed solutions once the API is deployed, especially with jQuery or whatever other JS framework you fancy.

Caviat

You will need to download the zip files or clone the Git repositories if you want to deploy ServiceStack to the GAC. Oh, and I’ll be making 1 tiny little change to the ServiceStack source code (BUT, you don’t have to if you are ok with the alternative, which I’ll describe later). So all in all, you can still make it work without this unfortunate thing, by accepting the alternative web.config mods (described later in this post), and NOT deploying the dlls to the GAC (deploy to the bin instead).

Project Structure

My final solution looks like this:

Deploying ServiceStack

To deploy to the GAC, I strongly typed SSPLoveSS.WebApi, as well as all the ServiceStack libraries, and then I froze the versions on all the libraries for this exercise as well.

To deploy the ServiceStack dlls, either use gacutil on your dev (if you strongly type them like I’m doing), or add the dlls to the SharePoint deployment package and then deploy to the GAC or the bin:

API Development

We’ll create an API that - RESTfully:

Lists all the groups in a web
Lists all the members of a group
Lists all the users in a web
Allows us to create a new group
Allows us to delete a group
Allows us to modify a group name

Our RESTful resources “could” look like this.

/api/groups: lists all groups in a web (verb=GET), create a group (verb=PUT)
/api/groups/{groupName}: deletes a group (verb=DELETE)
/api/groups/{groupName}/members: lists all members of a group (verb=GET), adds a member to a group (verb=POST)
/api/groups/{groupName}/members/{memberName}: removes a member from a group (verb=DELETE)

You certainly could do this! But, you don’t take advantage then of SharePoint 2010 built-in url rewriting that automatically gives you SPSite and SPWeb context, like it does for /_layouts/ and /_vti_bin/ urls. We want our API to be context specific to the SPWeb you are in, so let’s change the API resource urls to the following, which will inherit the SharePoint url rewriting magic (small price to pay for the benefit).

/_layouts/api/groups: lists all groups in a web (verb=GET), create a group (verb=PUT)
/_layouts/api/groups/{groupName}: deletes a group (verb=DELETE)
/_layouts/api/groups/{groupName}/members: lists all members of a group (verb=GET), adds a member to a group (verb=POST)
/_layouts/api/groups/{groupName}/members/{memberName}: removes a member from a group (verb=DELETE)

The web.config changes (just add this to the feature receiver to insert the changes during deployment, see source code download at the end of this post) are as follows:

In order to make the path “_layouts/api” work above, I had to insert 2 lines into the ServiceStack source, in the “ServiceStackHttpHandlerFactory” class. This class, as written in ServiceStack makes assumptions about the virtual directory format (as far as I can tell, after a cursory look), and these assumptions don’t work for us. My changes are as follows:

public static IHttpHandler GetHandlerForPathInfo(string httpMethod, string pathInfo, string requestPath, string filePath)
{
    var pathParts = pathInfo.TrimStart('/').Split('/');
    if (pathParts.Length == 0) return NotFoundHttpHandler;

    // BEGIN MOD for SharePoint
    if (pathParts.Length > 2 && pathParts[0].Equals("_layouts", StringComparison.OrdinalIgnoreCase))
        pathParts = pathParts.Skip(2).ToArray();
    // END MOD for SharePoint

    var handler = GetHandlerForPathParts(pathParts);
    if (handler != null) return handler;

    .... rest of method ....

}

The alternative to making these changes to the source code are to specify the individual Handlers in the web.config, instead of using the Handler factory class. This is the format used in the ServiceStack examples download. That would look something like this instead:

As far as building services, I won’t go over the details here, it’s extremely simple and ServiceStack has a very clear tutorial on how to do this. I highly suggest you follow it to get started. If you use Nuget, you can use the "Install-Package ServiceStack" command. Avoid the "ServiceStack.Host.AspNet" package in a .NET 3.5 project because the package download has some framework dependency issues with a couple 3rd party libraries that require .NET 4.0 (at the time of this writing, i.e.: Web Activator).

The services I implemented here to meet the use-case are straightforward, here’s what the “GroupService” looks like:

Group Service source code
 [CollectionDataContract(Name = "groups", ItemName = "group")]
public class GroupList : List<Group>
{
    public GroupList()
    {
    }
 
    public GroupList(IEnumerable<Group> collection)
        : base(collection)
    {
    }
}
 
[DataContract(Name = "group")]
public class Group
{
    public Group() { Members = new UserList(); }
    public Group(SPGroup group)
        : this()
    {
        this.Id = group.ID;
        this.Name = group.Name;
 
        this.Members = new UserList(group.Users.Cast<SPUser>().Select(u => new User(u)));
        this.MemberCount = Members.Count;
    }
 
    [DataMember(Name = "id", Order = 1)]
    public int Id { get; set; }
    [DataMember(Name = "name", Order = 2)]
    public string Name { get; set; }
    [DataMember(Name = "count", Order = 3)]
    public int MemberCount { get; set; }
    [DataMember(Name = "members", Order = 4)]
    public UserList Members { get; set; }
}
 
[RestService("/_layouts/api/groups", "GET,PUT")]
[RestService("/_layouts/api/groups/{Ids}", "GET,POST,DELETE")]
public class GroupRequest : Group
{
    public string Ids { get; set; }
}
 
public class GroupService : BaseService<GroupRequest>
{
    // Get 1 or more groups
    public override object OnGet(GroupRequest request)
    {
        if (!IsAuthenticated) return new HttpResult(HttpStatusCode.Unauthorized, "Requires authentication");
 
        try
        {
            var ids = string.IsNullOrEmpty(request.Ids)
                          ? null
                          : request.Ids.Split(',').Select(i => Convert.ToInt32(i)).ToArray();
 
            var groups = new List<Group>();
            var spGroups = CurrentWeb.Groups;
            foreach (SPGroup spGroup in spGroups)
            {
                if (ids == null || ids.Contains(spGroup.ID))
                    groups.Add(new Group(spGroup));
            }
 
            return new GroupResponse(CurrentWeb, groups, GetQueryStringValue("sort", "asc"));
        }
        catch
        {
            return new HttpResult(HttpStatusCode.BadRequest, "Invalid request");
        }
    }
 
    // Create a group
    public override object OnPut(GroupRequest request)
    {
        if (!IsAuthenticated) return new HttpResult(HttpStatusCode.Unauthorized, "Requires authentication");
 
        if(string.IsNullOrEmpty(request.Name))
            return new HttpResult(HttpStatusCode.BadRequest, "Name is missing");
 
        try
        {
            var web = CurrentWeb;
            var groupName = request.Name;
            var user = SPContext.Current.Web.CurrentUser;
 
            var groupArray = new string[] {groupName};
 
            web.AllowUnsafeUpdates = true;
 
            var groupCollection = web.SiteGroups.GetCollection(groupArray);
            if (groupCollection.Count == 0)
            {
                web.SiteGroups.Add(groupName, user, null, "The " + groupName + " group");
            }
            var spGroup = web.SiteGroups[groupName];
 
            var roleDef = web.RoleDefinitions.GetByType(SPRoleType.Contributor);
            var roles = new SPRoleAssignment(spGroup);
            roles.RoleDefinitionBindings.Add(roleDef);
            web.RoleAssignments.Add(roles);
 
            //// Assign to site
            spGroup.AddUser(user);
            spGroup.Update();
 
            web.AllowUnsafeUpdates = false;
 
            return new HttpResult(HttpStatusCode.OK, "SUCCESS");
        }
        catch (UnauthorizedAccessException uex)
        {
            return new HttpResult(HttpStatusCode.Forbidden, uex.Message);
        }
        catch (Exception ex)
        {
            return new HttpResult(HttpStatusCode.BadRequest, ex.Message);
        }
    }
 
    // Update the group
    public override object OnPost(GroupRequest request)
    {
        if (!IsAuthenticated) return new HttpResult(HttpStatusCode.Unauthorized, "Requires authentication");
 
        if (string.IsNullOrEmpty(request.Name))
            return new HttpResult(HttpStatusCode.BadRequest, "Name is missing");
 
        try
        {
            var ids = string.IsNullOrEmpty(request.Ids)
                          ? null
                          : request.Ids.Split(',').Select(i => Convert.ToInt32(i)).ToArray();
 
            if (ids == null || ids.Length == 0)
                return new HttpResult(HttpStatusCode.BadRequest, "Id is missing");
 
            CurrentWeb.AllowUnsafeUpdates = true;
 
            var spGroup = CurrentWeb.Groups.GetByID(ids[0]);
            spGroup.Name = request.Name;
            spGroup.Update();
 
            CurrentWeb.AllowUnsafeUpdates = false;
 
            return new HttpResult(HttpStatusCode.OK, "SUCCESS");
        }
        catch (UnauthorizedAccessException uex)
        {
            return new HttpResult(HttpStatusCode.Forbidden, uex.Message);
        }
        catch (Exception ex)
        {
            return new HttpResult(HttpStatusCode.BadRequest, ex.Message);
        }
    }
 
    // Delete 1 or more groups
    public override object OnDelete(GroupRequest request)
    {
        if (!IsAuthenticated) return new HttpResult(HttpStatusCode.Unauthorized, "Requires authentication");
        
        try
        {
            var ids = string.IsNullOrEmpty(request.Ids)
                          ? null
                          : request.Ids.Split(',').Select(i => Convert.ToInt32(i)).ToArray();
 
            if (ids == null || ids.Length == 0)
                return new HttpResult(HttpStatusCode.BadRequest, "Ids are missing");
 
            CurrentWeb.AllowUnsafeUpdates = true;
 
            foreach (var id in ids)
            {
                CurrentWeb.Groups.RemoveByID(id);
            }
            CurrentWeb.Update();
 
            CurrentWeb.AllowUnsafeUpdates = false;
 
            return new HttpResult(HttpStatusCode.OK, "SUCCESS");
        }
        catch (UnauthorizedAccessException uex)
        {
            return new HttpResult(HttpStatusCode.Forbidden, uex.Message);
        }
        catch (Exception ex)
        {
            return new HttpResult(HttpStatusCode.BadRequest, ex.Message);
        }
    }
}
 

I added an event receiver to the feature (see source code download) to configure web.config and the global.asax file to enable the ServiceStack handlers. And that’s it, we’re ready to deploy.

See it in Action

ServiceStack automatically gives you a full metadata screen that documents the web services. ServiceStack automatically supports JSON, XML, JSV, CSV, SOAP 1.1 and SOAP 1.2 response types.

ServiceStack automatically describes each method with supported VERBS and how to use it.

ServiceStack automatically gives you a nice HTML5 view of the data for GET requests, and links to call the other response types.

ServiceStack supports the standard HTTP headers to call services as desired. In this screenshot, the Content Type specified is application/json on a GET request for all groups in an SPWeb.

In the following screenshot, the Content Type specified is application/xml, using an OVERRIDE in the URL. The serialized XML obeys all the DataContract serialization attributes for naming and ordering conventions.

Here we create a SharePoint group using a PUT request.

Here we update a SharePoint group name using a POST request.

Here we delete a SharePoint group using a DELETE request.

ServiceStack also does great error handling, and gives you full control over what Status Codes you wish to return and any accompanying messaging.

Conclusion

So there we have it. These APIs are obviously completely compatible with jQuery and any other javascript framework, and they provide a great foundation for hosting a nice self-documenting API within SharePoint. Once the infrastructure file is changed, updating the API with new methods just takes a simple DLL update.

Well, hopefully you enjoyed this post as much as I enjoyed putting this little prototype project together. Thanks for reading, and please do check out the download and modify as you see fit. You’ll have to download the ServiceStack code/binaries from Github.

Download Code

By Matt C. | Posted in .NET, SharePoint | Tagged csv, json, jsv, rest, servicestack, sharepoint, soap, web api | Comments (3)

Scripting your SQL database (using SMO and the command line)

April 21, 2012 – 1:17 pm

When I’m developing, I often start with a “database first” design, so having a utility that can generate my database scripts and dump them into a directory comes in really handy. During a development cycle, I can then build into my application the ability to rebuild the database at runtime and seed the database with data. Once the application moves into production, on restart, it compares the directory of sql scripts with all the scripts it has already executed (stored in a table), and executes any new scripts as needed. Perhaps I’ll blog about how this can be done in an ASP.NET MVC application at some point in the near future (it’s actually very simple to do). If you’re interested in this now, there are some good sample applications that do this already. The nopCommerce application does this for example, although it uses EF CodeFirst and relies on the built-in CodeFirst script generator to create the SQL scripts (last time I checked at least). Personally, I prefer to have full control over the SQL schema generation process, so generating databases on the fly using EF CodeFirst is not something I’m personally very fond of (right now), but that’s a personal preference - the overall concept however is similar.

With that out of the way, this post is about generating SQL scripts, so on with it.

Requirements

Create a command line application that generates the needed SQL scripts to re-create a database (something that can be added as an external tool either to SQL Server Management Studio, or Visual Studio)
Provide the following argument options:
- Server and Database/Catalog to script
- Directory to dump the files into
- Override default names for the various SQL files: Tables, Stored Procedures, UDFs, and Data

The Console Application

When creating a simple command line application, the first thing you need is a way to parse arguments and to return to the user some “help” documentation on how to use the application commands. There’s no use in re-inventing the parsing algorithms for a simple application, there are plenty out there already written for you. In this case, check out the following two links:

THANKS TO Richard Lopes: http://www.codeproject.com/Articles/3111/C-NET-Command-Line-Arguments-Parser
IMPROVED BY Jake Ginnivan: http://jake.ginnivan.net/c-sharp-argument-parser

Once we wire this into our application, it’s just a matter of leveraging the argument parser for our specific needs. In our case, that looks something like this.

Command Line
  
class Program
{
    static void Main(string[] args)
    {
        var program = new Program(new Arguments(args), Console.Out);
        if (!program.Validate())
        {
            program.OutputHelp();
        }
        else
        {
            program.Execute();
        }
    }
 
    private Arguments _arguments;
    private TextWriter _logger;
 
    public Program(Arguments arguments, TextWriter logger)
    {
        this._arguments = arguments;
        this._logger = logger;
    }
 
    private bool Validate()
    {
        if (string.IsNullOrEmpty(_arguments.Single("server")) && string.IsNullOrEmpty(_arguments.Single("s")))
            return false;
 
        if (string.IsNullOrEmpty(_arguments.Single("database")) && string.IsNullOrEmpty(_arguments.Single("d")))
            return false;
 
        if (string.IsNullOrEmpty(_arguments.Single("directory")) && string.IsNullOrEmpty(_arguments.Single("dir")))
            return false;
 
        return true;
    }
 
    private void Execute()
    {
        var stacktrace = _arguments.IsTrue("stacktrace");
        try
        {
            var server = _arguments.Single("server") ?? _arguments.Single("s");
            var database = _arguments.Single("database") ?? _arguments.Single("d");
            var directory = _arguments.Single("directory") ?? _arguments.Single("dir");
 
            var output = _arguments["output"] ?? new Collection<string> { "tables", "udfs", "procedures", "data" };
            var verbose = _arguments.Exists("verbose") ? _arguments.IsTrue("verbose") : (_arguments.Exists("v") && _arguments.IsTrue("v"));
 
            var tablesFileName = _arguments.Single("tf") ?? "tables.sql";
            var udfsFileName = _arguments.Single("tu") ?? "udfs.sql";
            var proceduresFileName = _arguments.Single("tp") ?? "procedures.sql";
            var dataFileName = _arguments.Single("td") ?? "data.sql";
 
            string[] files;
            DatabaseScripter.Execute(server, database, directory, output.Contains("tables"), output.Contains("udfs"),
                          output.Contains("procedures"), output.Contains("data"), tablesFileName, udfsFileName,
                          proceduresFileName, dataFileName, (verbose ? _logger : null), out files);
 
            foreach (var file in files)
                _logger.WriteLine(file);
        }
        catch (Exception ex)
        {
            _logger.WriteLine("Exception: {0}", ex.Message);
            if (stacktrace)
                _logger.WriteLine("StackTrace: \n{0}", ex);
        }
    }
 
    private void OutputHelp()
    {
        _logger.WriteLine("\nMandatory arguments\n");
        _logger.WriteLine("{0,-20}{1}", "-server", "The database server.");
        _logger.WriteLine("{0,-20}{1}", "-database", "The name of the database (catalog).");
        _logger.WriteLine("{0,-20}{1}", "-directory", "The directory to output the database scripts to.");
 
        _logger.WriteLine("\nOptional arguments\n");
        _logger.WriteLine("{0,-20}{1}", "-verbose", "Output information while processing.");
        _logger.WriteLine("{0,-20}{1}", "-stacktrace", "Include the stack trace in the error message.");
        _logger.WriteLine("{0,-20}{1}", "-output", "This command can be specified multiple times.");
        _logger.WriteLine("{0,-20}{1}", "", "Value options: tables, udfs, procedures, data");
        _logger.WriteLine("{0,-20}{1}", "", "(by default, all options are selected).");
        _logger.WriteLine("{0,-20}{1}", "-tf", "'tables' script output file (default: tables.sql).");
        _logger.WriteLine("{0,-20}{1}", "-tu", "'udfs' script output file (default: udfs.sql).");
        _logger.WriteLine("{0,-20}{1}", "-tp", "'procedures' script output file (default: procedures.sql).");
        _logger.WriteLine("{0,-20}{1}", "-td", "'data' script output file (default: data.sql).");
    }
}
 

Scripting the Database

SQL Server Management Studio has a very nice built-in wizard that does everything we need.

The goal is to use the SMO API to basically reproduce what the wizard does above, all from a command line. The API gives you pretty much everything you need with the Scripter and ScriptingOptions objects.

Using these objects, we can easily script Tables, Stored Procedures, UDFs, and Data:

Scripter and ScriptingOptions
  var dbServer = new Server(server);
            var db = dbServer.Databases[database];
            var scriptingOptions =  new ScriptingOptions
                                   {
                                       AppendToFile = true,
                                       AnsiFile = true,
                                       AnsiPadding = true,
                                       ChangeTracking = true,
                                       ClusteredIndexes = true,
                                       ContinueScriptingOnError = false,
                                       DriAll = true,
                                       IncludeHeaders = true,
                                       IncludeIfNotExists = true,
                                       Indexes = true,
                                       ToFileOnly = true,
                                       WithDependencies = true
                                   };
            var scripter = new Scripter(dbServer) {Options = scriptingOptions};
 
            var paths = new List<string>();
            if (includeTables)
            {
                var fName = Path.Combine(directory, tablesFileName);
                ScriptTables(fName, db, scripter, logger);
                paths.Add(fName);
            }
            if (includeUdfs)
            {
                var fName = Path.Combine(directory, udfsFileName);
                ScriptUdfs(fName, db, scripter, logger);
                paths.Add(fName);
            }
            if (includeProcedures) 
            {
                var fName = Path.Combine(directory, proceduresFileName);
                ScriptProcedures(fName, db, scripter, logger);
                paths.Add(fName);
            }
            if (includeData) 
            {
                var fName = Path.Combine(directory, dataFileName);
                ScriptData(fName, db, scripter, logger);
                paths.Add(fName);
            }
 

Final Output

Now that everything is wired up, it’s time to test it out. Let’s go with the aspnetdb, something everybody is already pretty familiar with.

Here’s what the “help” screen looks like:

Now for the real thing:

And there you go. Now, you can use a command line to easily script your databases. All the code is provided below. Use, experiment, extend as you please.

Download Code

By Matt C. | Posted in .NET | Tagged command line, console application, smo, sql | Comments (5)

Adding LLBLGen to the Dapper performance benchmark test project

February 24, 2012 – 12:30 am

The Dapper project hosted on GitHub at https://github.com/SamSaffron/dapper-dot-net has a performance/benchmark test project that compares a number of ORMs and data access strategies. While the project only tests a very simple fetch algorithm, it’s a good starter indicator for performance.

I thought I would add one of my favorite ORMs (LLBLGen) to the mix, and see how it performs.

The Tests

Here’s the code I added to the PerformanceTests.cs class file. Use this code, in conjunction with the download further down to perform the tests yourself if you're familiar with LLBLGen.

Here are the LLBLGen Adapter Tests:

LLBLGen Adapter Tests

//LLBLGen Adapter
var adapter1 = new DataAccessAdapter(Program.connectionString, true, CatalogNameUsage.Clear, null);
adapter1.OpenConnection();
tests.Add(id => adapter1.FetchEntity(new PostEntity(id)), "LLBLGen Adapter (FetchEntity w/ KeepConnectionOpen)");
var adapter2 = new DataAccessAdapter(Program.connectionString, false, CatalogNameUsage.Clear, null);
adapter2.OpenConnection();
tests.Add(id => adapter2.FetchEntity(new PostEntity(id)), "LLBLGen Adapter (FetchEntity)");
var adapter3 = new DataAccessAdapter(Program.connectionString, true, CatalogNameUsage.Clear, null);
adapter3.OpenConnection();
tests.Add(id =>
{
var bucket = new RelationPredicateBucket();
bucket.PredicateExpression.Add(PostFields.Id == id);
var entities = new EntityCollection<PostEntity>();
adapter3.FetchEntityCollection(entities, bucket);
entities.First();
}, "LLBLGen Adapter (Predicate w/ KeepConnectionOpen)");
var adapter4 = new DataAccessAdapter(Program.connectionString, false, CatalogNameUsage.Clear, null);
adapter4.OpenConnection();
tests.Add(id =>
{
var bucket = new RelationPredicateBucket();
bucket.PredicateExpression.Add(PostFields.Id == id);
var entities = new EntityCollection<PostEntity>();
adapter4.FetchEntityCollection(entities, bucket);
entities.First();
}, "LLBLGen Adapter (Predicate)");

Here are the LLBLGen Self-Servicing Tests:

LLBLGen Self-Servicing Tests

//LLBLGen SelfServicing
CommonDaoBase.ActualConnectionString = Program.connectionString;
tests.Add(id => (new LLBLGenSS.EntityClasses.PostEntity()).FetchUsingPK(id), "LLBLGen SelfServicing (UC Fetch)");
tests.Add(id => new LLBLGenSS.EntityClasses.PostEntity(id, (IPrefetchPath)null), "LLBLGen SelfServicing (PK in constructor)");
tests.Add(id =>
{
var entities = new LLBLGenSS.CollectionClasses.PostCollection();
entities.GetMulti(LLBLGenSS.HelperClasses.PostFields.Id == id);
entities.First();
}, "LLBLGen SelfServicing (Predicate)");
var transactionManager = new LLBLGenSS.HelperClasses.Transaction(IsolationLevel.ReadUncommitted, "Test", Program.connectionString);
tests.Add(id =>
{
var entity = new LLBLGenSS.EntityClasses.PostEntity();
transactionManager.Add(entity);
entity.FetchUsingPK(id);
}, "LLBLGen SelfServicing (UC Fetch in Transaction)");

Now let’s see the results.

The Results

In plain text format:

Strategy	Time
Dynamic Mapper Query (buffered)	57ms
Mapper Query (non-buffered)	58ms
Mapper Query (buffered)	59ms
Dynamic Mapper Query (non-buffered)	59ms
hand coded	59ms
PetaPoco (Fast)	63ms
Dapper.Cotrib	64ms
OrmLite QueryById	64ms
PetaPoco (Normal)	67ms
Dynamic Massive ORM Query	69ms
BLToolkit	95ms
Simple.Data	98ms
Linq 2 SQL Compiled	106ms
LLBLGen Adapter (FetchEntity w/ KeepConnectionOpen)	121ms
SubSonic Coding Horror	123ms
NHibernate Session.Get	124ms
LLBLGen SelfServicing (UC Fetch in Transaction)	126ms
LLBLGen Adapter (Predicate w/ KeepConnectionOpen)	127ms
NHibernate SQL	128ms
Entity framework CompiledQuery	135ms
NHibernate HQL	141ms
LLBLGen Adapter (FetchEntity)	145ms
LLBLGen SelfServicing (UC Fetch)	148ms
LLBLGen SelfServicing (PK in constructor)	150ms
LLBLGen Adapter (Predicate)	155ms
LLBLGen SelfServicing (Predicate)	164ms
NHibernate Criteria	175ms
Soma	186ms
Linq 2 SQL ExecuteQuery	240ms
Linq 2 SQL	737ms
NHibernate LINQ	769ms
Entity framework ESQL	812ms
Entity framework ExecuteStoreQuery	829ms
Entity framework	1063ms
Entity framework No Tracking	1065ms
SubSonic ActiveRecord.SingleOrDefault	4874ms

Download

To run the LLBLGen test, just include the 2 folders in the attached zip file into the Dapper project, and reference the LLBLGen assemblies

Project:

References:

Download Code

Conclusion

While LLBLGen is not a Micro-ORM, the LLBLGen Adapter and SelfServicing runtime frameworks rank up there at the top performance-wise. I’ve worked with Entity Framework extensively in the past few months, including Code-First, even though that doesn’t make me an expert, but none-the-less, LLBLGen is more attractive now than ever before. We’ll see how EF 5.0 turns out. Meanwhile, NHibernate, PetaPoco, Dapper, and LLBLGen all look like good options.

In a separate study (updated on 4/20/2012)

I found the codebase I used a while back to compare basic CRUD operations between EF 4.1, Dapper and LLBGen 3.1. I took a quick stab at refactoring it a little with newer libraries and here are the findings.

The results are in milliseconds, and are averages taken over 5 iterations of 1000, 2000, and 5000 user records. Code is provided as a download. I make a distinction in my tests between BATCH and ATOMIC transactions.
A BATCH transaction is a transaction over the entire span of records (start_transaction foreach { do_record_update } commit_transaction); whereas, an ATOMIC transaction is a transaction on each individual record (foreach { start_transaction do_record_update commit_transaction }).

All frameworks “delete”, and “insert” about the same. Dapper clearly outperforms both LLBGen and EF in batch reading data out of a database (although, all frameworks retrieve over 5000 records in less than 90ms in my tests, which is probably good enough for most applications), unless you’re running a site like StackOverflow, in which case, you might as well switch to a search engine instead. LLBLGen is a near 2nd, EF takes 3rd place (with some concerns in the area of “atomic reads” and “updates”).

Of course, I make assumptions in my tests, and the database in these tests is not indexed, so this isn’t the final say on the matter, take it as you will Smile .

Separate Study Code

By Matt C. | Posted in .NET | Tagged ado.net, dapper, entity framework, llblgen, micro-orm | Comments (5)

Using Autofac in SharePoint 2010

February 1, 2012 – 10:00 pm

Up till now in my SharePoint 2010 projects, I’ve been using the very nice SharePoint Service Locator implementation, from the patterns & practices group. This has been really useful, and works great. If you’re not familiar with the service locator pattern, you can read up on it here. Using this pattern, it’s easy to build a lightweight common library that you pass out to your team (or as is often the case “teams”) of developers without them having to mess around with the implementation of every interface. After all, the implementation of the interfaces are registered and deployed on the SharePoint server using a feature, and that’s all developers need to know. In many instances, developers might even develop for SharePoint without even having SharePoint installed. The service locator allows you to build an API that often looks like this in my projects:

AppContext.Repositories.ContactsRepository.GetContact( … ), UpdateContact( … )
AppContext.Services.RegisterUser( … ), PurchaseProduct( … ) –> complex step that might work with multiple repositories and talk to AD, and the user profile manager
AppContext.Data.Call<storedprocedure>( … ) –> allow a direct call to a database, which happens sometimes in a custom service application

Why Autofac?

While Autofac provides service locator functionality with it’s ILifetimeScope and IContainer implementations, it also provides dependency injection of constructor parameters, and properties, and a fluent interface that allows late-bound resolution of injected parameters, and much more. Other containers you could look at are: StructureMap (another favorite of mine), Unity, Ninject, Spring.NET, Castle Windsor. Based on benchmarks though – see here and here – it seems that Autofac is one of the better performing containers. Another thing that is useful about using Autofac is the “InstancePerLifetimeScope()” registration extension that will automatically instantiate the registered type for every ASP.NET request, and properly dispose of the object at the end of the ASP.NET request, which makes it an ideal candidate to minimize round-trip connections to a database for example, or other persistent storage.

Autofac vs. SharePoint Service Locator

Whereas the SharePoint Service Locator stores it’s mapping of interfaces to their implementations as persisted objects in the database, Autofac wires things up in ASP.NET during the Application_Start event. Autofac also requires some additional configuration in the web.config file, so already, Autofac is going to demand a little more of a setup process.

The other core need is to allow developers to wire up additional mappings in any new features they develop, and it would be nice for there not to be a need to always modify the web.config file everytime a new feature with new mappings is developed. So storing mappings in SharePoint might be a good idea.

Wiring Autofac up

The following code is a Farm-scoped Feature EventReceiver that wires up Autofac on all the web front-ends within the farm. The code takes care of registering Autofac and unregistering Autofac. It primarily does 2 things:

Makes all configuration changes to the web.config file necessary to support Autofac
Alters the global.asax file on all web front-ends

IISRESET is required after installing the feature

Autofac integration feature
 [Guid("9578f389-4e0b-40ef-9e8e-5358003e64d4")]
public class AutofacIntegrationEventReceiver : SPFeatureReceiver
{
    public override void FeatureActivated(SPFeatureReceiverProperties properties)
    {
        // generally equivalent to: SPFarm.Local.Services.GetValue<SPWebService>()
        var webService = properties.Feature.Parent as SPWebService;
        if (webService == null)
        {
            throw new ArgumentNullException("properties.Feature.Parent");
        }
 
        // let's register an autofac module (saves the config in the farm properties)
        SPContainerBuilder.RegisterModule(typeof(Playground.PlaygroundModule2).AssemblyQualifiedName, null, null, true);
 
        // let's make the necessary changes to register autofac in web.config and global.asax
        var globalAsaxFiles = new List<string>();
        var zones = Enum.GetValues(typeof (SPUrlZone)).Cast<SPUrlZone>().ToArray();
        foreach (var webApp in webService.WebApplications)
        {
            if (webApp.IsAdministrationWebApplication)
                continue; // don't mess with central admin  
 
            // assuming every other web application "wants" to use Autofac
            // (this may not be the case, change logic here, or scope the feature differently)
 
            foreach (var entry in _autofacConfigEntries)
            {
                webApp.WebConfigModifications.Add(entry.Prepare());
            }
 
            webApp.WebService.Update();
            webApp.WebService.ApplyWebConfigModifications();
 
            var paths =
                zones.Select(z => Path.Combine(webApp.GetIisSettingsWithFallback(z).Path.ToString(), "global.asax"))
                    .Distinct().Where(File.Exists).ToArray();
            globalAsaxFiles.AddRange(paths);
        }
 
        // Now replace the global.asax file(s) - I dare you!! 
        try
        {
            var asaxFileContents =
                string.Format(
                    @"<%@ Assembly Name=""Microsoft.SharePoint""%><%@ Assembly Name=""{0}""%><%@ Application Language=""C#"" Inherits=""{1}"" %>",
                    typeof(IContainerProviderAccessor).Assembly.FullName,
                    typeof(Autofac.Integration.SharePoint.GlobalApplication).AssemblyQualifiedName);
            foreach (var asax in globalAsaxFiles)
            {
                File.WriteAllText(asax, asaxFileContents, Encoding.UTF8);
            }
        }
        catch
        {
            // Modify the global.asax files manually (this shouldn't happen)
        }
    }
 
    public override void FeatureDeactivating(SPFeatureReceiverProperties properties)
    {
        var webService = properties.Feature.Parent as SPWebService;
        if (webService == null)
        {
            // allow de-activation here
            return;
        }
 
        var globalAsaxFiles = new List<string>();
        var zones = Enum.GetValues(typeof (SPUrlZone)).Cast<SPUrlZone>().ToArray();
        foreach (var webApplication in webService.WebApplications)
        {
            var modsCollection = webApplication.WebConfigModifications;
 
            for (var i = modsCollection.Count - 1; i > -1; i--)
            {
                if (modsCollection[i].Owner == ConfigModsOwnerName)
                {
                    // Remove it and save the change to the configuration database  
                    modsCollection.Remove(modsCollection[i]);
                }
            }
            webApplication.Update();
 
            // Reapply all the configuration modifications
            webApplication.WebService.Update();
            webApplication.WebService.ApplyWebConfigModifications();
 
            var paths =
                zones.Select(z => Path.Combine(webApplication.GetIisSettingsWithFallback(z).Path.ToString(), "global.asax"))
                    .Distinct().Where(File.Exists).ToArray();
            globalAsaxFiles.AddRange(paths);
        }
 
        // Now replace the global.asax file(s)
        try
        {
            var asaxFileContents =
                @"<%@ Assembly Name=""Microsoft.SharePoint""%><%@ Application Language=""C#"" Inherits=""Microsoft.SharePoint.ApplicationRuntime.SPHttpApplication"" %>";
            foreach (var asax in globalAsaxFiles)
            {
                File.WriteAllText(asax, asaxFileContents, Encoding.UTF8);
            }
        }
        catch
        {
            // Modify the global.asax files manually (this shouldn't happen)
        }
    }
 
    #region Web.Config Modification Entries
    private const string ConfigModsOwnerName = "Autofac Integration For SharePoint";
    private readonly WebConfigEntry[] _autofacConfigEntries = 
        { 
            // configSections entry
            new WebConfigEntry( 
                "section[@name='autofac']" 
                ,"configuration/configSections" 
                ,"<section name=\"autofac\" type=\"" + typeof(Autofac.Configuration.SectionHandler).AssemblyQualifiedName + "\"/>" 
                ,SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode 
                ,false)  
 
            // autofac entry
            ,new WebConfigEntry(
                "autofac" 
                ,"configuration" 
                ,"<autofac/>" 
                ,SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode 
                ,false)   
 
            // autofac modules entry
            ,new WebConfigEntry(
                "modules" 
                ,"configuration/autofac" 
                ,"<modules/>" 
                ,SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode 
                ,false)
            // This is just a sample module
            ,new WebConfigEntry(
                "module[@type='" + typeof(Playground.PlaygroundModule).AssemblyQualifiedName + "']"
                ,"configuration/autofac/modules"
                ,"<module type=\"" + typeof(Playground.PlaygroundModule).AssemblyQualifiedName + "\"/>"
                ,SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode 
                , false)
 
            // autofac httpModules entries
            ,new WebConfigEntry( 
                "add[@name='ContainerDisposal']" 
                ,"configuration/system.web/httpModules" 
                ,"<add name=\"ContainerDisposal\" type=\"" + typeof(ContainerDisposalModule).AssemblyQualifiedName + "\" />"
                ,SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode 
                ,false) 
            ,new WebConfigEntry( 
                "add[@name='PropertyInjection']" 
                ,"configuration/system.web/httpModules" 
                ,"<add name=\"PropertyInjection\" type=\"" + typeof(PropertyInjectionModule).AssemblyQualifiedName + "\" />"
                ,SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode 
                ,false)
            ,new WebConfigEntry( 
                "add[@name='AttributeInjection']" 
                ,"configuration/system.web/httpModules" 
                ,"<add name=\"AttributeInjection\" type=\"" + typeof(AttributedInjectionModule).AssemblyQualifiedName + "\" />"
                ,SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode 
                ,false)
                
            // autofac iis7 modules entries
            ,new WebConfigEntry( 
                "add[@name='ContainerDisposal']" 
                ,"configuration/system.webServer/modules" 
                ,"<add name=\"ContainerDisposal\" type=\"" + typeof(ContainerDisposalModule).AssemblyQualifiedName + "\" preCondition=\"managedHandler\" />"
                ,SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode 
                ,false) 
            ,new WebConfigEntry( 
                "add[@name='PropertyInjection']" 
                ,"configuration/system.webServer/modules" 
                ,"<add name=\"PropertyInjection\" type=\"" + typeof(PropertyInjectionModule).AssemblyQualifiedName + "\" preCondition=\"managedHandler\" />"
                ,SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode 
                ,false)
            ,new WebConfigEntry( 
                "add[@name='AttributeInjection']" 
                ,"configuration/system.webServer/modules" 
                ,"<add name=\"AttributeInjection\" type=\"" + typeof(AttributedInjectionModule).AssemblyQualifiedName + "\" preCondition=\"managedHandler\" />"
                ,SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode 
                ,false)
 
            // autofac assemblies entries
            ,new WebConfigEntry( 
                "add[@assembly='" + typeof(Autofac.IContainer).Assembly.FullName + "']" 
                ,"configuration/system.web/compilation/assemblies" 
                ,"<add assembly=\"" + typeof(Autofac.IContainer).Assembly.FullName + "\" />"
                ,SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode 
                ,false)
            ,new WebConfigEntry( 
                "add[@assembly='" + typeof(Autofac.Configuration.ComponentElement).Assembly.FullName + "']" 
                ,"configuration/system.web/compilation/assemblies" 
                ,"<add assembly=\"" + typeof(Autofac.Configuration.ComponentElement).Assembly.FullName + "\" />"
                ,SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode 
                ,false)
            ,new WebConfigEntry( 
                "add[@assembly='" + typeof(IContainerProviderAccessor).Assembly.FullName + "']" 
                ,"configuration/system.web/compilation/assemblies" 
                ,"<add assembly=\"" + typeof(IContainerProviderAccessor).Assembly.FullName + "\" />"
                ,SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode 
                ,false)
                
            // add some safe control entries as well (assuming you'll be building some controls in Autofac.Integration.SharePoint)
            ,new WebConfigEntry( 
                "SafeControl[@Assembly='" + typeof(Autofac.Integration.SharePoint.GlobalApplication).Assembly.FullName + "']"
                ,"configuration/SharePoint/SafeControls" 
                ,"<SafeControl Assembly=\"" + typeof(Autofac.Integration.SharePoint.GlobalApplication).Assembly.FullName + "\" Namespace=\"Autofac.Integration.SharePoint\" TypeName=\"*\" Safe=\"True\" SafeAgainstScript=\"False\" />"
                ,SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode 
                ,false)
 
        };
 
    /// <summary> 
    /// Container to hold info about our modifications to the web.config. 
    /// </summary> 
    public class WebConfigEntry
    {
        public string Name;
        public string XPath;
        public string Value;
        public SPWebConfigModification.SPWebConfigModificationType ModificationType;
        public bool KeepOnDeactivate;
 
        public WebConfigEntry(string name, string xPath, string value,
            SPWebConfigModification.SPWebConfigModificationType modificationType, bool keepOnDeactivate)
        {
            Name = name;
            XPath = xPath;
            Value = value;
            ModificationType = modificationType;
            KeepOnDeactivate = keepOnDeactivate;
        }
 
        public SPWebConfigModification Prepare()
        {
            var modification = new SPWebConfigModification(Name, XPath)
            {
                Owner = ConfigModsOwnerName,
                Sequence = 0,
                Type = ModificationType,
                Value = Value
            };
            return modification;
        }
    }
    #endregion
}
 

After the event receiver installs Autofac … you’ll see the following summarized list of changes to the root web.config files on all front-end web applications:

web.config change summary
 <?xml version="1.0" encoding="utf-8"?>
<configuration>
  <configSections>
    <section name="autofac" type="Autofac.Configuration.SectionHandler, Autofac.Configuration, Version=2.5.2.830, Culture=neutral, PublicKeyToken=17863af14b0044da" />
  </configSections>
  <SharePoint>
    <SafeControls>
      <SafeControl Assembly="Autofac.Integration.SharePoint, Version=1.0.0.0, Culture=neutral, PublicKeyToken=3a3b1e988e6ba16f" Namespace="Autofac.Integration.SharePoint" TypeName="*" Safe="True" SafeAgainstScript="False" />
    </SafeControls>
  </SharePoint>
  <system.web>
    <httpModules>
      <add name="AttributeInjection" type="Autofac.Integration.Web.Forms.AttributedInjectionModule, Autofac.Integration.Web, Version=2.5.2.830, Culture=neutral, PublicKeyToken=17863af14b0044da" />
      <add name="ContainerDisposal" type="Autofac.Integration.Web.ContainerDisposalModule, Autofac.Integration.Web, Version=2.5.2.830, Culture=neutral, PublicKeyToken=17863af14b0044da" />
      <add name="PropertyInjection" type="Autofac.Integration.Web.Forms.PropertyInjectionModule, Autofac.Integration.Web, Version=2.5.2.830, Culture=neutral, PublicKeyToken=17863af14b0044da" />
    </httpModules>
    <compilation batch="true" debug="true" optimizeCompilations="true">
      <assemblies>
        <add assembly="Autofac, Version=2.5.2.830, Culture=neutral, PublicKeyToken=17863af14b0044da" />
        <add assembly="Autofac.Configuration, Version=2.5.2.830, Culture=neutral, PublicKeyToken=17863af14b0044da" />
        <add assembly="Autofac.Integration.Web, Version=2.5.2.830, Culture=neutral, PublicKeyToken=17863af14b0044da" />
      </assemblies>
    </compilation>
 </system.web>
  <system.webServer>
    <modules runAllManagedModulesForAllRequests="true">
      <add name="AttributeInjection" type="Autofac.Integration.Web.Forms.AttributedInjectionModule, Autofac.Integration.Web, Version=2.5.2.830, Culture=neutral, PublicKeyToken=17863af14b0044da" preCondition="managedHandler" />
      <add name="ContainerDisposal" type="Autofac.Integration.Web.ContainerDisposalModule, Autofac.Integration.Web, Version=2.5.2.830, Culture=neutral, PublicKeyToken=17863af14b0044da" preCondition="managedHandler" />
      <add name="PropertyInjection" type="Autofac.Integration.Web.Forms.PropertyInjectionModule, Autofac.Integration.Web, Version=2.5.2.830, Culture=neutral, PublicKeyToken=17863af14b0044da" preCondition="managedHandler" />
    </modules>
  </system.webServer>
  <autofac>
    <modules>
      <module type="Autofac.Integration.SharePoint.Playground.PlaygroundModule, Autofac.Integration.SharePoint, Version=1.0.0.0, Culture=neutral, PublicKeyToken=3a3b1e988e6ba16f" />
    </modules>
  </autofac>
</configuration>
 

You’ll notice that this implementation has the “modules” functionality already wired-up at the very end of the web.config file to add new modules (see here for more details). But this isn’t the only way you can add new modules. In the “Autofac Integration Code” above, I’ve added the ability to store module configuration in SharePoint (using a Farm property) that also gets added to the modules wired up during Application_Start.

The functions that developers can use in their feature event receivers to register (and unregister) Autofac modules into SharePoint are in the SPContainerBuilder class in the download, and have the following signatures:

public static void RegisterModule(string moduleType, IDictionary<string, string> properties, IDictionary<string, string> parameters, bool replaceIfExists){ … }
public static void RemoveModule(string moduleType){ … }

Developers integrating new features into SharePoint can leverage these two functions in their feature event receivers to add new Autofac mappings without modifying the web.config files going-forward.

By modifying the global.asax file, Autofac fires during Application startup and calls an initialize method on a static service locator class. The initialization of the Autofac container looks like the following.

Initializing the container

/// <summary>
/// This method is available for testing, or to use the ServiceLocator capability outside of
/// the SharePoint HttpContext (i.e.: console app, linqpad, powershell, etc...)
/// </summary>
/// <param name="configFile">Autofac configuration file</param>
/// <param name="additionalBuild">Any additional build information desired (will override the web.config and farm properties)</param>
public static void InitializeContainer(string configFile, Action<ContainerBuilder> additionalBuild)
{
    if (_initialized)
        return;
 
    if (!string.IsNullOrEmpty(configFile) && !File.Exists(configFile))
        throw new FileNotFoundException("The autofac configuration file could not be found.", configFile);
 
    var builder = new SPContainerBuilder();
 
    // register modules from config file
    builder.RegisterModule(string.IsNullOrEmpty(configFile)
                               ? new ConfigurationSettingsReader("autofac")
                               : new ConfigurationSettingsReader("autofac", configFile));
 
    // register modules from SPFarm.Properties
    var modules = SPContainerBuilder.GetFarmPropertyConfiguredModules();
    foreach (var module in modules)
        builder.RegisterModule(module);
 
    if (additionalBuild != null)
        additionalBuild.Invoke(builder);
 
    _containerProvider = new SPContainerProvider(builder.Build());
    _initialized = true;
}

Using Autofac within Application Pages, Web Parts, User Controls, Event Receivers

Application pages and user controls can use the standard injection techniques that come with Autofac with ASP.NET (in the Autofac.Integration.Web assembly).

Property Injection Example

[InjectProperties]
public partial class Play : LayoutsPageBase, IPlaygroundView
{
    private PlaygroundPresenter _presenter;
    public PlaygroundPresenterFactory PresenterFactory { get; set; }
 
    // these are here just to test property injection
    public ILogger Logger { get; set; }
    public IPlayInterface PlayInterface { get; set; }
 
    protected void Page_Load(object sender, EventArgs e)
    {
        _presenter = PresenterFactory(this);
    }

In SharePoint, there are various scenarios that need to be supported. The following is a non-comprehensive list, along with various usages that support each.

Presenter/Service Registration

protected override void Load(ContainerBuilder builder)
{
    // presenters and views
    //builder.RegisterType<UlsLogger>().AsImplementedInterfaces().SingleInstance(); // logger singleton
    builder.RegisterType<UlsLogger>().As<ILogger>().WithParameter("categoryName", "Autofac Logger").SingleInstance(); // logger singleton
    builder.RegisterType<PlaygroundPresenter>().AsSelf().InstancePerDependency();
    builder.RegisterGeneratedFactory<PlaygroundPresenterFactory>();
}

InstancePerLifetimeScopes

protected override void Load(ContainerBuilder builder)
{
    // InstancePerLifetimeScope implementation
    builder.RegisterType<PlayInterface>().As<IPlayInterface>().InstancePerLifetimeScope();
}

Service Locator example

// old-school detection:
var cpa = (IContainerProviderAccessor) HttpContext.Current.ApplicationInstance;
IContainerProvider cp = cpa.ContainerProvider;
_logger = cp.RequestLifetime.Resolve<ILogger>();
 
// or, an easier way:
this._logger = SPServiceLocator.GetRequestLifetime().Resolve<ILogger>();

Manual property injection

SPServiceLocator.GetRequestLifetime().InjectProperties(this);

Elevated Privileges

SPSecurity.RunWithElevatedPrivileges(delegate {
    AnotherLogger = SPServiceLocator.GetRequestLifetime().Resolve<ILogger>();
    if (AnotherLogger == null)
    {
        _view.ErrorMessage = "Elevated privileges was NOT successful";
    } else {
        _view.SuccessMessage = "Elevated privileges was successful";
        AnotherLogger.Log(typeof (PlaygroundPresenter), _view.SuccessMessage);
    }
});

when HttpContext is null

HttpContext context = HttpContext.Current;
HttpContext.Current = null;
try
{
    using (ILifetimeScope container = SPServiceLocator.NewDisposableLifetime())
    {
        var logger = container.Resolve<ILogger>();
        ...
    }
}
catch (Exception ex)
{
    _view.ErrorMessage = ex.Message;
}
HttpContext.Current = context;

In a background thread

 
public void TestInBackgroundThread()
{
    // page must be async for the following (which still hangs the page)
    var bw = new BackgroundWorker {WorkerSupportsCancellation = false, WorkerReportsProgress = true};
    bw.DoWork += BwDoWork;
    bw.ProgressChanged += BwProgressChanged;
    bw.RunWorkerCompleted += BwRunWorkerCompleted;
    bw.RunWorkerAsync();
}
 
private static void BwDoWork(object sender, DoWorkEventArgs e)
{
    ILifetimeScope container = null;
    try
    {
        container = SPServiceLocator.NewDisposableLifetime("worker");
        var bwLogger = container.Resolve<ILogger>();
 
        var worker = sender as BackgroundWorker;
 
        for (int i = 1; (i <= 10); i++)
        {
            // Perform a time consuming operation and report progress.
            Thread.Sleep(500);
            if (worker != null)
                worker.ReportProgress((i * 10));
            bwLogger.Log(typeof(PlaygroundPresenter), string.Format("bw worker progress: {0}%", (i * 10)));
            // watch this in ULS
        }
 
        e.Result = "Super, background worker has completed";
    }
    finally
    {
        if (container != null)
            container.Dispose();
    }
}

Using Autofac on the Server

You can reference and override the registrations from a console app, powershell, and/or Linqpad, and as long as you are on the server, access service implementations.

General Comments

If you’re interested in this code, go ahead and download the sample attached to this post. I’ve included the feature and feature event receiver project, and a library project which has SPContainerBuilder, SPServiceLocator, SPContainerProvider classes and more that help with managing Autofac capabilities.

BTW: I coded all this in 1 day, as a prototype, so I can’t promise it’s 100% full-proof, but it does work in my use-cases. Do download the code and play with it, and if you decide to improve this, let me know. Or take it into a new direction all-together, that’s fine too .

Download Code and Files

Thanks for reading as always.

By Matt C. | Posted in .NET, SharePoint | Tagged .net, autofac, di, ioc, sharepoint | Comments (1)

Extending Quartz.NET – A sequential plugin execution engine

January 27, 2012 – 12:15 am

This post is for those that want to learn a little bit about Quartz.NET, and/or need a tool to easily implement a cron-job capability, with cron-job scheduling that can be modified at runtime. In fact, you don’t need to know anything about Quartz.NET to get up and running with what’s below.

How I discovered Quartz.NET

Three or four years ago I stumbled on the book "Quartz Job Scheduling Framework: Building Open Source Enterprise Applications" by "Chuck Cavaness" while browsing the technology section of a Borders bookstore. I remember it peeking my interest. A year or two later, I heard about an open-source version of this engine implemented in .NET: "Quartz.NET - Enterprise Job Scheduler for .NET Platform". I have since then used this framework on many projects, including some very large global ecommerce projects handling hundreds of thousands of transactions. Needless to say, if you need a lightweight job scheduling framework that really packs a punch, and you have a windows server available, this thing might be perfect for you. If you’re not into windows, use the original version “Quartz Enterprise Job Scheduler”.

Putting Quartz.NET to good use

Three years ago, as my first little pet project on Quartz.NET, I decided to build a little stock analysis application that would:

keep a local database (at first, just a bunch of csv files, then later a relational database) updated with all the ticker symbols from the NYSE and NASDAQ
download a reasonable amount of level II intra-day price information for each ticker over the course of each trading day (using a couple feeds, but primarily the free Yahoo stock price feed)
allow me to write plugins that I could easily drop into a folder, to:

download company pro-forma statements and earnings reports
parse html from daily and weekly advice columns
do analytics on the data

The core framework took only a week-end to build with Quartz.NET (the plugins took a little longer ).
With a single plugin that would email me alerts, some custom reports, and just writing custom queries and T-SQL over the database, I had myself a nice little useful tool.

Rediscovering the project

I just recently found this little project from several years ago on a backup drive, and decided to take a look at it again. Noticing that Quartz.NET has gone through a couple evolutions, I thought I would upgrade it to the latest version, and put it out there for anybody that might be interested (minus the stock trading piece, maybe save that for another blog post), before it goes back into hibernation.

I am leaving the core logic alone, it's useful as is. Run with it if you like it. If I were to fully redevelop this, I would make better use of the Quartz.NET trigger and listener features, but those are not necessary to build a pretty nifty capability, as I think you'll see from this.
Let me just highlight the core capabilities I was originally out to satisfy from a cron-engine, for this personal project described above:

Keep management of the job engine to a minimum
Execute a primary job at specific intervals (every X number of seconds for example, or minutes, or hours)
The primary job loops sequentially through a number of other jobs that have their own timing parameters, and whose timing parameters can be changed at run-time
Each job executes a set of plugins sequentially
Key/value configuration settings can be specified globally, per job, and per plugin. Lower tiers can override settings inherited from a higher tier.
Plugins must be able to pass data between themselves, so that by chaining plugins one after another, one can use information established during a prior plugin's execution in it's own processing logic

Download Code and Files

Installing Quartz.NET as a Windows Service

Installing Quartz.NET is a breeze. Everything you need is in the zip file attached to this post. But you can also get the Quartz files directly from sourceforge at: http://sourceforge.net/projects/quartznet/files/quartznet/. I’m using version Quartz.NET 2.0 beta 2 in this post.

Step 1 – Extract the contents of the attached zip file into a directory of your choice - I'm using “C:\Quartz\”, then proceed to step 3. If you want to install from the sourceforge download, proceed to step 2.

Step 2 – Installing from source.

- Extract the contents from the “Quartz.NET-2.0-beta2.zip” file into a directory. I am extracting them into “C:\Quartz\src”.

- Open a VS.NET command prompt, and run the following commands:

- Check to make sure the log files msbuild.output.log and msbuild.server.output.log both end without any warnings or errors.

- Create 2 batch files to easily install and uninstall the Quartz server as a windows service:

- Compile the source of the project “Quartz.Server.Extensions.csproj” from the zip file attached to this post or just copy the “Quartz.Server.Extensions.dll” to your Quartz directory

- Overwrite the existing quartz_jobs.xml (the new one has the preconfigured Sequential job configuration in it) and quartz.server.exe.config file (this is optional, it just has some additional log4net settings in it) with those from the zip file

Step 3 – Start the service

Before you start, make sure that paths in the files quartz_jobs.xml, quartz.server.exe.config, install.bat, and uninstall.bat are correct. If you’re following along with this post, you don’t need to do anything.

Go to a command prompt (run it as Administrator preferably) and run the install.bat file. To uninstall the service, just run the uninstall.bat file.

You’ll now see that your service is running in your services console.

A “logs” directory was created and you can look inside the “quartz.log” file to track what’s going on with your server.

Your log file will look like this below. Notice the “ERROR” line, we’ll talk about that next.

Running the Sequential Job and some sample plugins

If you dig into the quartz_jobs.xml file, you’ll see a configuration section that looks like this. This is the job that runs all the plugins.

quartz_jobs.xml
 <job>
  <name>SequentialJobPool</name>
  <group>SequentialJobPoolGroup</group>
  <description>A job that runs a sequence of plugins on a regular basis</description>
  <job-type>Quartz.Server.Extensions.SequentialPluginJob, Quartz.Server.Extensions</job-type>
  <durable>true</durable>
  <recover>false</recover>
  <job-data-map>
    <entry>
      <key>PluginsDirectory</key>
      <value>c:\quartz\plugins</value>
    </entry>
    <entry>
      <key>PluginsConfigFile</key>
      <value>plugins-quartz-config.xml</value>
    </entry>
    <entry>
      <key>PluginsScheduleFile</key>
      <value>plugins-quartz-schedule.xml</value>
    </entry>
  </job-data-map>
</job>
<trigger>
  <simple>
    <name>SequentialJobPool-Trigger</name>
    <group>SequentialJobPool-TriggerGroup</group>
    <description>Fires the job to check for any plugin activity needed</description>
    <job-name>SequentialJobPool</job-name>
    <job-group>SequentialJobPoolGroup</job-group>
    <misfire-instruction>SmartPolicy</misfire-instruction>
    <repeat-count>-1</repeat-count>
    <repeat-interval>30000</repeat-interval>
  </simple>
</trigger>
 

You’ll notice a reference to a plugins directory.

Plugin DLLs can be either placed in the plugins directory, or in the root directory, but a valid plugins directory must be configured.
Let’s create the plugins directory, and restart the service.

Now, take a look at the log file:

And in the plugins directory there are 2 files: plugins-quartz-config.xml and plugins-quartz-schedule.xml …

plugins-quartz-config.xml
 <?xml version="1.0" encoding="utf-8"?>
<config>
  <!-- Main configuration file for the sequential plugin job used within Quartz -->
  <jobs count="1">
    <job id="1">
      <plugins count="1">
        <plugin type="Quartz.Server.Extensions.NullPlugin, Quartz.Server.Extensions">
          <settings count="2">
            <setting key="sampleSetting1" value="sampleSettingValue1" />
            <setting key="sampleSetting2" value="sampleSettingValue2" />
          </settings>
        </plugin>
      </plugins>
      <settings count="2">
        <setting key="sampleSetting1" value="sampleSettingValue1" />
        <setting key="sampleSetting2" value="sampleSettingValue2" />
      </settings>
    </job>
  </jobs>
  <settings count="2">
    <setting key="sampleSetting1" value="sampleSettingValue1" />
    <setting key="sampleSetting2" value="sampleSettingValue2" />
  </settings>
</config>
 

The first xml file - plugins-quartz-config.xml - describes the plugins that are executed. Each job (Sequential Job, not to be confused with a Quartz Job which is inside the quartz_jobs.xml) runs sequentially, and each plugin within the job runs sequentially.

Settings can override each other, from top down, starting at the jobs level (across all jobs), job level, and plugin level. These settings are all part of the context parameter passed into the plugin’s “Execute” function.

plugins-quartz-schedule.xml
 <?xml version="1.0" encoding="utf-8"?>
<schedules>
  <!-- The incrementType can be: Milliseconds, Seconds, Minutes, Hours, Days, None, Once -->
  <schedule jobId="1" increment="10000" incrementType="Milliseconds" lastRun="2012-01-27T00:46:34" nextRun="2012-01-27T00:46:44" />
</schedules>
 

The 2nd file - plugins-quartz-schedule.xml – describes the schedule for the job. These XML files are created as a sample when you run the SequentialJob for the first time. The nice thing about this file, is that it’s constantly changing. To turn off a job, just set the “nextRun” value of the jobId to a future date far into the future. Change the increment and incrementType at any time to change the schedule for this job.

Creating a plugin and adding it to a job

There’s only 1 simple interface to extend, in the Quartz.Server.Extensions assembly:

IPlugin
 public interface IPlugin
{
    void Execute(PluginContext pluginContext);
}
 

The PluginContext class has 2 public properties on it:

PluginContext
  
/// <summary>
/// These are the settings from the plugins-quartz-config.xml file.
/// <remarks>
/// If a setting key at the plugin level exists at a higher level, it will override the higher level key.
/// </remarks>
/// </summary>
public ReadOnlyDictionary<string, string> Settings { get; internal set; }
/// <summary>
/// This dictionary of data can be altered within your plugin. You can chain plugins together
/// and create business logic based on values inside the Data dictionary.
/// </summary>
public IDictionary<string, object> Data { get; private set; }
 

We’ll create a sample plugin, as follows.

Code Snippet
 using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Common.Logging;
 
namespace Quartz.Server.Extensions.Plugins
{
    public class SamplePlugin : IPlugin
    {
        private static readonly ILog log = LogManager.GetLogger(typeof(SamplePlugin));
 
        public void Execute(PluginContext pluginContext)
        {
            // get the settings & data
            var settings = pluginContext.Settings;
            var data = pluginContext.Data;
 
            // let's do a bunch of logging
            log.Info("Start " + this.GetType().Name);
 
            // log the settings
            int settingsCount = settings == null ? 0 : settings.Count();
            log.Info("With " + settingsCount + " settings:");
            if (settingsCount > 0)
            {
                foreach (string key in settings.Keys)
                    log.Info(string.Format(" {0}: {1}", key, settings[key]));
            }
 
            // log data dictionary values
            int dataCount = data == null ? 0 : data.Count();
            log.Info("With " + dataCount + " data records:");
            if (dataCount > 0)
            {
                foreach (string key in data.Keys)
                    log.Info(string.Format(" {0}: {1}", key, data[key]));
            }
 
            // change and/or add some data dictionary values
            data["Status"] = "Success";
            data["StatusDate"] = DateTime.Now;
 
            log.Info("End " + this.GetType().Name);
        }
    }
}
 

This sample is in the Quartz.Server.Extensions.Plugins.dll file. Let’s just move that file into the plugins directory now, and register it in the plugins-quartz-config.xml file.

new plugins-quartz-config.xml
 <?xml version="1.0" encoding="utf-8"?>
<config>
  <!-- Main configuration file for the sequential plugin job used within Quartz -->
  <jobs count="1">
    <job id="1">
      <plugins count="2">
        <plugin assembly="Quartz.Server.Extensions.Plugins" type="Quartz.Server.Extensions.Plugins.SamplePlugin">
          <settings count="1">
            <setting key="sampleSetting2" value="SamplePlugin setting value 2" />
          </settings>
        </plugin>
        <plugin type="Quartz.Server.Extensions.NullPlugin, Quartz.Server.Extensions">
          <settings count="1">
            <setting key="sampleSetting1" value="NullPlugin setting value 1" />
          </settings>
        </plugin>
      </plugins>
      <settings count="2">
        <setting key="sampleSetting1" value="sampleSettingValue1" />
        <setting key="sampleSetting2" value="sampleSettingValue2" />
      </settings>
    </job>
  </jobs>
  <settings count="2">
    <setting key="sampleSetting1" value="sampleSettingValue1" />
    <setting key="sampleSetting2" value="sampleSettingValue2" />
  </settings>
</config>
 

As soon as you alter this file, provided the DLL file was moved into the plugins directory, the job will pick it up and execute it in sequence. See the log output here:

Summary

This was an exercise in creativity, leveraging Quartz.NET, and a single Quartz.NET job to create a little sequential job plugin framework that one can alter at runtime by adding/removing plugins and changing job schedules without any pain whatsoever.

Adding plugins using the method above is a breeze, extend an interface, drop the dll into the folder, and add a <plugin> tag to an xml file, and voila. I have clients that have been running this code (possibly slightly tailored to their needs Winking smile ) for 2 to 3 years now to chain all kinds of events together like parsing text, doing mass PDF conversions of files sent in through SMTP, sending out email newsletters, and synchronizing user profiles between Active Directory and various CMS systems. It keeps cranking thanks to Quartz.NET. A variation of this was also used to process orders in a large ecommerce site, and send the orders to shipping companies and manufacturing plants around the world.

I’m sure there is MUCH to improve on this; so if you do experiment with this, and improve on it, please do let me know, and I’d be glad to post your updates here.

Download Code and Files

Thanks for reading as always.

By Matt C. | Posted in .NET | Tagged .net, C#, job scheduling engine, plugins, Quartz.NET | Comments (2)

How do you learn? A PMP Exam Case Study

December 27, 2011 – 12:19 pm

Learning how you learn

I think a team is at its best when all of it's members have an awareness of the learning methods and techniques that work best for them. Experts in the field of learning have uncovered common characteristics between novice and expert learners, such as in this study 'Novice to Expert.doc' at the University of Milwaukee, by Barbara Daley, which takes into account the learner's motivation, and drive for obtaining results. According to this study, the expert sees learning as an "active and self-initiated process", while the novice requires to be guided in the learning process, and can be sometimes easily overwhelmed with information and fear. It's interesting to me that one's ability to learn could possibly boil down in some ways to 'confidence'. In my experience, that confidence comes from experience, from trial and error. This is what I mean by: "learning to learn". Consider, for example, the different learning patterns of an artist, a businessman and a librarian. Watching an artist build beautiful works is a magical experience. How did they learn to do that? At what point did they realize they were 'good enough' to put their art on display? At what point did they realize this would be their vocation? What is their mental model? A businessman sees numbers behind things, and has learned to see the value of something in terms of how the market will value it. He has learned to gauge his ability to sell that very thing to his network. At what point did that businessman learn to trust in his instincts about the value of things? A librarian often knows a myriad of facts and figures. How does a person like that know so much? How does one person's brain contain so much information. This bookish-type of learner is most intimidating to me, because I just don't have that gift. There are many more types of learners, and those who fall into the same "type" can have very different learning methods as well. That might be the most fascinating part of all.

How do you learn? More importantly, have you learned how you learn?

Here's a personal analogy.

I moved around a lot as a kid, completing most of my K-12 years in France, and finishing the last two years of high school in the D.C. area. As a 3rd culture kid, I learned how to adapt across cultures, to speak different languages, and to blend in socially as best I could. In school I developed techniques to learn materials, as many of us have, by creating summaries that drew parallels to my experiences. This technique has proven invaluable for me over time, and it's a foundation for additional techniques I've crafted: creating summaries of summaries, special note-taking techniques and abbreviations, cross-summary references, and using numbers instead of words to remember things. I'm still learning about how I learn, and I feel like I have a long ways to go still. I am fascinated by the learning process. Understanding my own learning process has been a pivotal part of my life, one that helped me survive the ever-changing environments of my youth. One of the biggest factors for me in "learning to learn" has been learning to trust in my learning abilities. An "I can learn this" mentality. That understanding has enabled me to thrive in my career.

In 2006 I was working at a big consulting firm, and the PMP credential was popular. To bid on certain projects, we needed to provide proposals showing not only technical expertise, but also credentials, namely PMP-certified resources. The company was paying for all the directors and senior managers to take classes and intensives to pass the PMP exam. I was a new manager at the time, so I didn't get the privilege. But I wanted the credential. I had all the project-management prerequisites (number of hours, etc.), so I just needed to pass the exam. I prepared for the test, and the experience I had learning the material and what I learned about myself were more valuable to me than the material itself.

I selected two books from which to learn the material: PMP in Depth: Project Management Professional Study Guide for the PMP Exam, and A Guide to the Project Management Body of Knowledge: (Pmbok Guide). I bought the books one weekend, scheduled the test for the next, and studied on the weekdays between. Here's the learning process I used for this 'learning project':

I glanced through the books' indices and mapped out on paper the major points I needed to learn, including all the knowledge areas (inputs, outputs, tools & techniques), math equations and how to use them, keywords and definitions;
I mapped out a strategy and timeline for how I would learn the material, and what I would know by when. I focused on learning concepts, processes and knowledge areas first, and math equations last;
I summarized the material using acronyms, numbers, and a color-coded grid system for memorizing relationships between knowledge areas; and
I repeatedly quizzed myself on the content.

I spent the first 25 minutes of the test doing a brain-dump onto paper of the entire grid I memorized. This worked well. If you're interested in my grid system and the technique I used, it's attached to this post as a download, or you can view it below.

This is a typical process for me on any given 'learning project', but this method of learning I realize does not work for everyone. My wife, for instance, learns by putting things into their temporal context--she recalls past events as if they happened yesterday. I could ask her what was I doing on March 2, 2006 at 5 p.m., and she would figure it out within a minute by using historical landmarks embedded in her brain. I don't know how she does it. It's amazing. But it's frustrating, too, because I can't get away with anything. My dad, as another example, was a bookworm and had developed a mental cataloging system for his thousands of books, magazines and newspaper clippings that was otherworldly to me. On any subject I discussed with him, he could point to a half-dozen books, articles, down to the page number and paragraph on-the-fly, often citing the actual author of the article whether it be from the New York Times newspaper, a book, or encyclopedia. For him, understanding the authorship, and context of a piece of knowledge on any given thing was central to his learning process of the thing itself. For him, a map could be used not only to provide directions, but also to tell a story of a place. His method of learning was to take notes, boxes and boxes of notes, then to file the notes, and do further research related to ideas he had in his notes. He was successful because of his ability to learn, and help others learn about things of interest to them.

In Conclusion

Our ability to master the learning process can be a very exciting endeavor, not only for our careers, but more importantly, it greatly enriches our lives, and gets us further in touch with the things we care most about. Learning is a lifelong habit, and achievement. If this resonates to you, have some learning tips or ideas on the subject, feel free to share them in the comments of this post. Happy learning!

PMP Study Sheet and Learning Grid System

This is the grid system I used below on the 'PMP exam learning project' I wrote about above.

PMP Study Sheet Download

[--link rel='stylesheet' href='/wp-content/themes/sandbox-custom/scripts/pmp-sample.css' type='text/css' media='all' /--] [--script type='text/javascript' src='/wp-content/themes/sandbox-custom/scripts/pmp-sample.js'--][--/script--] [--script language="JavaScript1.2"--] $(document).ready(function() { toggleAll(); $(".ekap").click(function() { toggleAll(); }); $(".cp-sel").click(function() { var cpid = $(this).attr("class").replace("cp-sel cp-sel-", "cp"); $("." + cpid).toggle(); }); $(".mouse-mv").mouseover(function() { var cpid = $(this).text(); $(".cp-sel-" + cpid).parent().parent().children("td").css("border-bottom","1px solid #ff0000").css("background-color","#ffcccc"); }); $(".mouse-mv").mouseout(function() { var cpid = $(this).text(); $(".cp-sel-" + cpid).parent().parent().children("td").css("border-bottom","1px solid #000000").css("background-color","#efefef"); }); $(".init-hidden").show(); }); [--/script--]

ommunication Management

Communication Planning
» 25

Inputs:
	EEF, OPA, Proj. scope statement, Proj. mgt. plan (constraints, assumptions)
Tools & techniques:
	Communicatiions requirements analysis, Communications technology
Outputs:
	Communications management plan

Information Distribution
» 33

Inputs:
	Communications management plan
Tools & techniques:
	Communications skills, Information gathering and retrieval systems, Information distribution methods, Lessons learned process
Outputs:
	OPA (updates), Requested changes

Performance Reporting
» 28, 32, 36, 37, 39, 40, 41, 42

Inputs:
	Work performance information, Performance measurements, Forecasted completion, Quality control measurements, Proj. mgt. plan (performance measurement baseline), Approved change requests, Deliverables
Tools & techniques:
	Information presentation tools, Performance information gathering and compilation, Status review meetings, Time reporting system, Cost reporting systems
Outputs:
	Performance reports, Forecasts, Requested changes, Recommended corrective actions, OPA (updates)

Manage Stakeholders
» 43

Inputs:
	Communications mgt. plan, OPA
Tools & techniques:
	Communications methods, Issue logs
Outputs:
	Resolved issues, Approved change requests, Approved corrective actions, OPA (updates), Proj. mgt. plan (updates)

ost Management

Cost Estimating
» 7, 18, 22, 37

Inputs:
	EEF, OPA, Proj. scope statement, WBS, WBS dictionary, Proj. mgt. plan (schedule mgt. plan, staffing mgt. plan, risk register)
Tools & techniques:
	Analogous estimating, Determine resource cost rates, Bottom-up estimating, Parametric estimating, Proj. mgt. software, Vendor bid analysis, Reserve analysis, Cost of quality
Outputs:
	Activity cost estimating, Activity cost estimate supporint detail, Requested changes, Cost mgt. plan (updates)

Cost Budgeting
» 7, 37, 42

Inputs:
	Proj. scope statement, WBS, WBS dictionary, Activity cost estimates, Activity cost estimate supporting detail, Proj. schedule, Resource calendars, Contract, Cost mgt. plan
Tools & techniques:
	Cost aggregation, Reserve analysis, Parametric estimating, Funding limit reconciliation
Outputs:
	Cost baseline, Proj. funding requirements, Cost mgt. plan (updates), Requested changes

Cost Control
» 34, 37, 43

Inputs:
	Cost baseline, Proj. funding requirements, Performance reports, Work performance information, Approved change requests, Proj. mgt. plan
Tools & techniques:
	Cost change control system, Performance measurement analysis, Forecasting, Proj. performance reviews, Proj. mgt software, Variance management
Outputs:
	Cost estiamte (updates), Cost baseline (updates), Performance measurements, Forecasted completion, Requested changes, Recommended corrective actions, OPA (updates), Proj. mgt. plan (updates)

uman Resource Management

Human Resource Planning
» 26

Inputs:
	EEF, OPA, Proj. mgt. plan (activity resource requirements)
Tools & techniques:
	Organizational charts and positiion descriptions, Networking, Organizational theory
Outputs:
	Roles and responsibilities, Proj. organization charts, Staffing mgt. plan

Acquire Project Team
» 17, 18, 19, 27

Inputs:
	EEF, OPA, Roles and responsibilities, Proj. organization charts, Staffing mgt. plan
Tools & techniques:
	Pre-assignment, Negotiation, Acquisition, Virtual teams
Outputs:
	Project staff assignments, Resource availability, Staffing mgt. plan (updates)

Develop Project Team
» 28

Inputs:
	Proj. staff assignments, Staffing mgt. plan, Resource availability
Tools & techniques:
	General mgt. skills, Training, Team-building activities, Ground rules, Co-location, Recognition and rewards
Outputs:
	Team performance assessment

Manage Project Team
» 37, 43

Inputs:
	OPA, Project staff assignments, Roles and responsibiliities, Proj. organization charts, Staffing mgt. plan, Team performance assessment, Work performance information, Performance reports
Tools & techniques:
	Observation and conversation, Proj. performance appraisals, Conflict mgt., Issue log
Outputs:
	Requested changes Recommended corrective actions, Recommended preventive actions, OPA (updates), Proj. mgt. plan (updates)

ntegration Management

Develop Project Charter
» 2, 4

Inputs:
	Contract, SOW, EEF, OPA
Tools & techniques:
	Proj. selection methods, Proj. mgt. methodology, IS, Expert Judgment
Outputs:
	Project Charter

Develop Preliminary Project Scope Statement
» 4, 5, 23

Inputs:
	Project Charter, SOW, EEF, OPA
Tools & techniques:
	Proj. mgt. methodology, IS, Expert Judgment
Outputs:
	Preliminary project scope statement

Develop Project Management Plan
» 3, 4, 7, 9, 10, 13, 15, 21, 24, 26

Inputs:
	Preliminary project scope statement, Proj. mgt. processes, EEF, OPA
Tools & techniques:
	Proj. mgt. methodology, IS, Expert Judgment
Outputs:
	Proj. mgt. plan

Direct and Manage Project Execution
» 28, 29, 33, 34, 35, 36, 38, 39, 40, 41, 42

Inputs:
	Proj. mgt. plan, Approved corrective actions, Approved preventive actions, Approved change requests, Approved defect repair, Validated defect repair, Admin. closure procedure
Tools & techniques:
	Proj. mgt. methodology, IS
Outputs:
	Deliverables, Requested changes, Implemented change requests, Implemented corective actions, Implemented preventive actions, Implemented defect repair, Work performance information

Monitor & Control Project
» 37

Inputs:
	Proj. mgt. plan, Work performance information, Rejected change requests
Tools & techniques:
	Proj. mgt. methodology, IS, Earned value technique, Expert judgment
Outputs:
	Recommended corrective ations, Recommended preventive actions, Forecasts, Recoimmended defect repair, Requested changes

Integrated Change Control
» 5, 23, 24, 33, 43

Inputs:
	Proj. mgt. plan, Requested changes, Work performance information, Recommended preventive actions, Recommended corrective actions, Recommended defect repair, Deliverables
Tools & techniques:
	Proj. mgt. methodology, IS, Expert judgment
Outputs:
	Approved change requests, Rejected change requests, Proj. mgt. plan (updates), Proj. scope statement (updates), Approved corrective actions, Approved preventive actions, Approved defect repair, Validated defect repair, Deliverables

Close Project
» 44

Inputs:
	Proj. mgt. plan, Contract documentation (from Contract Administration), EEF, OPA, Work performance information, Deliverables
Tools & techniques:
	Proj. mgt. methodology, IS, Expert judgment
Outputs:
	Administrative closure procedure, Contractx closure procedure, Final product, service, or result, OPA (updates)

rocurement Management

Plan Purchases and Acquisitions
» 8

Inputs:
	EEF, OPA, Proj. scope statement, WBS, WBS dictionary, Proj. mgt. lan (risk register, risk-related contractual agreements, resource requirements, proj. schedule, activity cost estimates, cost baseline)
Tools & techniques:
	Make-or-buy analysis, Expert judgment, Contract types
Outputs:
	Procurement management plan, Contract statement of work, Make-or-buy decisions, Requested changes

Plan Contracting
» 30

Inputs:
	Procurement management plan, Contract statement of work, Make-or-buy decisions, Proj. mgt. plan (risk register, risk-related contractual agreements, resource requirements, proj. schedule, activity cost estimate, cost baseline)
Tools & techniques:
	Standard forms, Expert judgment
Outputs:
	Procurement documents, Evaluation criteria, Contract statement of work (updates)

Request Seller Responses
» 31

Inputs:
	OPA, Procurement management plan, Procurement documents
Tools & techniques:
	Bidder conferences, Advertising, Develop qualified sellers list
Outputs:
	Qualified sellers list, Procurement document package, Proposals

Select Sellers
» 17, 18, 19, 22, 36, 37

Inputs:
	OPA, Procurement mgt. plan, Evaluation criteria, Procurement document package, Proposals, Qualified sellers list, Proj. mgt. plan (risk register, risk-related contractual agreements)
Tools & techniques:
	Weighting system, Independent estimates, Screening system, Contract negotiation, Seller rating systems, Expert judgment, Proposal evaluation techniques
Outputs:
	Selected sellers, Contract, Contract mgt. plan, Resource availability, Procurement mgt. plan (updates), Requested changes

Contract Administration
» 37, 43

Inputs:
	Contract, Contract mgt. plan, Selected sellers, Performance reports, Approved change requests, Work performance information
Tools & techniques:
	Contract change control system, Buyer-conducted performance review, Inspections and audits, Performance reporting, Payment system, Claims administration, Records mgt. system, Information technology
Outputs:
	Contract documentation, Requested changes, Recommended corrective actions, OPA (updates), Proj. mgt. plan (procurement mgt. plan, contract mgt. plan updates)

Contract Closure
» (43)

Inputs:
	Procurement mgt. plan, Contract mgt. plan, Contract documentation, Contract closure procedure
Tools & techniques:
	Procurement audits, Records mgt. system
Outputs:
	Closed contracts, OPA (updates)

uality Management

Quality Planning
» 29, 37

Inputs:
	EEF, OPA, Proj. scope statement, Proj. mgt. plan
Tools & techniques:
	Cost-benefit analysis, Benchmarking, Design of experiments, Cost of quality (COQ), Additional quality planning tools
Outputs:
	Quality mgt. plan, Quality metrics, Quality checklists, Process improvement plan, Quality baseline, Proj. mgt. plan (updates)

Quality Assurance
» 37, 38

Inputs:
	Quality mgt. plan, Quality metrics, Process improvement plan, Work performance information, Approved change requests, Quality control measurements, Implemented change requests, Implemented corrective actions, Implemented defect repair, Implemented preventive actions
Tools & techniques:
	Quality planning tools and techniques, Quality audits, Process analysis, Quality control tools and techniques
Outputs:
	Requested changes, Recommended corrective actions, OPA (updates), Proj. mgt. plan (updates)

Perform Quality Control
» 37, 43

Inputs:
	Quality mgt. plan, Quality metrics, Quality checklists, OPA, Work performance information, Approved change requests, Deliverables
Tools & techniques:
	Cause and effect diagram (Ishikawa diagrams / fishbone diagrams), Control charts (defect analysis, are process variables within acceptable limits, upper and lower +/- 3 sigma standard deviation), Flowcharting (consecutive boxes), Histogram (bar chart showing frequency of characteristics), Pareto chart (type of histogram ordered by frequency of occurrence / best defect analysis tool to guide corrective action by fixing problems causing the most defects first - 80/20 rules [i.e.: number of defective cases on Y, types of problems histogram on X), Run chart (trend analysis to predict future), Scatter diagram (determine relationship between 2 variables), Statistical sampling, Inspection, Defect repair review
Outputs:
	Quality control measurements, Validated defect repair, Quality baseline (updates), Recommended corrective actions, Recommended preventive actions, Requested changes, Recommended defect repair, OPA (updates), Validated deliverables, Proj. mgt. plan (updates)

isk Management

Risk Management Planning
» 11, 18, 19

Inputs:
	EEF, OPA, Proj. scope statement, Proj. mgt. plan
Tools & techniques:
	Planning meetings and analysis
Outputs:
	Risk mgt. plan

Risk Identification
» 7, 12, 31

Inputs:
	EEF, OPA, Proj. scope statement, Risk mgt. plan, Proj. mgt. plan
Tools & techniques:
	Documentation reviews, Information gathering techniques, Checklist analysis, Assumptions analysis, Diagramming techniques
Outputs:
	Risk register

Qualitative Risk Analysis
» 13

Inputs:
	OPA, Proj. scope statement, Risk mgt. plan, Risk register
Tools & techniques:
	Risk probability and impact assessment, Probability and impact matrix (PIM), Risk data quality assessment, Risk categorization, Risk urgency assessment
Outputs:
	Risk register (updates)

Quantative Risk Analysis
» 14

Inputs:
	OPA, Proj. scope statement, Risk mgt. plan, Risk register, Proj. mgt. plan (proj. schedule mgt. plan, proj. cost mgt. plan)
Tools & techniques:
	Data gathering and representation techniques, Quantitative risk analysis and modeling techniques
Outputs:
	Risk register (updates)

Risk Response Planning
» 7, 31, 37, 41

Inputs:
	Risk mgt. plan, Risk register
Tools & techniques:
	Strategies for negative risk or threats, Strategies for positive risks or opportunities, Strategy for both threats and opportunities, Contingent response strategy
Outputs:
	Risk register (updates), Proj. mgt. plan (updates), Risk-related contractual agreements

Risk Monitoring and Control
» 37, 43

Inputs:
	Risk mgt. plan, Risk register, Approved change requests, Work performance information, Performance reports
Tools & techniques:
	Risk reassessment, Risk audits, Variance and trend analysis, Technical performance measurement, Reserve analysis, Status meetings
Outputs:
	Risk register (updates), Requested changes, Recommended corrective actions, Recommended preventive actions, OPA (updates), Proj. mgt. plan (updates)

cope Management

Scope Planning
» 5

Inputs:
	EEF, OPA, Proj. charter, Preliminary proj. scope statement, Proj. mgt. plan
Tools & techniques:
	Expert judgment, Templates, forms, standards
Outputs:
	Proj. scope mgt. plan

Scope Definition
» 3, 6, 7, 9, 10, 15, 21, 37

Inputs:
	OPA, Proj. charter, Preliminary proj. scope statement, Proj. scope mgt. plan, Approved change requests
Tools & techniques:
	Product analysis, Alternatives identification, Expert judgment, Stakeholder analysis
Outputs:
	Proj. scope statement, Requested changes, Proj. scope mgt. plan (updates)

Create WBS
» 7, 15, 21, 35, 37

Inputs:
	OPA, Proj. scope statement, Proj. scope mgt. plan, Approved change requests
Tools & techniques:
	WBS templates, Decomposition
Outputs:
	Proj. scope statement (updates), WBS, WBS dictionary, Scope baseline, Proj. scope mgt. plan (updates), Requested changes

Scope Verification
» 37, 40

Inputs:
	Proj. scope statement, WBS dictionary, Proj. scope mgt. plan, Deliverables
Tools & techniques:
	Inspection
Outputs:
	Accepted deliverables, Requested changes, Recommended corrective actions

Scope Control
» 37, 43

Inputs:
	Proj. scope statement, WBS, WBS dictionary, Proj. scope mgt. plan, Performance reports, Approved change requests, Work performance information
Tools & techniques:
	Change control system, Variance analysis, Replanning, Configuration management system
Outputs:
	Proj. scope statement (updates), WBS (updates), WBS dictionary (updates), Scope baseline (updates), Requested changes, Recommended corrective actions, OPA (updates), Proj. mgt. plan (updates)

ime Management

Activity Definition
» 16, 37

Inputs:
	EEF, OPA, Proj. scope statement, WBS, WBS dictionary, Proj. mgt. plan
Tools & techniques:
	Decomposition, Templates, Rolling wave planning, Expert judgment, Planning component
Outputs:
	Activity list, Activity attributes, Milestone list, Requested changes

Activity Sequencing
» 17, 37

Inputs:
	Proj. scope statement, Activity list, Activity attributes, Milestone list, Approved change requests
Tools & techniques:
	Precedence diagramming method (PDM), Arrow diagramming method (ADM), Schedule network templates, Dependency determination, Applying leads and lags
Outputs:
	Proj. schedule network diagrams, Activity list (updates) Activity attributes (updates), Requested changes

Activity Resource Estimating
» 7, 18, 20, 22, 37

Inputs:
	EEF, OPA, Activity list, Activity attributes, Resource availability, Proj. mgt. plan
Tools & techniques:
	Expert judgment, Alternatives analysis, Published estimating data, Proj. mgt. software, Bottom-up estimating
Outputs:
	Activity resource requirements, Activity attributes (updates), Resource breakdown structure, Resource calendars (updates), Requested changes

Activity Duration Estimating
» 19, 37

Inputs:
	EEF, OPA, Proj. scope statement, Activity list, Activity attributes, Activity resource requirements, Resource calendars, Proj. mgt. plan (risk register, activity cost estimates)
Tools & techniques:
	Expert judgment, Analogous estimating, Paraetric estimating, Three-point estimates, Reserve analysis
Outputs:
	Activity duration estimates, Activity attributes (updates)

Schedule Development
» 7, 22, 37, 39

Inputs:
	OPA, Proj. scope statement, Activity list, Activity attributes, Proj. schedule network diagrams, Activity resource requirements, Resource calendars, Activity duration estimates, Proj. mgt. plan (risk register)
Tools & techniques:
	Schedule network analysis, Critical path method, Schedule compression, What-if scenario analysis, Resource leveling, Critical chain method, Proj. mgt. software, Applying calendars, Adjusting leads and lags, Schedule model
Outputs:
	Proj. schedule, Schedule model data, Schedule baseline, Resource requirements (updates), Activity attributes (updates), Proj. calendar (updates), Requested changes, Proj. mgt. plan (schedule mgt. plan updates)

Schedule Control
» 37, 43

Inputs:
	Schedule mgt. plan, Schedule baseline, Performance reports, Approved change requests
Tools & techniques:
	Progress reporting, Schedule change control system, Performance measurement, Proj. mgt. software, Variance analysis, Schedule comparison bar charts
Outputs:
	Schedule model data (updates), Schedule baseline (updates), Performance measurements, Requested changes, Recommended corrective actions, OPA (pdates), Activity list (updates), Activity attributes (updates), Proj. mgt. plan (updates)

Legend

ACWP	Actual cost of work performed
ADM	Activity diagramming method
BAC	Budget at completion
BCR	Benefit cost ratio
BCWP	Budget cost of work performed
BCWS	Budget cost of work scheduled (Planned value)
CPI	Cost performance index
CPIF	Cost plus incentive fee
CPF	Cost plus fee (fee increases with services rendered) --> aka: CPPC
CPFF	Cost plus fixed fee
CPPC	Cost plus percentage of cost
EAC	Estimate at completion
EEP	Enterprise environmental factors (org. culture, proj. mgt. information system, HR pool)
EMV	Expected monetary value Decision tree analysis (helps to show how to make a decision between alternatives).
ETC	Estimate to complete
EV	Earned value (aka: BCWP)
FFP	Firm fixed price
IS	Project management information system
MR	Management Reserve (cost estimating) The difference between the maximum funding and the end of the cost baseline is Management Reserve
OBS	Organizational breakdown structure Arranged according to an organization's existing departments, units, or teams. Activities are listed under each organizational unit.
OPA	Organization process assets (policies, procedures, standards, guidelines, defined processes, historical information, lessons learned)
PDM	Precedence diagramming method(activity sequencing technique) Method of constructing a project schedule network diagram that uses boxes or rectangles (nodes) to represent activities and connects them with arrows that show the dependencies. Also called "Activity-on-node (AON)" 4 types of dependencies (or precedence relationships): - Finish-to-start (most common: initiation of successor activity depends upon the completion of the predecessor activity) - Finish-to-finish - Start-to-start - Start-to-finish
PIM	Probability and impact matrix Risks are prioritized according to their potential implications for meeting the project's objectives (assessed against Cost, Time, Scope, Quality). Each risk is rated on its probability of occurring and impact on an objective if it does occur. The organization's thresholds for low, moderate or high risks are shown in the matrix and determine whether the risk is scored as high, moderate, or low for that objective.
PV	Planned value (aka: BCWS)
RACI	Form of a RAM chart (see: RAM)
RAM	Responsibility assignment matrix Example: RACI (Responsible, Accountable, Consult, Inform)
RBS	Risk breakdown structure Lists the categories and sub-categories within which risks may arise for a typical project. Remind participants of the many sources from which project risk could arise. A relative scale representing probability values from "very unlikely" to "almost certainty" could be used.
RBS	Resource breakdown structure Used to breakdown the project by types of resources.
SOW	Statement of Work
SPI	Schedule performance index
SV	Schedule variance
TCPI	To complete performance index (efficiency index)
WBS	Work breakdown structure

Formulas

EV (BCWP)	EV = BAC * ( WC / TW ) --> Work completed / Total work to be performed
CV (ACWP)	CV = EV - AC
CPI	CPI = EV / AC --> should be > 1

PV (BCWS)	PV = BAC * ( TP / TST ) --> Time passed / Total scheduled time
SV	SV = EV - PV
SPI	SPI = EV / PV --> should be > 1

ETC	ETC = BAC - EV
EAC	EAC = ETC + AC
TCPI	( BAC - EV / BAC - AC )

BCR	BCR = Benefit / Cost --> should be > 1


	Constrained optimization methods: Linear, Non-linear, Dynamic, Integer, Multiple object programming
	Probability distributions: Beta (most common), Triangular, Normal, Uniform
	Risk response strategies: S.E.E. vs. A.T.M. (Share vs. Transfer / Exploit (acceptance) vs. Avoid / Enhance vs. Mitigate)
	Tuckman model: 5 stages of development --> Forming, Storming, Norming, Performing, Adjourning
	Management strategies: Avoidance (use for non-critical periods to cool off), Competition (= forcing, win/lose), Compromising (lose-win/lose-win), Accommodation (lose/win, risk of losing credibility & influence in the future), Collaboration (win/win)
	Contract types: FFP (lump sum, risk to buyer & seller), CPF (max risk to buyer, aka: CPPC), CPFF, CPIF, T&M (when you don't know the quantity of procured items)
	Controlling quality: Prevention (eliminate errors) vs. Inspection (ensure compliance), Attribute sampling (compare results w/ standard) vs. variable sampling (measure degree of conformity), Common cause (predictable variations) vs. special cause (removable non-inherent defects), Control limits (expected variation via mean of a normal distribution) vs. tolerances (acceptable range)

PMP Study Sheet Download

By Matt C. | Posted in Miscellaneous | Tagged learning, pmp | Comments (0)

Custom Bookmarker with Chrome Extension

December 16, 2011 – 9:36 pm

A couple days ago, I launched a “Good Reads” section on this blog, where I could quickly capture some bookmarks for blogs I read using Google Reader. I wrote a couple PHP pages to capture the URL, Title and Blog RSS Feed source using a simple GET command, and then display them using the nice jquery plugin jquery.dataTables.js on a WordPress page.

It occurred to me, I might also just want to bookmark other pages as well, not just from Google Reader. So that’s when I thought about writing an extension.

Wow, I was very surprised! Chrome makes it super easy to write browser extensions. In 20 minutes I had a little bookmarker extension in place.

Here’s how it works.

The Code

Create 3 files in a folder of your choice:

index.html
index.js
manifest.json

index.js

 
function sendSelection(i, t) {
  var url = i.linkUrl ? i.linkUrl: i.pageUrl;
  var title = i.selectionText;
  sendTo(url,title);
}
 
function sendPage(i, t) {
  var url = i.url;
  var title = i.title;
  sendTo(url,title);
}
 
function sendTo(u, t) {
  var str = 'Would you like to bookmark the selection: \n';
  str += '- URL: ' + u + '\n';
  str += '- Title: ' + t + '\n';
  if(confirm(str)) {
    var sendToUrl = "http://www.mattjcowan.com/wp-admin/good-read.php?u=" + u + "&t=" + t + "&s=chrome&v=2";
    chrome.tabs.create({"url": sendToUrl });
  };
}
 
var contexts = ["page","selection"]; 
 
for (var i = 0; i < contexts.length; i++) {
  var context = contexts[i]; 
  var title = "Send '" + context + "' to mattjcowan.com";
  if(context == 'selection') { 
    chrome.contextMenus.create({"title": title, "contexts":[context], "onclick": sendSelection});
  };
  if(context == 'page') { 
    chrome.contextMenus.create({"title": title, "contexts":[context], "onclick": sendPage});
  };
}

index.html

<script src="index.js"></script>

manifest.json

{
  "name": "Send To (mattjcowan.com)",
  "description": "Sends the page information to mattjcowan.com",
  "version": "0.1",
  "permissions": ["contextMenus","tabs"],
  "background_page": "index.html",
  "content_security_policy": "default-src 'self'"
}

And there you go.

The Extension

Navigate in your Chrome browser to “chrome://extensions”, and upload your new extension:

The Result

Now, using a simple “Right-click”, I can quickly post a link or page to my “Good Reads” section of the blog.

And it magically appears in my bookmarks:

Nice!

By Matt C. | Posted in Miscellaneous | Tagged chrome | Comments (0)

LLBLGen Pro, Entity Framework 4.1, and the Repository Pattern

September 25, 2011 – 9:43 pm

Introduction

If you’ve been hanging out in the .NET universe, you’ve probably heard of LLBLGen Pro. It’s one of my favorite power tools. If you’re new to LLBLGen, visit http://www.llblgen.com. LLBLGen is an ORM mapping tool, similar to CodeSmith Tools and DataObjects.Net, other excellent power tools. One of the unique things I like about LLBLGen is that it supports an array of persistence frameworks, such as the home-grown, extremely powerful LLBLGen SelfServicing and Adapter frameworks, the Entity Framework (EF, STE, and POCO), Linq to Sql, NHibernate and FluentNHIbernate. LLBLGen supports every major relational database vendor I have ever worked with (MS Access, SQL Server, Oracle, PostgreSql, Firebird, IBM DB2, MySql, and Sybase). LLBLGen Pro supports both a database-first approach, and a model-first approach, and the next version (v3.5) will support the Code-First approach. It also has an advanced templating and code-generation engine and designer that: allows you to rename entities and entity fields; configure your own database mappings; supports advanced EF data models; allows you to create model-only relationships; create typed views on stored procedures and new and existing entity relationships; lets you create code generation libraries of your own, and override any template you want that comes out-of-the-box. And that's just the beginning. In this post, I’d like to have fun with LLBLGen Pro, so I’m going to:

Use the model-first approach in LLBGen Pro designer to create a simple membership API with some table-per-type inheritance;
Build some code-generation templates that extend the out-of-the-box EF templates in LLBLGen to model our API after the repository pattern;
Make use of SQL stored procedures;
Extend the generated code with some extension methods; and
Introduce more surprises along the way.

But first, a word about the repository pattern. Recently I’ve observed some backlash against it, and I don’t want that to distract you.

Repository Pattern (NIH)

This pattern almost needs no introduction. If you are new to this pattern, or itching for some reading, check out the following links:

The repository pattern is very popular, and there is no single implementation. If you’re curious, here are sample open-source applications that show different versions and usages of this pattern.

A hot debate

Abstracting your data access behind clean repository and unit of work interfaces seems simple, but before you go down this road, check your motivations. If you do this because you think one day you’ll want to “swap” out your DAL implementation with another--for instance, go from EF, to NHibernate, or LLBLGen Entity Adapter Framework, or custom DAO--step back and mull the decision a bit more. ORM designers have carefully crafted many features for you to use. Abstracting all of them into your repository façade for a hypothetical future technology shift begs the question: Why are you designing with the current technology in the first place? Check out some interesting voices in the debate: Repository is the new Singleton, The false myth of encapsulating data access in the DAL, Review: Microsoft N Layer App Sample, part VII-Data Access Layer is GOOD for you, Farewell to Repositories (I think). For example, if you use the LLBLGen Adapter framework, building your own Repository layer on top of it doesn’t make sense. This adapter framework has an incredible predicate and deep-fetch system built-in to it. It has a unit of work, and appropriately separates entities from the persistence framework so you can create a rich n-tier, manager, service layer, or whatever you like. So building a repository layer on top of an existing ORM framework may simply duplicate existing functionality. Don’t do it just because you can, or because you’re a purist. Ok, so now, if the fear has been put into you, good. At least you’re going to make a conscious decision to use or not use this pattern in your application. All that said, I still personally like using this pattern more often then not. The repository pattern is easy to implement, and with a code-generation framework like LLBLGen, you can build into your repositories all kinds of things, customizations, thanks to the rich designer and some creative thinking. Using the pattern also helps me think through my implementation a little more, and thinking is fun, and that makes it worth it to me. I almost always use the repository pattern when developing against SharePoint lists for example, because in SharePoint, lists are all over the place, and I can use a simple repository interface (and the Service Locator Pattern) to easily read/write to the lists, regardless of which site the list is in. By creating a factory for the unit of work and the repository, I can also use the repository pattern to read/write data to different persistence stores, which I have had to do on occasion. So all that to say, have fun with it.

Case Study

Building the model

LLBLGen supports two different types of paradigms: “Database First”, and “Model First” code generation. The first approach consists of working with an existing database, and modeling your entity graph on top of it. The second approach consists of modeling your entities within the LLBLGen modeling tool, and working with LLBLGen generated SQL scripts to create and alter your database schema (a migrations approach that relies on SQL). LLBLGen handles creating update scripts for you, as it tracks the state of your model as you model it over time. You can also generate a complete “CREATE” script for your database at any point in time. Check out some intro videos to LLBLGen: http://www.llblgen.com/pages/videos.aspx, and an older video by the wizard of data access Frans Bouma (a principal developer of LLBLGen): http://weblogs.asp.net/fbouma/archive/2010/04/28/llblgen-pro-v3-0-with-entity-framework-v4-0-12m-video.aspx.

Use-case

We are going to create a User entity, Group entity, and Role entity that all inherit from an abstract Principal base entity. In our scenario, the Id and Name of a Principal is always unique. The User entity will have some unique fields of it’s own. Every principal (regardless of type) can have settings (similar to a dictionary of properties for that principal), and preferences, although preferences can be shared between principals. Finally, principals are containers that can contain other principals (regardless of type), and can be contained within any number of other principals (regardless of type).

The Model

Using the LLBLGen Pro modeling tool, I come up with the following.

By making the Principal entity a base entity for Group, User, and Role, I can model the many-to-many relationship between Principal and Preferences, and the one-to-many relationship between Principal and Settings as such, and these relationships are passed on to the child entities. Creating the container relationships so that any principal can contain other principals is a little more complex because recursion is involved. In order to handle recursive queries of varying depths, we'll incorporate some stored procedures with CTE queries into our LLBLGen Pro generated model.

Generating code

The first step in generating code is to determine what persistence framework we’re going to use. So we validate the integrity of the model using the menus: Project > Validate. Then we fix any errors that show up in the error list. Then we map the model to a persistence store using the menus: Project > Add Relational Model Data Storage for a Database. We pick SQL Server. Once we’ve chosen our database type, we can either manually map all the fields to a schema, or we can let the tool do it all for us. In our case we will let the tool do it for us, and we can do that using: Project > Automap Unmapped Entities. Finally, we generate our Visual Studio solution using the menus: “Project > Generate Source Code...”. On the “Task queue to execute” tab, we’ll pick the SD.EntityFramework.v4 (Proxy friendly POCO entities) preset (which is the template bundle we’re going to use), and then we click “Start generator (Normal)”. Here is our solution in the LLBLGen Pro Entity Explorer, and the generated code in Visual Studio.

LLBLGen Pro Entity Explorer		Visual Studio Solution Explorer

At this point, this is essentially a similar output as what the EF designer would produce along with the T4 POCO templates. One of the benefits of LLBLGen here is the extent to which you can customize the entities in the designer, and then quickly update the code. LLBLGen can easily handle hundreds of entities. Well, we’ve just started to have fun, now let’s get creative.

Building Repository Classes with LLBLGen

Our repository implementation will consist of an IUnitOfWork interface in the “Model” project (which is NOT persistence aware), and a UnitOfWork class (in our case the EF context class) which will manage adds/edits/deletes of entities and coordinate updates against the persistence store. You can read about the Unit of Work pattern here. The implementation will also involve generating a base IRepository interface and a set of I<entity>Repository interfaces for each entity within the “Model” project, and their implementations in the “Persistence” project. Let’s generate a folder within each of these projects called “RepositoryClasses”. Building out code generation templates in LLBLGen involves a couple things:

Creating a templatebindings file that represents a virtual bundle of all our custom templates, with each template’s physical mapping to disk
Creating the templates themselves that generate the code.
Creating a preset file that houses the execution instructions of which templates to fire, in which order, and the parameters to pass to the templates. This includes instructions to create folders to contain the classes and add those folders to the Model and Persistence projects.

Upon re-generating the code, the visual studio solution looks as follows. Notice the new directories and repository classes.

Generating the database

To build the database, I’ll use the “Project > Generate Database Schema Create Script (DDL SQL)…” menus, and run the SQL output in SQL Management studio. After creating the database schema, in the LLBLGen Pro designer I will lync the database schema to the model we have created so that LLBLGen can keep them in sync. Built-in to the templates, I have added code that will generate methods in the IUnitOfWork interface and it’s implementation to call the stored procedures in the database, using the connection information belonging to the EF context. We will now create some stored procedures that use CTE recursion to query for which users, groups, roles are members of each other. The CTE query to retrieve members of a principal for example looks like this:

Code Snippet

CREATE PROCEDURE [dbo].[ChildPrincipalMemberships]
  @parentId int,
  @childType int,
  @recurse bit
AS 
  SET NOCOUNT ON
    IF (@recurse = 1)
    BEGIN
        ;WITH child_memberships_CTE (ParentPrincipalId, ChildPrincipalId, Level) AS
        (
             -- Anchor cte definition
             SELECT pm.ParentPrincipalId, pm.ChildPrincipalId, 0 AS Level
             FROM PrincipalMembers AS pm
             WHERE pm.ParentPrincipalId = @parentId
             UNION ALL
             -- Recursive cte definition
             SELECT pm.ParentPrincipalId, pm.ChildPrincipalId, cte.Level + 1
             FROM PrincipalMembers AS pm
             INNER JOIN child_memberships_CTE AS cte ON pm.ParentPrincipalId = cte.ChildPrincipalId
        )
        SELECT d.Id, Principal.Name, Principal.Description, d.Email, d.FullName, d.Birthdate, child_memberships_CTE.Level, 1 as MemberType
        FROM [User] as d
            INNER JOIN Principal ON d.Id = dbo.Principal.Id
            INNER JOIN child_memberships_CTE ON Principal.Id = child_memberships_CTE.ChildPrincipalId
        WHERE (@childType = 1 OR @childType = 0)
        UNION ALL
        SELECT d.Id, Principal.Name, Principal.Description, NULL as Email, NULL as FullName, NULL as Birthdate, child_memberships_CTE.Level, 2 as MemberType
        FROM [Group] as d
            INNER JOIN Principal ON d.Id = dbo.Principal.Id
            INNER JOIN child_memberships_CTE ON Principal.Id = child_memberships_CTE.ChildPrincipalId
        WHERE (@childType = 2 OR @childType = 0)
        UNION ALL
        SELECT d.Id, Principal.Name, Principal.Description, NULL as Email, NULL as FullName, NULL as Birthdate, child_memberships_CTE.Level, 3 as MemberType
        FROM [Role] as d
            INNER JOIN Principal ON d.Id = dbo.Principal.Id
            INNER JOIN child_memberships_CTE ON Principal.Id = child_memberships_CTE.ChildPrincipalId
        WHERE (@childType = 3 OR @childType = 0)
    END
    ELSE
    BEGIN
        SELECT d.Id, Principal.Name, Principal.Description, d.Email, d.FullName, d.Birthdate, 0 as Level, 1 as MemberType
        FROM [User] as d
            INNER JOIN Principal ON d.Id = dbo.Principal.Id
            INNER JOIN PrincipalMembers ON Principal.Id = PrincipalMembers.ChildPrincipalId
        WHERE PrincipalMembers.ParentPrincipalId = @parentId AND (@childType = 1 OR @childType = 0)
        UNION ALL
        SELECT d.Id, Principal.Name, Principal.Description, NULL as Email, NULL as FullName, NULL as Birthdate, 0 as Level, 2 as MemberType
        FROM [Group] as d
            INNER JOIN Principal ON d.Id = dbo.Principal.Id
            INNER JOIN PrincipalMembers ON Principal.Id = PrincipalMembers.ChildPrincipalId
        WHERE PrincipalMembers.ParentPrincipalId = @parentId AND (@childType = 2 OR @childType = 0)
        UNION ALL
        SELECT d.Id, Principal.Name, Principal.Description, NULL as Email, NULL as FullName, NULL as Birthdate, 0 as Level, 3 as MemberType
        FROM [Role] as d
            INNER JOIN Principal ON d.Id = dbo.Principal.Id
            INNER JOIN PrincipalMembers ON Principal.Id = PrincipalMembers.ChildPrincipalId
        WHERE PrincipalMembers.ParentPrincipalId = @parentId AND (@childType = 3 OR @childType = 0)
    END 

Now, we can just refresh the model in the LLBLGen designer and ask it to create calling methods for these new stored procedures. After re-generating the code, the calling methods now appear as part of our unit of work.

Code Snippet

public interface IUnitOfWork : ISql, IDisposable
{
    void Commit();
    // Incorporate stored procedure calls in the unit of work (or create a separate interface for these custom actions)
    ///<summary>Calls the stored procedure '[dbo].[ChildPrincipalMemberships]' and returns its results in a single DataSet</summary>
    ///<param name="parentId">Parameter mapped onto the stored procedure parameter '@parentId'</param>
    ///<param name="childType">Parameter mapped onto the stored procedure parameter '@childType'</param>
    ///<param name="recurse">Parameter mapped onto the stored procedure parameter '@recurse'</param>
    ///<returns>A single DataSet with all result data returned by the called stored procedure</returns>
    DataSet GetChildPrincipalMembershipsResults(System.Int32 parentId, System.Int32 childType, System.Boolean recurse);
    ///<summary>Calls the stored procedure '[dbo].[ParentPrincipalMemberships]' and returns its results in a single DataSet</summary>
    ///<param name="childId">Parameter mapped onto the stored procedure parameter '@childId'</param>
    ///<param name="parentType">Parameter mapped onto the stored procedure parameter '@parentType'</param>
    ///<param name="recurse">Parameter mapped onto the stored procedure parameter '@recurse'</param>
    ///<returns>A single DataSet with all result data returned by the called stored procedure</returns>
    DataSet GetParentPrincipalMembershipsResults(System.Int32 childId, System.Int32 parentType, System.Boolean recurse);
    // Not a real big fan of this method, but need it for EF support
    IObjectSet<TEntity> CreateSet<TEntity>() where TEntity : class;
}

Using the Code

We are ready to use the code. I’ve attached a console project to the download that shows how to use the generated code. The console project in the download provides samples for the following:

Using the repositories with an IoC framework (I created a real simple IoC implementation for the purpose of this demo in the download, I would use one of the popular DI containers most likely in a real implementation) - so don't gag when you see IoC.Resolve<IABC> below.
Deleting entities specifically and using predicates, including Cascade deletes.
Creating new entities (with 1 to many and many to many associated entities) as part of the same transaction.
Auto-incrementing behavior upon UnitOfWork commit.
Eager loading of related entities (both with Paths, and Lambdas).
Running SQL directly against the underlying persistence store.
Executing statically typed stored procedures.

Code Snippet

class Program
{
    static void Main(string[] args)
    {
        IoC.Initialize();
        using (var uow = IoC.Resolve<IUnitOfWork>())
        {
            var preferenceRepository = IoC.Resolve<IPreferenceRepository>(uow);
            var groupRepository = IoC.Resolve<IGroupRepository>(uow);
            var roleRepository = IoC.Resolve<IRoleRepository>(uow);
            var userRepository = IoC.Resolve<IUserRepository>(uow);
            var memberRepository = IoC.Resolve<IPrincipalMemberRepository>(uow);
            //------------------------------------------------------------------------------------------------
            // TEST 1 -  Delete all records
            //------------------------------------------------------------------------------------------------
            // When deleting, to enable Cascade delete, you must load related entities into the context (ARGHH...)
            // OR, just modify your SQL/DATABASE to implement ON CASCADE DELETE
            //------------------------------------------------------------------------------------------------
            preferenceRepository.DeleteMulti(preferenceRepository.FetchAll(new[] { Preference.Includes.CascadeDeleteDependencies }));
            // OR, use the DeleteMulti with predicate argument, which will automatically delete dependencies based on the
            // UseCascadeDelete flag in LLBLGen designer (equivalent to method above)
            groupRepository.DeleteMulti(x => x.Id > 0);
            roleRepository.DeleteMulti(x => x.Id > 0);
            userRepository.DeleteMulti(x => x.Id > 0);
            memberRepository.DeleteMulti(x => x.Id > 0);
            //------------------------------------------------------------------------------------------------
            // TEST 2 - Create some records
            //------------------------------------------------------------------------------------------------
            // Create a couple records, along with some 1 to many, and many to many related items all at once
            //------------------------------------------------------------------------------------------------
            var user1 = User.CreateUser("someone", "someone@somewhere.com", "", "Some One", DateTime.Now.AddYears(-20));
            user1.Settings.Add(new Setting { Name = "Doctor", Value = "Good" }); // add 1 to many related item
            user1.Preferences.Add(new Preference { Name = "EyeColor", Value = "Blue" }); // add many to many related item
            var user2 = User.CreateUser("noone", "noone@nowhere.com", "", "No One", DateTime.Now.AddYears(-40));
            user2.Settings.Add(new Setting { Name = "Doctor", Value = "Evil" }); // add 1 to many related item
            user2.Preferences.Add(new Preference { Name = "EyeColor", Value = "Green" }); // add many to many related item
            userRepository.CreateMulti(new[] {user1, user2});
            uow.Commit(); // Commit all this to the database
            Console.WriteLine("PASS - Delete and Create");
            //------------------------------------------------------------------------------------------------
            // TEST 3 - AutoIncrementing behavior test
            //------------------------------------------------------------------------------------------------
            // Check that the AutoIncrementing IDs for objects and related items were all created and are now active
            // on committed objects
            //------------------------------------------------------------------------------------------------
            if (user1.Id > 0 &&
                user1.Settings.First().Id > 0 &&
                user1.Preferences.First().Id > 0 &&
                userRepository.Count() > 0)
            {
                Console.WriteLine("PASS - AutoIncrementing IDs");
            }
            else
            {
                Console.WriteLine("FAIL - AutoIncrementing IDs");
            }
            //------------------------------------------------------------------------------------------------
            // TEST 4 - Inside UOW prefetch test of committed related item
            //------------------------------------------------------------------------------------------------
            // A related item that was committed within a Unit of Work should not have to be eager loaded
            // on subsequent fetches, as it should live in the context
            //------------------------------------------------------------------------------------------------
            if (userRepository.FetchUsingEmail("someone@somewhere.com").Name == "someone" &&
                userRepository.FetchUsingName("someone").Email == "someone@somewhere.com" &&
                userRepository.FetchUsingName("someone").Settings.Count == 1 &&
                userRepository.FetchUsingName("someone", new[] { User.Includes.Paths.Settings }).Settings.Count == 1)
            {
                Console.WriteLine("PASS - Inside UOW eager loading behavior after commit");
            }
            else
            {
                Console.WriteLine("FAIL - Inside UOW eager loading behavior after commit");
            }
        }
        //------------------------------------------------------------------------------------------------
        // TEST 5 - Eager loading tests
        //------------------------------------------------------------------------------------------------
        // 5a - No eager loading
        //------------------------------------------------------------------------------------------------
        using (var uow = IoC.Resolve<IUnitOfWork>())
        {
            // Create a user
            var userRepository = IoC.Resolve<IUserRepository>(uow);
            var test5a = userRepository.FetchUsingName("someone").Settings.Count == 0;
            Console.WriteLine((test5a ? "PASS" : "FAIL") + " - No eager loading test");
        }
        //------------------------------------------------------------------------------------------------
        // 5b - Eager loading with lambdas
        //------------------------------------------------------------------------------------------------
        using (var uow = IoC.Resolve<IUnitOfWork>())
        {
            // Create a user
            var userRepository = IoC.Resolve<IUserRepository>(uow);
            var users = userRepository.FetchUsingName("someone", new[] { User.Includes.OneToMany.Settings, User.Includes.ManyToMany.Preferences });
            var test5b = users.Settings.Count == 1 && users.Preferences.Count == 1;
            Console.WriteLine((test5b ? "PASS" : "FAIL") + " - Eager loading with lambdas");
        }
        //------------------------------------------------------------------------------------------------
        // 5c - Eager loading with paths
        //------------------------------------------------------------------------------------------------
        using (var uow = IoC.Resolve<IUnitOfWork>())
        {
            // Create a user
            var userRepository = IoC.Resolve<IUserRepository>(uow);
            var users = userRepository.FetchUsingName("someone", new object[] { User.Includes.Paths.Settings, User.Includes.Paths.Preferences, User.Includes.Paths.Preferences_Principals, User.Includes.Paths.Settings_Principal });
            var test5c = users.Settings.Count == 1 && users.Preferences.Count == 1 && users.Preferences.First().Principals.Count > 0;
            Console.WriteLine((test5c ? "PASS" : "FAIL") + " - Eager loading with paths");
            var test5a = userRepository.FetchUsingName("someone").Settings.Count == 0;
            Console.WriteLine((test5a ? "FAIL" : "PASS") + " - No eager loading test (for already fetched item with eager load)");
        }
        //------------------------------------------------------------------------------------------------
        // TEST 6 - Run some SQL directly against the database
        //------------------------------------------------------------------------------------------------
        using (var uow = IoC.Resolve<IUnitOfWork>())
        {
            var count = (int)uow.ExecuteScalarQuery(CommandType.Text, "SELECT count(Id) FROM [User]");
            Console.WriteLine((count > 0 ? "PASS" : "FAIL") + " - SQL test");
        }
        //------------------------------------------------------------------------------------------------
        // TEST 7 - Execute statically typed stored procedure call
        //------------------------------------------------------------------------------------------------
        using (var uow = IoC.Resolve<IUnitOfWork>())
        {
            var groupRepository = IoC.Resolve<IGroupRepository>(uow);
            var roleRepository = IoC.Resolve<IRoleRepository>(uow);
            var userRepository = IoC.Resolve<IUserRepository>(uow);
            var membershipRepository = IoC.Resolve<IPrincipalMemberRepository>(uow);
            var groups = new List<Group>();
            var roles = new List<Role>();
            var users = new List<User>();
            for (var i = 1; i < 11; i++)
            {
                groups.Add(Group.CreateGroup("Sample Group " + i));
                roles.Add(Role.CreateRole("Sample Role " + i));
                users.Add(User.CreateUser("user" + i, "user" + i + "@company.com", "", "User " + i, null));
            }
            groupRepository.CreateMulti(groups);
            roleRepository.CreateMulti(roles);
            userRepository.CreateMulti(users);
            // commit so that IDs are updated
            uow.Commit();
            // Create a bunch of hierarchical memberships
            var principalMembers = new[]
                                       {
                                           PrincipalMember.CreatePrincipalMember(users[0].Id, groups[0].Id),
                                           PrincipalMember.CreatePrincipalMember(users[0].Id, groups[1].Id),
                                           PrincipalMember.CreatePrincipalMember(users[0].Id, groups[2].Id),
                                           PrincipalMember.CreatePrincipalMember(users[0].Id, groups[3].Id),
                                           PrincipalMember.CreatePrincipalMember(users[0].Id, groups[4].Id),
                                           PrincipalMember.CreatePrincipalMember(users[0].Id, groups[5].Id),
                                           PrincipalMember.CreatePrincipalMember(users[0].Id, groups[6].Id),
                                           PrincipalMember.CreatePrincipalMember(users[1].Id, groups[0].Id),
                                           PrincipalMember.CreatePrincipalMember(users[1].Id, groups[2].Id),
                                           PrincipalMember.CreatePrincipalMember(users[1].Id, groups[4].Id),
                                           PrincipalMember.CreatePrincipalMember(users[1].Id, groups[8].Id),
                                           PrincipalMember.CreatePrincipalMember(users[1].Id, groups[9].Id),
                                           PrincipalMember.CreatePrincipalMember(groups[1].Id, roles[0].Id),
                                           PrincipalMember.CreatePrincipalMember(groups[1].Id, roles[2].Id),
                                           PrincipalMember.CreatePrincipalMember(groups[2].Id, roles[2].Id),
                                           PrincipalMember.CreatePrincipalMember(groups[2].Id, roles[3].Id),
                                           PrincipalMember.CreatePrincipalMember(users[1].Id, roles[4].Id),
                                           PrincipalMember.CreatePrincipalMember(users[1].Id, roles[8].Id),
                                           PrincipalMember.CreatePrincipalMember(roles[6].Id, roles[5].Id),
                                           PrincipalMember.CreatePrincipalMember(users[7].Id, roles[5].Id),
                                           PrincipalMember.CreatePrincipalMember(users[8].Id, roles[6].Id)
                                       };
            membershipRepository.CreateMulti(principalMembers);
            uow.Commit();
            // Extension method sample: role 5 contains user 8 and role 6, role 6 contains user 7
            var role5 = roles[5];
            if (role5.GetMembers(PrincipalType.Any, false).Count() == 2 && // no recursion, any type of principal
                role5.GetMembers(PrincipalType.User, false).Count() == 1 && // no recursion, only users
                role5.GetMembers(PrincipalType.User, true).Count() == 2) // recurse, only users
            {
                Console.WriteLine("PASS - Called typed stored procedure with parameters");
            }
            else
            {
                Console.WriteLine("FAIL - Problem with typed stored procedure call");
            }
        }
        Console.WriteLine("ALL DONE");
        Console.Read();
    }
}

Extending the code

One creative way of using the stored procedures is to use extension methods on the entities. See for example the following extension method on the principal entity that uses the stored procedure defined earlier to retrieve it’s members:

Code Snippet

public enum PrincipalType
{
    Any = 0,
    User = 1,
    Group = 2,
    Role = 3
}
public static class ExtensionMethodSamples
{
    public static IEnumerable<Principal> GetMembers(this Principal principal, PrincipalType constraintType, bool recurse)
    {
        var principals = new List<Principal>();
        using (var uow = IoC.Resolve<IUnitOfWork>())
        {
            var dt = uow.GetChildPrincipalMembershipsResults(principal.Id, (int)constraintType, recurse).Tables[0];
            foreach (var row in dt.Rows.Cast<DataRow>())
            {
                switch(Convert.ToInt32(row["MemberType"]))
                {
                    case 1:
                        principals.Add(User.MapUser(
                            row.Field<int>(0),
                            row.Field<string>(1),
                            row.Field<string>(3),
                            row.Field<string>(2),
                            row.Field<string>(4),
                            row.Field<DateTime?>(5)));
                        break;
                    case 2:
                        principals.Add(Group.MapGroup(
                               row.Field<int>(0),
                               row.Field<string>(1),
                               row.Field<string>(2)));
                        break;
                    case 3:
                        principals.Add(Role.MapRole(
                               row.Field<int>(0),
                               row.Field<string>(1),
                               row.Field<string>(2)));
                        break;
                    default:
                        break;
                }
            }
        }
        return principals;
    }
    public static IEnumerable<Principal> GetMemberships(this Principal principal, PrincipalType constraintType, bool recurse)
    {
        var principals = new List<Principal>();
        using (var uow = IoC.Resolve<IUnitOfWork>())
        {
            var dt = uow.GetParentPrincipalMembershipsResults(principal.Id, (int)constraintType, recurse).Tables[0];
            foreach (var row in dt.Rows.Cast<DataRow>())
            {
                switch (Convert.ToInt32(row["MemberType"]))
                {
                    case 1:
                        principals.Add(User.MapUser(
                            row.Field<int>(0),
                            row.Field<string>(1),
                            row.Field<string>(3),
                            row.Field<string>(2),
                            row.Field<string>(4),
                            row.Field<DateTime?>(5)));
                        break;
                    case 2:
                        principals.Add(Group.MapGroup(
                               row.Field<int>(0),
                               row.Field<string>(1),
                               row.Field<string>(2)));
                        break;
                    case 3:
                        principals.Add(Role.MapRole(
                               row.Field<int>(0),
                               row.Field<string>(1),
                               row.Field<string>(2)));
                        break;
                    default:
                        break;
                }
            }
        }
        return principals;
    }
}

You could use extension methods to add / remove members as well, or just use the IPrincipalMemberRepository class to create the record directly.

Conclusion

I hope this has been entertaining and that you have found this post useful. I’d love to hear your comments. The entire set of templates, the LLBLGen project file, and the visual studio solution are available for download below. Use the supplied download as you wish, the intent for it is to help you have fun with LLBLGen, the Entity Framework, and the Repository Pattern.

Download Code

By Matt C. | Posted in .NET | Tagged .net, entity framework, llblgen | Comments (9)

Hello world!

September 9, 2011 – 9:36 pm

Hello world!

My name is Matt. I love lots of things, and one of these is coding. To kick off this blog, here are some of my favorite things:

If you know me, shoot me an email and give me some suggestions on things to blog about

See you soon.

By Matt C. | Posted in .NET | Comments (5)

Background

The End-Goal

Getting Started

API Development

See it in Action

Conclusion

Requirements

The Console Application

Scripting the Database

Final Output

The Tests

The Results

Download

Conclusion

In a separate study (updated on 4/20/2012)

Why Autofac?

Autofac vs. SharePoint Service Locator

Wiring Autofac up

Using Autofac within Application Pages, Web Parts, User Controls, Event Receivers

Using Autofac on the Server

General Comments

How I discovered Quartz.NET

Putting Quartz.NET to good use

Rediscovering the project

Installing Quartz.NET as a Windows Service

Running the Sequential Job and some sample plugins

Creating a plugin and adding it to a job

Summary

Learning how you learn

In Conclusion

PMP Study Sheet and Learning Grid System

Legend

Formulas

The Code

The Extension

The Result

Introduction

Repository Pattern (NIH)

A hot debate

Case Study

Building the model

Use-case

The Model

Generating code

Building Repository Classes with LLBLGen

Generating the database

Using the Code

Extending the code

Conclusion

Pages

Categories

Archives

Search

RSS Feeds

Meta