Having fun with code
  • Blog
  • Favorites
    • Favorite Books
    • Favorite Libraries
    • Favorite Tools
  • Utilities
    • SP 2010 Colors
  • Contact
  • Home /
  • .NET /
  • Extending Quartz.NET – A sequential plugin execution engine

Extending Quartz.NET – A sequential plugin execution engine

January 27, 2012 / Matt C. / .NET
Three or four years ago I stumbled on the book "Quartz Job Scheduling Framework: Building Open Source Enterprise Applications" by "Chuck Cavaness", while browsing the technology section of a Borders bookstore. I remember it peeking my interest. A year or two later, I heard about an open-source version of this engine implemented in .NET called Quartz.NET. This post is for those that want to learn a little bit about Quartz.NET, and/or need a tool to easily implement a cron-job capability, with cron-job scheduling that can be modified at runtime. In fact, you don’t need to know anything about Quartz.NET to get up and running with what's in this post. This post is for those that want to learn a little bit about Quartz.NET, and/or need a tool to easily implement a cron-job capability, with cron-job scheduling that can be modified at runtime. In fact, you don’t need to know anything about Quartz.NET to get up and running with what’s below.

How I discovered Quartz.NET

Three or four years ago I stumbled on the book "Quartz Job Scheduling Framework: Building Open Source Enterprise Applications" by "Chuck Cavaness" while browsing the technology section of a Borders bookstore. I remember it peeking my interest. A year or two later, I heard about an open-source version of this engine implemented in .NET: "Quartz.NET - Enterprise Job Scheduler for .NET Platform". I have since then used this framework on many projects, including some very large global ecommerce projects handling hundreds of thousands of transactions. Needless to say, if you need a lightweight job scheduling framework that really packs a punch, and you have a windows server available, this thing might be perfect for you. If you’re not into windows, use the original version “Quartz Enterprise Job Scheduler”.

Putting Quartz.NET to good use

Three years ago, as my first little pet project on Quartz.NET, I decided to build a little stock analysis application that would:
  • keep a local database (at first, just a bunch of csv files, then later a relational database) updated with all the ticker symbols from the NYSE and NASDAQ
  • download a reasonable amount of level II intra-day price information for each ticker over the course of each trading day (using a couple feeds, but primarily the free Yahoo stock price feed)
  • allow me to write plugins that I could easily drop into a folder, to:
    • download company pro-forma statements and earnings reports
    • parse html from daily and weekly advice columns
    • do analytics on the data
The core framework took only a week-end to build with Quartz.NET (the plugins took a little longer :-)). With a single plugin that would email me alerts, some custom reports, and just writing custom queries and T-SQL over the database, I had myself a nice little useful tool.

Rediscovering the project

I just recently found this little project from several years ago on a backup drive, and decided to take a look at it again. Noticing that Quartz.NET has gone through a couple evolutions, I thought I would upgrade it to the latest version, and put it out there for anybody that might be interested (minus the stock trading piece, maybe save that for another blog post), before it goes back into hibernation. I am leaving the core logic alone, it's useful as is. Run with it if you like it. If I were to fully redevelop this, I would make better use of the Quartz.NET trigger and listener features, but those are not necessary to build a pretty nifty capability, as I think you'll see from this. Let me just highlight the core capabilities I was originally out to satisfy from a cron-engine, for this personal project described above:
  • Keep management of the job engine to a minimum
  • Execute a primary job at specific intervals (every X number of seconds for example, or minutes, or hours)
  • The primary job loops sequentially through a number of other jobs that have their own timing parameters, and whose timing parameters can be changed at run-time
  • Each job executes a set of plugins sequentially
  • Key/value configuration settings can be specified globally, per job, and per plugin. Lower tiers can override settings inherited from a higher tier.
  • Plugins must be able to pass data between themselves, so that by chaining plugins one after another, one can use information established during a prior plugin's execution in it's own processing logic
[button link="http://sdrv.ms/ZHyf0h" color="primary" target="_blank" size="large" title="Code Download" icon_before="download"]Code Download[/button]

Installing Quartz.NET as a Windows Service

Installing Quartz.NET is a breeze. Everything you need is in the zip file attached to this post. But you can also get the Quartz files directly from sourceforge at: http://sourceforge.net/projects/quartznet/files/quartznet/. I’m using version Quartz.NET 2.0 beta 2 in this post. Step 1 – Extract the contents of the attached zip file into a directory of your choice - I'm using “C:\Quartz\”, then proceed to step 3. If you want to install from the sourceforge download, proceed to step 2. Step 2 – Installing from source. - Extract the contents from the “Quartz.NET-2.0-beta2.zip” file into a directory. I am extracting them into “C:\Quartz\src”. - Open a VS.NET command prompt, and run the following commands: image - Check to make sure the log files msbuild.output.log and msbuild.server.output.log both end without any warnings or errors. image - Create 2 batch files to easily install and uninstall the Quartz server as a windows service: image - Compile the source of the project “Quartz.Server.Extensions.csproj” from the zip file attached to this post or just copy the “Quartz.Server.Extensions.dll” to your Quartz directory - Overwrite the existing quartz_jobs.xml (the new one has the preconfigured Sequential job configuration in it) and quartz.server.exe.config file (this is optional, it just has some additional log4net settings in it) with those from the zip file Step 3 – Start the service Before you start, make sure that paths in the files quartz_jobs.xml, quartz.server.exe.config, install.bat, and uninstall.bat are correct. If you’re following along with this post, you don’t need to do anything. Go to a command prompt (run it as Administrator preferably) and run the install.bat file. To uninstall the service, just run the uninstall.bat file. image You’ll now see that your service is running in your services console. image A “logs” directory was created and you can look inside the “quartz.log” file to track what’s going on with your server. Your log file will look like this below. Notice the “ERROR” line, we’ll talk about that next. image

Running the Sequential Job and some sample plugins

If you dig into the quartz_jobs.xml file, you’ll see a configuration section that looks like this. This is the job that runs all the plugins.
<job>
  <name>SequentialJobPool</name>
  <group>SequentialJobPoolGroup</group>
  <description>A job that runs a sequence of plugins on a regular basis</description>
  <job-type>Quartz.Server.Extensions.SequentialPluginJob, Quartz.Server.Extensions</job-type>
  <durable>true</durable>
  <recover>false</recover>
  <job-data-map>
    <entry>
      <key>PluginsDirectory</key>
      <value>c:\quartz\plugins</value>
    </entry>
    <entry>
      <key>PluginsConfigFile</key>
      <value>plugins-quartz-config.xml</value>
    </entry>
    <entry>
      <key>PluginsScheduleFile</key>
      <value>plugins-quartz-schedule.xml</value>
    </entry>
  </job-data-map>
</job>
<trigger>
  <simple>
    <name>SequentialJobPool-Trigger</name>
    <group>SequentialJobPool-TriggerGroup</group>
    <description>Fires the job to check for any plugin activity needed</description>
    <job-name>SequentialJobPool</job-name>
    <job-group>SequentialJobPoolGroup</job-group>
    <misfire-instruction>SmartPolicy</misfire-instruction>
    <repeat-count>-1</repeat-count>
    <repeat-interval>30000</repeat-interval>
  </simple>
</trigger>
You’ll notice a reference to a plugins directory. Plugin DLLs can be either placed in the plugins directory, or in the root directory, but a valid plugins directory must be configured. Let’s create the plugins directory, and restart the service. image Now, take a look at the log file: image And in the plugins directory there are 2 files: plugins-quartz-config.xml and plugins-quartz-schedule.xml …
<?xml version="1.0" encoding="utf-8"?>
<config>
  <!– Main configuration file for the sequential plugin job used within Quartz –>
  <jobs count="1">
    <job id="1">
      <plugins count="1">
        <plugin type="Quartz.Server.Extensions.NullPlugin, Quartz.Server.Extensions">
          <settings count="2">
            <setting key="sampleSetting1" value="sampleSettingValue1" />
            <setting key="sampleSetting2" value="sampleSettingValue2" />
          </settings>
        </plugin>
      </plugins>
      <settings count="2">
        <setting key="sampleSetting1" value="sampleSettingValue1" />
        <setting key="sampleSetting2" value="sampleSettingValue2" />
      </settings>
    </job>
  </jobs>
  <settings count="2">
    <setting key="sampleSetting1" value="sampleSettingValue1" />
    <setting key="sampleSetting2" value="sampleSettingValue2" />
  </settings>
</config>
The first xml file - plugins-quartz-config.xml - describes the plugins that are executed. Each job (Sequential Job, not to be confused with a Quartz Job which is inside the quartz_jobs.xml) runs sequentially, and each plugin within the job runs sequentially. Settings can override each other, from top down, starting at the jobs level (across all jobs), job level, and plugin level. These settings are all part of the context parameter passed into the plugin’s “Execute” function.
<?xml version="1.0" encoding="utf-8"?>
<schedules>
  <!– The incrementType can be: Milliseconds, Seconds, Minutes, Hours, Days, None, Once –>
  <schedule jobId="1" increment="10000" incrementType="Milliseconds" lastRun="2012-01-27T00:46:34" nextRun="2012-01-27T00:46:44" />
</schedules>
The 2nd file - plugins-quartz-schedule.xml – describes the schedule for the job. These XML files are created as a sample when you run the SequentialJob for the first time. The nice thing about this file, is that it’s constantly changing. To turn off a job, just set the “nextRun” value of the jobId to a future date far into the future. Change the increment and incrementType at any time to change the schedule for this job.

Creating a plugin and adding it to a job

There’s only 1 simple interface to extend, in the Quartz.Server.Extensions assembly:
public interface IPlugin
{
    void Execute(PluginContext pluginContext);
}
The PluginContext class has 2 public properties on it:
/// <summary>
/// These are the settings from the plugins-quartz-config.xml file.
/// <remarks>
/// If a setting key at the plugin level exists at a higher level, it will override the higher level key.
/// </remarks>
/// </summary>
public ReadOnlyDictionary<string, string> Settings { get; internal set; }
/// <summary>
/// This dictionary of data can be altered within your plugin. You can chain plugins together
/// and create business logic based on values inside the Data dictionary.
/// </summary>
public IDictionary<string, object> Data { get; private set; }
We'll create a sample plugin, as follows.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Common.Logging;
 
namespace Quartz.Server.Extensions.Plugins
{
    public class SamplePlugin : IPlugin
    {
        private static readonly ILog log = LogManager.GetLogger(typeof(SamplePlugin));
 
        public void Execute(PluginContext pluginContext)
        {
            // get the settings & data
            var settings = pluginContext.Settings;
            var data = pluginContext.Data;
 
            // let's do a bunch of logging
            log.Info("Start " + this.GetType().Name);
 
            // log the settings
            int settingsCount = settings == null ? 0 : settings.Count();
            log.Info("With " + settingsCount + " settings:");
            if (settingsCount > 0)
            {
                foreach (string key in settings.Keys)
                    log.Info(string.Format(" {0}: {1}", key, settings[key]));
            }
 
            // log data dictionary values
            int dataCount = data == null ? 0 : data.Count();
            log.Info("With " + dataCount + " data records:");
            if (dataCount > 0)
            {
                foreach (string key in data.Keys)
                    log.Info(string.Format(" {0}: {1}", key, data[key]));
            }
 
            // change and/or add some data dictionary values
            data["Status"] = "Success";
            data["StatusDate"] = DateTime.Now;
 
            log.Info("End " + this.GetType().Name);
        }
    }
}
This sample is in the Quartz.Server.Extensions.Plugins.dll file. Let’s just move that file into the plugins directory now, and register it in the plugins-quartz-config.xml file.
<?xml version="1.0" encoding="utf-8"?>
<config>
  <!– Main configuration file for the sequential plugin job used within Quartz –>
  <jobs count="1">
    <job id="1">
      <plugins count="2">
        <plugin assembly="Quartz.Server.Extensions.Plugins" type="Quartz.Server.Extensions.Plugins.SamplePlugin">
          <settings count="1">
            <setting key="sampleSetting2" value="SamplePlugin setting value 2" />
          </settings>
        </plugin>
        <plugin type="Quartz.Server.Extensions.NullPlugin, Quartz.Server.Extensions">
          <settings count="1">
            <setting key="sampleSetting1" value="NullPlugin setting value 1" />
          </settings>
        </plugin>
      </plugins>
      <settings count="2">
        <setting key="sampleSetting1" value="sampleSettingValue1" />
        <setting key="sampleSetting2" value="sampleSettingValue2" />
      </settings>
    </job>
  </jobs>
  <settings count="2">
    <setting key="sampleSetting1" value="sampleSettingValue1" />
    <setting key="sampleSetting2" value="sampleSettingValue2" />
  </settings>
</config>
As soon as you alter this file, provided the DLL file was moved into the plugins directory, the job will pick it up and execute it in sequence. See the log output here: image

Summary

This was an exercise in creativity, leveraging Quartz.NET, and a single Quartz.NET job to create a little sequential job plugin framework that one can alter at runtime by adding/removing plugins and changing job schedules without any pain whatsoever. Adding plugins using the method above is a breeze, extend an interface, drop the dll into the folder, and add a <plugin> tag to an xml file, and voila. I have clients that have been running this code (possibly slightly tailored to their needs Winking smile ) for 2 to 3 years now to chain all kinds of events together like parsing text, doing mass PDF conversions of files sent in through SMTP, sending out email newsletters, and synchronizing user profiles between Active Directory and various CMS systems. It keeps cranking thanks to Quartz.NET. A variation of this was also used to process orders in a large ecommerce site, and send the orders to shipping companies and manufacturing plants around the world. I’m sure there is MUCH to improve on this; so if you do experiment with this, and improve on it, please do let me know, and I’d be glad to post your updates here. [button link="http://sdrv.ms/ZHyf0h" color="primary" target="_blank" size="large" title="Code Download" icon_before="download"]Code Download[/button] Thanks for reading as always. .net, C#, job scheduling engine, plugins, Quartz.NET

4 comments on “Extending Quartz.NET – A sequential plugin execution engine”

  1. george says:
    October 15, 2012 at 6:20 am

    Would morning matt, i’m planning to perform, very heavy operations and i want to use quarz. In my database i have a table called pendingJobs. I woluld like to use quartz, and have one job that only read lines from database and generate new jobs like:

    • Matt C. says:
      October 15, 2012 at 9:49 am

      Sorry george, looks like your comment was cut off.

  2. SohelElite says:
    May 2, 2012 at 8:42 am

    Much better example to play with Quartz.NET in fact its simpler than the official website documentation and tutorial. Thanks Matt.

    • Matt C. says:
      May 2, 2012 at 9:57 pm

      Very cool. Glad it was useful.

Categories

  • .NET (20)
  • ASP.NET MVC (4)
  • JAMstack (2)
    • Headless CMS (2)
  • Miscellaneous (2)
  • Python (1)
  • REST (3)
  • SharePoint (8)
  • WordPress (1)

Tags

.net ado.net autofac binding C# chrome code generation command line console application csv dapper di entity framework integration ioc job scheduling engine json jsv learning llblgen load balancing micro-orm mycorp odata orm people editor people picker picker controls picker dialog plugins pmp python Quartz.NET rest saf service application servicestack sharepoint smo soap sql sqlalchemy tornado web server validation web api

Archives

  • May 2020 (2)
  • November 2013 (1)
  • June 2013 (1)
  • May 2013 (1)
  • March 2013 (1)
  • December 2012 (1)
  • November 2012 (1)
  • October 2012 (3)
  • September 2012 (2)
  • June 2012 (2)
  • May 2012 (1)
  • April 2012 (1)
  • February 2012 (2)
  • January 2012 (1)
  • December 2011 (2)
  • September 2011 (2)
(c) 2013 Having fun with code - "FunCoding" theme designed by mattjcowan using the Theme Blvd framework