How I discovered Quartz.NET
Three or four years ago I stumbled on the book "Quartz Job Scheduling Framework: Building Open Source Enterprise Applications" by "Chuck Cavaness" while browsing the technology section of a Borders bookstore. I remember it peeking my interest. A year or two later, I heard about an open-source version of this engine implemented in .NET: "Quartz.NET - Enterprise Job Scheduler for .NET Platform". I have since then used this framework on many projects, including some very large global ecommerce projects handling hundreds of thousands of transactions. Needless to say, if you need a lightweight job scheduling framework that really packs a punch, and you have a windows server available, this thing might be perfect for you. If you’re not into windows, use the original version “Quartz Enterprise Job Scheduler”.Putting Quartz.NET to good use
Three years ago, as my first little pet project on Quartz.NET, I decided to build a little stock analysis application that would:- keep a local database (at first, just a bunch of csv files, then later a relational database) updated with all the ticker symbols from the NYSE and NASDAQ
- download a reasonable amount of level II intra-day price information for each ticker over the course of each trading day (using a couple feeds, but primarily the free Yahoo stock price feed)
- allow me to write plugins that I could easily drop into a folder, to:
- download company pro-forma statements and earnings reports
- parse html from daily and weekly advice columns
- do analytics on the data
Rediscovering the project
I just recently found this little project from several years ago on a backup drive, and decided to take a look at it again. Noticing that Quartz.NET has gone through a couple evolutions, I thought I would upgrade it to the latest version, and put it out there for anybody that might be interested (minus the stock trading piece, maybe save that for another blog post), before it goes back into hibernation. I am leaving the core logic alone, it's useful as is. Run with it if you like it. If I were to fully redevelop this, I would make better use of the Quartz.NET trigger and listener features, but those are not necessary to build a pretty nifty capability, as I think you'll see from this. Let me just highlight the core capabilities I was originally out to satisfy from a cron-engine, for this personal project described above:- Keep management of the job engine to a minimum
- Execute a primary job at specific intervals (every X number of seconds for example, or minutes, or hours)
- The primary job loops sequentially through a number of other jobs that have their own timing parameters, and whose timing parameters can be changed at run-time
- Each job executes a set of plugins sequentially
- Key/value configuration settings can be specified globally, per job, and per plugin. Lower tiers can override settings inherited from a higher tier.
- Plugins must be able to pass data between themselves, so that by chaining plugins one after another, one can use information established during a prior plugin's execution in it's own processing logic
Installing Quartz.NET as a Windows Service
Installing Quartz.NET is a breeze. Everything you need is in the zip file attached to this post. But you can also get the Quartz files directly from sourceforge at: http://sourceforge.net/projects/quartznet/files/quartznet/. I’m using version Quartz.NET 2.0 beta 2 in this post. Step 1 – Extract the contents of the attached zip file into a directory of your choice - I'm using “C:\Quartz\”, then proceed to step 3. If you want to install from the sourceforge download, proceed to step 2. Step 2 – Installing from source. - Extract the contents from the “Quartz.NET-2.0-beta2.zip” file into a directory. I am extracting them into “C:\Quartz\src”. - Open a VS.NET command prompt, and run the following commands:
- Check to make sure the log files msbuild.output.log and msbuild.server.output.log both end without any warnings or errors.
- Create 2 batch files to easily install and uninstall the Quartz server as a windows service:
- Compile the source of the project “Quartz.Server.Extensions.csproj” from the zip file attached to this post
or just copy the “Quartz.Server.Extensions.dll” to your Quartz directory
- Overwrite the existing quartz_jobs.xml (the new one has the preconfigured Sequential job configuration in it) and quartz.server.exe.config file (this is optional, it just has some additional log4net settings in it) with those from the zip file
Step 3 – Start the service
Before you start, make sure that paths in the files quartz_jobs.xml, quartz.server.exe.config, install.bat, and uninstall.bat are correct. If you’re following along with this post, you don’t need to do anything.
Go to a command prompt (run it as Administrator preferably) and run the install.bat file. To uninstall the service, just run the uninstall.bat file.
You’ll now see that your service is running in your services console.
A “logs” directory was created and you can look inside the “quartz.log” file to track what’s going on with your server.
Your log file will look like this below. Notice the “ERROR” line, we’ll talk about that next.
Running the Sequential Job and some sample plugins
If you dig into the quartz_jobs.xml file, you’ll see a configuration section that looks like this. This is the job that runs all the plugins.<job>
<name>SequentialJobPool</name>
<group>SequentialJobPoolGroup</group>
<description>A job that runs a sequence of plugins on a regular basis</description>
<job-type>Quartz.Server.Extensions.SequentialPluginJob, Quartz.Server.Extensions</job-type>
<durable>true</durable>
<recover>false</recover>
<job-data-map>
<entry>
<key>PluginsDirectory</key>
<value>c:\quartz\plugins</value>
</entry>
<entry>
<key>PluginsConfigFile</key>
<value>plugins-quartz-config.xml</value>
</entry>
<entry>
<key>PluginsScheduleFile</key>
<value>plugins-quartz-schedule.xml</value>
</entry>
</job-data-map>
</job>
<trigger>
<simple>
<name>SequentialJobPool-Trigger</name>
<group>SequentialJobPool-TriggerGroup</group>
<description>Fires the job to check for any plugin activity needed</description>
<job-name>SequentialJobPool</job-name>
<job-group>SequentialJobPoolGroup</job-group>
<misfire-instruction>SmartPolicy</misfire-instruction>
<repeat-count>-1</repeat-count>
<repeat-interval>30000</repeat-interval>
</simple>
</trigger>
You’ll notice a reference to a plugins directory.
Plugin DLLs can be either placed in the plugins directory, or in the root directory, but a valid plugins directory must be configured.
Let’s create the plugins directory, and restart the service.
Now, take a look at the log file:
And in the plugins directory there are 2 files: plugins-quartz-config.xml and plugins-quartz-schedule.xml …
<?xml version="1.0" encoding="utf-8"?>
<config>
<!– Main configuration file for the sequential plugin job used within Quartz –>
<jobs count="1">
<job id="1">
<plugins count="1">
<plugin type="Quartz.Server.Extensions.NullPlugin, Quartz.Server.Extensions">
<settings count="2">
<setting key="sampleSetting1" value="sampleSettingValue1" />
<setting key="sampleSetting2" value="sampleSettingValue2" />
</settings>
</plugin>
</plugins>
<settings count="2">
<setting key="sampleSetting1" value="sampleSettingValue1" />
<setting key="sampleSetting2" value="sampleSettingValue2" />
</settings>
</job>
</jobs>
<settings count="2">
<setting key="sampleSetting1" value="sampleSettingValue1" />
<setting key="sampleSetting2" value="sampleSettingValue2" />
</settings>
</config>
The first xml file - plugins-quartz-config.xml - describes the plugins that are executed. Each job (Sequential Job, not to be confused with a Quartz Job which is inside the quartz_jobs.xml) runs sequentially, and each plugin within the job runs sequentially.
Settings can override each other, from top down, starting at the jobs level (across all jobs), job level, and plugin level. These settings are all part of the context parameter passed into the plugin’s “Execute” function.
<?xml version="1.0" encoding="utf-8"?> <schedules> <!– The incrementType can be: Milliseconds, Seconds, Minutes, Hours, Days, None, Once –> <schedule jobId="1" increment="10000" incrementType="Milliseconds" lastRun="2012-01-27T00:46:34" nextRun="2012-01-27T00:46:44" /> </schedules>The 2nd file - plugins-quartz-schedule.xml – describes the schedule for the job. These XML files are created as a sample when you run the SequentialJob for the first time. The nice thing about this file, is that it’s constantly changing. To turn off a job, just set the “nextRun” value of the jobId to a future date far into the future. Change the increment and incrementType at any time to change the schedule for this job.
Creating a plugin and adding it to a job
There’s only 1 simple interface to extend, in the Quartz.Server.Extensions assembly:public interface IPlugin
{
void Execute(PluginContext pluginContext);
}
The PluginContext class has 2 public properties on it:
/// <summary>
/// These are the settings from the plugins-quartz-config.xml file.
/// <remarks>
/// If a setting key at the plugin level exists at a higher level, it will override the higher level key.
/// </remarks>
/// </summary>
public ReadOnlyDictionary<string, string> Settings { get; internal set; }
/// <summary>
/// This dictionary of data can be altered within your plugin. You can chain plugins together
/// and create business logic based on values inside the Data dictionary.
/// </summary>
public IDictionary<string, object> Data { get; private set; }
We'll create a sample plugin, as follows.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Common.Logging;
namespace Quartz.Server.Extensions.Plugins
{
public class SamplePlugin : IPlugin
{
private static readonly ILog log = LogManager.GetLogger(typeof(SamplePlugin));
public void Execute(PluginContext pluginContext)
{
// get the settings & data
var settings = pluginContext.Settings;
var data = pluginContext.Data;
// let's do a bunch of logging
log.Info("Start " + this.GetType().Name);
// log the settings
int settingsCount = settings == null ? 0 : settings.Count();
log.Info("With " + settingsCount + " settings:");
if (settingsCount > 0)
{
foreach (string key in settings.Keys)
log.Info(string.Format(" {0}: {1}", key, settings[key]));
}
// log data dictionary values
int dataCount = data == null ? 0 : data.Count();
log.Info("With " + dataCount + " data records:");
if (dataCount > 0)
{
foreach (string key in data.Keys)
log.Info(string.Format(" {0}: {1}", key, data[key]));
}
// change and/or add some data dictionary values
data["Status"] = "Success";
data["StatusDate"] = DateTime.Now;
log.Info("End " + this.GetType().Name);
}
}
}
This sample is in the Quartz.Server.Extensions.Plugins.dll file. Let’s just move that file into the plugins directory now, and register it in the plugins-quartz-config.xml file.
<?xml version="1.0" encoding="utf-8"?>
<config>
<!– Main configuration file for the sequential plugin job used within Quartz –>
<jobs count="1">
<job id="1">
<plugins count="2">
<plugin assembly="Quartz.Server.Extensions.Plugins" type="Quartz.Server.Extensions.Plugins.SamplePlugin">
<settings count="1">
<setting key="sampleSetting2" value="SamplePlugin setting value 2" />
</settings>
</plugin>
<plugin type="Quartz.Server.Extensions.NullPlugin, Quartz.Server.Extensions">
<settings count="1">
<setting key="sampleSetting1" value="NullPlugin setting value 1" />
</settings>
</plugin>
</plugins>
<settings count="2">
<setting key="sampleSetting1" value="sampleSettingValue1" />
<setting key="sampleSetting2" value="sampleSettingValue2" />
</settings>
</job>
</jobs>
<settings count="2">
<setting key="sampleSetting1" value="sampleSettingValue1" />
<setting key="sampleSetting2" value="sampleSettingValue2" />
</settings>
</config>
As soon as you alter this file, provided the DLL file was moved into the plugins directory, the job will pick it up and execute it in sequence. See the log output here:
Summary
This was an exercise in creativity, leveraging Quartz.NET, and a single Quartz.NET job to create a little sequential job plugin framework that one can alter at runtime by adding/removing plugins and changing job schedules without any pain whatsoever. Adding plugins using the method above is a breeze, extend an interface, drop the dll into the folder, and add a <plugin> tag to an xml file, and voila. I have clients that have been running this code (possibly slightly tailored to their needs
) for 2 to 3 years now to chain all kinds of events together like parsing text, doing mass PDF conversions of files sent in through SMTP, sending out email newsletters, and synchronizing user profiles between Active Directory and various CMS systems. It keeps cranking thanks to Quartz.NET. A variation of this was also used to process orders in a large ecommerce site, and send the orders to shipping companies and manufacturing plants around the world.
I’m sure there is MUCH to improve on this; so if you do experiment with this, and improve on it, please do let me know, and I’d be glad to post your updates here.
[button link="http://sdrv.ms/ZHyf0h" color="primary" target="_blank" size="large" title="Code Download" icon_before="download"]Code Download[/button]
Thanks for reading as always.
Would morning matt, i’m planning to perform, very heavy operations and i want to use quarz. In my database i have a table called pendingJobs. I woluld like to use quartz, and have one job that only read lines from database and generate new jobs like:
Sorry george, looks like your comment was cut off.
Much better example to play with Quartz.NET in fact its simpler than the official website documentation and tutorial. Thanks Matt.
Very cool. Glad it was useful.