quartz
Development

Using Quartz.NET to implement a background worker in a cloud system

Quartz.NET is a full-featured, open source job scheduling system that can be used from smallest apps to large scale enterprise systems (http://quartznet.sourceforge.net). It is a part of the open source Java job scheduling framework, Quartz.

The main components of Quartz.NET are:
  • The scheduler instance
  • The job (detail)
  • The job trigger
THE SCHEDULER INSTANCE

A scheduler instance acts as a server to execute scheduled jobs and/or to schedule the jobs themselves. We can schedule jobs using an instance and they will be executed by the instance if it has been started.
In order for the scheduler to function properly it needs to be configured. There are three ways to configure a scheduler:

a. The Application Configuration File

This is the method we are currently using for configuring both the executing instance and the scheduling one.
A new section must be added to the configSections in the configuration file:

<section name="quartz" type="System.Configuration.NameValueSectionHandler, System, Version=1.0.5000.0,Culture=neutral, PublicKeyToken=b77a5c561934e089" />

This allows us to add the quartz configuration section which looks like this (this example is for an instance which should be used for executing jobs, not only scheduling them):

<quartz>
    <add key="quartz.scheduler.instanceName" value="SchedulerQuartzServer" />
    <add key="quartz.scheduler.instanceId" value="instance_one" />
    <add key="quartz.threadPool.threadCount" value="10" />
    <add key="quartz.threadPool.threadPriority" value="Normal" />
    <add key="quartz.jobStore.misfireThreshold" value="60000" />
    <add key="quartz.jobStore.type" value="Quartz.Impl.AdoJobStore.JobStoreTX, Quartz" />
    <add key="quartz.jobStore.useProperties" value="false" />
    <add key="quartz.jobStore.dataSource" value="default" />
    <add key="quartz.jobStore.tablePrefix" value="QRTZ_" />
    <add key="quartz.jobStore.clustered" value="true" />
    <add key="quartz.jobStore.lockHandler.type" value="Quartz.Impl.AdoJobStore.SimpleSemaphore, Quartz" />
    <add key="quartz.dataSource.default.connectionStringName" value="BILLINGDB_CONN_STRING" />
    <add key="quartz.dataSource.default.provider" value="SqlServer-20" />
    <add key="quartz.scheduler.exporter.type" value="Quartz.Simpl.RemotingSchedulerExporter, Quartz" />
    <add key="quartz.scheduler.exporter.port" value="555" />
    <add key="quartz.scheduler.exporter.bindName" value="QuartzScheduler" />
    <add key="quartz.scheduler.exporter.channelType" value="tcp" />
    <add key="quartz.scheduler.exporter.channelName" value="httpQuartz" />
</quartz>

The instance name is the name of the scheduler instance and it will be used to save jobs. This name must be common to all scheduler instances, allowing us to schedule a job with one instance and execute it with another. In order to be able to run multiple scheduler instances in the same application or on the same machine (one to execute and one or more to schedule) we need to change for each one the instanceId, the port and the bindName. Otherwise, there will be an error on trying to schedule/execute jobs.

The threadPool settings control how many threads will be used for the instance. For example we can set the thread pool to 2 which means only 2 jobs will be executing at the same time.

It may be recommended to have multiple executing instances (one with many threads for small jobs, one with few threads for heavier jobs…), in which case the instanceName has to be the same when scheduling and executing (e.g. HeavyQuartz for the heavy ones and LightQuartz for the small ones).

The jobStore settings configure where and how to store the jobs themselves. The given example uses a database to store the jobs but we can also choose to work in memory by setting the job store type to RAM or not setting it at all (it is the default job type). Another possibility is working with an xml file.

The tablePrefix sets the prefix which is used for the quartz table names in the database, so this way we can have multiple scheduler instances mapping to different tables. The connectionStringName is the name of the connection string which has to be found in the configuration file in the connectionStrings section.

The scheduler is instantiated in this case in the following way:

schedulerFactory = new StdSchedulerFactory(); //getting the scheduler factory
instance = schedulerFactory.GetScheduler(); //getting the instance
b. The Quartz Properties File

This is another option and it needs to have a file in the application host called quartz.config. The settings are the same but the format is different:

# You can configure your scheduler in either <quartz> configuration section
# or in quartz properties file
# Configuration section has precedence
quartz.scheduler.instanceName = ServerScheduler

# configure thread pool info
quartz.threadPool.type = Quartz.Simpl.SimpleThreadPool, Quartz
quartz.threadPool.threadCount = 10
quartz.threadPool.threadPriority = Normal
...

The scheduler is instantiated in this case in the following way:

schedulerFactory = new StdSchedulerFactory(); //getting the scheduler factory
instance = schedulerFactory.GetScheduler(); //getting the instance
c. Hardcoding

The quartz instance can also be configured in the code by using a NameValueCollection:

var properties = new NameValueCollection();
properties["quartz.scheduler.instanceName"] = "ServerScheduler";
properties["quartz.scheduler.proxy"] = "true";
properties["quartz.threadPool.threadCount"] = "0";
...
schedulerFactory = new StdSchedulerFactory(properties); //getting the scheduler factory
instance = schedulerFactory.GetScheduler(); //getting the instance
THE JOB (DETAIL)

The job detail holds the data associated with a job. Each job to be scheduled and executed must have a corresponding class implementing the IJob interface. The only method to be implemented is Execute which will be launched when the time comes:

public void Execute(IJobExecutionContext context)

The context holds all the necessary data and objects related to the job, including the quartz instance itself. This also allows chaining jobs by scheduling a job at the end of another.

A job can have associated job data which is saved as a dictionary:

JobDataMap jdm = new JobDataMap();
jdm.Add(EMAIL_TYPE, emailType);
jdm.Add(CUSTOMER_ID, customerId);
jdm.Add(SUBSCRIPTION_ID, subscriptionId);
jdm.Add(SITE_ID, siteId);
jdm.Add(COUNT, count);
jdm.Add(DAY, day);

var job = JobBuilder.Create<ChainedEmailJob>()
      .WithIdentity("Job_Name”, EMAIL_JOB_GROUP)
      .SetJobData(jdm)
      .Build();

The JobDataMap also allows keeping more complex objects which are serialized. However, in this case we must make sure that we maintain back compatibility for those classes (to avoid problems on deserializing after updating a class).

THE JOB TRIGGER

The job trigger allows us to control the timing of the scheduler, when to execute it, how many times, what happens on misfire. The following example creates a trigger which will launch only once, two days from now.

// Instantiate a DateTimeOffset value in UTC with the necessary offset based on the provided days
DateTime utcTime = DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Utc);
DateTimeOffset targetTime = new DateTimeOffset(utcTime);
targetTime = new DateTimeOffset(utcTime.AddDays(2));

//create a trigger to be launched only once
trigger = (ISimpleTrigger)TriggerBuilder.Create()
.WithIdentity("Trigger", EMAIL_TRIGGER_GROUP)
.StartAt(targetTime)//fire the trigger at the computed date
.WithSimpleSchedule(x => x
.WithRepeatCount(0)//fire it only once
.WithMisfireHandlingInstructionFireNow())//if the trigger does not fire on time then fire it the next time it is possible
.Build();

More complex triggers can be created using cron triggers which allow a finer control (for example, every two minutes, between 12:40 – 13:00 on every Tuesday).

More details on the Quartz.NET can be found at http://quartznet.sourceforge.net and particularly at http://quartznet.sourceforge.net/tutorial/index.html.

DEVELOPMENT ISSUES

This section is used to highlight some of the issues that may come up when working with Quartz.Net, as well as some recommendations.

Backward Compatibility

When making changes to existent quartz jobs we have to make sure that they are backward compatible in terms of:

  • The stored data – the job data map allows us to store data for the job to access when it is executed. The danger when changing the job class is to add new data to the job data map. This way, if there are any old instances of the job in the database, the new code will try to get the new data which does not exist for the old jobs and an exception will be raised. This case must be covered.
  • The job class name or namespace. Another piece of data which is stored in the database is the job name and namespace, so again if this is changed old jobs will fail to execute.

Tracking/Debugging Quartz Jobs

It may be difficult to debug quartz jobs because of the multithreaded model.

Debugging Guidelines

Debug

Being launched in the scheduler the debug instance has to be on the scheduler as well to get into the job code. To enter services code the debug instance has to be on the business service layer.

Tracking Triggers

The main goal of quartz jobs is to take the load off the main user thread for smaller response times as well as a larger spread of the CPU usage (by executing the same task over a larger period of time). The point is that not all triggers lead to heavy work. In some cases the job only checks if anything needs to be done, the result is false so the job is done.

The number of triggers/jobs that are created can be quite large (as in the case of creating a subscription), so tracking them becomes difficult. This applies mostly to one time jobs such as emails or webhooks, other jobs such as assessing subscriptions or updating the cc status have only one instance.

1. Quartz Thread Count

Limit the quartz thread count to 1 (in order for quartz to execute a single job at any given time). To limit the thread count change the “quartz.threadPool.threadCount” property either in the application configuration or the configuration code (depending on how the qserver was configured).

<add key="quartz.threadPool.threadCount" value="5" />

Sometimes the job may not be launched on time because of the quartz delays or because the CPU is busy with some thing else. In order to increase the probability of launching jobs on time increase the thread priority to Normal:

<add key="quartz.threadPool.threadPriority" value="Normal" />

2. Quartz Database

The database tables to follow are QRTZ_JOB_DETAILS and QRTZ_TRIGGERS.

A trigger’s NEXT_FIRE_TIME represents the next time when the trigger will be fired to execute it’s job and it is expressed in ticks.

The following utility sql script can be used to convert ticks to datetime and vice versa:

Declare @To varchar = ''	-- DO NOT CHANGE

-------[ CONVERTING Ticks to DateTime ]-------------------------------------------------------------
Declare @TickOf19000101 bigint = 599266080000000000 -- DO NOT CHANGE
Declare @SourceTickValue bigint
Declare @Minutes float		  

--####### set the source tick value here ########
Set @SourceTickValue = 635095980000000000 
--###############################################

SET @Minutes = (@SourceTickValue - @TickOf19000101) * POWER(10.00000000000,-7) / 60

Select @SourceTickValue as [Ticks], @To as [To], DATEADD(MI, @Minutes, '1900-01-01') as [DateTime] 
----------------------------------------------------------------------------------------------------


-------[ CONVERTING DateTime to Ticks ]-------------------------------------------------------------
declare @ticksPerDay bigint = 864000000000 -- DO NOT CHANGE
declare @date datetime

--####### set the source date value here ########
set @date = '2013-07-16 19:00:00.000'
--###############################################

declare @date2 datetime2 = @date

declare @dateBinary binary(9) = cast(reverse(cast(@date2 as binary(9))) as binary(9))
declare @days bigint = cast(substring(@dateBinary, 1, 3) as bigint)
declare @time bigint = cast(substring(@dateBinary, 4, 5) as bigint)

select @date as [DateTime], @To as [To], @days * @ticksPerDay + @time as [Ticks]
----------------------------------------------------------------------------------------------------

You can change the next fire time to a value in the future in order to postpone the execution of the job.

Triggers can also be deleted, in which case the job should be deleted as well (to avoid consistency problems). This allows us to single out particular triggers and make them easier to track.

The current executing jobs are those jobs with a status which is different from WAITING:

SELECT * FROM [BillingDB].[dbo].[QRTZ_TRIGGERS]
WHERE TRIGGER_STATE != 'WAITING'
GO

This script also retrieves triggers that may have been executed already and are completed, but these triggers still take one thread of the available ones, so technically it is still executing.

In the case of one time jobs (adding emails, webhooks) the trigger name contains some ids, depending on the job type (site id, customer id, subscription id). Check the code in order to see what data is put into the name.

Quartz Issues

Some issues were observed while working with the Quartz scheduler. Some triggers that were left behind were no longer launched by the scheduler. This was reported on several machines but seldom. During a major deploy the jobs which accumulated for the scheduler on the Production server were not launched, in large numbers. On a close investigation of the issue the conclusion was that the jobs were not launched in time and they were considered misfired. All the jobs are created with the instruction to be launched immediately in case of misfire, however, the instruction most probably failed.

As part of the solution we increased the number of threads assigned to the scheduler, in order to avoid large numbers of triggers accumulating in times of heavy data processing. We also increased the misfire threshold from its 60 seconds default, to an entire day.