Python job scheduling: 6 ways to schedule and execute jobs

Scheduling jobs is a crucial aspect of most modern applications, whether it’s polling APIs or databases, monitoring system health or auto-scaling. 

Even auto-scaling systems such as Kubernetes and Apache Mesos require periodic checks to ensure the status of deployed applications.

To ensure tasks are decoupled from business logic, decoupled execution queues like Redis queues are used. This article will explore different ways to schedule and execute Python jobs using mini tutorials, including simple loops, simple threaded loops, schedule library, Python crontab and RQ scheduler as decoupled queues.

Method 1: The simple loop

Scheduling a job with simple loops is a straightforward approach involving an infinitely running while loop that periodically calls a function. Although there are more efficient methods, it can do the job.

You can introduce time delays using the sleep function from the built-in time module. However, this method is less popular because it isn’t as visually appealing and readable as other methods. To schedule Python jobs with simple loops, you can use the time module in Python to create a loop that runs a certain function at specified intervals. Here’s an example code:

import time

def my_job():
    # function to run as a job
    print("This is my job")

while True:
    my_job()
    time.sleep(60) # run the job every 60 seconds

In the above example, the my_job function will be executed every 60 seconds using a simple loop and time.sleep() method. You can adjust the time interval as per your requirements.

Common problem:

This method is unsuitable for long-running or complex tasks as it may block the main thread and make your application unresponsive.

Method 2: Simple threaded loops

To solve the blocking problem, use simple threaded loops with Python’s threading module. This method is similar to simple loops, but instead of using a single loop, you create a new thread for each task you want to run.

Here are the general steps:

1. Define the function that you want to run periodically.
2. Create a thread for the function using the threading module.
3. Start the thread and set a time delay for how often the function should run using the sleep function from the time module.

Here’s a snippet of the code:

import threading
import time

def job():
   print("This is a scheduled job")

def run_threaded(job_func):
   job_thread = threading.Thread(target=job_func)
   job_thread.start()

while True:
   run_threaded(job)
   time.sleep(60)

In this example, we define the job function we want to run every 60 seconds and then define a run_threaded function that creates a new thread for the job function and starts it.

Finally, we use an infinite loop to repeatedly call the run_threaded function and add a time delay of 60 seconds between each call. This way, the job function will be executed every minute.

Common problem:

Once a thread is initiated and running, the main program or thread cannot change or modify its behavior. To allow the thread to respond to certain conditions or events, additional resources or logic may need to be added to the program to monitor these specific scenarios and take appropriate actions based on them.

Method 3: A schedule library

While loops aren’t visually appealing, a schedule library fixes this problem. 

import schedule
import time

def job():
   print("I'm working...")

schedule.every(10).seconds.do(job)

while True:
   schedule.run_pending()
   time.sleep(1)

This example defines a function job we want to run every 10 seconds. We then use schedule.every(10).seconds.do(job) to schedule this function to run every 10 seconds. Finally, we use while True: to continuously run the schedule.run_pending() function to check if any scheduled jobs need to be run and run them if necessary.

This code is much cleaner and easier to read than using a while loop. Also, it’s easier to modify the code to schedule multiple jobs or change an existing job’s schedule.

Common problem:

A common problem with the Python schedule library is that it may stop running when the process or program using the library exits or terminates. This can result in missed job executions or incomplete schedules. To avoid this issue, running the schedule library in a separate thread or process that continues to run even if the main program terminates is recommended.

Method 4: Python crontab

A Python crontab is a way to schedule tasks to run automatically on a specified time interval using the same syntax as the UNIX cron utility. Cron is a time-based job scheduler that runs on Unix-like operating systems, and Python crontab is a Python package that provides a simple interface to create cron-like schedules for executing Python code. Python crontab is especially useful on Linux systems where cron is natively available, but similar jobs scheduling can also be implemented on Windows systems using Task Scheduler or third-party tools.

With Python crontab, you can specify the interval, specific time of day, day of the week or month of the year a particular script or program should run. The scheduling is done by defining a set of rules using the crontab syntax, which specifies the exact timing of when a task should be executed.

Here’s an example of how the crontab syntax works:

# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday;
# │ │ │ │ │ 7 is also Sunday on some systems)
# │ │ │ │ │
# │ │ │ │ │
# * * * * * command to execute

In the above example, the first five columns specify the time intervals for running the command, while the last column is the command to be executed at the specified intervals.

To use Python crontab, you first need to install it via pip. Once installed, you can create a crontab object and use its schedule method to add a task to be executed at the specified intervals.

Here’s an example code snippet that schedules a Python script to run every day at 6 AM:

from crontab import CronTab

# Create a new crontab object
cron = CronTab(user='username')

# Add a new cron job to run the script every day at 6 AM
job = cron.new(command='python /path/to/script.py')
job.setall('0 6 *')

# Write the job to the user's crontab
cron.write()

This code creates a new crontab object for the specified user and adds a new cron job running the specified Python script daily at 6 AM. The setall method sets the time interval using the crontab syntax, and the write() method writes the job to the user’s crontab.

Common problem:

The write() method must be manually executed to save the schedules in python-crontab, as the library does not have an auto-save feature.

Method 5: RQ scheduler

To execute certain tasks that can’t be performed immediately, it is necessary to create a queue and organize tasks according to a queue system like LIFO or FIFO. A tool called python-rq allows this by using Redis as a broker to queue jobs. The rq-scheduler module uses a scheduler class to organize and manage jobs, offering advanced control over job intervals and execution timing. The information for a new job is saved in a hash map, including created_at, enqueued_at, origin, data and description.

Once queued, jobs can be executed by a program known as a worker. Workers have an entry in the Redis cache and are responsible for dequeuing jobs and updating their status in Redis. Jobs can be queued whenever necessary, but to schedule them, rq-scheduler is needed.

Here’s an example of using RQ scheduling in Python:

from datetime import datetime, timedelta
from redis import Redis
from rq_scheduler import Scheduler

# create a connection to Redis
redis_conn = Redis(host='localhost', port=6379)

# create a scheduler object
scheduler = Scheduler(connection=redis_conn)

# define the job function
def my_job():
   print("Hello, world!")

# schedule the job to run every minute
scheduler.schedule(
   scheduled_time=datetime.utcnow(),  # start immediately
   func=my_job,
   interval=timedelta(minutes=1)
)

In this example, we first create a connection to Redis and then create a scheduler object using that connection. We define a simple job function that prints “Hello, world!” and then schedule it to run every minute using the scheduler.schedule() method, which takes the planned time, job function and interval as parameters. The datetime.utcnow() function starts the job immediately.

Common problem:

RQ scheduling requires a separate worker process to execute the jobs, which may not be feasible for small applications or systems with limited resources. Also, since the jobs are executed by separate worker processes, it may require more work to manage resources and ensure optimal performance under heavy load.

How to use Python with RunMyJobs by Redwood

RunMyJobs by Redwood is a Python job scheduling and workflow management system that executes jobs across multiple platforms.

To use Python with RunMyJobs, follow these general steps:

1. Install the RunMyJobs Python client library. You can install it using pip by running pip install runmyjobs.
2. Create a Python script that defines your job. This script should include the code you want to run and any necessary dependencies.
3. Upload your Python script and any necessary dependencies to RunMyJobs. You can do this using the runmyjobs command-line tool, which is included with the RunMyJobs Python client library.
4. Create a job definition in RunMyJobs that references your Python script. This job definition should specify any configuration options, such as the environment in which your script should run.
5. Submit your job to RunMyJobs. You can do this using the runmyjobs command-line tool.
6. Monitor the status of your job in RunMyJobs. You can use the runmyjobs command-line tool or the RunMyJobs web interface.

Also, you can use the “pip install schedule” command to install the schedule library in Python. Once your job is complete, you can retrieve the output from RunMyJobs and use it as needed. These steps are only general guidelines and vary depending on your specific use case and environment.

Explore your options for Python schedulers

Python functions as a high-level programming language used for general-purpose programming, including web development, data science, machine learning and scientific computing.

Python offers various methods, including simple loops, threaded loops, Schedule library, Python crontab and RQ scheduler. Simple loops may not be suitable for long-running or complex tasks. Threaded loops can help solve blocking problems but require additional resources. The schedule library is visually appealing and easy to modify, but it may stop running when the program exits. Python crontab is an excellent option for scheduling tasks on a specified time interval.

The best Python scheduling method depends on task complexity, time interval and required monitoring.
Use a tool like RunMyJobs by Redwood to automate the execution and monitoring of Python workflows and job dependencies.