Creating AWS CDK Eventbridge Scheduler for Glue Data Quality Ruleset Runs

Amazon Web Services (AWS) provides a variety of tools and services for building and managing data processing workflows in the cloud. Two of these services are AWS Glue and AWS EventBridge. AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy to move data between data stores. AWS EventBridge is a serverless event bus that makes it easy to connect applications together using data from your own applications, Software-as-a-Service (SaaS) applications, and AWS services.

In this article, we will show you how to use AWS CDK (Cloud Development Kit) to create an EventBridge scheduler for AWS Glue Data Quality ruleset runs. This will allow you to automatically run your Data Quality rulesets on a schedule, without having to manually trigger them.

Prerequisites

Before you begin, you will need to have the following:

• An AWS account
• The AWS CLI installed and configured
• The AWS CDK installed and configured
• Node.js and npm installed
• An AWS Glue Data Quality ruleset created

Creating the AWS CDK Stack

To create the AWS CDK stack, you will need to create a new CDK project and install the necessary dependencies. You can do this by running the following commands:

$ cdk init app --language=javascript
$ cd my-app
$ npm install @aws-cdk/aws-events-targets @aws-cdk/aws-events @aws-cdk/aws-glue @aws-cdk/core

This will create a new CDK project with the necessary dependencies installed. Next, you will need to create a new CDK stack by creating a new file called my-stack.js in the lib directory.

Creating the EventBridge Rule

To create the EventBridge rule, you will need to use the @aws-cdk/aws-events and @aws-cdk/aws-events-targets modules. The following code shows how to create a new EventBridge rule that runs every day at 12:00 AM:

const rule = new events.Rule(this, 'MyRule', {
 schedule: events.Schedule.expression('cron(0 0 * * ? *)'),
});

Next, you will need to add a target to the rule. In this case, the target will be a Lambda function that runs the AWS Glue Data Quality ruleset. You can create the Lambda function using the @aws-cdk/aws-lambda module, as shown in the following code:

const lambdaFunction = new lambda.Function(this, 'MyFunction', {
 runtime: lambda.Runtime.NODEJS_12_X,
 handler: 'index.handler',
 code: lambda.Code.fromAsset('lambda'),
});

This code creates a new Lambda function with the runtime set to Node.js 12.x, the handler set to index.handler, and the code loaded from the lambda directory. You will need to create the lambda directory and add your Lambda function code to it.

Once you have created the Lambda function, you can add it as a target to the EventBridge rule, as shown in the following code:

rule.addTarget(new events_targets.LambdaFunction(lambdaFunction));

Creating the AWS Glue Data Quality Ruleset

To create the AWS Glue Data Quality ruleset, you will need to use the @aws-cdk/aws-glue module. The following code shows how to create a new AWS Glue Data Quality ruleset:

const ruleset = new glue.DataQualityRuleset(this, 'MyRuleset', {
 rules: [
 {
 name: 'MyRule',
 statement: glue.DataQualityRuleStatement.rowCountIsGreaterThan(0),
 },
 ],
});

This code creates a new AWS Glue Data Quality ruleset with a single rule called MyRule. The rule checks that the row count is greater than 0.

Running the Ruleset in the Lambda Function

To run the ruleset in the Lambda function, you will need to use the AWS SDK for JavaScript in Node.js. The following code shows how to run the ruleset in the Lambda function:

const AWS = require('aws-sdk');
const glue = new AWS.Glue();

exports.handler = async (event) => {
 const params = {
 DataQualityRulesetName: 'MyRuleset',
 JobName: 'MyJob',
 };

 await glue.startDataQualityJob(params).promise();
};

This code uses the startDataQualityJob method of the glue service to start a new Data Quality job using the ruleset called MyRuleset and the job called MyJob. You will need to create the Data Quality job using the AWS Management Console or the AWS CLI.

Deploying the CDK Stack

To deploy the CDK stack, you can use the following command:

$ cdk deploy

This will create the EventBridge rule, the Lambda function, and the AWS Glue Data Quality ruleset in your AWS account.

In this article, we have shown you how to use AWS CDK to create an EventBridge scheduler for AWS Glue Data Quality ruleset runs. This allows you to automatically run your Data Quality rulesets on a schedule, without having to manually trigger them. We hope this article has been helpful in getting you started with using AWS CDK to manage your AWS resources.

References