An alternative method to webhooks might be pushing an event/message onto an AWS SQS queue and having any consumer deal with it however they want.
Unfortunately the other application might not use Laravel, or, if it does, the job name might not be compatible with your project.
I will outline with the use terraform to setup a SQS queue, and provide some commands to push raw messages, and the few steps required to set up the queue, and a job to process messages in Laravel.
AWS
We just need to create a simple queue, (not FIFO) and I use the default settings.
The next steps are to allow us to create messages through the cli, and optionally, create the queue using terraform.
Create a profile and set the config and credentials:
aws --profile=ally-api-webhooks configure
When enqueueing a message (or anything else, for that matter) with awscli it closes out to a pager (less by default) which is super annoying, so the following command will disable it (alternatively you could use an environment variable AWS_PAGER).
aws --profile=ally-api-webhooks configure set cli_pager ''
Terraform (optional)
The terraform script is simple, since SQS is relatively simple.
variables.tf:
variable "aws_profile" {
type = string
default = "ally-api-webhooks"
}
variable "aws_region" {
type = string
default = "eu-west-2"
}
variable "queue_name" {
type = string
default = "webhook_queue"
}
main.tf:
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.0"
}
}
}
provider "aws" {
profile = var.aws_profile
region = var.aws_region
shared_config_files = [pathexpand("~/.aws/config")]
shared_credentials_files = [pathexpand("~/.aws/credentials")]
}
resources.tf:
resource "aws_sqs_queue" "webhook_queue" {
name = var.queue_name
}
resource "aws_sqs_queue_policy" "webhook_queue_policy" {
queue_url = aws_sqs_queue.webhook_queue.id
policy = <<POLICY
{
"Version": "2008-10-17",
"Id": "webhook_queue_policy",
"Statement": [
{
"Sid": "webhook_queue",
"Effect": "Allow",
"Principal": {
"AWS": "${data.aws_caller_identity.current.account_id}"
},
"Action": [
"SQS:*"
],
"Resource": "arn:aws:sqs:eu-west-2:${data.aws_caller_identity.current.account_id}:"
}
]
}
POLICY
}
data.tf:
data "aws_caller_identity" "current" {}
data "aws_region" "current" {}
output.tf:
output "queue_url" {
value = aws_sqs_queue.webhook_queue.url
}
output "region" {
value = data.aws_region.current.name
}
output "SQS_PREFIX" {
value = replace(aws_sqs_queue.webhook_queue.url, "/${var.queue_name}", "")
}
output "SQS_QUEUE" {
value = var.queue_name
}
The core commands for terraform are:
terraform init
terraform fmt
terraform plan
terraform apply
This might take a minute, arguably quicker to create through the web UI.
It will also give you some values to set as environment variables for queue config later on.
Project (optional)
This is for example purposes, so I create a fresh install.
docker run \
--rm \
--tty \
--interactive \
--volume="$(pwd)/src:/app" \
--volume="${COMPOSER_HOME:-$HOME/.composer}:/tmp" \
--workdir=/app \
thecodingmachine/php:7.4-v4-cli \
composer create-project laravel/laravel=^8 src
Then to get a shell in the container:
docker run \
--rm \
--tty \
--interactive \
--volume="$(pwd)/src:/app" \
--volume="${COMPOSER_HOME:-$HOME/.composer}:/tmp" \
--workdir=/app \
thecodingmachine/php:7.4-v4-cli \
bash
Driver
I use primitivesense/laravel-raw-sqs-connector to interpret arbitrary SQS messages to be handled by a Job in Laravel world.
composer require primitivesense/laravel-raw-sqs-connector
After install, add their service provider into your providers in config/app.php:
<?php
return [
'providers' => [
/*
* Package Service Providers...
*/
\PrimitiveSense\LaravelRawSqsConnector\RawSqsServiceProvider::class,
],
];
Queue
Duplicate the sqs connection in config/queue.php, renaming it to sqs-plain.
Change the driver from sqs to raw-sqs - this new driver is from the package.
config/queue.php:
<?php
return [
'connections' => [
'sqs-plain' => [
// custom driver from the service provider
'driver' => 'raw-sqs',
'key' => env('AWS_ACCESS_KEY_ID'),
'secret' => env('AWS_SECRET_ACCESS_KEY'),
'prefix' => env('SQS_PREFIX', 'https://sqs.us-east-1.amazonaws.com/your-account-id'),
'queue' => env('SQS_QUEUE', 'default'),
'suffix' => env('SQS_SUFFIX'),
'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
],
],
];
Job
Create a new job to handle the SQS messages, i.e. php artisan make:job ProcessWebhook.
Make the following changes:
app/Jobs/ProcessWebhook.php:
<?php
namespace App\Jobs;
-use Illuminate\Bus\Queueable;
-use Illuminate\Contracts\Queue\ShouldQueue;
-use Illuminate\Foundation\Bus\Dispatchable;
-use Illuminate\Queue\InteractsWithQueue;
-use Illuminate\Queue\SerializesModels;
+use PrimitiveSense\LaravelRawSqsConnector\RawSqsJob;
-class ProcessWebhook implements ShouldQueue
+class ProcessWebhook extends RawSqsJob
{
- use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
public function handle()
{
+ // TODO: handle the job, use $this->getData() to get SQS message contents
}
}
Finally, set job_class to \App\Jobs\ProcessWebhook::class in your sqs-plain queue connection in config/queue.php, i.e.:
<?php
return [
'connections' => [
'sqs-plain' => [
// custom driver from the service provider
'driver' => 'raw-sqs',
'job_class' => \App\Jobs\ProcessWebhook::class,
'key' => env('AWS_ACCESS_KEY_ID'),
'secret' => env('AWS_SECRET_ACCESS_KEY'),
'prefix' => env('SQS_PREFIX', 'https://sqs.us-east-1.amazonaws.com/your-account-id'),
'queue' => env('SQS_QUEUE', 'default'),
'suffix' => env('SQS_SUFFIX'),
'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
],
],
];
You should be good to consume messages now.
Producer
Since the scenario is to process an arbitrary SQS payload with a Job, we will use aws sqs send-message to enqueue a ‘job’.
cat <<EOF | aws \
--profile=ally-api-webhooks \
sqs \
send-message \
--queue-url=$(terraform output -raw queue_url) \
--message-body file:///dev/stdin
{
"request_headers": [],
"request_body": {
"from": "$(hostname)",
"at": "$(date)"
}
}
EOF
Consumer
The last step is to run the queue.
Monitor/verify configuration:
php artisan queue:monitor sqs-plain:webhook_queue
Process the message:
php artisan queue:work --once --queue=webhook_queue sqs-plain
