PagerDuty alert task

Open and resolve PagerDuty incidents when Synnax statuses change.

A PagerDuty alert task sends the statuses that you choose to PagerDuty. For example, when an Arc automation sets the “Engine 1” status to Error, the task triggers a PagerDuty alert, and the on-call engineer gets a page. When the status changes to Success, the task resolves the alert.

Before you start, you need a PagerDuty service with an Events API v2 integration, and the statuses to send. To get the 32-character integration key, also called the routing key, open Services -> Service Directory in PagerDuty, select the service, then Integrations -> Add Integration -> Events API v2.

How the task sends events

Status variant	PagerDuty
Error	An `error` event, or `critical` with Treat error as critical.
Warning	A `warning` event.
Info	An `info` event.
Success	Resolves the alert.
Loading, Disabled	Nothing.

The status key is the PagerDuty dedup key, so a later change updates the same open alert.
Error, warning, and info all send a trigger event. The settings of the PagerDuty service decide whether the alert opens an incident and how urgent it is.
The summary of the event is the status message, followed by : and the description when there is one. The source is the status name, or the key when the name is empty. The details are the status details.
The task sends only changes that happen while it runs. It does not send the current value of a status when it starts.
Deleting the task or a status does not resolve open incidents. Set the statuses to Success first.
When PagerDuty rejects an event, the task shows an error and keeps running.

The Synnax Core runs the task and sends the events, so the computer that runs the Core needs internet access to events.pagerduty.com. The task follows the standard task lifecycle.

Create a PagerDuty alert task

Run the “Create PagerDuty alert task” command from the command palette.
Enter the Routing key.
Leave Connect from set to the Core.
Click Add alert. Under Trigger, select the Status to watch.
Optional: turn on Treat error as critical, and fill in Component, Group, and Class under Incident.
Add an alert for each status.
Click (Start).

The task creates no channels.

Create and run an alert task

import synnax as sy

client = sy.Synnax()

STATUS_KEY = "engine-1"
client.statuses.set(
    sy.Status(
        key=STATUS_KEY,
        name="Engine 1",
        variant="success",
        message="Running",
    )
)

alert_task = sy.pagerduty.AlertTask(
    name="Engine alerts",
    routing_key="<your-32-character-integration-key>",
    auto_start=False,
    alerts=[
        sy.pagerduty.Alert(
            status=STATUS_KEY,
            errors_critical=True,
            component="engine-1",
            group="engines",
            class_="engine-failure",
        ),
    ],
)
client.tasks.configure(alert_task)

with alert_task.run():
    # Opens an incident
    client.statuses.set(
        sy.Status(
            key=STATUS_KEY,
            name="Engine 1",
            variant="error",
            message="Not responding",
            description="The connection to the controller is lost.",
        )
    )
    sy.sleep(5)
    # Resolves the incident
    client.statuses.set(
        sy.Status(
            key=STATUS_KEY,
            name="Engine 1",
            variant="success",
            message="Running",
        )
    )
    sy.sleep(3)

import { pagerduty } from "@synnaxlabs/client";

await client.statuses.set({
  key: "engine-1",
  name: "Engine 1",
  variant: "success",
  message: "Running",
});

const [rack] = await client.racks.retrieve({ integration: "pagerduty" });
const alertTask = await client.tasks.create({
  name: "Engine alerts",
  rack: rack.key,
  type: "pagerduty_alert",
  config: pagerduty.taskConfigZ.parse({
    routingKey: "<your-32-character-integration-key>",
    alerts: [
      {
        status: "engine-1",
        errorsCritical: true,
        component: "engine-1",
        group: "engines",
        class: "engine-failure",
      },
    ],
  }),
});
await alertTask.start();

Task options

Console label	Python field	Default	Notes
Routing key	`routing_key`	-	The 32-character Events API v2 integration key.
Connect from	-	The Core	The Core runs the task. Python and TypeScript use the Core’s rack.
Auto start	`auto_start`	Off	Starts the task when you configure it.
Alerts	`alerts`	-	At least one alert must be enabled.

Alert reference

Console label	Python field	Default	Notes
Status	`status`	-	The key of the status to watch.
Enabled check box	`disabled`	Off	A disabled alert is left out of the task.
Treat error as critical	`errors_critical`	Off	Sends Error as `critical` instead of `error`.
Component	`component`	Empty	The part of the system, such as `"engine-1"`.
Group	`group`	Empty	A group of components, such as `"engines"`.
Class	`class_`	Empty	The kind of event, such as `"engine-failure"`. `class` in TypeScript.

Troubleshooting

`must be exactly 32 characters`

Cause: The routing key is not 32 characters long.
Fix: Copy the integration key again from the Events API v2 integration in PagerDuty.

`at least one alert must be enabled`

Cause: The task has no alerts, or every alert is disabled.
Fix: Add an alert, or enable one.

`Failed to send PagerDuty event`

Cause: The Core could not send the event to PagerDuty. The description gives the cause. The task keeps running and sends the next change.
Fix: PagerDuty rejected the event -> check the routing key.

The Core could not reach events.pagerduty.com -> check the internet access of the computer that runs the Core.

No incident opens

Cause: The task sends only changes that happen while it runs, and only for the status keys in its alerts.
Fix: The task is not running -> start the task.

The automation sets a different key -> set the key of an alert. An Arc status.set with a name that matches no status creates a new status with a new key.

For other problems, see tasks.

Arc alarms

Client libraries