# PagerDuty alert task

> Open and resolve PagerDuty incidents when Synnax statuses change.

A PagerDuty alert task sends the [statuses](https://docs.synnaxlabs.com/reference/alarms/statuses) that you choose to PagerDuty. For example, when an Arc automation sets the “Engine 1” status to **Error**, the task triggers a PagerDuty alert, and the on-call engineer gets a page. When the status changes to **Success**, the task resolves the alert.

Before you start, you need a PagerDuty service with an Events API v2 integration, and the statuses to send. To get the 32-character integration key, also called the routing key, open **Services** -> **Service Directory** in PagerDuty, select the service, then **Integrations** -> **Add Integration** -> **Events API v2**.

## How the task sends events

| Status variant    | PagerDuty                                                         |
| ----------------- | ----------------------------------------------------------------- |
| Error             | An `error` event, or `critical` with **Treat error as critical**. |
| Warning           | A `warning` event.                                                |
| Info              | An `info` event.                                                  |
| Success           | Resolves the alert.                                               |
| Loading, Disabled | Nothing.                                                          |

* The status key is the PagerDuty dedup key, so a later change updates the same open alert.
* Error, warning, and info all send a trigger event. The settings of the PagerDuty service decide whether the alert opens an incident and how urgent it is.
* The summary of the event is the status message, followed by `: `and the description when there is one. The source is the status name, or the key when the name is empty. The details are the status details.
* The task sends only changes that happen while it runs. It does not send the current value of a status when it starts.
* Deleting the task or a status does not resolve open incidents. Set the statuses to **Success** first.
* When PagerDuty rejects an event, the task shows an error and keeps running.

The Synnax Core runs the task and sends the events, so the computer that runs the Core needs internet access to `events.pagerduty.com`. The task follows the standard [task lifecycle](https://docs.synnaxlabs.com/reference/hardware/tasks#task-lifecycle).

## Create a PagerDuty alert task

[Enterprise](https://docs.synnaxlabs.com/reference/get-started/editions "Needs Synnax Enterprise")

**Console**

1. Run the “Create PagerDuty alert task” command from the [command palette](https://docs.synnaxlabs.com/reference/get-started/ui-overview#search-and-command-palette).
2. Enter the **Routing key**.
3. Leave **Connect from** set to the Core.
4. Click **Add alert**. Under **Trigger**, select the **Status** to watch.
5. Optional: turn on **Treat error as critical**, and fill in **Component**, **Group**, and **Class** under **Incident**.
6. Add an alert for each status.
7. Click (**Start**).

The task creates no channels.

[](https://synnax.nyc3.cdn.digitaloceanspaces.com/docs/device-drivers/pagerduty/alert-task/create-dark.mp4#t=0.001)

**Python**

Create and run an alert task

```python
import synnax as sy

client = sy.Synnax()

STATUS_KEY = "engine-1"
client.statuses.set(
    sy.Status(
        key=STATUS_KEY,
        name="Engine 1",
        variant="success",
        message="Running",
    )
)

alert_task = sy.pagerduty.AlertTask(
    name="Engine alerts",
    routing_key="<your-32-character-integration-key>",
    auto_start=False,
    alerts=[
        sy.pagerduty.Alert(
            status=STATUS_KEY,
            errors_critical=True,
            component="engine-1",
            group="engines",
            class_="engine-failure",
        ),
    ],
)
client.tasks.configure(alert_task)

with alert_task.run():
    # Opens an incident
    client.statuses.set(
        sy.Status(
            key=STATUS_KEY,
            name="Engine 1",
            variant="error",
            message="Not responding",
            description="The connection to the controller is lost.",
        )
    )
    sy.sleep(5)
    # Resolves the incident
    client.statuses.set(
        sy.Status(
            key=STATUS_KEY,
            name="Engine 1",
            variant="success",
            message="Running",
        )
    )
    sy.sleep(3)
```

**TypeScript**

```typescript
import { pagerduty } from "@synnaxlabs/client";

await client.statuses.set({
  key: "engine-1",
  name: "Engine 1",
  variant: "success",
  message: "Running",
});

const [rack] = await client.racks.retrieve({ integration: "pagerduty" });
const alertTask = await client.tasks.create({
  name: "Engine alerts",
  rack: rack.key,
  type: "pagerduty_alert",
  config: pagerduty.taskConfigZ.parse({
    routingKey: "<your-32-character-integration-key>",
    alerts: [
      {
        status: "engine-1",
        errorsCritical: true,
        component: "engine-1",
        group: "engines",
        class: "engine-failure",
      },
    ],
  }),
});
await alertTask.start();
```

## Task options

| Console label | Python field  | Default  | Notes                                                              |
| ------------- | ------------- | -------- | ------------------------------------------------------------------ |
| Routing key   | `routing_key` | -        | The 32-character Events API v2 integration key.                    |
| Connect from  | -             | The Core | The Core runs the task. Python and TypeScript use the Core’s rack. |
| Auto start    | `auto_start`  | Off      | Starts the task when you configure it.                             |
| Alerts        | `alerts`      | -        | At least one alert must be enabled.                                |

## Alert reference

| Console label           | Python field      | Default | Notes                                                                 |
| ----------------------- | ----------------- | ------- | --------------------------------------------------------------------- |
| Status                  | `status`          | -       | The key of the status to watch.                                       |
| Enabled check box       | `disabled`        | Off     | A disabled alert is left out of the task.                             |
| Treat error as critical | `errors_critical` | Off     | Sends **Error** as `critical` instead of `error`.                     |
| Component               | `component`       | Empty   | The part of the system, such as `"engine-1"`.                         |
| Group                   | `group`           | Empty   | A group of components, such as `"engines"`.                           |
| Class                   | `class_`          | Empty   | The kind of event, such as `"engine-failure"`. `class` in TypeScript. |

## Troubleshooting

### `must be exactly 32 characters`

* Cause

  The routing key is not 32 characters long.

* Fix

  Copy the integration key again from the **Events API v2** integration in PagerDuty.

### `at least one alert must be enabled`

* Cause

  The task has no alerts, or every alert is disabled.

* Fix

  Add an alert, or enable one.

### `Failed to send PagerDuty event`

* Cause

  The Core could not send the event to PagerDuty. The description gives the cause. The task keeps running and sends the next change.

* Fix

  * **PagerDuty rejected the event** -> check the routing key.
  * **The Core could not reach `events.pagerduty.com`** -> check the internet access of the computer that runs the Core.

### No incident opens

* Cause

  The task sends only changes that happen while it runs, and only for the status keys in its alerts.

* Fix

  * **The task is not running** -> start the task.
  * **The automation sets a different key** -> set the key of an alert. An Arc `status.set` with a name that matches no status creates a new status with a new key.

For other problems, see [tasks](https://docs.synnaxlabs.com/reference/hardware/tasks).