Errors & Retrying

How to deal with errors and write reliable tasks.

When an uncaught error is thrown inside your task, that task attempt will fail.

You can configure retrying in two ways:

1. In your trigger.config file you can set the default retrying behavior for all tasks.
2. On each task you can set the retrying behavior.

By default when you create your project using the CLI init command we disabled retrying in the DEV environment. You can enable it in your trigger.config file.

A simple example with OpenAI

This task will retry 10 times with exponential backoff.

• openai.chat.completions.create() can throw an error.
• The result can be empty and we want to try again. So we manually throw an error.

/trigger/openai.ts

import { task } from "@trigger.dev/sdk/v3";
import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

export const openaiTask = task({
  id: "openai-task",
  //specifying retry options overrides the defaults defined in your trigger.config file
  retry: {
    maxAttempts: 10,
    factor: 1.8,
    minTimeoutInMs: 500,
    maxTimeoutInMs: 30_000,
    randomize: false,
  },
  run: async (payload: { prompt: string }) => {
    //if this fails, it will throw an error and retry
    const chatCompletion = await openai.chat.completions.create({
      messages: [{ role: "user", content: payload.prompt }],
      model: "gpt-3.5-turbo",
    });

    if (chatCompletion.choices[0]?.message.content === undefined) {
      //sometimes OpenAI returns an empty response, let's retry by throwing an error
      throw new Error("OpenAI call failed");
    }

    return chatCompletion.choices[0].message.content;
  },
});

Combining tasks

One way to gain reliability is to break your work into smaller tasks and trigger them from each other. Each task can have its own retrying behavior:

/trigger/multiple-tasks.ts

import { task } from "@trigger.dev/sdk/v3";

export const myTask = task({
  id: "my-task",
  retry: {
    maxAttempts: 10,
  },
  run: async (payload: string) => {
    const result = await otherTask.triggerAndWait("some data");
    //...do other stuff
  },
});

export const otherTask = task({
  id: "other-task",
  retry: {
    maxAttempts: 5,
  },
  run: async (payload: string) => {
    return {
      foo: "bar",
    };
  },
});

Another benefit of this approach is that you can view the logs and retry each task independently from the dashboard.

Retrying smaller parts of a task

Another complimentary strategy is to perform retrying inside of your task.

We provide some useful functions that you can use to retry smaller parts of a task. Of course, you can also write your own logic or use other packages.

retry.onThrow()

You can retry a block of code that can throw an error, with the same retry settings as a task.

/trigger/retry-on-throw.ts

import { task, logger, retry } from "@trigger.dev/sdk/v3";

export const retryOnThrow = task({
  id: "retry-on-throw",
  run: async (payload: any) => {
    //Will retry up to 3 times. If it fails 3 times it will throw.
    const result = await retry.onThrow(
      async ({ attempt }) => {
        //throw on purpose the first 2 times, obviously this is a contrived example
        if (attempt < 3) throw new Error("failed");
        //...
        return {
          foo: "bar",
        };
      },
      { maxAttempts: 3, randomize: false }
    );

    //this will log out after 3 attempts of retry.onThrow
    logger.info("Result", { result });
  },
});

If all of the attempts with retry.onThrow fail, an error will be thrown. You can catch this or let it cause a retry of the entire task.

retry.fetch()

You can use fetch, axios, or any other library in your code.

But we do provide a convenient function to perform HTTP requests with conditional retrying based on the response:

/trigger/retry-fetch.ts

import { task, logger, retry } from "@trigger.dev/sdk/v3";

export const taskWithFetchRetries = task({
  id: "task-with-fetch-retries",
  run: async ({ payload, ctx }) => {
    //if the Response is a 429 (too many requests), it will retry using the data from the response. A lot of good APIs send these headers.
    const headersResponse = await retry.fetch("http://my.host/test-headers", {
      retry: {
        byStatus: {
          "429": {
            strategy: "headers",
            limitHeader: "x-ratelimit-limit",
            remainingHeader: "x-ratelimit-remaining",
            resetHeader: "x-ratelimit-reset",
            resetFormat: "unix_timestamp_in_ms",
          },
        },
      },
    });
    const json = await headersResponse.json();
    logger.info("Fetched headers response", { json });

    //if the Response is a 500-599 (issue with the server you're calling), it will retry up to 10 times with exponential backoff
    const backoffResponse = await retry.fetch("http://my.host/test-backoff", {
      retry: {
        byStatus: {
          "500-599": {
            strategy: "backoff",
            maxAttempts: 10,
            factor: 2,
            minTimeoutInMs: 1_000,
            maxTimeoutInMs: 30_000,
            randomize: false,
          },
        },
      },
    });
    const json2 = await backoffResponse.json();
    logger.info("Fetched backoff response", { json2 });

    //You can additionally specify a timeout. In this case if the response takes longer than 1 second, it will retry up to 5 times with exponential backoff
    const timeoutResponse = await retry.fetch("https://httpbin.org/delay/2", {
      timeoutInMs: 1000,
      retry: {
        timeout: {
          maxAttempts: 5,
          factor: 1.8,
          minTimeoutInMs: 500,
          maxTimeoutInMs: 30_000,
          randomize: false,
        },
      },
    });
    const json3 = await timeoutResponse.json();
    logger.info("Fetched timeout response", { json3 });

    return {
      result: "success",
      payload,
      json,
      json2,
      json3,
    };
  },
});

If all of the attempts with retry.fetch fail, an error will be thrown. You can catch this or let it cause a retry of the entire task.

Advanced error handling and retrying

We provide a handleError callback on the task and in your trigger.config file. This gets called when an uncaught error is thrown in your task.

You can

• Inspect the error, log it, and return a different error if you’d like.
• Modify the retrying behavior based on the error, payload, context, etc.

If you don’t return anything from the function it will use the settings on the task (or inherited from the config). So you only need to use this to override things.

OpenAI error handling example

OpenAI calls can fail for a lot of reasons and the ideal retry behavior is different for each.

In this complicated example:

• We skip retrying if there’s no Response status.
• We skip retrying if you’ve run out of credits.
• If there are no Response headers we let the normal retrying logic handle it (return undefined).
• If we’ve run out of requests or tokens we retry at the time specified in the headers.

tasks.ts

import { task } from "@trigger.dev/sdk/v3";
import { calculateISO8601DurationOpenAIVariantResetAt, openai } from "./openai.js";

export const openaiTask = task({
  id: "openai-task",
  retry: {
    maxAttempts: 1,
  },
  run: async (payload: { prompt: string }) => {
    const chatCompletion = await openai.chat.completions.create({
      messages: [{ role: "user", content: payload.prompt }],
      model: "gpt-3.5-turbo",
    });

    return chatCompletion.choices[0].message.content;
  },
  handleError: async (payload, error, { ctx, retryAt }) => {
    if (error instanceof OpenAI.APIError) {
      if (!error.status) {
        return {
          skipRetrying: true,
        };
      }

      if (error.status === 429 && error.type === "insufficient_quota") {
        return {
          skipRetrying: true,
        };
      }

      if (!error.headers) {
        //returning undefined means the normal retrying logic will be used
        return;
      }

      const remainingRequests = error.headers["x-ratelimit-remaining-requests"];
      const requestResets = error.headers["x-ratelimit-reset-requests"];

      if (typeof remainingRequests === "string" && Number(remainingRequests) === 0) {
        return {
          retryAt: calculateISO8601DurationOpenAIVariantResetAt(requestResets),
        };
      }

      const remainingTokens = error.headers["x-ratelimit-remaining-tokens"];
      const tokensResets = error.headers["x-ratelimit-reset-tokens"];

      if (typeof remainingTokens === "string" && Number(remainingTokens) === 0) {
        return {
          retryAt: calculateISO8601DurationOpenAIVariantResetAt(tokensResets),
        };
      }
    }
  },
});

Preventing retries

Using AbortTaskRunError

You can prevent retries by throwing an AbortTaskRunError. This will fail the task attempt and disable retrying.

/trigger/myTasks.ts

import { task, AbortTaskRunError } from "@trigger.dev/sdk/v3";

export const openaiTask = task({
  id: "openai-task",
  run: async (payload: { prompt: string }) => {
    //if this fails, it will throw an error and stop retrying
    const chatCompletion = await openai.chat.completions.create({
      messages: [{ role: "user", content: payload.prompt }],
      model: "gpt-3.5-turbo",
    });

    if (chatCompletion.choices[0]?.message.content === undefined) {
      // If OpenAI returns an empty response, abort retrying
      throw new AbortTaskRunError("OpenAI call failed");
    }

    return chatCompletion.choices[0].message.content;
  },
});

Using try/catch

Sometimes you want to catch an error and don’t want to retry the task. You can use try/catch as you normally would. In this example we fallback to using Replicate if OpenAI fails.

/trigger/myTasks.ts

import { task } from "@trigger.dev/sdk/v3";

export const openaiTask = task({
  id: "openai-task",
  run: async (payload: { prompt: string }) => {
    try {
      //if this fails, it will throw an error and retry
      const chatCompletion = await openai.chat.completions.create({
        messages: [{ role: "user", content: payload.prompt }],
        model: "gpt-3.5-turbo",
      });

      if (chatCompletion.choices[0]?.message.content === undefined) {
        //sometimes OpenAI returns an empty response, let's retry by throwing an error
        throw new Error("OpenAI call failed");
      }

      return chatCompletion.choices[0].message.content;
    } catch (error) {
      //use Replicate if OpenAI fails
      const prediction = await replicate.run(
        "meta/llama-2-70b-chat:02e509c789964a7ea8736978a43525956ef40397be9033abf9fd2badfe68c9e3",
        {
          input: {
            prompt: payload.prompt,
            max_new_tokens: 250,
          },
        }
      );

      if (prediction.output === undefined) {
        //retry if Replicate fails
        throw new Error("Replicate call failed");
      }

      return prediction.output;
    }
  },
});