One-off Dyno Metrics

This add-on is operated by MISC CO BV

Natively monitor your Heroku Scheduler jobs and any other detached one-off dyno.

One-off Dyno Metrics

Last updated July 04, 2024

The One-off Dyno Metrics add-on allows you to natively monitor and investigate detached one-off dynos. Consult different types of metrics to diagnose issues on detached one-off dynos started by you, your application, Heroku, or a Heroku add-on like Advanced Scheduler.

This service is an extension of the existing Application Metrics provided in the Heroku Dashboard. You can configure threshold alerting to actively monitor one-off dyno-related issues. In addition to raw metrics and alerts, One-off Dyno Metrics provides guidance and recommendations on how to correct specific one-off dyno problems.

One-off Dyno Metrics is language-agnostic, so it can integrate with apps in any programming language. All features are accessible from the One-off Dyno Metrics Dashboard.

Alert email notifications sent by One-off Dyno Metrics contain a part of the one-off dyno’s command. Don’t put sensitive data directly in the command. Use config vars instead.

Provisioning the Add-on

Attach One-off Dyno Metrics to a Heroku application via the CLI:

See a list of all plans available here.

$ heroku addons:create one-off-metrics
Creating one-off-metrics on sharp-mountain-4005... free
Your add-on has been provisioned successfully

After installing One-off Dyno Metrics, it’s fully integrated and configured. It picks up all relevant data automatically from your app.

Dashboard

The One-off Dyno Metrics dashboard allows you to investigate one-off dynos running in your Heroku app’s environment using different types of metrics. In addition, you can create and manage the monitors for your one-off dynos.

You can access the dashboard via the CLI:

$ heroku addons:open one-off-metrics
Opening one-off-metrics for sharp-mountain-4005

or by visiting the Heroku Dashboard and selecting the application in question. Select One-off Dyno Metrics from the Add-ons menu.

One-off Dynos

One-off dynos are dynos that run a specific command until it exits. They run alongside other dynos, exactly like the app’s web, worker, and other formation dynos. One-off dynos are ideal for performing one-off administrative or maintenance tasks for your Heroku app. They can also execute some task periodically using a scheduler add-on like Heroku Scheduler.

Learn more about one-off dynos here.

One-off Dyno Metrics only considers metrics from detached one-off dynos. It doesn’t include any attached one-off dynos started by a user, for example, via the Heroku Console, and consequently doesn’t set off any alerts.

One-off Dyno Metrics

Gathered Metrics

Events

The Event plot displays one-off dyno lifecycle events per specific one-off dyno.

Execution Time

The Execution Time plot displays summary metrics for one-off dyno execution times.

Throughput

The throughput plot displays the number of one-off dynos broken down by exit status (0, null, or anything greater than 0). One-off dynos that result in a null status code are considered to have timed out. One-off dynos that result in anything greater than 0 (zero) are considered to have failed. One-off dynos that result in a 0 (zero) status code are considered to have succeeded.

Concurrency

The concurrency plot displays summary metrics of the number of one-off dynos running at the same time.

Data Resolution

There are 4 levels of data resolution in One-off Dyno Metrics:

  • 1 minute (Past 3 hrs)
  • 10 minutes (72-48 hrs ago, 48-24 hrs ago, and past 24 hrs)
  • 30 minutes (Past 72 hrs)
  • 1 hour (Past 7 days)

Data Filter

One-off Dyno Metrics enables you to analyse the metrics data of a subset of your one-off dynos by filtering based on one-off dyno attributes.

Either filter only by command by providing (part of) the applicable command, or explicitly filter based on one or more specific attributes by using the attribute:value syntax.

Filtering through the input field in the Dashboard applies to all plots at once. This allows you to correlate metrics data of different plots.

Attributes

Name Description Example
command command used to start this process "python scrape.py"
state current status of process (either: running, succeeded, failed, or timed out) "timed out"
type type of process "scheduler"
name the name of this process on this dyno "scheduler.4321"
size dyno size (either: free, hobby, standard-1X, standard-2X, performance-m, performance-l, private-s, private-m or private-l) "performance-l"

You can use 2 different match types, an exact match, or a partial match. Exact matches include double quotes. For example, "scheduler" only matches if it’s exactly equal to the test string. Partial matches, e.g sched, don’t include double quotes and match if the test string contains the filter value.

Prepend - to exclude metrics data containing a certain test string e.g. -type:release, -state:succeeded, or -command:"python scrape.py"

Real-world Examples

To only get metrics data related to one-off dynos running command echo “Hello World!”:

echo "Hello World!"

To filter out one-ff dynos running a command containing Hello World:

-Hello World

To only get metrics data related to one-off dynos running command echo “Hello World!” that either failed or timed out:

command:echo "Hello World!" state:failed state:timed out

To only get metrics data related to one-off dynos running command echo “Hello World!” that did not succeed:

command:echo "Hello World!" -state:succeeded

To only get metrics data related to one-off dynos of size Performance-L that are currently running:

state:running size:performance-l

To only get metrics data related to one-off dynos started by Heroku Scheduler that succeeded:

type:"scheduler" state:succeeded

To only get metrics data related to a one-off dyno with a specific name:

name:scheduler.1234

Alternatively, metrics data can be filtered in a specific plot by using the plot’s legend buttons. This does not affect the metrics data displayed in the other plots.

Monitors and Threshold Alerting

One-off Dyno Metrics allows you to create different types of monitors that check specific aspects of one-off dynos and open alerts based on various settings.

Alert email notifications contain a part of the one-off dyno’s command. Don’t put sensitive data directly in the command. Use config vars instead.

Types of Monitors

One-off Dyno Command Failure

Monitor the one-off dyno exit status. Given a sensitivity setting of 1, this type of monitor opens an alert whenever a one-off dyno fails to successfully execute a specific command. An exit status greater than 0 (zero) is considered a failure.

One-off Dyno Command Timeout

Monitor whether a one-off dyno fails to exit on its own. Given a sensitivity setting of 1, this type of monitor opens an alert whenever something forces a one-off dyno executing a specific command to exit.

One-off Dyno Command Execution Time

Monitor the one-off dyno execution time. Given a sensitivity setting of 1, this type of monitor opens an alert whenever a one-off dyno takes longer to execute a specific command than your threshold setting.

One-off Dyno Command Concurrency

Monitor the number of one-off dynos executing a specific command. Given a sensitivity setting of 1, this type of monitor opens an alert whenever the number of concurrent one-off dynos is at or exceeds your threshold setting.

One-off Dyno Total Concurrency

Monitor the total number of one-off dynos running inside of your Heroku app’s environment. Given a sensitivity setting of 1, this type of monitor opens an alert whenever the total number of concurrent one-off dynos is at or exceeds your threshold setting.

Monitors exist independently from one another. A single one-off dyno can meet the alert conditions of different monitors, setting off multiple alerts.

Sensitivity Setting

You can configure a sensitivity setting for every monitor. This setting determines how many consecutive times a monitor must meet the alert conditions before actually opening an alert.

Filter Settings

You can configure filter settings for every monitor. These filters determine which one-off dynos a specific monitor accounts for while discarding the others.

You can set the filter using 2 different match types, an exact match, or a partial match. Exact matches include double quotes. For example, "filter value" only matches if it’s exactly equal to the test string. Partial matches, e.g filter value, don’t include double quotes and match if the test string contains the filter value.

Both partial and exact match types are case insensitive.

Dyno Command Filter

The Dyno Command Filter enables you to configure a monitor to either account for or discard a one-off dyno based on its command.

For example:

Filtering based on the one-off dyno’s command, .py matches all one-off dynos with a command containing the string .py.

Filtering with "node src/jobs/migrate.js" in double quotes matches all one-off dynos with a command exactly equal to node src/jobs/migrate.js.

Dyno Type Filter

The Dyno Type Filter enables you to configure a monitor to either account for or discard a one-off dyno based on its type.

For example:

Filtering based on the one-off dyno’s type, scheduler matches all one-off dynos with a type containing the string scheduler.

Filtering with "scheduler" in double quotes matches all one-off dynos with a type exactly equal to scheduler.

The first example matches both one-off dynos created by Heroku Scheduler and Advanced Scheduler (which creates one-off dynos with type advanced-scheduler). The latter only matches one-off dynos created by Heroku Scheduler.

Dyno Size Filter

The Dyno Size Filter enables you to configure a monitor to either account for or discard a one-off dyno based on its size.

For example:

Filtering based on the one-off dyno’s size, performance matches all one-off dynos with a size of either Performance-M or Performance-L. "performance" in double quotes doesn’t match any one-off dyno because the dyno size performance doesn’t exist.

By using the monitor’s filter settings, multiple monitors of the same type can exist beside one another. This flexibility allows you to account for different one-off dynos that are running in your Heroku app’s environment.

Alert Notifications

By default, the distribution for email notifications is to all app owners and collaborators for non-org accounts, and admins for people in a Heroku Enterprise org. Alternatively, you can add additional email addresses, such as for email-based PagerDuty integration.

Although One-Off Dyno Metrics supports multiple additional addresses, the preferred approach is to use your email platform to create an alias. You can manage group membership outside of the alert notification framework.

Using One-off Dyno Metrics with Heroku Scheduler

You can use One-off Dyno Metrics to monitor your Heroku Scheduler jobs.

To create a monitor that only accounts for one-off dynos started by Heroku Scheduler, set the custom Dyno Type filter to "scheduler". This filter ensures that all one-off dynos that don’t have a type scheduler get discarded by this monitor.

For more fine-grained control, you can also configure a monitor to filter based on the dyno command and/or the size of the dyno.

Using One-off Dyno Metrics with Other Scheduler Add-ons

You can use One-off Dyno Metrics to monitor any job started by a Scheduler add-on that uses one-off dynos under the hood.

To create a monitor that only accounts for one-off dynos started by a particular scheduler add-on, use the custom Dyno Type filter. Set the filter to the dyno type used by that add-on. You can derive this value from the names of the one-off dynos started by that add-on.

For example, Advanced Scheduler starts dynos named advanced-scheduler.xxxx, making the dyno type advanced-scheduler. Setting the monitor’s custom Dyno Type filter to "advanced-scheduler" ensures that all one-off dynos that don’t have this type get discarded by this monitor.

For more fine-grained control, you can also configure a monitor to filter based on the dyno command and/or the size of the dyno.

Using One-off Dyno Metrics with heroku run:detached

You can use One-off Dyno Metrics to monitor any one-off dyno started with the Heroku CLI’s heroku run:detached command.

To create a monitor that only accounts for these one-off dynos running in the background, set the custom Dyno Type filter to "run". This filter ensures that all one-off dynos not started by the Heroku CLI get discarded by this monitor.

For more fine-grained control, you can also configure a monitor to filter based on the dyno command and/or the size of the dyno.

Using One-off Dyno Metrics with Heroku Platform API’s Dyno Create Endpoint

You can use One-off Dyno Metrics to monitor one-off dynos created by the Heroku Platform API’s Dyno Create endpoint.

To create a monitor that only accounts for these specific one-off dynos, use the custom Dyno Type filter. Set the filter to the value of the type property in the request body used to create the one-off dyno. This filter ensures that all other one-off dynos get discarded by this monitor.

For more fine-grained control, you can also configure a monitor to filter based on the dyno command and/or the size of the dyno.

Using One-off Dyno Metrics with Release Phases

You can use One-off Dyno Metrics with release phase tasks.

To create a monitor that only accounts for one-off dynos related to a new release of an app, set the custom Dyno Type filter to "release". This filter ensures that all other one-off dynos get discarded by this monitor.

For more fine-grained control, you can also configure a monitor to filter based on the dyno command and/or the size of the dyno.

Migrating Between Plans

Application owners must carefully manage the migration timing to ensure proper application function during the migration process.

Use the heroku addons:upgrade command to migrate to a new plan.

$ heroku addons:upgrade one-off-metrics:newplan
-----> Upgrading one-off-metrics:newplan to sharp-mountain-4005... done, v18 ($49/mo)
       Your plan has been updated to: one-off-metrics:newplan

Removing the Add-on

You can remove One-off Dyno Metrics via the CLI:

This action destroys all associated data and can’t be undone!

$ heroku addons:destroy one-off-metrics
-----> Removing one-off-metrics from sharp-mountain-4005... done, v20 (free)

One-off Dyno Guidance

Troubleshooting One-off Dyno Failures

Whenever a one-off dyno exits with an exit status greater than 0 (zero), it’s considered to have failed. There can be several reasons why the one-off dyno exited with an error code.

Issues can occur during the provisioning of the one-off dyno. Check your one-off dyno’s app logs for any information, as these issues get logged most of the time.

command not found in your one-off dyno’s app logs indicates that the command provided for the one-off dyno doesn’t exist. Try investigating why. Did you make a typo or forget to push your changes to your Heroku app’s environment?

Check for application-level errors. Review your one-off dyno’s app logs for any errors, warnings, or additional information. Try to reproduce it locally and start debugging from there.

Troubleshooting One-off Dyno Timeouts

Whenever something forces a one-off dyno to exit instead of exiting on its own, it’s considered to have timed out. Sometimes this forced exit is harmless, for example, when a user kills the one-off dyno using the Heroku CLI.

A forced exit can also mean that your one-off dyno keeps running indefinitely until it reached its TTL (Time-To-Live). Upon creation, a one-off dyno gets a TTL value assigned in seconds. Heroku automatically kills the dyno after the number of seconds has passed.

The following example shows what appears in your one-off dyno’s app logs when Heroku kills a dyno in this way :

Cycling
State changed from up to complete
Stopping all processes with SIGTERM
Process exited with status 143

The one-off dyno’s command either took longer to run than its TTL, or it failed to exit on its own after performing the required actions. Test your command locally and make sure it exits properly before running it using a one-off dyno.

When using the Heroku Scheduler add-on, it creates a one-off dyno with a TTL value that is equal to the job’s interval. For example, for a job executed every 10 minutes, it creates one-off dynos with a TTL value of 600 seconds (10 minutes).

Support

Submit all One-off Dyno Metrics support and runtime issues via one of the Heroku Support channels. Any non-support issues or product feedback is welcome at support@oneoffmetrics.com.

Feedback

Log in to submit feedback.