Private Spaces DNS Service Discovery
Last updated April 13, 2020
Table of Contents
Processes running in Heroku Private Spaces can communicate easily with one another by using DNS to obtain the IP address of other dynos in the same space. This makes it easy to create and deploy microservices. For example, suppose you want to create an image-processing app that consists of several small services, including:
- A web process for accepting jobs
- A worker process for processing incoming images
- An agent process for handling back-office interactions
With DNS Service Discovery, processes in systems like this have a simple way to reach one another to perform tasks. When the web process accepts a new job, it can alert the back-office service via its DNS name that a new job has arrived. It can then send the job to the worker process via its DNS name as well.
Unlike dynos in the Common Runtime, dynos in Private Spaces can freely exchange TCP and UDP traffic on a private network. A process type of any kind (web or worker) can choose to listen on any available port, and other dynos in the space can connect to that port on the dyno host.
To communicate with another process, you need to know the IP address of the dyno(s) the process is running on, along with the port number the process is listening on. Typically you know the port number in advance, because server processes listen on a fixed port number. It is not possible, however, to know the dyno’s IP address in advance. To resolve this, Heroku provides a DNS-based naming scheme for dynos. This lets you find and connect to any dyno in a Private Space by specifying:
- The name of the process’s associated Heroku app
- The process type (such as
web
orworker
) of the process you want to connect to - Optionally, the dyno number to connect to (useful when a process is running on more than one dyno)
Setup: Enable DNS lookups
DNS Service Discovery is enabled by default on all applications created after May 31, 2017. For existing applications, you need to enable this by running:
$ heroku features:enable spaces-dns-discovery --app <your app>
If the app is already running, you need to restart it before this setting takes effect:
$ heroku restart --app <your app>
After the feature is enabled, your app’s dynos have a slightly different DNS configuration that enables DNS lookups. There is a slight risk that this change might cause issues for your application. See known technical considerations below. If you encounter any such issues, please disable the feature and restart your app:
$ heroku features:disable spaces-dns-discovery --app <your app>
$ heroku restart --app <your app>
Usage
Binding to the private network
All dyno-to-dyno communication in Private Spaces takes places via the private network. To communicate via the private network, your process needs to bind to its dyno’s private IP address on a port between 1024
and 65535
. The HEROKU_PRIVATE_IP
environment variable contains your dyno’s private IP address.
All processes (not just web
) can bind to ports on the private network
The app and process you want to use with Private Space Service Discovery should not use the value of the $PORT
environment variable. $PORT
varies for different dynos powering the same process types and changes when dynos are restarted. Instead, pick a static port (like 5000
) and use that. For example (using Node.js and Express):
const express = require('express')
const PORT = 5000
express()
.get('/', (req, res) => res.send('Hello World!'))
.listen(PORT, () => console.log(`Listening on ${ PORT }`))
This could be deployed as the foo
process type by using this Procfile
:
foo: node index.js
After your process binds to the private IP address, it’s reachable on the chosen port via its DNS name and port. The above example would be available on this URL:
http://foo.[APP_NAME].app.localspace:5000/
Resolving other processes
DNS Service Discovery uses the following naming scheme:
[DYNO_NUMBER (optional)].[PROCESS_TYPE].[APP_NAME].app.localspace
For example, performing the following nslookup
from a one-off dyno returns IP addresses for all dynos running the web
process for an app named sushi
:
$ nslookup web.sushi.app.localspace
web.sushi.app.localspace. 0 IN A 10.10.10.11
web.sushi.app.localspace. 0 IN A 10.10.10.10
web.sushi.app.localspace. 0 IN A 10.10.10.9
If your process connects to a DNS name with the format shown above, the DNS registry routes it to one of the associated process’s dynos in round-robin fashion. If you instead want to connect to a specific dyno running the associated process, you can specify its dyno number at the beginning of the DNS name.
This example demonstrates performing an nslookup
specifically for the IP address of dyno number 2 running the sushi
app’s web
process:
$ nslookup 2.web.sushi.app.localspace
2.web.sushi.app.localspace. 0 IN A 10.10.10.10
Note that the last two components of the naming scheme (app
and localspace
) are constant for all DNS names.
Environment variables
Dynos with DNS Service Discovery enabled have the following environment variables set:
Name | Contents |
---|---|
HEROKU_DNS_FORMATION_NAME |
The round-robin DNS name for the dyno’s process type |
HEROKU_DNS_DYNO_NAME |
The dyno’s DNS name |
HEROKU_DNS_APP_NAME |
The dyno’s application DNS name |
HEROKU_PRIVATE_IP |
The dyno’s private IP |
Unlike HEROKU_DNS_DYNO_NAME
and HEROKU_DNS_FORMATION_NAME
, HEROKU_DNS_APP_NAME
does not resolve by itself. It is provided to simplify determining the dyno’s application-level hostname.
DNS registry details
Update frequency
The DNS registry’s set of dyno locations is updated every 10 seconds. Take this timeframe into account when designing apps that use DNS Service Discovery. Different dynos might observe these registry updates at slightly different times, but results should converge after about 10 seconds.
TTLs
DNS Service Discovery records always have a time-to-live (TTL) of zero seconds, meaning that the requester should not itself cache the information. It is important that your application is properly configured to respect the TTLs in the DNS response, so it does not cache these local requests.
Technical considerations
Routing
Networking using Private Space DNS Service Discovery uses raw TCP/IP. Even if you are running an HTTP service, there’s no Heroku routing layer, no router logs or measurements and no port-binding related boot timeouts. This can hamper troubleshooting HTTP services deployed as non-web processes, but this can also allow you to run non-HTTP services in Heroku Private Spaces.
IPv6
When you enable DNS Service Discovery for an app, IPv6 is disabled for the app’s dynos. Although Private Spaces only offers connectivity via IPv4, your app’s behavior nevertheless might change when IPv6 is no longer available on the dyno. For example, Go apps need to Listen via tcp4
instead of tcp
. Code similar to the following:
conn, err := net.Dial("tcp", "golang.org:80")
ln, err := net.Listen("tcp", ":8080")
needs to be replaced with the following:
conn, err := net.Dial("tcp4", "golang.org:80")
ln, err := net.Listen("tcp4", ":8080")