Stork 2.0: Open Source DHCP Management Tool
ISC launched the Stork management application almost exactly five years ago, in November 2019, as an experimental tool.
Read post
DNS over HTTPS Update
It has been a long time since our last blog on the BIND 9 DNS-over-HTTPS (DoH) implementation. Here is an update on the considerable progress made since then. Although we will discuss user-visible changes here, most of the changes happened under the hood: our implementation is now more mature and standards-compliant.
In some prior versions of the BIND 9.17 development branch, it was impossible to build BIND without DNS-over-HTTPS support. This is no longer required as of BIND 9.17.17. Thus, libnghttp2 is no longer a build time dependency for BIND. Use --disable-doh as the ./configure script option to build without DoH.
Here are the topics we cover in this blog:
1. We have added connection limits to protect the server from excessive and abusive traffic.
2. We have simplified other DoH syntax
3. We provide some tips on testing your DoH server
4. We also describe another alternative for TLS offloading, using HAProxy in either TCP or HTTP mode
New Configuration Option - listener-clients
The first new option is http-listener-clients. It works very similarly to tcp-clients, but with one notable exception: it sets the limit on a per-listener basis, not globally. The http-listener-clients option sets the default per-listener quota size, which can be overridden by a listener-clients option within an http clause. Such an approach adds flexibility to the configuration, making more complex configurations possible. For example, one could make BIND listen on a public and private interface with different quota values.
tls key-cert-pair {
key-file "key.pem";
cert-file "cert.pem";
};
http local {
...
listener-clients 300;
...
};
options {
...
http-listener-clients 300;
...
listen-on tls key-cert-pair http local { 10.53.0.1; }; // DoH
...
};
The default value for http-listener-clients is 300. Setting it to 0 disables the quota facility, which is useful for testing and benchmarking purposes. We settled for the value 300 for now because this value is large enough to serve some clients while not large enough to let the server be abused too much, taking into consideration that it might need to serve clients over other DNS transports. Because hardware and software configurations might differ widely, we suggest you do your own measurements to find an appropriate value for your deployment. A significant number of these connections will remain idle for some time after the clients completed the name resolution, which brings us to the following point.
DNS-over-HTTPS works differently than other transports. In particular, the clients (primarily WEB-browsers) tend to keep connections open for longer periods compared to, Do53 or DNS over TCP. They do so to reduce latency during name resolution: if there is an existing HTTP/2 connection to the DoH server, they will issue a request over that connection rather than opening a new one. That brings another point - if you have plans to serve DNS-over-HTTPS, you might want to set tcp-initial-timeout, tcp-keepalive-timeout, and tcp-idle-timeout to the minimum values that work for you. Setting them to tcp-initial-timeout 100, tcp-keepalive-timeout 100 and tcp-idle-timeout 100 (ten seconds) is a good setting to try.
Before (http-)listener-clients was introduced, DoH connections shared a quota with other TCP clients. Idle HTTP clients would have prevented other TCP clients from being served. That was the initial motivation for adding these new DoH-specific quotas.
New Configuration Option - streams-per-connection
A second new option is http-streams-per-connection. This option sets the hard limit of the concurrent HTTP/2 streams per connection. After the hard limit is reached, the HTTP/2 session will be closed by the server. The http-streams-per-connection option sets the default number of streams per-connection, which can be overridden by a streams-per-connection option within an http clause.
tls key-cert-pair {
key-file "key.pem";
cert-file "cert.pem";
};
http local {
...
streams-per-connection 100;
...
};
options {
...
http-streams-per-connection 100;
...
listen-on tls key-cert-pair http local { 10.53.0.1; }; // DoH
...
};
The default limit for the number of streams per connection is 100. We choose this value because there is libnghttp2-based software that expects it to have at least that many simultaneous HTTP/2 requests made at once. With this in mind, we would suggest leaving the default value. However, one could consider lowering this number, as no sensible client would make that many concurrent requests. Setting it to 0 disables the limit, but doing so is strongly discouraged for anything but benchmarking in a controlled testing environment.
The combination of (http-)streams-per-connection and (http-)listener-clients allow the administrator to limit the load HTTP clients can create on a server. listener-clients manages the number of active HTTP/2 connections, while (http-)streams-per-connection limits the number of simultaneous streams in an HTTP/2 connection. By doing so, one limits the maximum amount of resources the server would use per HTTP/2 connection.
HTTP Clause Syntax Simplified
Everything within an http clause is now optional. As a result, there is no need to specify endpoints (absolute HTTP paths) within an http clause. When the endpoints option is omitted, the standard, default value is used for it (/dns-query). By the way, the syntax of absolute HTTP paths is now strictly verified in the configuration and should comply with the portion of the grammar given in the RFC3986 (see the path-absolute production definition in Appendix A for the details).
How to Test Your DoH Server
Although we said that you should test your configuration in order to find the right values for the options discussed above, we have to admit that at this point BIND itself cannot provide you with enough feedback to help establish the best values. We plan to fix that eventually. For now, we could recommend using HTTP/2 benchmarking software for this purpose and rely on the feedback given by it.
One such tool is h2load by the libnghttp2 authors. The tool can be used to test the HTTP transport using both GET and POST methods, as they are required to be implemented by a compliant server.
To test a server in GET mode, one needs a DNS request in wire format encoded in base64url encoding. Get one directly from the RFC8484 section 4.1.1 (as we did in this example) or extract one from a packet trace and manually encode in base64url. (For example, see the Export Packet Bytes functionality in Wireshark to save binary data).
Example Test Query
h2load -t 8 -c 300 -m 100 -n 1000000 https://doh.example.com/dns-query?dns=AAABAAABAAAAAAAAA3d3dwdleGFtcGxlA2NvbQAAAQAB
The example above will test the server https://doh.example.com with load simulating 300 concurrent clients (connections), opening up to 100 concurrent streams per connection using 8 threads doing 1000000 requests in total. The text after ?dns= is the DNS message encoded in base64url (A record for www.example.com). The tool will give feedback, including the number of queries per second in addition to statistics regarding the received HTTP status codes (2xx error codes in HTTP indicate success). Ideally, if the request data is in the right format, all of the queries should be answered with 2xx status codes.
To test a server using POST requests, we need a file containing a DNS request in wire format. The tool will make requests with the file data in their body. We also need to make sure that the header indicating the data type is included. Otherwise, BIND will treat such requests as malformed ones.
Example Test Query using HTTP POST
h2load -t 8 -c 300 -m 100 -n 1000000 -d ~/path/to/request_data_file.bin -H "content-type:application/dns-message" https://doh.example.com/dns-query
The example is equivalent to the one above, except that it uses HTTP POST instead of GET.
The added options and tools discussed so far are not directly related to DNS. In fact a DNS server serving DoH queries is like a regular HTTP/2 server which happens to serve only DNS queries, rather than web pages. Any experience working and configuring HTTP servers is helpful in managing DoH.
Encryption Offloading Using HAProxy
In our first blog post we described how NGINX could be used for terminating TLS. There is another popular solution for encryption offloading (also known as TLS termination) and HTTP load balancing - HAProxy. We recommend reading the related parts of the previous update where we discuss the reasons for TLS termination in the context of DoH.
HAProxy can be used in conjunction with BIND in two modes: TCP-mode and HTTP-mode. However, they provide different levels of flexibility. In TCP mode, HAProxy takes unencrypted traffic from BIND, which acts as a backend server. Then it applies encryption on top of data received via TCP without examining what is inside the TCP segments it encrypts. You can do TLS offloading this way by adding the following lines into the HAProxy configuration file:
frontend doh-in-tls
mode tcp
timeout client 10s
# Here we specify "h2" as the ALPN token to be selected.
# It is crucial for DoH to work.
bind *:443 v4v6 tfo ssl crt /path/to/cert.pem alpn h2
default_backend doh-server-plain-tcp
backend doh-server-plain-tcp
mode tcp
timeout connect 10s
timeout server 10s
# Address where BIND listens for unencrypted HTTP/2 requests
server doh-server 10.53.53.53:80
In the example above, in the frontend section, we instruct HAProxy to listen on port 443 (default HTTPS port) for TCP connections that are proxied to a BIND server acting as backend. The BIND server’s address is 10.53.53.53, where it listens on port 80 (default plain HTTP port) for unencrypted DNS requests over HTTP/2 (as described in the backend section).
Here one can see a significant advantage on HAProxy over NGINX when doing TLS termination - it allows specifying Application-Layer Protocol Negotiation (ALPN) tokens, which is vital for DoH.
Proxying of TCP connections is the most performant way to do TLS encryption offloading. It is recommended if you want to dedicate a whole domain name (like https://doh.example.com) specifically to the DNS server intended to serve DNS-over-HTTPS.
Let’s discuss proxying in HTTP mode now. First, we will create a configuration very similar to the one above. For regular DoH clients, it will work in the same way, albeit maybe with a slight performance loss.
frontend doh-in-https
mode http
timeout client 10s
# Here we specify "h2" as the ALPN token to be selected.
# It is crucial for DoH to work.
bind *:443 v4v6 tfo ssl crt /path/to/cert.pem alpn h2,http/1.1
default_backend doh-server-plain-http2
backend doh-server-plain-http2
mode http
timeout connect 10s
timeout server 10s
# Address where BIND listens for unencrypted HTTP/2 requests
server doh-server 10.53.53.53:80 proto h2
However, there is one important difference: now we have gained the ability to serve DNS-over-HTTPS queries over HTTP 1.1. By adding http/1.1 to the list of acceptable ALPN tokens, we are leveraging the HAProxy’s ability to convert between different HTTP protocol versions. That is something impossible to do in TCP mode.
If we want to, we could make HAProxy serve plain HTTP/1.1 by adding another frontend statement, similar to the following one:
frontend doh-plain-http1-1
mode http
timeout client 10s
bind *:80 v4v6 tfo # listen on TCP port 80 (plain HTTP)
default_backend doh-server-plain-http2
One could ask: what is the benefit of providing HTTP 1.1 if the specification strictly requires HTTP/2 or higher? HTTP 1.1 might be needed if one has HTTP 1.1 reverse proxying infrastructure in place, where the frontend can serve multiple HTTP versions (including HTTP/2), while the backend is expected to serve HTTP 1.1. There are, for example, CDN providers which still work this way.
We also expect that, at some point, HAProxy will get HTTP/3 support, making it possible to serve DNS-over HTTP/3 in a similar way (the so-called DoH3). We may consider extending our DoH support to cover this case natively, too. Although it is too early to discuss it at this point, as there is no final HTTP/3 specification available yet.
We realise that running a DoH server behind existing HTTP caching reverse proxying infrastructure might be important to some of our users. To fully cover this case, though, we need to properly set the Cache-Control HTTP header. We have a Gitlab issue to keep track of this issue. If this is important to you please add your comments and use cases there.
Improving Privacy while Offloading TLS
Now it is time to discuss another situation where proxying in HTTP mode will be useful. Let’s imagine that you have a site, e.g. https://site.example.com, and you want to serve DNS queries under the standard path (https://site.example.com/dns-query) while using the domain name for the rest of the site. HAProxy allows one to do that in HTTP proxying mode with the feature called HTTP Routing. Here is a sample configuration implementing this approach:
frontend in-route-http-doh
mode http
timeout client 10s
# Here we specify "h2" as the ALPN token to be selected.
# It is crucial for DoH to work.
bind *:443 v4v6 tfo ssl crt /path/to/cert.pem alpn h2,http/1.1
# Serve the data from the WEB-server by default
default_backend local-web-server
# If path in a request starts with /dns-query,
# use the DoH server as the backend
acl dns-query path_beg -i /dns-query
use_backend doh-server-plain-http2 if dns-query
backend local-web-server
mode http
timeout connect 15s
timeout server 15s
# WEB server’s address and port (plain HTTP 1.1)
server web-server 127.0.0.1:80
# For HTTPS 1.1 use
#server <IP address>:<port> ssl verify none
# For HTTP/2 (encrypted) use
#server <IP address>:<port> ssl verify none proto h2
In the configuration given above, there are two backends. One is the WEB-server serving the site; it runs on the local address 127.0.0.1, port 80. It is the default one. Another backend is the DoH server, serving the DNS queries. It is used when a request containing /dns-query as the HTTP path is made, as instructed in the frontend section.
A configuration like this has one crucial, non-technical benefit: it enables plausible deniability for end-users making DNS queries against the DoH server. In this case, a third party analysing the traffic cannot distinguish the DNS queries over HTTP from regular requests of the site content, as at most only the site name is transmitted in clear text in SNI part of the TLS session packets, but not request paths. As a result, the end-user cannot be accused of doing DNS queries against an unauthorised DNS server in restricted environments.
As concealing DNS requests in regular HTTP traffic was one of the main goals of developing DoH, we recommend using a similar configuration to the DNS operators interested in deploying DoH, should they want to do TLS termination with HAProxy. We believe that a slight performance drop when using such a configuration is well-worth the end-users’ privacy benefits.
In this example the local-web-server backend configuration contains examples of using encryption between frontend and backend. That is exactly the case where BIND’s ephemeral certificates feature may be used.
This would be a good time to test in your environment
We have only scratched the surface regarding the usage of HAProxy for doing TLS encryption offloading for DNS-over-HTTPS. As you can see, it is a capable tool, so if you plan to use it might be worth it to look at its documentation to end up with a configuration ideally suited for your environment. We used it a lot with our DoH implementation with good enough results to recommend it as a TLS termination solution for BIND, especially when BIND lacks some TLS-specific configuration options. That is one of the areas in which we want to improve. (See Gitlab #2795 and Gitlab # 2796).
Our DNS-over-HTTPS implementation in BIND has come a long way since it was introduced in our development branch at the beginning of this year. Although we don’t recommend running production services on the development branch, our DoH implementation is certainly mature enough for testing. Consider testing it by running the latest development release instance configured as a DoH forwarder.
Here is an example configuration for a DoH Forwarder
# Private Key and certificate pair for TLS
tls local-tls {
key-file "/path/to/private_key.pem";
cert-file "/path/to/fullchain_cert.pem";
};
# Disable remote control capabilities
controls { };
options {
allow-recursion {any;};
max-cache-size 5%;
# Forward all queries to the local resolver.
# You may consider replacing it with a resolver’s IP
# address.
forwarders { 127.0.0.1; };
# Disable DNS over port 53 (Do53)
listen-on-v6 { none; };
listen-on { none; };
# Listen for encrypted HTTP/2 queries on
# all IPv4 and IPv6 addresses
listen-on port 443 tls local-tls http default {any;};
listen-on-v6 port 443 tls local-tls http default {any;};
};
We want to share one tip on how to quickly test if your DNS-over-HTTPS deployment will work well with WEB-browsers: as soon as you have finished configuring the DNS-over-HTTPS server instance, visit the URL for DNS queries (e.g. https://doh.example.com/dns-query). If you see a blank page, then most probably you got it right. At the very least, this quick test ensures that the encryption configuration has been done correctly.
If you are considering deploying DoH using BIND, it is the right time to give it a try. We would love to hear from you should you find any problems or something lacking. We hope to hear from you in the cases when our implementation works well for you, too. In the meantime, we will continue improving the implementation as well as work on other BIND features. Stay tuned!
