Quad9 and the BIND 9 Subscription Edition

A Conversation with Quad9

We recently had an extended chat with the Quad9 team to discuss their experience with BIND 9. John Todd, Danielle Deibler, and Corey Mosher were kind enough to spend nearly two hours talking with us.

Quad9 operates a large public resolver at 9.9.9.9, of course, but also 9.9.9.10, and 9.9.9.11. They use multiple open source DNS software systems to resolve queries. Incoming queries are first triaged by dnsdist from PowerDNS, then routed to either Unbound, PowerDNS Recursor, or BIND 9. Their systems are deployed in 152 POPs around the world; typically these are operated by Packet Clearing House in Internet IX locations. PCH provides Quad9 with co-location, transit, and local hands support.

Who are Quad9’s users?

Quad9 has gotten a lot of coverage in the technical press. They are listed in many FAQs and How-Tos and are easily findable via search engines. However, although they have an Android app, most of Quad9’s users are not private individuals, but are smaller and medium-sized ISPs and enterprises where a single decision can shift a large number of users. John estimated that 85% of their traffic is forwarded to them by ISP or enterprise forwarding caches.

The Quad9 service was established as a non-profit, privacy-protecting open DNS resolver. Their very explicit privacy policy is a big draw for users concerned about privacy. However, John thinks that most of their enterprise users are really attracted by Quad9’s security filtering that blocks malware and phishing sites.

Requirement for EDNS Client-Subnet Identifier

One of their first big users was the City of New York, which is using the Quad9 service for both their free city-wide public wifi and within city agencies. The public wifi project required EDNS Client Subnet Identifier, a DNS feature that uses the end user’s subnet information to return an address that may be customized for that location. This feature was developed for efficient access to local content cached by CDNs. ECS introduces additional privacy concerns, because the end-user subnet information is shared upstream, so Quad9 requires an explicit opt-in for the service.

“We needed a highly robust implementation of ECS, that had a good implementation of the standards, so that was the reason we used BIND.” - Danielle Deibler

Quad9 is running the BIND 9 Subscription Edition, which is a limited-access version of BIND that implements EDNS Client-Subnet Identifier. Queries from users who opt-in to the ECS service are routed to 9.9.9.11, where they are handled by BIND resolvers. ISC recommends that users deploying ECS implement a white list to control where the additional information is included. In the case of the massive Quad9 service, managing a whitelist was too onerous, so they include the ECS information with every query handled by the 9.9.9.11 service.

The main technical reason for whitelisting destinations for EDNS client-subnet information is that, until quite recently, some DNS systems didn’t understand this information; it could cause them to send back errors, or even fail to respond at all. ISC’s Mark Andrews has been lobbying for improved EDNS compatibility in the industry for years, and in 2019, a group of DNS software developers, commercial vendors, and operators collaborated in a joint effort to force a “clean-up” of these non-compliant systems. According to Quad9, DNS Flag Day 2019 helped tremendously with reducing the problems they saw with EDNS compatibility.

Also called their “CDN Friendly” resolver service, Quad9’s 9.9.9.11 service also supports DNSSEC validation. ISC recommends DNSSEC validation as a best practice, so we were glad to hear they are offering this to all their users.

Technical Feedback

We wondered if Quad9 had seen any technical problems with their BIND 9-based ECS service, since the ECS feature is very complex. It is literally designed to increase the resolver cache, because it stores customized responses on a per client-subnet basis. Operating an ECS resolver at a huge scale could potentially consume a lot of resources, so we pressed Quad9 for details. However, although the Quad9 team uses instances with limited memory for the ECS implementations,they haven’t had any problems. They tested the BIND ECS implementation pretty thoroughly before they deployed it and found it to be very solid. (We can’t resist sharing Corey’s compliment below.)

“Your ECS implementation just worked right out of the box. That’s it!" - Corey Mosher

The resolver ECS feature is popular among our BIND support subscribers. Some larger enterprises are using ECS to customize answers for internal DNS on a per-department basis. Another popular application is similar to the Quad9 application, forwarding some traffic to a resolver service with extra security filtering.

Finally, we discussed plans for the future. Quad9 has been running a DNS over TLS service (the new standard for encrypted DNS) since 2017. ISC is in the process of developing this feature in BIND, and is committed to releasing it in 2020. We are interested in benefiting from Quad9’s operational experience, and in collaborating on the continued development of standards for DNS privacy and encryption.

If you think that your organization could benefit from the features of ECS in BIND 9 the way Quad9 does, please contact us for a support subscription quote.

From the Quad9 FAQ

What is EDNS Client-Subnet?

EDNS Client-Subnet is a method that includes components of end-user IP address data in requests that are sent to authoritative DNS servers. This means that there is privacy “leakage” for recursive resolvers that send EDNS Client-Subnet data, where components of the end user’s IP address are transmitted to the remote site. While this is typically used to improve the performance of Content Distribution Networks, we have determined that Client-Subnet data falls into a grey area of personally identifiable information, and we do not transmit that data in our default service. In some circumstances, this may result in suboptimal routing between CDN origins and end users. We do support a secure service that sends Client-Subnet data.

Secure IPv4: 9.9.9.11 Provides: Security blocklist, DNSSEC, EDNS Client-Subnet sent. If your DNS software requires a Secondary IP address, please use the secure secondary address of 149.112.112.11.

Secure IPv6: 2620:fe::11 Provides: Security blocklist, DNSSEC, EDNS Client-Subnet sent. If your DNS software requires a Secondary IP address, please use the secure secondary address of 2620:fe::fe:11.

About the BIND 9 S-Edition

The BIND 9 S-Edition is available to ISC support subscribers only. We provide it in both source code and pre-packaged images. The BIND 9 S-Edition is based on an extended-support version for stability, with added commercial features like ECS from our private repository, and other selected new features and fixes backported from our leading edge branch. For more information about how to subscribe for BIND support from ISC, including the BIND Subscription Edition, contact us at info@isc.org.