An Architectural Perspective on the Interaction between Web Clients and the DNS

There is a fundamental flaw in the way that some web server operators use the DNS to identify the IP address of the web servers to which they connect. This article attempts to explain and justify this assertion.

Until quite recently the website for a domain name would usually be hosted on a server named “www” on that domain. Sometimes the server wasn’t really named that, but a CNAME record sufficed to direct users to the right server, even if that was, strictly speaking, a breach of the semantics of the CNAME record.

These days, however, the corporate marketing folks want to publish “bare” domain names in their URLs. This causes problems if the site is hosted on a third-party CDN because CDNs usually only publish the hostnames and not the IP addresses of their servers; the DNS data model prohibits placing a CNAME at the apex of a domain name (i.e. at the zone cut) because the domain’s NS and SOA records have to go there. This is, however, just a corollary of the fact that you can’t place a CNAME record alongside any other DNS resource record. If I want to use a CNAME record to direct the web content for research.example.com to a CDN, I can’t also place an MX record there.

Back in 2009, the IAB published RFC 5507 “Design Choices When Expanding the DNS”. The title is perhaps misleading, but in general it discusses the issue of the different ways that the DNS can be used to map from a service offered by a domain to the relevant information about that service. In summary there are three practical ways that a service identifier can be used in the DNS:

1. append a prefix to the domain name (e.g. SRV underscore records);

2. put something service specific in the data (e.g. TXT SPF records); or

3. use a service-specific resource record (RR) type.

Generally, the document recommends the latter course. It’s far easier to obtain new RR types than it used to be.

Websites did #1 via their use of the “www” prefix but really did so by accident. It wasn’t part of any standard, but it was a very widely used convention, and it worked. That “www” prefix was the service identifier; it was a pretty blatant hint that the A and AAAA records were there for web use. That prefix meant “direct me to the web service for this domain.” However, when you try to visit a URL without a “www” prefix, there’s no longer a service identifier – your browser treats the domain part of the URL as a hostname but you’re still really looking for a service. The tendency towards prefix-less URLs whilst still requiring lookups for the A or AAAA records of the given domain ignores that fundamental distinction.

My proposal then, is a type #3 service identifier for web use – the HTTP resource record. Its format looks exactly like a CNAME, but it can co-exist alongside other resource records. Think of it as an “MX record for web clients,” albeit without the priority field. It is expected (albeit not mandatory) that when asked for an HTTP record, DNS resolvers would return any corresponding A and AAAA records from cache. If the data isn’t already in cache, resolvers may go ask for it before returning the HTTP answer.

All that said, there are two downsides to my proposal, although the long term result should be a much cleaner architecture that is consistent with RFC 5507 and also solves all of the issues with CNAME.

The first downside is that web clients would have to explicitly ask for the HTTP record, and that means client codebase changes. That said, I think the implementation effort would be significantly less than that needed for a DNS over HTTP (DoH) client, and it’s a one-time hit.

The second downside is that if an HTTP record is returned but the additional A and AAAA records from cache are not, the web client then has to make a second A / AAAA request for the hostname returned by the HTTP record. I expect this to be only a short-term issue, though – most resolver implementers deploy new RR types very quickly after IANA assignment of the code point.

I know this won’t be painless, but I honestly believe that there is no way to fix the apex CNAME issue through magic tricks in recursive or authoritative servers. Doing so puts too much burden on the DNS, and fails to resolve the architectural flaw discussed above. Web clients need to start using an application-specific service identifier in the DNS just like every other Internet service.