The Origins of Web Proxy Auto Discovery (WPAD)
Back in 1997, the Internet and World Wide Web were much earlier in their evolution. Without widespread adoption of DHCP, DNS TXT or SD, discovering proxy servers was a challenge.
For those wondering how the current WPAD solution came to be, I’m one of the original co-authors, and served as Program Manager for WinInet during Internet Explorer 5, where it was first implemented. The DNS Devolution discovery scheme was supposed to be a short-term solution until SVRLOC and DHCP became more widely deployed. 25 years later, it remains unfinished business.
Recently I came across the ACM paper, humorously titled “Waiting Patiently for an Announced Disaster”[1], written by Marc Dacier and Elyssa Boulila, cc’d on this email, the latest in a series of security papers on WPAD vulnerabilities.
Let’s see if we can make change before we get another spanking from the security police 😊
I’ve prepared some thoughts on its origins and suggestions for evolution.
Origins
I started thinking about this while working at UPS, in the early 1990s when we were setting up web browsing from corporate desktops.
In UPS's case, that meant not just IT workers, but also the labor workforce, truck drivers, shipping center supervisors, etc. in various UPS centers. We needed to avoid drowning the IT help desk with calls complaining "The Internet is down!", responding with "No, the Internet is not down. Your proxy settings just aren't configured correctly. Here's how you fix..." We also needed proxy authentication, which didn't exist in HTTP 1.0, but that's another story.
Even for IT workers, when they were at work, they needed to use the corporate proxy, and then when they took their laptops home, they needed to go direct (or at least stop using the corporate proxy which was unavailable). They shouldn't have to manually flip back and forth every day. Or when they went to a meeting at a different company, a WG meeting hosted by another company, or an IETF meeting, they could automatically pick up relevant proxy settings for that network.
WPAD was focused on the discovery mechanism to find the URL for the configuration file. Netscape Navigator 3.0 already had an option for the user to manually configure a URL for a PAC file.
Configuration file formats were JavaScript PAC, and IE configuration files, which were essentially Windows INI files. For those who grimace at JavaScript, it is important to consider that things could have been worse 😊
Initial Proposals
I 1997, I was a developer on the Netscape Proxy Server team. I began drafting a proposal. At the same time, Stuart Kwan @ Microsoft was doing the same based on DHCP.
We coordinated, and I drafted an ID which included the DHCP option as well as SRVLOC, which were the preferred discovery choices, and sent it to the IETF HTTP Working Group aka httpwg list in March 1997.
https://lists.w3.org/Archives/Public/ietf-http-wg/1997JanMar/0686.html
Reading the thread itself also provides insights into the thinking of the other httpwg members. You'll note in the original thread, the discovery proposals were based on SVRLOC and DHCP.
Stuart Kwan’s DHCP ID is:
https://datatracker.ietf.org/doc/html/draft-kwan-proxy-client-conf-00
Others weighed in on scenarios:
Joel N Weber @ MIT stated:
When people at my school screwed the routing so that the machines behind the firewall could talk to the mail server, but not outside machines, I changed about three teacher's machines manually. Those teachers didn't have a clue what a proxy is; and it would be no easier for them to tell the browser to use automagic* setup than it is for them to say to use 204.130.130.62 port 80.
[snip]
I personally would rather see DHCP used instead of DNS I think.
But I need to read the DHCP spec before I comment further...
https://lists.w3.org/Archives/Public/ietf-http-wg/1997JanMar/0698.html
* Refers to pre-WPAD manual PAC configuration in Navigator 3.0
Stuart Kwan @ MSFT stated:
…I unplug my laptop and take it on a visit to Netscape. When I plug into the Netscape corporate network…
https://lists.w3.org/Archives/Public/ietf-http-wg/1997JanMar/0713.html
SVRLOC
My initial preference was to use SVRLOC. However, at the time SVRLOC was new, the WG was still figuring things out, and it was pre-adoption. Furthermore, the teams that would build implementations were far away from deciding what shape their application facing APIs would take.
Josh Cohen stated:
Another fundamental issue here is if you buy into the service location protocol ideas. I do. They are trying to solve the problem of "how do I use a standard method to look up arbitrary services on a network". Granted, by following their DNS recommendations ( which is only one of many ways to advertise services in their world ), its not perfect. In time, I hope that we could support their multicast discovery protocol as well.
I suppose I could have specified that in the draft, but since the rest of the serverloc stuff is so new, and virtually undeployed as of yet, I figured this gives us a solution to a big problem, today, with protocols and APIs software implementors in the realm of the 'world wide web' are commonly using.
https://lists.w3.org/Archives/Public/ietf-http-wg/1997JanMar/0708.html
.
DHCP
DHCP was well-specified, and deployment was increasing, however APIs for applications to ask for random DHCP options were not yet available. Similarly, emerging DHCP server products didn’t always allow an IT admin to configure arbitrary options easily or at all.
Vinod Valloppillil @ MSFT stated:
On windows, for ex., you can look at DHCP options quite easily in the registyr but there aren't API's (yet) for querying RR's from the DNS (unless you want to count on NSLOOKUP output on the commandline)
https://lists.w3.org/Archives/Public/ietf-http-wg/1997JanMar/0712.html
Stuart Kwan @ MSFT stated:
3) It is true that DHCP is not necessarily widely available today, but if anything SRVLOC is less available. The DHCP method at least gives you something you can use now.
4) The fact that there is no cross-platform API to retrieve DHCP options is interesting, but does not block implementation. While the DHCP WG investigates this problem, use the platform-specific method for retrieving options. Please note that there is no cross-platform standard API for retrieving TXT RRs from DNS.
https://lists.w3.org/Archives/Public/ietf-http-wg/1997JanMar/0713.html
Stuart Kwan @ MSFT stated:
I am also not opposed to storing this information in two places. I am only concerned that we solve the automatic configuration problem.
https://lists.w3.org/Archives/Public/ietf-http-wg/1997JanMar/0715.html
DNS
Due to the above constraints, DNS was another discovery option. TXT records were considered a cleaner approach than A records, but they were new and not widely deployed.
Joel N Weber @ MIT stated:
It should be noted that albert.gnu.ai.mit.edu (the mail server that processes all my incoming mail) had a version of named which couldn't handle TXT records up until about a month ago. So even that solution is not really completely compatible with existing sites.
However, if you want to go the name server route, I wonder why you couldn't make the URL something hardcoded like http://www-ns-pac/proxy.ins That would mean that you have to use port 80 with a standardized path, but that's less strange than the TXT or SRVRLOC records.
https://lists.w3.org/Archives/Public/ietf-http-wg/1997JanMar/0714.html
A hardcoded domain name A record and path was the only solution that would always be available, and became the “fallback” case.
CARP/ICP
At the time, we were also unsure of how proxy deployments would evolve for caching efficiency as many people were using dialup, and IP capable smartphones were still in the future. Contemporary efforts were Internet Cache Protocol (ICP)[2], and Cache Array Routing Protocol (CARP)[3].
Josh Cohen stated:
I want to make the point that as caches as being deployed more frequently and with more complexity, ie hierarchies and dynamic ICP type protocols, the configurations need to be dynamic and complex. Well beyond what a nontechnical user should need to know, and extremely difficult to provide a UI to let a user specify it manually.
https://lists.w3.org/Archives/Public/ietf-http-wg/1997JanMar/0705.html
CARP is what we call sharding today. An algorithm would compute a hash of the URL and treat the cache array as a hash table. Requests would be forwarded to the cache for that hash bucket.
ICP is a UDP based protocol between caches that resembles how hierarchical DNS caching works. Most of the functionality was inter-cache via UDP, but a client function could determine the best first-hop in the hierarchy.
We thought these scenarios were important enough to support, and the generalities of JavaScript solved the browser’s side of the problem. One could implement CARP hashing, or ICP bootstrap in JavaScript.
Implementation in IE5
Internet Explorer 5 released Beta 1 in June 1998, Beta 2 in November 1998, and final in March 1999.
At the time the workable discovery solutions were DHCP and DNS A. The implementation prioritized DHCP and fell-back to DNS A.
DNS Devolution
Somewhere along this path, we came up with the DNS Devolution scheme. We were wise enough to know that new TLDs would come, someday, but naïve in thinking it would be a rare occurrence. It’s not like we’re ever going to see
http://ietf.rocks
Oops.
[1] https://dl.acm.org/doi/10.1145/3565361 “Waiting Patiently for an Announced Disaster”, Dacier, Bouilila
[2] https://en.wikipedia.org/wiki/Internet_Cache_Protocol
[3] https://en.wikipedia.org/wiki/Cache_Array_Routing_Protocol