EQX Webcast System
Background
The community surrounding Apple is famous for its insatiable appetite for the latest news and rumors about new products. As a result, websites covering Apple news are often inundated with visitors after major stories are published, no more so than during the Keynote presentations made by Steve Jobs during MacWorld San Francisco in January and the Worldwide Developer Conference in June. These keynotes are usually covered by websites providing a transcript updated regularly throughout the keynote
Traditionally, websites have coped with visitors during the keynotes by swapping to text-only pages, using HTML Meta Refresh to reload the page at set intervals, usually every three minutes, for example, MacRumors' coverage from MacWorld in January 2005. As a result, the sites lose their identity and branding, and cannot make best use of the additional traffic for site promotion.
Even with all this simplification the sheer number of visitors can still take the server down. To provide the best coverage during live events an improved system is required.
System Aims
- Support an unlimited number of visitors
- Retain the site's normal layout throughout the keynote
- Provide visitors with seamless updates, more akin to a standard application than a website
- Provide visitors with more frequent updates
- Simplify the administration of updates
System Overview
An overview of the system architecture, click to view full-size version
Basic Design
The aims required a system that could update the transcript without reloading unnecessary objects (i.e. images, CSS & existing transcript text).
The ability to load information into a page without reloading other objects is something that has come to the forefront of web development in recent months thanks to the Javascript function XMLhttprequest and the related methods for manipulating data.
The transcript is split into two parts - when the visitor loads the page any existing transcript text is included in the HTML and then all new updates are delivered via the XML file loaded by Javascript at a set interval - 60 seconds by default. Each line in the transcript is given a unique ID to ensure that it is only included once - when the XML file is loaded the IDs contained in it are checked against the most recent line and are ignored if they have already been included.
This technique requires a browser capable of supporting XMLhttprequest, and obviously Javascript must be enabled. Although the majority of recent browsers have support for this it is not universal, requiring the basic Meta Refresh page to be provided for unsupported browsers.
To select the correct page for the browser, the home page is the basic Meta Refresh version, which will reload the whole page at the set interval. Within this page there is a Javascript function that checks for support of XMLhttprequest - if support is present then it immediately redirects to the Javascript-based page. If no support is present, or if Javascript is disabled, the page will continue to reload using Meta Refresh.
If a non-supported browser should load the Javascript-based page directly they will either be redirected back to the Meta Refresh page (if they have Javascript enabled but are using a non-supported browser) or be warned that the page will not reload.
Administration
The new system requires 3 different files to be updated -
- The Meta Refresh page
- The Javascript page
- The XML file
All 3 files need to be updated simultaneously each time a new line is added to the transcript making manual updates infeasible. A database back-end is used to store the updates, with each line of the transcript being stored as a separate row in the database, together with its unique ID and the time it was posted.
The use of a database provides the solution to another aim of the system - the ability to support an unlimited number of visitors. The required files can be generated on as many servers as required and the servers placed in a load balanced configuration.
The administration interface also provides some configuration options, primarily allowing the administrator to dynamically change the time that clients wait before refreshing the XML file
Infrastructure
As this system is retaining the normal layout of the site, all the necessary objects (CSS, images & Javascript) also need to be loaded. This increases the load placed on the web servers considerably in comparison to the old system, which only had a single text file to display. Even with the ability to use multiple servers, the web server software needs to be improved.
By using PHP to generate the necessary HTML & XML files every ten seconds, rather than each time the file is requested, it is possible to use small & simple web servers to deliver the public site, without the overhead of database usage. Two web server software packages are employed - lighttpd and thttpd - with Apache still being used for the administration interface.
The efficiency of these packages over Apache is considerable and allows quite modest hardware to be used to serve a large number of visitors. Benchmarks of lighttpd compared to various other packages can be found on their website. In our tests, thttpd was able to sustain in excess of 3,300 hits per second, whereas Apache managed only 600 hits per second for 20 seconds, before overloading the server due to memory usage. [1]
The use of these more specialised web servers allows a vast number of simultaneous visitors to be supported - as the system stands at the moment, in excess of 150,000.
IRC
IRC servers have been used for a number of years to provide updates during keynotes. Their ability to relay real-time text to a large number of visitors is well suited to this application.
The database-based nature of this webcast system allows flexible delivery of the updates. A simple script based on SmartIRC was written to automatically deliver updates to the IRC channel without manual intervention.
Implementation
For the MacRumors.com coverage of the 2006 World Wide Developers Conference Keynote this system was deployed at MacRumorsLive.com and employs a redundant group of 6 webservers, 8 IRC servers and administration servers running the MySQL databases. The servers are hosted at The Planet and EV1Servers, both located in Texas, US, and EQSN located in Glasgow, UK. Additional IRC server capacity kindly donated by RnJ.com.
This system successfully delivered live updates of the 2006 Macworld San Francisco Keynote speech to over 100,000 people simultaneously.
About
This site was created by Scotland-based developer Equiknox for MacRumors.com and employs hosting services delivered by Tecknohost.
[1] Test performed using default install of thttpd & Apache on a 1.7Ghz Celeron with 512MB of RAM. Apache could have been improved by optimizing modules installed etc. but the difference would still have been considerable.