EQX Webcast System
The community surrounding Apple is famous for its insatiable appetite for the latest news and rumors about new products. As a result, websites covering Apple news are often inundated with visitors after major stories are published, no more so than during the Keynote presentations made by Steve Jobs during MacWorld San Francisco in January and the Worldwide Developer Conference in June. These keynotes are usually covered by websites providing a transcript updated regularly throughout the keynote
Traditionally, websites have coped with visitors during the keynotes by swapping to text-only pages, using HTML Meta Refresh to reload the page at set intervals, usually every three minutes, for example, MacRumors' coverage from MacWorld in January 2005. As a result, the sites lose their identity and branding, and cannot make best use of the additional traffic for site promotion.
Even with all this simplification the sheer number of visitors can still take the server down. To provide the best coverage during live events an improved system is required.
An overview of the system architecture, click to view full-size version
The aims required a system that could update the transcript without reloading unnecessary objects (i.e. images, CSS & existing transcript text).
The new system requires 3 different files to be updated -
All 3 files need to be updated simultaneously each time a new line is added to the transcript making manual updates infeasible. A database back-end is used to store the updates, with each line of the transcript being stored as a separate row in the database, together with its unique ID and the time it was posted.
The use of a database provides the solution to another aim of the system - the ability to support an unlimited number of visitors. The required files can be generated on as many servers as required and the servers placed in a load balanced configuration.
The administration interface also provides some configuration options, primarily allowing the administrator to dynamically change the time that clients wait before refreshing the XML file
By using PHP to generate the necessary HTML & XML files every ten seconds, rather than each time the file is requested, it is possible to use small & simple web servers to deliver the public site, without the overhead of database usage. Two web server software packages are employed - lighttpd and thttpd - with Apache still being used for the administration interface.
The efficiency of these packages over Apache is considerable and allows quite modest hardware to be used to serve a large number of visitors. Benchmarks of lighttpd compared to various other packages can be found on their website. In our tests, thttpd was able to sustain in excess of 3,300 hits per second, whereas Apache managed only 600 hits per second for 20 seconds, before overloading the server due to memory usage. 
The use of these more specialised web servers allows a vast number of simultaneous visitors to be supported - as the system stands at the moment, in excess of 150,000.
IRC servers have been used for a number of years to provide updates during keynotes. Their ability to relay real-time text to a large number of visitors is well suited to this application.
The database-based nature of this webcast system allows flexible delivery of the updates. A simple script based on SmartIRC was written to automatically deliver updates to the IRC channel without manual intervention.
For the MacRumors.com coverage of the 2006 World Wide Developers Conference Keynote this system was deployed at MacRumorsLive.com and employs a redundant group of 6 webservers, 8 IRC servers and administration servers running the MySQL databases. The servers are hosted at The Planet and EV1Servers, both located in Texas, US, and EQSN located in Glasgow, UK. Additional IRC server capacity kindly donated by RnJ.com.
This system successfully delivered live updates of the 2006 Macworld San Francisco Keynote speech to over 100,000 people simultaneously.
 Test performed using default install of thttpd & Apache on a 1.7Ghz Celeron with 512MB of RAM. Apache could have been improved by optimizing modules installed etc. but the difference would still have been considerable.