NBN: Click Frenzy's "Click Fail" and Lessons for the Coalition

Just past midnight, as the day turned into 1st of June, 2000, my boss and I packed up and headed home. It had not just been a long day, but a long six months for me: we had just successfully managed the first major Government On-line Transaction, GST Registrations, "over the hump" and through its busiest day, without failing and without making the headlines. We were pleased, our managers relieved.

The lessons from that day apply to Internet Adoption generally and the NBN in particular. Something that the Coalition might take on-board with their NBN policy.



The system specified by the ATO and implemented by Wizard, since liquidated, had failed three iron-clad rules of performance analysis :

• On-line Projects and Critical Systems require Performance Plans as much, or more, than Functional Specifications and Security Plans. This implies you have a trained, experienced Performance Analyst involved. I filled that role when I arrived late onto the project notionally to do Systems Admin.

• What you don't know will kill you: Integral to the System Planning and Design phase is collection demand data, developing a load model and forecasting peak traffic over the Design Period. The Performance Plan, when exposed to open-ended demand, must include Overload Protection: mine was "Busy Tone". Within six weeks of arriving, I'd created a demand model and predicted a peak-load of twenty times their Design Goal: correct to 5% on that busiest day. When the ATO and Wizard did not respond to this forecast or to an early overload event, I designed/specified "Busy Tone" which our managers backed and Wizard grudgingly implemented, delivering untested at the last possible minute. Wizard disregarded a specific design requirement (NO database access for Busy Tone test) and post-event analysis showed the hasty implementation and this disregard almost caused a catastrophic system collapse.

• Your system needs to collect and report critical performance metrics in real-time for adequate Systems Administration, Operations and Troubleshooting/Diagnosis. You cannot manage what you don't measure. I designed and implemented an adequate tool in early January 2000. Whilst it didn't provide automatic alerts and alarms, it was instrumental in detecting and explaining an overload event in February and selling management that morning on a $100,000 system upgrade.
•  It also became Management's Best Friend: during the final three weeks, when demand regularly exceeded our capacity, "Busy Tone" activated when needed, and the managers could see for themselves if the systems they were accountable for, were performing or not. They slept evenings and kept out of my hair, knowing they'd done everything they could and it had worked.

From what I saw reported and screen images, "Click Frenzy" had quite sophisticated Overload Protection, it was active and doing its job perfectly: allowing the systems to usefully work at Maximum Capacity preventing them degrading into "Thrashing" - running at 100% but not completing any useful work. This behaviour was first documented in the 1960's with Virtual Memory and a definitive theoretical description and solution provided by Peter J Denning for his PhD.

What went wrong at "Click Frenzy" was the same as what we saw with the GST Registrations in 2000:

The traffic/demand forecasts were off by an order-of-magnitude (10 times) or more.

This is quite common on the Internet for systems with open-ended demand...
It's part of what it means to be a disruptive technology: you cannot forecast demand.

Think about the Industries that the Internet, as a disruption, has redefined, removed or invented in just the last 10 years:

• Newspapers are closing at an alarming rate. Locally, Fairfax has seen its "Rivers of Gold" go.
• On-line Advertising has taken off, pushing Google into one of the most valuable stocks around.
• Print publishing has radically changed: magazines, news, books, academic publishing, even Encyclopaedia Britannica, have had to change what they do and their business models. 
• Music, Video, Movies and even TV/radio have all been seriously affected by either pirating, free publishing (Youtube etc) or by on-line merchants: iTunes and Amazon.
• Online Retailing is starting to become a major competitor to "Bricks and Mortar" operations.
• "Click Frenzy" demonstrated just how fast adoption of "etailing" is progressing.
• Skype and Tablets have become the default "videophone" - where all calls are free.
• Fixed-line phones and national/international toll-call traffic is down, everywhere.
• Online Banking, Insurance, Markets/Trading and Finance is coming to dominate those sectors, not just changing the businesses cost base, but increasing the speed of transactions and customer control and services.
• Mobile Devices, with 3G/4G/WiFi connectivity, have burgeoned in just the last 5 years, creating Application Stores, displacing Desktop/laptop computers and spawning whole new business models. This trend has years and years to run.
• Education, one of the early adopters of All Things Internet, is now suffering its own Disruption: MOOC's - Massive Online Open Courses.

Since 1996, there has only been one constant with the Growth of the Internet: It's been faster than anyone has predicted. This was the driver behind the "Dot Boom" - and the inability of businesses to "capture the lightening in a bottle", the cause of the "Dot Bust".

NBN Lessons for the Coalition

The On-line GST Registrations, commissioned by the Howard Government, had no Cost/Benefit Analysis. They were a natural extension of a Policy: go on-line.

If they'd run a Cost/Benefit Analysis on the initial forecasts (1,000/day, 20,000 on-line, 1M total), then the $2-3M project would've been shuttered before it began. Instead we saw 20,000/day, 600,000 on-line from a total of 3.3M registrations. At $5/registration, the project was brilliantly cost-effiective.

The replacement system produced within 6 months for $100,000 total, and handling 50-60% of on-going registrations (30,000/mth, IIRC), was even better cost-justified.

Cost/Benefit Analyses (CBA's) are a good, necessary, business tool when properly applied.
If you don't have demand/traffic forecasts that you know to be accurate, then "Garbage In, Garbage Out" applies: the analyses aren't just useless and a waste of time and resources, they are actively counter-productive because they provide an illusion of competency and preparedness: a fools paradise of certainty.

Cost/Benefit Analyses are profoundly useful when evaluating low-growth, high-volume Industries, like Mining. But even there, they have limits. Recently we've seen Twiggy Forrest's "FMG" forced to recapitalise and restructure its finances because of the sudden, but not unexpected peak of the mining boom, whilst Nathan Tinkler's paper empire, based on high Coal prices, is crashing down around his ears. No CBA's would've prevented either of these changes of fortune when externalities intervened. A more potent example is the massive failures in all sectors, especially the CBA-driven finance and investment sector, in the 2008 Global Financial Crisis.
CBA's didn't protect any investors who lost money, in fact the reverse: based on false assumptions and ignoring inherent risk and volatility, they actively duped investors.

This is amply demonstrated by one of Mr Turnbull's own anecdotes: not only did an (entrepreneurial) investment he made fail, the operation went bust within 180 days. If Cost/Benefit Analyses are universally applicable, how could such an astute and seasoned investor as Mr Turnbull get it so wrong? Because the forecasts of revenues and expenses were completely wrong.

The current Coalition, especially Mr Turnbull and Fletcher, should take note of the Howard Government: not everything can, or should, be subject to Cost/Benefit Analyses.

This is the first reason demanding predictive Cost/Benefit Analyses for the NBN is both spurious and fallacious: all we know about future broadband connection demand, market penetration and traffic use is: We Don't Know.

So I disagree with Paul Fletcher's observation that if anyone came to him with a business proposal, like the NBN, without a Cost/Benefit Analyses, he'd fire them on the spot. Exactly because the Internet is still in an unpredictable, explosive growth phase and "disruptive technologies" are emerging all too frequently, the fixed straight-jacket of a Cost/Benefit Analysis is not just wrong, it's the worst justification to bring to the table.
A Business Plan, that's a good idea, and that, not CBA's, is what the best Venture Capital firms in Silicon Valley demand.

By the Turnbull/Fletcher logic, they would not have funded iPods, iPhones, iPads, Google, Android, Skype, Facebook, Ebay, Amazon or the plethora of other hugely successful Internet businesses that are only 5-10 years old.
How did Apple under Steve Jobs become the biggest company on the planet? By understanding Innovation and Entrepreneurship, not through the "respected and well-understood methodology" of Cost/Benefit Analyses demanded as a Fix-Everything by the Coalition.

For National Infrastructure Projects, whilst the input costs are well-know, restrictive definitions of "Benefits" are inappropriate. As the Coalition frequently reminds everyone, the job of Government is not to run Businesses, so Projects, especially basic Infrastructure, cannot be justified or analysed on narrowly defined criteria: only the receipts, as if it were a business.

A Government can justify spending Public Monies on Public/Common Goods, like Infrastructure Projects, even where there is NO direct revenue resulting, but where Social or Economic Benefits do accrue broadly. Otherwise we'd have no Education, Health, Roads or Defence spending.

The ALP, quite correctly, in their 2007 pre-election NBN Policy document, laid out the best evidence for the Economic and Social Benefits that the Australian Economy would gain from Universal Broadband access to the Internet. That justification was both necessary and sufficient for proceeding with an NBN: the gains in the National Interest, the benefit to the Australian Economy, would dwarf any costs incurred by the Government.

Mr Turnbull himself now opines that "very fast broadband [is] a bedrock requirement for a successful digital economy. And a successful economy, full stop." Whilst glib and "self-evident", Mr Turnbull has failed to provide any reasonable evidence for this assertion. Perhaps his reliance on intuition or "business acumen" and the exactly wrong financial justifications, like CBA's, is why he complains of at least one spectacular entrepreneurial investment failure.

The Coalition, in Abbott/Turnbull/Fletcher, don't just not understand the difference in nature between, and tools needed to properly analyse, Business Projects and Government Infrastructure, they seem to not understand the wide set of tools needed to examine and justify unpredictable, high-growth sectors like the Internet and its supporting access infrastructure, an NBN.

Lastly, Paul Budde has recently raised an essential role that neither the ALP nor Coalition has discussed about the NBN:

Multi-Factorial Realised Cost/Benefits Studies

The whole point of investing in, and building, any National Core Infrastructure is not "because", but that there are tangible National Benefits to be realised.
If those projected Benefits are not actively tracked, isn't that a failure of Public Governance and Accountability?

If nobody is watching and Accounting for the Benefits:

• How can we, the taxpaying public funding it all, know if XYZ Project (Roads, Health, NBN, ...) was a good idea at all?
• If the Benefits, Economic, Social, Environmental and Human-Capital, are not measured, they cannot be managed.
• How do we, the taxpayers, or the Government know if too little has been invested? If the project is under-capitalised?
• How can we know if too much is being spent and returns are low or Nil or additional services or markets?
• How can we know if niche markets are under/over served, or if valuable markets have emerged that are being ignored?
• How can we know if services are priced too high, preventing adoption, critical-mass and full available benefits flowing to the Economy? Governments have always had a "pump priming" role of temporary subsides to invigorate new markets.
• How can we know if any of our original assumptions and project goals were Not Quite Right and need to be redefined, or "tweaked"?
• How can we know if something better has come along and we need to scale-back our plans or new, disruptive technologies/services/products (like the iPhone/iPad), have completely changed the Rules of the Game, and we need to scale-up our investment and bring-forward goals to realise available economic and social benefits?

Both the Coalition have neglected the most essential part of any Project: tracking and review.

Because the National Benefits accruing from any NBN are multi-faceted, not just economic, they are far broader than merely the receipts for services collected by NBN Co. Any Realised Cost/Benefit Study of an NBN must look much more widely than the revenues of NBN Co and its RSP's.

To adequately address this on-going role, a permanent authority, like the Audit Office, needs to exist to independently and objectively evaluate the benefits flowing from Government Expenditure. It needs to combine the hard-headed accounting of the Audit Office with the Economic Research, Modelling and Evaluation of the Productivity Commission and the objective data gathering of the Bureau of Statistics.

If the Coalition supported Realised Cost/Benefit Analyses, for all Government Infrastructure, they would both understand the nature of the problem and the necessary tools needed for objective evaluation. This would be a transformation of Australian Government processes far greater and long-lasting than even the NBN.

It answers the central Taxpayer Question: What did we just get for our tax-dollar?

It's a question that can only be asked in hindsight, not with false forecasts of Costs and Benefits.