What the ABS can learn about online humility
Last night’s #Censusfail debacle provided many lessons for organisations in how to manage large scale online events, says Paul Wallbank, not the least of which is to never promise what you can’t deliver.
As it turned out, reliability of the site on the night turned out to be the big communications challenge for the ABS because it turned out it wasn’t particularly well prepared as hubris and overconfidence took over from reality and caution.
The ABS told Fairfax last week it was confident there would not be an internet meltdown. When asked about the ability of the online Census database to cope with such high traffic numbers, an ABS spokesman said online could handle “1,000,000 form submissions every hour. That’s twice the capacity we expect to need.”
Catering for a million submissions an hour capacity wasn’t enough and the ABS’ communications team was caught unprepared with its responses as people complained on Twitter about the website being unavailable, and for responses being a template “online form and website are operating smoothly as expected. Please try again.”
The big questions:
Was there really a DDos attack or is that just arse covering?
If there was, is the data entered before the attack now out in the wild?
DDos attacks are just greatly increasing the amount of queries to load the website, overwhelming the server hosting the website with traffic, causing the website to crash. Doesn’t steal data, I believe (not an IT expert, could be wrong).
@AJ – A DDoS doesn’t even need to crash the server to take out the website. It might just saturate one or more internet links which lead to the server, thereby taking the site offline either for everyone or a subset of visitors.
Come on anonymous. A DDOS attack isn’t a compromised site.
It literally refers to an attack that prevents a site from talking to genuine users by flooding it with fake requests. It’s like a thousand people standing in the bank and yelling at the teller to stop them working. it’s loud and nothing gets done, but the vault is entirely secure.
Thanks for the insights on dos attacks, I based my second question on a quote from the ABS Chief:
“But the site was shut down after a “gap” in the system’s security measures was found during a fourth attack, Mr Kalisch said.”
Perhaps the teller left the key in the outside of the door when she unlocked the bank?
How would data be exposed by a DDoS attack? Sounds like you have no idea what a DDoS attack is…
How is data compromised ?
Because a DDOS attack overwhelms the system and allows a routine to create a way into the system while it tries to cope with the attack.
This is in fact the purpose of a DDOS attack.
The system didn’t break down from the attack though, it was closed down in response to a large volume of packets from an apparently similar source. It’s quite likely that the packets where legitimate and simply routed through a foreign server when the local ones became saturated.
IBM’s software is junk in other words.
This is so badly written – do you have an editor or did they fail last night as well?
Visnja,
badly written?? You response displays a striking similarity to the failed, arrogant traits that Paul has alluded to. You need to go to Specsavers…. on your way to your job at ABS HQ.
This is one of the better articles on the census fail i have read.
The problem is it’s written by just another blogger who thinks he knows better.
The Census failed we get it. Government stuffs up; what else is new? The megaphone in an echo chamber giving us just another headache.
An even bigger problem was the failure of ABS this morning to describe what was happening.
No-one was available to comment on breakfast TV so the networks just ran with their own commentators, most of whom spoke of “hacking” and idly speculated that “maybe the scare campaign was right” with no alternative perspective. This just reinforced the community concerns about security of information and will probably compound non-participation once the site is back up and running.
Surely ABS should have had a team of co-ordinated spokespeople hitting every media outlet from 5am, not just saying “We’ll give a press conference at 9.”
Why do bureaucrats (and politicians, for that matter) never learn that they should NOT dabble or meddle in areas requiring expertise that is way beyond their own experience and qualifications? Major fails like this one just push their credibility through the floor.
The bureaucrats didn’t dabble in it, they paid IBM $9m+ create this catastrophe.
It’s called the Dunning-Kruger effect. Look for it on your favourite Wikipedia site.
Essentially, it’s when you don’t know enough to even realise that you’re out of your depth.
AKA “talking out of the wrong orifice”
Unfortunately, they’ve been getting lessons in government from Donald Trump.
What do they care? Just fine people if they don’t complete them (which they have broadcasted by loudspeaker, although now I notice they are saying no fines due to their website being down, so I’ll give them that), punish wilfully, no need to learn, you’ve got the power, give me your details about your life, we’ll keep them, all ours, gotta give it to us etc etc etc.
Learn? Who’s going to make them!
#DoItAgainOnAugust10
If we assume there really was an overseas DDoS attack (and there is DDoS attack traffic monitoring evidence on Twitter to suggest there wasn’t), considering this is a survey on Australian households I don’t understand why they didn’t scrub overseas traffic before it even hit Australia. That seems like a no-brainer.
Perhaps I should tender for the $9m+ contract next time!
It really IS an exciting time to be an Australian.
Well… that escalated quickly ?
They claim that they thought they could handle “1,000,000 form submissions every hour. That’s twice the capacity we expect to need.” says it all. Some pretty basic analysis suggests maybe 7 million households on the East coast would try to complete between 7pm and 9pm – that’s an average load of over three times their estimated load, and it would be likely to spike to at least twice that again through normal traffic fluctuations – and any competent web traffic analyst should have been able to tell them that.
The DDOS explanation sounds dodgy to me – The normal load you would have expected would have been 6-10 times what they planned for.
Chris, I agree completely. They pretty quickly added to the story that Defence Signals Directorate weren’t able to determine the source of these four attacks. This story started to change as the morning went on, though. I can imagine that DSD weren’t very happy about being made to look incompetent. There would’ve been some hastily arranged phone conversations between those two organisations.
BBC’s digitisation project went wrong. Boss got fired.
BA’s move into Heathrow Terminal 5 went wrong. Boss got fired.
Abs #censusfail…
You know that old phrase nobody can get fired for using IBM?
It’s going to be interesting to find out if that’s true.
Denial of service (DDOS) attacks have been known to be used as a way of masking other activities. While IT staff are focused on scrambling to keep their service safe and reliable from one side, the real intention is going on where they are not focused.
Although I doubt a large offshore DDOS attack was the culprit as statistics for the period show low level activity towards the domain from anywhere outside Australian IP ranges, there was definitely some sort site penetration that led to whole service being taken offline by the ABS and IBM manually, at very short notice.
I’m amused by the many ‘official’ statements that the DDos “didn’t succeed” (assuming it really *was* a DDoS, not merely an overloaded server struggling to cope with a load far greater than it was designed for).
Yes, it DID SUCCEED. A DDoS is intended to do nothing more than stop people accessing a web site. Due to the combined efforts of the overload AND the on-site staff, people were unable to reach the census web site.
Clearly, the DDoS succeeded.
Whether there were other factors at play (such as a misdirection) we don’t yet know. But I remain very nervous about the statements that no data was taken… seems very ‘Dunning Kruger” to me (see comment above).