|
Case Studies
Active/Active Payment Processing at
Swedbank
(3,1) Swedbank uses active/active Base24 to support credit cards and
POS terminals.
agileTel Runs Active/Active with
Linux and MySQL (8,10) Two data centers for VoIP are
kept synchronized with Tungsten data replication.
Amazon S3 Storage Taken Down
by Fat Finger (12,3) A command to remove a small
number of servers was erroneously entered.
Apollo 11 - Continuous
Availability, 1960s Style (4,9) NASA's safety-critical
computer systems put men on the moon four decades ago.
Asymmetric Active/Active
at Banco de Credito (2,11) Using an symmetric
configuration saves programming changes.
Bank Chooses
"Sizzling-Hot-Takeover" Data Replication for its BASE24 Business
Continuity Solution (11,4) BoV
uses Shadowbase.
Bank-Verlag - the Active/Active Pioneer
(1,3)
Bank-Verlag went active/active two decades ago with IBM/Tandem.
Bank-Verlag - An Update
(5,8)
Bank-Verlag replicates processed transactions in its active/active system.
BANKSERV Goes Active/Active (2,4) A banking
switching service in South Africa moves Base24 into active/active.
Banks Use Synchronous
Replication for Zero RPO (5,2) Triplexed data centers give fast
recovery time with zero data loss.
Can Applications Achieve Seven
Nines of Availability? (13,3) Software can, and active/active hardware
systems can.
Casa Ley Upgrades to Active/Active
OmniPayments (8,1) One of Mexico's largest grocery
chains installs active/active financial transaction switch.
Cellular Provider Goes Active/Active
for Prepaid Calls (3,9) NonStop active/active system
keeps prepaid calls moving in Africa.
Commerzbank Survives 9/11 with
OpenVMS Clusters (4,7) With an active/active backup 30
miles away, getting their people there did it.
Community College Learns From SAN
Disaster (2,2) A disastrous SAN failure leads to dual
redundancy.
Controlling Amtrak Trains in the
Northeast Corridor (11,11) The Amtrak train control system
used Tandems and was developed by SAI.
CPA at Aqueduct, Belmont, and Saratoga (2,1) Race track wagering
can never fail, or else riots start.
Do You Know
Where Your Train Is? (1,1) A transit authority goes active/active
for train tracking.
Dr. Timothy Chou Keynote Speaker at 2016
NSTBC on the Internet of Things (12,2) His talk
focused on things that are the backbone of the planet.
European Bank's Active/Active ATM
Network (4,6) In this active/active system, ATM
failover via DNS rerouting has its problems.
Faster Payments - Bringing Payment
Processing Into the 21st Century (5,6) VocaLink
uses active/active to provide 24x7 real-time payments.
Going Prices for Personal
Information (9,1) Did you know that your date of birth is worth
$11 to hackers?
Google and Symantec Spar Over
Digital Certificates (12,4) Google looses
faith in Symantec over the issuance of improper digital certificates.
Grocery Store Chain Achieves
Continuous Availability with OmniReplicator (10,1) Running
Active/Active ensures operation during server failures.
Handelsbanken Turns to Parallel
Sysplex (4,10) Sweden's Handelsbanken goes active/active to
protect its online banking and ATM network.
High Availability, 1970s
Style (11,10)
MiniData Services, a small payroll processor, used dual processors back in
the 70s to ensure continued operation.
Hospitals Under
Ransomware Attacks (11,7) Hospitals are paying thousands
of dollars to gain access to maliciously encrypted files.
How
Does Google Do It? (3,2) Google processes tens of
gigabytes of data in minutes on their massive clusters.
How Does Google Do It
(part 2) (7,11) Google exposes how it distributes massive
applications across thousands of servers while saving energy.
How the Ukraine Power Grid Was
Hacked (11,3) Hackers disabled power for 230,000 Ukrainian
customers for six hours in December 2015.
HP's Active/Active Home Location Register (1,2) The
brains of a cellular network can never go down.
HP's OpenCall INS Goes Active/Active
(2,6)
Replication lets OpenCall INS run active/active with collision detection and
resolution.
HPE Spins Off Software Assets
to Micro Focus (11,9) HPE spin-merges its non-core
software assets to Micro Focus.
Large Hadron Collider Running Out of
Disk (11,10)
The LHC has been more reliable than estimated, doubling its generation of
data.
Lloyds Banking Group Outage
- A Correction (12,01) The 2014 Lloyd's Bank outage was caused by
multiple CPU failures in a NonStop system.
Major Bank Uses Active/Active
to Avoid Hurricanes (2,10) Fast failover is used to switch
users out of hurricane path.
Major ISP Migrates from Sybase to
NonStop with No Downtime (3,11) Hundreds of millions of
accounts migrated and verified.
Major U.S. Bank Replaces BASE24
(7,6)
Opsol's OmniPayments keeps bank's NonStop systems running after ACI's sunset
announcement.
Payment Authorization - A Journey
from DR to Active/Active (2,12) A start with DR
leads this company to active/active and application integration.
QEI Provides Active/Active SCADA with
OpenVMS
(2,9) Electrical substation monitoring that never goes down.
RBS - A Poster Child for Outages
(11,1)
RBS and its associated banks, NatWest and Ulster Bank, have had six outages
in the last three and a half years.
Real-Time Fraud Detection
(4,12)
Credit-card switching service catches fraud on-the-fly - a great example of
real-time business information.
Ring-of-Fire Bank Beats
Earthquakes with Active/Active (7,7) Moving
from tape backup to active/active can be a series of controlled steps.
Royal Bank of Canada Goes Active/Active
for ATM/POS (6,7) Eliminates planned outages and reduces
unplanned downtime from hours to minutes.
Stock
Exchange Speeds Clearing with Data Replication (9,6) Manual
entry of trades is eliminated to provide same-day clearing.
Tackling Switchover
Times
(1,1)
If active/active is too big a step to take now, work on reducing your
switchover times.
The Bogus Missile Launch (13,1)
Hawaiian residents received a warning that a missile was headed their
way. False alarm.
The End of Custom Software
(12,11)
The need for custom software has passed. Existing software packages exist for
almost any need.
Telecom Italia's Active/Active Mobile Service (2,3)
Italy's biggest cell phone network is supported by active/active.
Tour Operator Optimizes
Look-to-Book Ratio (6,8) An asymmetric active/active
system unloads query processing from the OLTP master node.
Try Doing This Today
(8,8)
In the early days of computing, we built major systems with kilobytes of
memory and megahertz processor speeds.
UK National Health Service - Blood and
Transplant
(3,10) An OpenVMS split-site cluster guarantees the availability
of the UK's blood supply.
U.S. Bank Critiques Active/Active
(4,5) A NonStop active/active user shares experience and advice
to those who would follow.
Wells Fargo's Pioneering
Active/Active ATM Network (5,9) Dual networks with ATM
collocation provides continuous availability.
Yahoo!'s Users Are Victims of Massive
Hacks (12,5)
Data from one and a half billion accounts have been stolen from Yahoo.
Never Again
123-Reg Deletes Hundreds of
its Hosted Websites (11,4) A
maintenance-script error deletes virtual private servers hosting websites
from its network.
160516 (11,6) A
NonStop date routine bug laid dormant for over thirty years, and then struck
on May 16, 2016.
1961 - North Carolina Nuked (Almost) (9,2) A
B-52 carrying 400-megaton hydrogen bombs disintegrates, and three out of four
safety mechanisms fail.
911
Service Down for Six Hours Due to a Software Bug (9,11) The entire
state of Washington and seven other states had no 911 service.
$280 Million in
Cryptocurrency Lost Due to Bug (12,11) The
money was destroyed by bugs in a popular digital wallet service.
A Massive Attack on the U.S.
Government (10,6) Chinese hackers steal personal information
of over four-million government employees.
A Personal Failover Fault
(8,3)
- A failed PC forces a system recovery during a slide presentation. Failover
failed. The BCP saved the day.
Active/Active
Save #1 - Coffee Pot Takes Down Node (1,2) When the
coffee pot was plugged in - Surprise!
Amazon Christmas Present - Buy
for a Pence (10,1) A glitch in a price-setting utility reduced
many items on Amazon to one pence.
Amazon's Cloud Downed by Fat
Finger (6,5) A maintenance technician's error takes down
an entire Availability Zone for four days.
Amazon Downed by Memory
Leak (7,11)
A memory leak in an innocuous program cascades into major systems,
taking down the AWS cloud for hours.
American Airlines Grounded by iPad
Glitch (10,5) Pilots could not access charts stored on
iPads and had to return to the gates.
American Eagle's Eight-Day
Outage (5,9) Lack of recovery and failover testing takes
down online sales for the $3 billion retailer for over a week.
Anthem Loses 80 Million Records
to Hackers (10,3) Access was gained to the second largest
health insurer in the U.S. by stolen credentials.
Ashley Madison Cheats Exposed
(10,8)
The identifications of those looking for an extra-marital affair are now
posted for all to see.
Australia's Painful
Banking Outages (7,3) Australia's four major banks suffer
multiple outages as they upgrade their aging infrastructure.
Australia's Telstra Downed By
Fat Finger (11,3) A human error dropped mobile and home phone
services for 1.5 million customers for four hours.
Bank of England Suffers Major
Outage (10,4) After warning British banks that their core
legacy systems are not robust, BoE suffers its own outage.
BlackBerry Gets Juiced (2,5)
Poor testing leads to no service for North American subscribers.
BlackBerry Messenger
Down for Days (6,10) RIMs BBM texting service went
down for over four days when it suffered a failover fault.
BlackBerry Takes Another Dive (3,3)
Deja Vu. Poor testing once again leads to no North American service.
BlackBerry - OMG, it's Déjà Vue (5,1)
BlackBerry has now accumulated seven major outages in five years, providing
an availability of three 9s.
British Airways Downed by Fat
Finger (12,6)
A technician turns off the UPS in a BA data center, taking the airline down
for a weekend.
Cascading Software Bugs Take
Down Google Compute Engine (11,4) Google's GCE takes a
worldwide hit from three software bugs.
Commonwealth
Bank of Australia - A Correction (7,4) We make a
correction to our article entitled, "Australia's Painful Banking
Outages."
Console Command Takes
Down Active/Active System (1,3) Stop applications on one node,
stop other node. Oops!
Crimea Loses Power for Two Weeks
(10,12)
Ukrainian saboteurs destroy the power feeds delivering electricity to Crimea
from Ukraine.
DDoS Attacks on U.S. Banks
Continue (8,2) Islamic hactivists resume their attacks
to get blasphemous video removed from the Internet,
Delta Air Lines Cancels 2,100 Flights
Due to Power Outage (11,9) A fire in Delta's power room
takes down its data center.
Do the Russians Have Your Tax
Returns? (10,8) The tax returns of over 300,000 Americans
have been stolen from an IRS web site.
Don't Wait for the Other Shoe to Drop (2,2)
When a spare component fails, fix it fast. Don't tempt Murphy.
eBay's Slow Response to Data Hack
(9,7)
After learning that its user database had been hacked, it took eBay two weeks
to send out notifications.
Facebook Suffers
Self-Inflicted Outage (10,3) This outage was the worst
Facebook outage in four years, but Facebook still has great uptime.
FedEx Exposes Details on Thousands (13,2)
An Amazon S3 server set for public access contained passports and
drivers’ licenses of customers.
Fire Knocks Out Samsung (9,7) Why did a
fire in Samsung's backup data center take down its production data center?
Fire Suppression Suppresses WestHost
for Days (5,5) Never test a fire suppression system by
triggering it.
Fire Takes Down Atlanta
Airport (12,12) A fire in the tunnel carrying both the
primary power and the backup power feeds destroyed all power.
First Stuxnet - Now the Flame
Virus (7,6)
Deemed more serious than Stuxnet, Flame takes over PCs and listens in on
conversations.
Gatwick
Airport Reverts to Whiteboards (13,8)
A broken fiber cable prevented airport displays from working, forcing
Gatwick to post gate departures on whiteboards.
GitLab Suffers Massive Backup Failure
Due to a Fat Finger (12,4) A technician accidentally
deleted 300 GB of data from a primary database.
Go Daddy Takes Down Millions of Web
Sites (7,9)
Any domain registered with Go Daddy was downed by a DNS network failure.
Google Troubles - A Case Study
in Cloud Computing (4,10) Even the 900-pound Gorilla can have
problems keeping its services up.
Hacked AP Tweet Crashes Markets
(8,5)
Phony AP tweet reporting that Obama was injured in an explosion crashes
markets in seconds.
Haiti's Cell-Phone Network
Costs Lives (5,3) Following Haiti's disastrous earthquake, many
people couldn't call for help from beneath the rubble.
Happy Valentine's Day - But
No Flowers (10,3) Global Payments financial-transaction
network went down for a day and a half over Valentine's.
Has Gmail Become Gfail? (4,3) Google's Gmail
service has been down for hours six times over the last eight months.
Heartbleed - The Worst
Vulnerability Ever (9,4) A flaw in OpenSSL allows hackers
to get private keys, user names, and passwords.
History's Largest DDoS Attack?
(8,4)
Spam black-lister Spamhaus is taken down for days by a disgruntled spammer
via a massive DDoS attack.
Hostway's Web Hosting Service Goes
Down for Days (2,9) Small online stores offline for
up to a week.
How 'Fat' Are Your Fingers?
(12,7)
Human errors have taken down many systems. Many of these outages are
described herein.
How Many 9s in Amazon? (3,7)
Even giants fall. Amazon's S3 and EC2 services and online retail store go
offline for hours.
Hubble Trouble (4,1) A
failover fault when recovering from an instrument controller failure almost
loses Hubble.
Hurricane Irma Causes
Massive Power Outages (12,9) Over half of Florida residents
and most Puerto Rican residents lose power.
Hurricane Sandy (7,12) 2012's
Hurricane Sandy flooded lower Manhattan, taking out tens of thousands of web
sites for weeks.
Innocuous Fault Leads to Weeks
of Recovery (3,12) A simple disk mirror failure propagates into
weeks of recovering lost data for a major bank.
Iowa Data Center Taken Down
by Fire (9,4) The State of Iowa CIO gives a detailed
description of the data-center recovery from the fire.
IRS Goof Costs U.S. Taxpayers
$300m + (2,1) Turning off the old system before testing the
new one is dumb.
ISIS Turns to Cyber Warfare
(10,4)
ISIS has opened a new front on its war with the West, hacking websites,
Facebook pages, and Twitter accounts.
Islamic Hacktivists Attack U.S.
Banks (7,10)
Several banks taken down for a day in protest of YouTube video,
"Innocence of Muslims."
Joyent Cloud Downed by Administrator
(9,6)
A mistyped command reboots all servers in the Joyent cloud.
JPMC Three-Day Outage Caused by
Replication Corruption (5,11) Corruption of primary SAN by
Oracle bug also takes down standby SAN.
Knight Capital Destroyed by Software
Bug (7,8)
A high-frequency trading bug costs Knight $440 million, forcing it to sell
out to a consortium.
Lightning Downs
Amazon - Not! (6,9) An Amazon European Availability
Zone is taken down by hardware, software, and human faults.
London Stock Exchange
PC-Trading System Down for a Day (3,10) Traders fume at
commission loss on one of the most hectic trading days.
Malicious Apple Chargers
(8,8)
Researchers find a way to infect iPhones and iPads via the USB charging port.
Malware Attacks Apple Apps
(10,10)
Chinese developers use infected Apple utility to create apps sold in the
Chinese iOS App Store.
Marketo Goes Down, Revived By
Customer (12,8) Marketo forgot to renew its domain, but a
customer renewed it for them.
Medical Center's Multiday
Outage (6,2) An attempt to achieve high availability on a
limited budget leads to non-availability.
Meltdown and Spectre
Security Threats (13,1) These threats steal data from a CPU’s cache
left there by speculative execution.
Metro-North Taken Down by
Redundant Power Failures (8,10) The primary power cable fails
while the other one is being upgraded.
Microsoft's Azure Cloud Goes
Down - Again (9,12) A faulty upgrade and
improper deployment brought down Azure for eleven hours.
Military GPS Disabled by Upgrade
(5,6)
A failed upgrade to support the next GPS generation takes down 10,000 military
GPS receivers.
Mizuho Bank Down for Ten Days (6,6) A
flood of earthquake donations by mobile phones overwhelmed the bank's evening
batch runs.
More Never Agains
(3,8)
Over two dozen disastrous outages for the first half of 2008 are recounted.
More Never Agains II
(4,2)
System downtime problems have moved from the power lines to the networks.
More Never Agains III
(4,7)
Add the cloud to power and network problems creating over two dozens outages on which we report.
More Never Agains IV
(5,2)
Network, hardware/software problems highlight outages for the last half of
2009.
More Never Agains V
(5,7)
Over half of our 30 horror stories took down hosting providers. A failover
plan is a must.
More Never Agains VI
(7,4)
Software bugs and recovery faults highlighted the outages in the first quarter
of 2012.
More Never Agains VII
(7,9)
Power outages were the main cause of these failures.
More Never Agains VIII
(8,2)
Security threats are becoming more prevalent, with the Chinese evidently
leading the charge.
More Never Agains IX
(8,7)
Environmental faults lead the list, followed by updates gone awry.
Mt. Gox, Largest Bitcoin Exchange,
Goes Belly Up (9,3) Hackers steal 855,000 bitcoins
worth over $500 million, forcing Mt. Gox into bankruptcy.
Nasdaq Taken Down by Software Flaw
(8,9)
A blast of messages from the NYSE disables Nasdaq's quote reporting system
and causes a failover fault.
National Australia
Bank Customers Down for Days (5,12) A bad batch update
disables critical customer services for two weeks.
Network Crashes (12,10)
Networks have not always been as robust as they are now. ARPANET had several
failures in the late 1900s.
New York City's New 911 System Goes
Down Four Times (8,6) After extensive testing, the city
lost 911 service four times in two days.
Northern Virginia's 911 Service
Down for Four Days (7,12) 90 mph winds and air in the
generator's fuel lines takes down a Verizon 911 hub.
On-Demand Software Utility Hits
Availability Bump (2,10) A utility is expected to be
always up, but this one didn't make it.
Our Power Grid Must Be
Reliable and Resilient (12,9) Reliable power is power without
interruption. Resilient power is quick restoration of power.
Oracle's Ticking Time Bomb
(7,2)
An obscure bug in the Oracle database could take down an entire data center
if not patched immediately.
Orca - The Outage That May Change
History (7,11) The Republican 2012 Get-Out-The-Vote system
flopped from the beginning.
PayPal Fault Takes Merchants Offline (4,9) A network
fault forces small online merchants to close shop for hours.
Poor Documentation Snags
Google (5,4) A data center goes down, and failover to the
backup data center goes awry.
Pop! Goes the Weasel! (11,6) A weasel
gnaws through a high voltage line and takes down the CERN Large Hadron
Collider.
Rackspace - Another Hosting Service
Bites the Dust (2,12) A truck driver wipes out web sites
for a day or more.
Royal Bank of Scotland Offline for Two
Weeks (7,7)
Falling back from a failed upgrade failed and took three U.K. banks down.
Royal Bank of Scotland
Suffers Multiple Outages (8,12) RBS, NatWest, and Ulster fail
three times in eighteen months due to aging systems.
RushCards Deny Funds to the Unbanked
(10,11)
Prepaid debit cards upon which may depend went inactive for ten days due to a
technical glitch.
Salesforce Takes a Dive (11.7) A
hardware bug followed by a firmware bug takes Salesforce down for six days.
Shellshock - The Bash Vulnerability
(9,10)
A critical code-injection vulnerability has been discovered in the popular
Bash shell
Sidekick: Your Data is in 'Danger' (4,11) A
million smart-phone users lose all of their contacts, calendars, and photos.
Singapore Bank Downed
by IBM Error (5,8) An undocumented new procedure
takes down all DBS Bank systems for hours.
Skype Holiday Present - Down for a Day
(6,1)
Skype overload takes down its peer-to-peer network of hundreds of thousands
of supernodes.
So You Think
Your System is Robust? (2,8) So did these major
enterprises, all of which went down in the first six months of 2007.
So You
Think Your System is Reliable (3,1) Horror stories from the
second half of 2007 focus on power and branch failures.
Software Bug
Causes Train Wreck (1,1) A software bug, controller
diversion, and engineer inattention combine to cause a train collision.
Sony PlayStation Taken Down for
Weeks by Hackers (6,5) Hackers steal 100 million accounts
from Sony, requiring weeks to repair security defenses.
Southwest Airlines' Router
Grounds 2,300 Flights (11,8) A router failed in an unusual
way, and the outage was not recognized by its backup.
Spamhaus Attacker
Caught (8,5) The mastermind behind the ten-day 300 gbps attack
on Spamhaus arrested in Spain and extradited to The Netherlands.
Stuxnet - The World's First
Cyberweapon (6,3) Stuxnet is the first worm to attack a control
system and destroy machinery.
Sydney's M5 Tunnel Closed Again by
Computer Glitch (3,11) Six times in six years is too
much for New South Wales.
Target Compromises Millions of Payment
Cards (9,1)
The magnetic stripe data on payment cards is stolen from 40 (updated to
110) million cards.
The $45 Million ATM
Heist (8,5)
Hackers clone prepaid debit cards and net $45 million in ten hours from ATMs
around the world.
The Alaska Permanent Fund and the $38
Billion Keystroke (2,4) What do you do when your active and
backup disks are wiped out and your tapes won't read?
The Case of the
Flying Cable
(1,1)
A technician loses control of an under-floor cable and lets it hit a power
strip.
The FAA's Availability Woes
(4,12)
Application and network failures plague air travelers. Where is NextGen - the
next generation airspace system?
The Government OPM Hack Gets
Worse (10,7)
OPM now acknowledges that personal information belonging to 20 million
Americans was stolen.
The Great 2003 Northeast Blackout
and the $6 Billion Software Bug (2,3) A hot day, an untrimmed tree,
and a monitoring system bug cost power customers $6 billion.
The Internet Hits a Capacity
Limit (9,8)
Verizon exceeds an address limitation in the Border Gateway Protocol and
slows the Internet to a crawl.
The Planet Blows Up (3,9) A
massive electrical eWannaxplosion takes out
thousands of hosting servers at a major dedicated hosting provider.
The State of Virginia - Down for
Days (5,10)
A maintenance error takes down 26 state agencies for up to a week.
TWC Internet Outage Affects
Millions (9,9) A human error by the U.S.'s second largest
cable provider disables its Internet service for hours.
Twitter Taken Down by DDoS Attack
(4,8)
The Twitter, Facebook, and LiveJournal social sites are taken down to silence
a Georgian blogger.
Triple Redundancy Failure on
the Space Station (2,11) A single point of failure takes down
a triplexed critical computer.
U-2 Spy Plane Crashes FAA Computer
(9,5)
Overflying Los Angeles at 60,000 feet, a U-2 wreaks havoc on air traffic in
the U.S. Southwest.
U.S. Internet Traffic Comes
to a Halt (11,11) A DDoS attack on a major DNS server
prevents access to web sites.
Verizon 4G Network Down for Two Days
(6,6)
Verizon's "always reliable" 4G network brought down by software bug
- no 3G backup.
Verizon Cloud Down for Forty
Hours (10,2)
Verizon takes its cloud down for two days to install a zero-downtime
upgrades.
VMware's Cloud Foundry Flounders
(6,7)
A storage fault caused by a power outage is followed by a bigger fault caused
by a fat finger.
Vodafone Downed by Burglars
(6,4)
Thieves sledgehammer their way into a Vodafone exchange and steal computers
and network equipment.
VoIP PBX Succumbs
to Overconfiguration (2,6) Why extra processing power made
this PBX less reliable.
WannaCry Ransomware Global Attack
(12,6)
WannaCry malware infects Windows 7 machines, encrypts files, demands $300
ransom.
What?
No Internet?
(3,2) A multiple cable break isolates North Africa, the Middle East,
and India.
What if GPS Fails? (12,1) GPS can fail
for a number of reasons. A failure would be anything from an inconvenience to
a catastrophe for many.
Why Back Up? (4,4) The
malicious act of an IT manager deletes his company's database and forces the
company to close its doors.
Will You Have Internet Access
After July 9, 2012? (7,5) A recent FBI sting took down
rogue DNS servers and substituted good servers until July 9th.
Windows Azure Cloud Succumbs to Leap
Year (7,3)
As the clock ticked to February 29th, the Azure Cloud went down for 32 hours.
Windows Azure Downed by a Single Point
of Failure (8,11) Azure developers could not run new
applications after a failed Microsoft update.
Best Practices
2010 NonStop Availability Award
(5,10)
This year's winner is Bank-Verlag with runners up Belgacom and VocaLink.
2013 NonStop Advanced
Technical Bootcamp (8,9) To be held in San Jose in
November, the NonStop Bootcamp is the premier NonStop annual meeting.
2014 HP NonStop Technical
Boot Camp (9,10) The 2014 NonStop Boot Camp will be held in
San Jose from November 16th through November 19th.
2015 - The Year of the Leap
Second (10,2) One second will be added at midnight, June
30th, potentially bringing down many systems.
911 Systems Experience
Unacceptable Availability (9,3) Statistics show that the
availability of 911 systems is, at best, dismal.
Accounting for Non-Accountants
(12,4)
Explaining the double-entry book-keeping system to software engineers.
Achieving Fast Failover in
Active/Active Systems - Part 1 (4,8) Using user and
network redirection to failover in subseconds.
Achieving Fast Failover in
Active/Active Systems - Part 2 (4,9) Using
server redirection to failover in subseconds.
Active/Active
Data Centers (8,12) Entire data centers are now
going active/active with instant reliable failover.
Adding High Availability
to the Cloud (9,8) Critical applications on internal
servers, other applications in the cloud with data replication.
Applying Predictive
Analytics to Power Backup (10,12) Predictive analytics can
forecast UPS and generator failures for proactive maintenance.
Availability Best
Practices
(2,1) Tips from those who have achieved near-continuous availability.
Avoiding Capacity
Exhaustion (7,7) A strikingly simple graphic display
forecasts capacity peaks by the hour over the year.
Avoiding Notworks
(4,1)
A network that doesn't work in a "notwork." Protect your network
with a good SLA.
Backup Is More Than Backing Up
(4,5) Backing up a database is an exercise in futility if you
can't restore the database.
Boeing 787 Could Lose All
Power (10,6)
A software bug in the generator control units could cause pilots to lose
control of the 787 Dreamliner.
Build to Fail (9,11) NetFlix survived an Amazon reboot by proactively ensuring
that its applications would continue to work under any failure.
California Fires Destroy HP Archives
(12,11)
An archive of historic documents created by Hewlett and Packard destroyed by
fire in Sonoma County.
Can 10,000 Chickens Replace Your
Tractor? (1,3) Save money by replacing your mainframe with
clusters - Not!
Can a Country Shut Down Its
Internet? (11,10) Some countries only have one or two ISPs,
and have on occasion shut down their Internet access.
Can You Trust the Compute Cloud?
(3,8)
What will it take to make cloud computing the data utility of the future?
Chillerless Data Centers
(4,11)
Google and Yahoo! locate new data centers in the north country to take
advantage of "free cooling."
Choosing a Business
Continuity Solution - Part 1 (6,7) What measures of
availability are important to your organization?
Choosing a Business
Continuity Solution - Part 2 (6,8) Data replication is the
fundamental force behind system availability.
Choosing a Business
Continuity Solution - Part 3 (6,9) Data replication leads to
several highly available architectures.
Choosing a Business
Continuity Solution - Part 4 (6,10) Choosing a highly
available architecture to meet your availability needs.
Cloud-to-Cloud Backup (13,2)
Application data stored in a cloud can be backed up in another cloud.
Continuous Availability Featured at HPTF
2009 (4,6)
Presentations include many continuous availability and high availability
talks.
CryptoLocker - Destructive
Ransomware (8,11) CryptoLocker encrypts your files until you
pay a ransom.
Cyber Threats Surpass Terrorism
(8.3)
The U.S. government says that in 2013, cyber threats surpassed terrorism as
the top security concern.
Dark DR - Avoid its Costs With
Active/Active (12,8) The passive side of an
active/passive redundant system is 'dark' - it is doing no work.
Data Centers Consume
Inordinate Amounts of Energy (11,8) Data centers are among
the fastest growing users of electricity.
Data Center Cooling Nature's Way
(5,5)
Data centers cut electric bills in half by replacing chillers with air
economizers.
Data Center in a Box (4,7) Your next
visit to a data center may be to the warehouse district.
Data Center Monitoring
with Open-Source Nagios (6,11) Including NonStop systems in
open-source "single pane of glass" monitoring.
Data Deduplication (6,2) Data
deduplication can reduce backup storage and disaster-recovery bandwidth
requirements by a factor of 20:1.
Data Deduplication in the
Cloud (13,3) As the cloud becomes used more for data
storage, deduplication in the cloud becomes more important.
DDoS Attacks on the Rise
(8,4)
2012 saw a 53% rise in DDoS attacks with greatly increased malicious
bandwidth.
Department of Homeland Security:
Disable Java (8,2) A serious vulnerability in Java 7
means that it should be removed from browsers.
Digest Managing
Editor Speaks About Staggered Systems (11,12) Dr. Bill
speaks at the 2016 NonStop Technical Boot Camp.
Document Your System (1,2) Documentation
is a necessary evil. Let's focus on the "necessary" and not the
"evil."
Does Data Replication
Eliminate the Need for Backups? (5,11) Data
replication protects operations; data backup protects data.
DRJ's Fall World 2010
Business Continuity Conference (5,8) A
week-long conference in September, 2010, focusing on Business Continuity.
DRJ's Spring World 2011
Business Continuity Conference (6,2) A
week-long conference in March, 2011, focusing on Business Continuity.
DRJ's Fall World 2011
Business Continuity Conference (6,8) A
week-long conference in September, 2011, focusing on Business Continuity.
DRJ's Spring World 2012
Business Continuity Conference (7,1) A
week-long conference in March, 2012, focusing on Business Continuity.
Enterprise
Availability Architectures for Business-Critical Services (8,6)
Achieving the proper balance of availability and cost.
Equifax Hacked for Data on 143
Million Customers (12,9) This was one of the largest
breaches of consumer data in history.
Facebook Turns Off an Entire Data
Center to Test Resiliency (11,11) Perhaps the ultimate in
disaster recovery testing.
FBI Warns Employees Are New
Targets (7,11) The FBI warns that cybercriminals are moving
from corporate IT systems to corporate employees.
Flywheel UPS Systems (9,1) Flywheel
UPS systems have many advantages over battery UPS systems.
Future Dates Spell Problems for IT
(12,3)
Date/time integers roll over in 2038 and in 2042.
Google's Extreme-Green Data
Centers (3,12) Wave motion and seawater may power and cool
data centers in the future.
Handling Data Collisions in
Asynchronous Replication (5,9) An update on data collision
avoidance, detection, and resolution.
High Availability Topics at HP
Discover 2011 (6,5) Over two dozen presentations on
high-availability topics will be presented in Las Vegas in June, 2011.
High-Voltage Transformers -
The Power Grid's Achilles Heel (11,3) A major
catastrophe can knock out large transformers in our power grid.
How Can Data Centers
Manage a Drought? (10,5) California's four-year drought
brings datacenter water usage to the forefront.
How Do Your Readiness
Plans Stack Up? (6,1) Compare your disaster recovery
plans with those of 300 other companies.
HP CloudSystem (7,2) Companies
can convert their current IT assets into a private cloud that can burst into
public clouds.
HP Discover 2011 (6,3) HP
Discover 2011 is HP's major annual marketing and educational event, held in
Las Vegas June 6th to June 10th, 2011.
HP's Project Odyssey - Migrating
Mission Critical to x86 (7,3) HP is moving HP-UX high
availability features to Intel's Xeon x86 chip.
HP Blows Up Data
Center
(2,8) An explosive demonstration of fast recovery.
HP's Cloud Recovery-as-a-Service
(7,6)
HP's cloud-based recovery service provides fast RTOs and short RPOs with no
upfront capital expenditures.
Human Triple Whammy - NYSE, UA,
WSJ (10,7)
70% of system outages involve a human error to some extent. Humans need
redundancy too.
Humanizing Three 9s
(2,9)
What if we lived in a world of three 9s?
IBM Builds Super-Dense 5-Nanometer
Chip (12,7)
With transistors packed more closely, the chip achieves a 40% boost in
performance.
IE Vulnerability Allows
Remote Code Execution (9,5) A zero-day Internet Explorer
vulnerability allows attackers to take over your systems.
ING Bank Bown Ten Hours Due to Fire-Suppression Test (11,9) The
noise from the test of its fire suppression system damages hard disks.
Interview with Ron LaPedis on NonStop
with XP Storage (2,5) How to improve NonStop reliability by
using a SAN.
IPv6 Is Here - Like It or Not
(6,4)
Some tips from a father of the Internet on the simple ways to convert from
IPv4 to IPv6.
Is Preventive Maintenance
Preventive? (7,10) Major IT faults have been caused by
preventive maintenance errors. Is PM worth it?
ISO 22301 - The New Business
Continuity Management Standard (7,10) The
first business continuity specification to be issued by ISO.
Japan's Data Centers
Survive Their Big One (10,10) What lessons can we learn from how
Japan's data centers survived the great earthquake?
Katrina - The Harsh Teacher (2,6) The most
powerful Gulf storm in 200 years showed us how unprepared we were for such a
disaster.
Keylogger
Found on HP Laptops (12,12) A tool to log all keystrokes is in the
Synaptics touchpad driver on hundreds of models of HP laptops.
Let's Share Outage
Information for the Benefit of All (9,4) We can
learn a great deal from the experience of others to resolve system outages.
Load Shedding (7,12) If your
system begins to overload, how do you determine what load to shed?
Magnetic Tape Makes a
Comeback (12,10) With so much data being created by mobile devices
and sensors, magnetic tape is returning as viable storage.
Malware as a Service (6,12) Powerful
hacking software is becoming just a click away.
Managing Your Private
Cloud (10,4)
A cloud needs constant managing as its application grow and as application
mix changes
Maximizing
Availability in Everyday Systems (5,7)
Even if you don't have a redundant system, there are things you can do to
minimize outages.
Microrebooting for Fast Recovery (2,3) An
application of Recovery-Oriented Computing.
Migrating IBM Power
System to HPE Open Systems (11,5) Moving applications from
Power Systems to open systems can save up to 50%
Mobile Device Threats to
Corporate Networks (8,7) Mobile devices are a convenience
for employees but a security threat for corporations.
Multifactor
Authentication (10,9) Passwords can be stolen. Add
another identification factor to improve your security.
NonStop Boot Camp is Coming in
October (7,8) The NonStop Community will gather in San Jose
from October 14th through October 16th, 2012.
On Blogs and Discussion Groups
(2,10)
Online forums can be a big boost to your professional growth.
Openness Was Not Always Assured (13,3)
Years ago, systems were not open and could not interoperate.
OpenStack - The Open Cloud
(7,4)
A major open-source initiative may take us one step closer to a true
worldwide compute utility.
OpenVMS Boot Camp Is Coming
in March (8,2) It will be held for four days from March 18th
through March 21 in Bedford, Massachusetts.
Oracle Releases Massive Security
Patch for Java (8,5) Following DHS recommendation to
disable Java, Oracle releases 42 critical security updates.
Protect Your Data Center
From Flooding (11,1) Sea levels are rising. Now is
the time to concern yourself with protecting your data center.
Protecting Big Data - Erasure
Coding (10,11) Erasure coding can protect thousands of
disks with only an efficient few.
Recovery-Oriented
Computing
(2,2) If recovery time can be made small enough, users will perceive a
faultless system.
Reliable Multicasting (3,1)
How to get messages over LAN and WAN multicast networks without message loss.
Retail Web Sites Losing Millions to
Poor Response Time (7,1) Slowness is worse than downtime -
it makes people hate your site.
Roll-Your-Own
Replication Engine - Part 1 (5,1) What does it take to build
your own replication engine? Lots!
Roll-Your-Own
Replication Engines - Part 2 (5,2) Issues with asynchronous
and synchronous replication engines.
Rules of Availability
- Part 1
(3,3) The first set of common rules of availability from our books, Breaking
the Availability Barrier.
Rules of Availability
- Part 2
(3,5) More common rules of availability from our books, Breaking the
Availability Barrier.
Rules of Availability
- Part 3
(3,7) Concluding the common rules of availability from our books, Breaking
the Availability Barrier.
Saving Data Centers by
Creating Chaos (13,2) Enterprises are breaking their systems on
purpose to smoke out weaknesses.
Serverless Computing (13,1)
Allows one to build and run applications without thinking about
servers. It’s all handled by the cloud.
Should Hope Trump Failover?
(9,2)
Too often, the decision to failover is delayed because failover is more risky
than trying to recover the production system.
Software Documentation
(11,2)
The Waterfall method of heavy documentation has given way to Agile software
methodology with minimum documentation.
Swapping
Data Replication Engines with Zero Downtime (12,12)
A replication engine can be upgraded or changed without an outage.
Superstorm Sandy Survivors (8,6) How three
companies in the path of Sandy kept their systems and services operational.
Synchronous
Replication Recovery Strategies (5,3) Bringing
a failed database copy back on line under synchronous replication.
Telstra Plague With Series of Outages
(11,6)
Australia's major telecommunication company had several outages in early
2016.
The 25 Most Exploitable
Programming Errors (8,2) A detailed list of the
programming errors that expose the most vulnerable security holes.
The Big One - Are You Ready?
(10,9)
The Cascadia Subduction Zone is about to unleash the mother of all
earthquakes in the Pacific Northwest.
The Importance of Failover
Testing (12,7) If you don't test your failover procedures,
your backup may not come up when you need it.
The Looming Threat of a Solar
Superstorm (11,7) A solar superstorm that could take power out
for months is overdue.
The Smarts Behind EMV Smart Cards:
Part 1 - Online Processing (9,11) Smart cards are finally coming
to the U.S. How do they work?
The Smarts Behind EMV Smart Cards: Part
2 - Offline Processing (9,12) Authorizing smart card
transactions offline.
The Value of
Availability
(6,6) Downtime costs are based on the likelihood, duration, impact, and
cost of each risk factor taken individually.
The Zero Outage Industry Standard
Organization (12,2) ZOISA focuses on minimizing the
risk of IT outages.
Transaction-Oriented
Computing (2,4) Old art to some, new to others, transaction
processing is the foundation for high availability.
Twitter Earthquake Detector
(5,4)
The U.S. Geological Service is mining tweets to get instant notification of
earthquakes.
Using
Dark Fiber (13,8) Fiber optic networks provide high
bandwidth, scalability, high availability, near-zero latency, and enhanced
security.
VRRP - Virtual Router Redundancy
Protocol
(3,10) Adding transparent failure detection and failover at the first
hop.
What Really Caused
the Windows Azure Outage? (7,5) The Windows Azure cloud
was taken down by a simple leap-year bug.
Why Your Business Continuity
Plan May Be Inadequate (9,9) Is your business plan
satisfactory, well-tested, and well-supported?
Windows Server 2003
Nearing End-of-Life (9,7) Microsoft will terminate support for
Windows Server 2003 in July, 2015.
Windows XP is No Longer PCI Compliant
(9,6)
With no security patches forthcoming, Windows XP systems are no longer
compliant with PCI DSS.
Windows XP Retirement a Hackers'
Boon (8,10)
Hackers are holding zero-day attacks until Microsoft no longer provides
security fixes.
With 100% Uptime, Do I Need a
Business Continuity Plan? (1,1) You'd better believe it.
Yahoo Hack Sets a Record - 500
Million Accounts (11,9) Details from a half-billion
customer accounts were stolen from the Yahoo database.
Availability Topics
911 Systems Are Failing Too
Often (11,5)
Though 911 systems should be considered highly mission-critical, they fail
all too often.
Active/Active Full Day Seminar at HPTF
(4,4)
Dr. Bill speaks on active/active theory and practice at the 2009 HPTF
conference.
Active/Active on
Commodity Servers (8,9) Why has active/active technology
not made it into the commodity server world?
Active/Active Versus Clusters (2,5)
For high availability, clusters are mature; but active/active systems provide
greater reliability.
Active/Active Systems - A
Taxonomy
(3,9) Classifying the many ways to build an active/active system.
Adding Availability to
Performance Benchmarks (2,9) Recovery time is the proper
metric to use for an availability benchmark.
Airlines' Aging Technology
Is Taking Its Toll (11,10) Outdated IT systems can strand thousands
of passengers and cost airlines millions of dollars.
Airline Outages Continue to
Ground Passengers (12,3) Airline systems are based on old
technology and are prone to failure.
Amazon's
Availability Zones (6,11) Critical applications can run
reliably in the cloud by distributing them across Amazon Availability Zones.
All About
Continuous Processing Architectures (1,1) CPA can
get you arbitrarily close to 100% uptime.
Anatomy of a DDoS Attack
(8,4)
DDoS attacks take down web sites by aiming traffic at various levels in the
Internet protocol.
Anti-Virus - A Single Point of
Failure? (5,5) McAfee's malicious anti-virus update takes
down millions of computers in a flash.
Are Our Power Grids
Vulnerable? (8,12) The danger weather of bringing down our
power grids is being compounded by the danger of terrorism.
Asynchronous Replication
Engines (1,2) These engines power most of today's active/active
systems.
Availability in the Cloud (13,1)
The cloud embodies the ides of anywhere and anytime access to service,
tools, and data.
Availability
Organizations (10,10) Many of the organizations that
offer availability advice and services are reviewed.
Availability
versus Performance (2,8) Is it time to trade higher availability for
reduced performance?
Banks Worldwide Suffer from IT
Legacy (9,2) Reliance on decades-old applications are
causing a rash of banking outages.
Can Hackers Take Down Our
Power Grid? (11,1) Ukraine lost much of its power in December,
2015. Are other countries at risk?
Can You Trust Your Public
Cloud? (9,6) With a history of massive failures, public
clouds require extra care for critical applications.
Can An Airliner Be Hacked?
(10,5)
Airliner avionics are often connected to the same Intranet as the passenger
in-flight entertainment systems.
Choosing a Database of
Record
(3,11) Which database copy in an active/active network is the
"single version of truth?"
Cloud
Resiliency (12,12) Cloud computing has dramatically changed
they way we think about application resiliency.
Collision Detection and Resolution
(2,4)
What do you do if you can't avoid collisions when using bidirectional
replication?
Comparing Clouds with
CloudHarmony (10,1) Amazon, Google win with five 9s;
Azure, CenturyLink lose with three 9s.
Court Decides - HP 1, Oracle 0
(7,8)
Judge finds Oracle arguments a Seinfeld sitcom, orders continued Oracle
support of HP Itanium servers.
Cyber Security and Downtime
(11,5)
Various forms of malware can take down your application.
Defining Active/Active
(4,12)
Can we agree on what are active/active architectures? Add your comments to
this ongoing effort.
Defining
Active/Active - Revision 1 (5,1) Revision 1 of our
definition based on suggestions posted to our LinkedIn Continuous
Availability Forum.
Eavesdropping on the Internet (4,3) A
vulnerability in the Border Gateway Protocol allows nefarious sites to read
your Internet traffic.
Eliminate Those
Single Points of Failure (12,10) In order for a system to be
highly available, all components must be redundant.
Epidemic of Certificate-Related
Outages (4,2) Organizations are letting their digital
certificates expire, thus losing access to public encryption keys.
Failsafe (11,8) Failsafe
makes the point that sometimes software lies.
Fault Tolerance for Virtual
Environments - Part 1 (3,3) How virtualization can
significantly reduce data center capital and operating costs.
Fault Tolerance for Virtual
Environments - Part 2 (3,4) Operating system and bare
metal hypervisors.
Fault Tolerance for Virtual
Environments - Part 3 (3,6) Hardening virtual
environments with failover and fault-tolerance.
Fire Extinguishers Can
Cause Data Center Outages (12,11) The noise of the sirens
can be loud enough to damage hard disks.
Fire Suppressant's Impact on
Hard Disks (6,2) Fire alarm sirens in the data center are
fingered as the culprit in hard-disk damage.
Fog Computing Improves
Application Availability (12,6) By distributing critical
processing to cloudlets, cloud outages can be masked.
FS-ISAC: Financial Services -
Information Sharing & Analysis Center (7,10) A
member-owned industry forum for sharing security threats.
Hardware Replication (2,1)
Replicating at the hardware level does not maintain database consistency.
Heartbleed Attacks
Androids (9,5) Heartbleed flaw in Android apps leaves
millions of Androids vulnerable to data loss.
Help! My Data
Center is Down! - Part 1: Power Outages (6,10) Unusual
data center outages caused by power failures.
Help! My Data
Center is Down! - Part 2: Storage Outages (6,11) Unusual
data center outages caused by storage system failures.
Help! My Data
Center is Down! - Part 3: Internet Outages (6,12) Unusual
data center outages caused by Internet failures.
Help! My Data
Center is Down! - Part 4: Intranet Outages (7,1) Unusual
data center outages caused by intranet failures.
Help! My Data
Center is Down! - Part 5: Upgrades (7,2) Unusual
data center outages caused by upgrades gone wrong.
Help! My Data Center
is Down! - Part 6: The Human Factor (7,3) Unusual
data center outages caused by fat fingers.
Help! My Data
Center is Down! - Part 7: Lessons Learned (7,4) The lessons we
can learn from the data center failures of Parts 1 to 6.
How Does Failover Affect Your
SLA? (9,12)
Recovering to a backup system takes time and reduces availability.
How HPE Is Making Blockchain
Resilient (12,7) HPE is porting blockchain to its NonStop
servers to provide continuous uptime.
HP Clarifies the Future of OpenVMS
(8,7)
OpenVMS will be supported by HP for years to come.
Hurricane Harvey's Hit on
Houston Spurs NextGen 911 (12,9) After Houston flooded,
911 service was overloaded. NextGen 911 is needed.
Hypoxic Fire-Prevention
Systems (6,1) Why drown your servers after a fire breaks
out? Keep the fire from starting in the first place.
Improving Power Availability with
Microgrids (11,11) A microgrid can disconnect from the main
electric grid to keep power flowing..
Is the Cost of Converting to
Active/Active Worth It? (4,11) Offsetting the cost of
conversion with the cost of downtime.
It's Official! Leap
Day Caused the Windows Azure Outage (7,5)
Incrementing the year by one to get next year's date took down the Azure
cloud.
Jim Gray - In Memoriam (3,7)
The database pioneer that set the stage for active/active systems is lost at
sea.
Leslie Lamport Wins Turing Award
for Distributed Computing (9,3) Lamport contributed to the
theory and practice of concurrent distributed systems.
Let's Get an Availability Benchmark
(2,6)
Great performance is meaningless if the system in unavailable.
Leveraging
Virtualization for Availability (5,12) With
many eggs in one basket, system availability becomes all that more important.
Linux Leap-Second Bug Takes
Down Data Centers (7,8) A leap second added on June
30, 2012, takes down unpatched Linux systems worldwide.
Media Communication
During a Crisis (6,5) Don't create a second crisis by
letting the press publish erroneous and damaging stories.
Migrating Your Application to Active/Active (2,3) What
must you do to prepare your application for an active/active environment?
My Jeep Wasn't Hacked! (10,08)
Security researchers were able to hack into and remotely control Jeep Grand
Cherokees via the infotainment system.
Napa
Wildfires Wipe Out Communications (13,8) During a fierce windstorm, wildfires in
Northern California burned over 245,000 acres.
OpenVMS Support To Continue
Indefinitely (9,8) HP does U-turn, licenses VSI to
support OpenVMS indefinitely on all future HP platforms.
Ponemon on Live
Threat Analysis (8,11) Intelligence data about cyber
threats happening right now is crucial to stopping cyberattacks.
Power Outages Caused by
Animals (11,7) Squirrels are more of a threat to our power
grid than hackers.
Recovery-as-a-Service (8,2) RaaS
provides backup service in the cloud for critical applications.
Reducing
Pharmaceutical Pollution (8,1) Monitoring of pharmaceutical
processing practices to reduce pollution requires high availability
computing.
Remembering Ken Olsen - An IT
Icon (6,3)
The founder of Digital Equipment Corp., Ken (1926 - 2011) brought interactive
computing to the individual.
Should Cyber Victims Be
Fined? (12,8) The U.K. is fining victims of cybercrime if
they are an essential infrastructure operator.
Social Media Availability and
Performance (6,4) Social media is becoming critical in our
daily lives. It is time for it to grow up.
Sophos Security Threat Report
2013 (8,6)
The major security threats of 2013 for businesses and individuals.
Spamalytics (4,10)
How good are our spam filters, and why does spam till pay?
Stratus Bets $50,000 That
You Won't Be Down (5,1) Buy an ftServer by February 26,
2010, and Stratus will give you $50K if it fails in the first six months.
Stratus Puts $50,000 Where
its Mouth Is - Again (6,12) Stratus' offer to pay you
$50,000 if your ftServer/vSphere application fails expires 12/31/11.
Stratus Puts
$50,000 Where its Mouth Is - an Update (7,2)
Stratus extends its $50,000 ftServer/vSphere availability offer for another
year to 12/31/12.
Synchronous Replication
(1,3)
Avoid data collisions and data loss following a node failure.
The Availability Matrix
(6,1)
Simplify your data center availability configurations using the independence
of RTO and RPO.
The Causes of Outages
(8,3)
250 Never Again stories tell us the proportion of outages due to hardware,
software, humans, networks, and other faults.
The Continuing Struggle with Legacy
Systems (11,12) Decades old legacy systems deny enterprises
flexibility.
The Darknet (9,9) Stolen
financial credentials are sold on the Darknet, a private network with
connections only between trusted peers.
The Dawn of Fault Tolerant
Computing (11,04)
In the early 1980s, there were several fault-tolerant systems. Only Tandem
and Stratus survived.
The Fragile Cloud (4,6)
This new computing paradigm might ultimately replace corporate data centers if
it can ever be made reliable.
The Fragile Internet
(4,5) Can you trust your mission-critical applications solely to
the Internet? We think not.
The Future is
Decentralized Data Storage (13,2) Decentralized storage is a model of online
storage in which data is stored on multiple computers.
The History of
Fault Tolerance (1,2) The fault-tolerant marketplace was hot
in 1984.
The Malware Threat to Android
(7,9)
With its major market share and unvetted apps, Android is the prime smart
phone target for hackers.
The IPv4 Doomsday (4,8) The
Internet Protocol Version 4 is about to run out of its four billion addresses
in two years. What now?
The Ubiquitous Internet
(4,7)
1.5 billion users, 200 million web sites, and one-million viruses depend on
the Internet
The U.S. Government's
IT Fossils (11,2) The U.S. Government spends 75% of its IT
budget on supporting legacy systems.
Time Synchronization for
Distributed Systems - Part 1 (2,11) How does NTP calculate
the time offset from a time server?
Time Synchronization for
Distributed Systems - Part 2
(2,12)
How NTP minimizes time offset errors?
Time
Synchronization for Distributed Systems - Part 3 (3,2) Logical
clocks offer an option for synchronizing systems.
Transaction Replication
(2,2)
A simple approach to active/active systems has scalability issues.
Tussling with the Word
"Redundant" (3,12) "Redundant" doesn't
always translate the same to those in different countries.
Unintended Acceleration and EMI
(5,4)
If testing doesn't show it, does that prove that EMI can't make an engine
computer misbehave?
United Airlines Bug Bounty
Program (10,7) United Airlines is offering frequent flyers
millions of miles for discovering security vulnerabilities.
Upgrades Can Take Your Down
(10,9)
Each upgrade must be carefully planned and tested. Be prepared to roll it
back if it doesn't work.
Using an Availability Benchmark (3,10)
Making use of a recovery time benchmark to influence your system choice.
VSI Releases First New
Version of OpenVMS - (10,6) A year after its founding,
VMS Software Inc. releases OpenVMS version 8.4-1H1.
WestHost Fire Suppression
Test Fiasco - An Update (5,9) Why did the accidental activation
of the fire suppression system destroy so many disks?
What is
Active/Active? (1,1)
Active/active architectures can give subsecond recovery following a
failure.
What is Reliability?
(5,6)
We can get rid of marketeering by quantifying highly-reliable computer
systems by their reliability parameters.
What is Reliability? (13,3)
An update on our earlier article.
What is the Availability
Barrier? (5,3)
Mean
time to recover. Let it fail but fix it fast.
What's Your Concern - MTR or
MTBF? (5,11)
Recovery time is for users, failure intervals are for system operators,
availability is for management.
Windows 7 Mainstream Support Ends
(10,2)
With mainstream support ending, Windows 7 users must upgrade to Windows 10, a
free upgrade for a year.
Worsing on Worsening (4,2) A 1967
chewing-out of IBM's Field Service staff resounds still today.
Recommended Reading
2014 Verizon Data Breach Investigations
Report (9,9) Most security incidents can be categorized
into nine categories.
2015 Verizon Data Breach
Investigations Report (10,8) Most data breaches can be prevented
with simple measures.
A Look at Today's Data Center
Availability (10,7) The Veeam report looks at the
state of today's data centers to meet their RTO and RPO SLAs.
Aberdeen's 2008 Business Continuity
Survey
(3,4) A look at 150 small to large companies and their BC/DR plans and
processes.
Archive Storage - Disk or Tape?
(5,11)
Disk provides fast recovery from backups, and tape provides economical
long-term archiving.
Beyond Redundancy
(7,5)
How geographic redundancy can improve service availability and reliability of
computer-based systems.
Big Switch: Rewiring the World
from Edison to Google (6,9) The cloud compute utility is
following in the tracks of the electric utility.
Blueprints for High Availability:
Designing Resilient Distributed Systems (2,5) All you
ever wanted to know about clusters.
Breaking the
Availability Barrier (3,5) Everything you ever
wanted to know about active/active systems - theory, implementation, and
practice.
Business
Continuity from A to Z (5,12) The online book explores the
responsibilities of the stakeholders in the business continuity plan.
Business
Continuity Planning: IT Examination Handbook (1,1) What better way to learn
about BCP than from the auditor's handbook.
Business Continuity
Today (4,3)
This freely-available living eBook covers a broad range business availability
topics.
Continuous Availability Systems
Design Guide
(2,1) What to do if you want to move to CPA.
Disaster Recovery as a Service
(12,9)
Gartner reviews several companies that provide DRaaS,
currently a $2 billion business.
Distributed Systems:
Principles and Paradigms (3,1) A thorough treatment of
requirements for distributed system transparency.
Fire in the Computer Room , What Now? (2,6) Are you
prepared for a total loss of your data center because of a fire or other
disaster?
Flash Boys: A Wall Street Revolt
(9,5)
A group of traders sets up the IEX stock exchange to prevent predatory
practices such as front-running.
Hactivism
(11,2)
An analysis of an Anonymous attack leads to some key strategies to protect a
company's website from hactivist attacks.
High Availability IT Services
(10,1)
An easy to read discussion of the roles that people, processes, and products
play in high availability IT systems.
High Availability Network
Fundamentals (4,4) A practical guide to
predicting network availability (especially for the mathematically
challenged).
High-Performance IT
Services (11,8) A non-mathematical discussion of how to
achieve high performance in mission-critical IT systems,
HP NonStop for Dummies
(9,10)
Finally! A Dummies book for NonStop systems is here, giving a high-level look
at these fault-tolerant systems.
HPE Cyber Risk Report
2016 (11,3)
The report examines the vulnerabilities that leave organizations exposed to
data breaches.
Hyperconverged
Infrastructure (12,10) HPE has released a Dummies book
describing its Hyperconverged infrastructure product, Simplivity.
Internet Disruptions
(12,5)
Nearly every company has had an Internet Disruption in the past year.
IT Disaster Recovery Planning
for Dummies (10,9) A disaster recovery plan is mandatory to
survive a disaster, but many companies go without one.
Megaplex: An Odyssey of Innovation
(4,12)
Tandem is 35 years old. The Standish Group looks back on 35 years of
availability innovation.
Megaplex Modeling: The
Future of NonStop Demand (5,10) Standish Group envisions
critical and non-critical applications sharing the same blades.
Migrating Legacy Systems: Gateways,
Interfaces, & the Incremental Approach (2,3) Legacy
systems must be decomposed to migrate to active/active.
Mission-Critical
Network Planning (4,9) A broad review of redundancy in
servers, networks, storage, data centers, and power.
Multiple Processor
Systems for Real-Time Applications (2,10) A classic
treatise on distributed systems that is still pertinent two decades later.
Pandemic Response Planning (4,10)
How will your company continue operations if the Swine Flu hits with a
vengeance?
Reliability and
Availability of Cloud Computing (10,2) The
factors contributing to downtime and how to design clouds for five 9s of
availability.
Roadmap to the Megaplex
(5,7)
The six steps that will modernize your vertical NonStop applications for the
open world of horizontal services.
Tandem Computers Unplugged:
A People's History (7,7) Tandem from 1975 till 1997 as seen
through the eyes of its employees.
Targeted - The Story of a
Pathological Serial Work Place Bully (10,6) A
workplace bully costs the job of the author at three companies.
TCP/IP Illustrated, Volume 1: The
Protocols (4,11) The "bible" of the TCP/IP Protocol
Suite, the glue that binds active/active systems.
The Business and Economics of
Linux and Open Source (1,3) Open source demystified for the
reluctant manager.
The Cost of Data Breaches
(11,1)
The Ponemon Institute finds that the average cost of a data breach in 2014
was USD $3.79 million.
The Cost of Data Center Outages
(10,12)
The Ponemon Institute finds that the average cost of a datacenter outage in
2013 was $690,000.(12,2)
The Disaster Recovery Journal (5,2) The
resource for business continuity professionals.
The High Availability Design
Spectrum - Part 1 (11,12) Dr. Terry Critchley sets forth
his first four observations on designing for high availability.
The
High Availability Design Spectrum - Part 2 (12,1) Dr.
Terry Critchley sets forth his next nine observations on designing for high
availability.
The
High Availability Design Spectrum - Part 3 (12,2) Dr. Terry
Critchley sets forth his next six observations on designing for high
availability.
The
High Availability Design Spectrum - Part 4 (12,3) Dr.
Terry Critchley sets forth his final four observations on designing for high
availability.
The Unified
Modeling Language User Guide (1,2) UML is now the accepted standard
for fast and easy documentation of systems and procedures.
Towards Zero Downtime: High
Availability Blueprints (2,8) A close look at
installing Microsoft clusters and cluster-aware applications.
Transaction Processing:
Concepts and Techniques (2,4) The classic book on transaction
processing systems, by Jim Gray and Andreas Reuter.
Unix Backup and Recovery (2,2)
Backing up is a pain, but it is the restore that counts.
Product Reviews
Ace Data Recovery
(9,12)
Ace recovers lost data from hard disks, RAID arrays, solid-state drives, tape,
mobile devices, and cloud environments.
Attunity Integration Suite
(5,12)
Data access, data federation, and data movement combine to make data and
services available across the enterprise.
Attunity Replicate (9,8) Attunity
Replicate is an appliance-based data-replication engine that supports
active/active across heterogeneous databases.
Carbonite - The Online File Copy
Utility (12,6) Carbonite automatically copies new and
modified files to its own online storage for later retrieval.
CenturyLink Targets Six
Nines (11,6)
The U.S. telco giant is aiming to offer its customers six nines of
reliability in its SLAs.
Critical Date Testing -
Leap Day and More (7,5) Many products exist to test
applications for proper processing of critical dates.
EMC's SRDF Data-Replication Engine
(6,4)
Maintain a consistent asynchronous or synchronous target copy of a database
with no server involvement.
Everbridge Emergency Notification
(7,9)
Preplanned mass notification to employees, customers, and suppliers is a must
during an emergency.
FalconStor RecoverTrac
- Automated Disaster Recovery (7,6) Build your own recovery
cloud supporting heterogeneous environments.
Fault-Tolerant Windows and Linux from
Stratus (2,9) ftServers provide transparent fault-tolerant
operation.
Fault Tolerance with everRun from
Stratus (10,12) Two x86 servers coupled with synchronous
replication provide fault tolerance with no data loss.
FileSync and CSR Synchronize NonStop
Systems: Part 1 - FileSync (6,10) FileSync replicates
changed files or file changes between systems.
FileSync and CSR
Synchronize NonStop Systems: Part 2 - CSR (6,11) Command
Stream Replicator repeats operator actions on remote systems.
Flexible Availability Options with
GoldenGate's TDM (2,2) Implement a variety of
data-sharing topologies with TDM's data replication facilities.
GRIDSCALE - A Virtualized Distributed
Database
(3,7) Like presentation and application servers, pooled database
servers for the three-tier architecture.
How Much
Will Active/Active Cost Me? (1,1) The cost of downtime can swing
your decision.
HPE Helion Private Cloud and Cloud
Broker Services (11,2) HPE provides an OpenStack
private cloud and services to implement hybrid clouds.
HP NonStop Servers Migrated to x86
(10,5)
NonStop servers now combine commodity technology with open software.
HP
Serviceguard Cluster Arbitration and Fencing Mechanisms (9,1) Quorum
systems handle partitioning of clusters in the face of network faults.
HP's NonStop Blades (3,8) NonStop
fault-tolerant fundamentals come to HP's c-Class blades.
HP's NonStop Synchronous Gateway
(4,6)
Finally, NonStop synchronous data replication might be on its way.
HP's Reliable Transaction Router
(5,5)
Reliable transaction messaging services between Windows, Linux, OpenVMS, and
HP-UX systems.
HP's ServiceGuard Clustering Facility (2,5)
Managing HP-UX and Linux clusters.
Introduction to HP
Serviceguard Clusters (9,3) HP's clusters protect
applications from failures and service interuptions.
Master/Slave Replication
with Continuent's Tungsten (4,5) Asynchronous replication
between MySQL and Oracle.
MySQL Clusters Go Active/Active (1,3)
Clusters of storage nodes are kept in sync by synchronous replication.
Nagios Open-Source Monitoring for HP
NonStop (8,3) Manage your NonStop systems along with your
Windows, Linux, and Unix systems with Nagios.
Neverfail for Windows Applications
(5,6)
Automated failover is provided for popular Windows applications like
Exchange, SharePoint, SQL Server, IIS.
NonStop AutoSYNC - Eliminating
Configuration Drift (6,8) Backup system configurations must
be kept synchronized with their production systems.
OpenVMS Active/Active Split-Site
Clusters
(3,6) OpenVMS Clusters provide active/active operation with synchronous
replication.
OpenVMS Emulation on PCs (8,9) vtAlpha
and vtVAX emulate Alpha and VAX hardware on multicore x86 PCs with no
software changes.
Oracle Data Replication (6,9) Data
Guard, Streams, or GoldenGate - Which replication engine should be used when?
Parallel Sysplex - Fault
Tolerance from IBM (3,4) IBM's Parallel Sysplex offers offers localized active/active availability.
Pathway - HPE NonStop's Application
Environment (11,5) Pathway provides transparent scalability and
fault tolerance for NonStop applications.
Penguin Computing Offers
Beowulf Clustering on Linux (2,1) NASA's Beowulf clustering
is available on Linux with Penguin's HPC servers.
Prolexic - A DDoS Mitigation
Services Provider (8,4) Prolexic protects companies from
DDoS attacks via a network of scrubbing centers.
Protect your
mission-critical applications with HP Serviceguard Solutions for Linux
(8,10)
A capability of HP's Insight Manager.
Raima's High-Availability Embedded
Database (6,12) A microprocessor embedded database with SQL capabilities
offering five 9s availability.
Redundant Load Balancing for
High Availability (8,7) Loadbalancer.org's redundant load
balancers eliminate a single point of Intranet failures.
Replicating Windows and Linux
Environments with Double-Take (4,8) Replicate entire servers
with incremental file-system updates.
Scaling MySQL with Continuent's uni/cluster (3,11)
Synchronous replication of update queries and distribution of read queries.
SchoonerSQL Brings Five 9s to
MySQL (7,1)
Significant extensions to MySQL to improve its availability and replication
performance.
Shadowbase - The Active/Active
Solution (2,3) Shadowbase provides fast data replication as
well as online copy and database resynchronization.
Shore Micro's 100-Microsecond
Link Failover (6,3) Field-Programmable Gate Arrays
protect redundant Ethernet links with 100 usec. failover.
solidDB - a Five 9s Memory-Resident
Database
(3,5) Server memory is getting so large, why not keep your
database in high speed memory?
Stratus Avance Brings Availability to
the Edge (4,2) If downtime in a branch office costs as
little as $1,000 per hour, Avance can pay for itself in a year.
Stratus Continues its
$50,000 Uptime Guarantee (10,3) Stratus has not paid a cent on
its guarantee for five years.
Stratus' ftServer Flexes Its
Recovery Muscle (5,8) Independent testing measures
scalability and demonstrates no impact due to catastrophic failure.
Surviving DNS DDoS Attacks
(8,11)
The Secure64 DNS Authority server detects and blocks DDoS traffic while
continuing to respond to DNS queries
TANDsoft FileSync Adds
Deduplication (9,2) The bandwidth and time needed for
file synchronization is greatly improved with data deduplication.
Testing for Y2038 and Y2028
(12,5)
Two utilities for testing for critical date/times are reviewed - Softdate and
OPTA2000.
The Uptime Institute
(10,10)
The Uptime Institute provides Tier-based standards for data center topology
and operational sustainability.
Time Synchronization for
NonStop Servers (02,11) NTP products from Bowden
Systems and HP for NonStop servers.
Verisign DDoS Mitigation
Services (8,12) Verisign's cloud mitigation service defends
against even the largest DDoS attacks.
Virtual Tape - Getting Rid of a
Troublesome Medium (1,2) The backup paradigm is changing.
Goodbye, tape.
Virtual Tape for NonStop Servers with
ETI-NET's EZX-BackBox (2,6) Virtual tape made super-fast with
deduplication.
Virtual Transactions with NonStop
AutoTMF (2,4) Converting nontransactional applications to transactional
applications.
Virtualized Time from TANDsoft (4,1) The
OPTA2000 Time Simulator lets multiple applications run on the same
NonStop system with different clocks.
VMTurbo - Managing Virtualization
(10,4)
VMTurbo is a management tool that automatically reconfigures virtual
environment to maintain SLAs.
VSI to Port OpenVMS to x86
(10,11)
VMS Software, the support organization for OpenVMS, will port the operating
system to run on x86-64 servers.
Windows Server Failover
Clustering (5,4) Microsoft's successor to MSCS adds simplified
cluster management and improved geographical dispersion.
The
Geek Corner
Calculating
Availability - Redundant Systems (1,1) Some useful rules come out
of the derivation of the availability equation.
Calculating
Availability - Repair Strategies (1,2) Your repair
policy can have a significant impact on your system availability.
Calculating
Availability - The Three Rs (1,3) Node repair, node
recovery, and system restore are all required.
Calculating
Availability - Hardware/Software Faults (2,1)
Most faults don't need a repair.
Calculating
Availability - Failover (2,2) When a system is failing over, it
is often effectively down, thus reducing availability.
Calculating
Availability - Failover Faults (2,3) Failovers can fail also.
Calculating
Availability - Environmental Faults (2,4) How to handle
hurricanes, power failures, and riots when calculating availability.
Calculating
Availability - Cluster Availability (2,5) How does the
availability of a cluster compare to that of an active/active system?
Calculating Availability - Nodes,
Subsystems, and Systems (2,6) When is a node a system, and when
is it a subsystem?
Calculating
Availability - Failure State Diagrams (2,9)
Formalizing our intuitive derivations.
Calculating
Availability - Heterogeneous Systems - Part 1
(3,3) Probability 101 in preparation for analyzing systems with
heterogeneous nodes.
Calculating
Availability - Heterogeneous Systems - Part 2
(3,5) The availability of redundant systems with different nodal
availabilities.
Calculating
Availability - Heterogeneous Systems - Part 3 (3,6)
Analyzing complex configurations of system components.
Calculating
Availability - Heterogeneous Systems - Part 4 (3,8)
Demonstrating that systems with century uptimes can be configured.
Calculating RPO (5,3) An RPO is
the probability that data loss following a node failure is less than a
specified amount. How can we verify this?
Configuring to Meet a
Performance SLA - Part 1 (3,12) What size server is needed to
provide a response time of 200 msec. 98% of the time?
Configuring to Meet a
Performance SLA - Part 2 4,1) Comparing the performance of
single-server systems to multiserver systems.
Configuring to Meet a
Performance SLA - Part 3 (4,2) Answering the SLA specification
for servers with exponential service times.
Configuring to Meet a
Performance SLA - Part 4 (4,3) Answering the SLA specification
for servers with arbitrary service times.
Configuring to Meet a
Performance SLA - Part 5 (4,4) Answering the SLA specification
for multiple servers in tandem.
Dr. Bill's Doctoral Thesis (12,8) Dr. Bill
researched Linear Decision Functions, With Application to Pattern
Recognition" for his doctor's thesis.
Estimating
Data Collision Rates (2,8) Can you go active/active with a
tolerable level of data collisions?
Failure
State Diagrams - Repair Strategies (2,10) The real story
behind sequential repair and parallel repair.
Failure State
Diagrams - Recovery Following Repair (2,12) The formal
analysis of the impact of having to recover a node after its repair.
Failure
State Diagrams - Hardware/Software Faults Revisited (3,2)
Our intuitive results were a little simplistic.
Google Will Help You Manage the
2016 Leap Second (11,12) You can use Google's NTP server
to get the leap second as 'smeared time.'
Improving Availability via
Staggered Systems (12,2) Optimize availability by
minimizing the probability that both systems will fail simultaneously.
Is Parallel Repair Really
Better Than Sequential Repair? (3,4) A Digest reader points out that
that depends upon the repair time distribution.
Modem Memories (10,11) Dr.
Bill developed one of the first AT&T modems, the DataPhone 103, back in
1958.
Not Only the Y2038 Problem -
There's a Y2028 Problem (12,5) The Y2028 problem was caused by
a 'temporary' Y2K work-around.
Random Events
Have No Memory
(9,11)
MTR and MTBF are random events characterized by the exponential and Poison probability
distributions.
Reliability Diagrams
(6,7)
Complex systems can be analyzed via reliability diagrams as sets of parallel
(redundant) and serial components.
Repair Strategies
(9,10)
The repair strategy used by an enterprise can have a significant impact on
application downtime.
SAP on VMware High Availability
Analysis (7,12) Our availability analysis is used to predict
the availability of VMware ESXi clusters.
Simplifying Failover
Analysis - Part 1 (5,10) User's aren't down just because two
nodes fail. They are also down waiting for a backup system to take over.
Simplifying Failover
Analysis - Part 2 (6,6) Extending failover analysis to
complex multinode systems.
So You Want to Mine Bitcoins?
(9,4)
A lot of money can be made by generating bitcoins, but only for a few.
Software Reliability
Models (8,8) Modeling software reliability is a complex
task upon which there is not much agreement.
The Cost of RPO and RTO
(7,9)
What is the optimum architecture for minimizing the costs of downtime and
lost data?
The Fallacy of Classic Availability
Theory (12,1) MTBF and MTR are memoryless variables. Use
MTTF instead.
What's
That Nerd Logo? (1,1)
Our logo, ff2, really has a
meaning. Find out why it describes active/active architectures.
Why Are Active/Active Systems So
Reliable?
(3,9) Analyzing the impact of resubmitting transactions rather than
bringing up a backup system.
Writing
Patent Applications (13,8) One of my writing services is to prepare
patent applications for my customers. In this article, we talk about the
format for patent applications.
Tweets
@availabilitydig - The
August, 2013, Twitter Feed of Outages (8,8)
@availabilitydig - The September,
2013, Twitter Feed of Outages (8,9)
@availabilitydig - The
October, 2013, Twitter Feed of Outages (8,10)
@availabilitydig - The
November, 2013, Twitter Feed of Outages (8,11)
@availabilitydig - The
December, 2013, Twitter Feed of Outages (8,12)
@availabilitydig - The
January, 2014, Twitter Feed of Outages (9,1)
@availabilitydig - The
February, 2014, Twitter Feed of Outages (9,2)
@availabilitydig - The
March, 2014, Twitter Feed of Outages (9,3)
@availabilitydig - The
April, 2014, Twitter Feed of Outages (9,4)
@availabilitydig - The
May, 2014, Twitter Feed of Outages (9,5)
@availabilitydig - The
June, 2014, Twitter Feed of Outages (9,6)
@availabilitydig - The
July, 2014, Twitter Feed of Outages (9,7)
@availabilitydig - The
August, 2014, Twitter Feed of Outages (9,8) (9,8)
@availabilitydig - The
September, 2014, Twitter Feed of Outages (9,9)
@availabilitydig - The
October, 2014, Twitter Feed of Outages (9,10)
@availabilitydig - The
November, 2014, Twitter Feed of Outages (9,11)
@availabilitydig - The
December, 2014, Twitter Feed of Outages (9,12)
@availabilitydig - The
January, 2015, Twitter Feed of Outages (10,1)
@availabilitydig - The
February, 2015, Twitter Feed of Outages (10,2)
@availabilitydig - The
March, 2015, Twitter Feed of Outages (10,3)
@availabilitydig - The April,
2015, Twitter Feed of Outages (10,4)
@availabilitydig - The
May, 2015, Twitter Feed of Outages (10,5)
@availabilitydig - The
June, 2015, Twitter Feed of Outages (10,6)
@availabilitydig - The
July, 2015, Twitter Feed of Outages (10,7)
@availabilitydig - The
August, 2015, Twitter Feed of Outages (10,8)
@availabilitydig - The
September, 2015, Twitter Feed of Outages (10,9)
@availabilitydig - The
October, 2015, Twitter Feed of Outages (10,10)
@availabilitydig - The
November, 2015, Twitter Feed of Outages (10,11)
@availabilitydig - The
December, 2015, Twitter Feed of Outages (10,12)
@availabilitydig - The
January, 2016, Twitter Feed of Outages (11,1)
@availabilitydig - The
February, 2016, Twitter Feed of Outages (11,2)
@availabilitydig - The
March, 2016, Twitter Feed of Outages (11,3)
@availabilitydig - The
April, 2016, Twitter Feed of Outages (11,4)
@availabilitydig - The
May, 2016, Twitter Feed of Outages (11,5)
@availabilitydig - The
June, 2016, Twitter Feed of Outages (11,6)
@availabilitydig - The
July, 2016, Twitter Feed of Outages (11,7)
@availabilitydig - The
August, 2016, Twitter Feed of Outages (11,8)
@availabilitydig - The
September, 2016, Twitter Feed of Outages (11,9)
@availabilitydig - The
October, 2016, Twitter Feed of Outages (11,10)
@availabilitydig - The
November, 2016, Twitter Feed of Outages (11,11)
@availabilitydig - The
December, 2016, Twitter Feed of Outages (11,12)
@availabilitydig - The
January, 2017, Twitter Feed of Outages (12,01)
@availabilitydig - The
February, 2017 Twitter Feed of Outages (12,02)
@availabilitydig - The
March, 2017 Twitter Feed of Outages (12,03)
@availabilitydig - The
April, 2017 Twitter Feed of Outages (12,04)
@availabilitydig - The
May, 2017 Twitter Feed of Outages (12,05)
@availabilitydig - The
June, 2017 Twitter Feed of Outages (12,06)
@availabilitydig - The
July, 2017 Twitter Feed of Outages (12,07)
@availabilitydig - The
August, 2017 Twitter Feed of Outages (12,08)
@availabilitydig - The
September, 2017 Twitter Feed of Outages (12,09)
@availabilitydig - The
October, 2017 Twitter Feed of Outages (12,10)
@availabilitydig - The
November, 2017 Twitter Feed of Outages (12,11)
@availabilitydig - The
December, 2017 Twitter Feed of Outages (12,12)
@availabilitydig - The
January, 2018 Twitter Feed of Outages (13,01)
@availabilitydig – The February, 2018 Twitter Feed of Outages
(13,01)
(x)
(y)
|
