Gov. Bob McDonnell wants an “independent third party” to investigate the collapse of a Virginia government computer system nearly a week ago that continues to paralyze some agencies.
“I am not pleased that our employees and citizens have experienced this disruption in service,” McDonnell said yesterday in ending his public silence on the crisis.
Six of 26 agencies hit by the outage Wednesday still were not fully up and running, despite a promise by state computer chief Samuel A. Nixon Jr. to have their service restored by yesterday at 8 a.m.
The VITA update as of yesterday at 5:00 pm said that 3 agencies – the DMV, the Department of Taxation, and the State Board of Elections – are still offline. While the VITA site hasn’t given much detail, some items are starting to come to light. According to the story in the Richmond Times-Dispatch, the failure in the SAN was specifically in a storage device built by EMC, a DMX-3. Supposedly, a circuit board failure in the unit caused the array to crash. Now, the word is that the load transferred to a backup unit which then failed in precisely the same way. EMC is supposedly adamant that this is unprecedented. Yeah, I’ll bet it is.
The widespread outage of state agency computer systems continues today. The Virginia Department of Motor Vehicles has issued a statement that they will be unable to renew driver’s licenses at their services locations again today. They offered no insight as to when they expect to be able to resume this rather critical service. They did make certain to say that when service is restored, you can expect seriously long lines and major delays. (At least, they’ve implied that so heavily as to be an explicit announcement.)
Over at the Virginia Information Technologies Agency (VITA) they are posting updates on the progress they’re making. As of 10:00pm Sunday, 29 August 2010, they reported that they were “continuing to work to restore services to agencies.” From their update page:
Update on storage outage
10 p.m., Sunday, August 29, 2010
Throughout the weekend, teams have been working steadily and deliberately to ensure the restoration process is complete and that all data is verified following the networked storage system failure experienced last week. We appreciate the cooperation and patience of every agency and citizen affected by this issue. Below is an update regarding the substantial progress that has been made over the weekend and that we expect will continue to be made through the night:
• Successful repair to the storage system hardware is complete, and all but three or possibly four agencies out of the 26 agency systems have been restored. Agencies continue to perform verification testing. • Progress continues, but work is not yet complete for the three or four agencies that have some of the largest and most complex databases. These databases make the restoration process extremely time consuming. The unfortunate result is the agencies will not be able to process some customer transactions until additional testing and validation are complete. • According to the manufacturer of the storage system, the events that led to the outage appear to be unprecedented. The manufacturer reports that the system and its underlying technology have an exemplary history of reliability, industry-leading data availability of more than 99.999% and no similar failure in one billion hours of run time. • While most issues have been resolved over the weekend, some issues may continue as the impacted systems are tested and validated. State agencies should report any issues to the VITA Customer Care Center (VCCC) at (866) 637-8482 or vccc@vita.virginia.gov. Additional staff will be available to handle any increase in call volume. Please note: E-mail should not be used to report critical issues or outages impacting an agency. To report a critical issue, please call the VCCC directly.
The outage, they say, “resulted from a damaged networked storage system” which now directly identifies the SAN part of their data center as the location of the outage. That makes sense, considering the presence of EMC in the troubleshooting team. That second bullet point is especially worrisome, though, from a design standpoint. About the only reason you’d need to take this incredibly long to repair damage to a database is if 1) the indexes of the database itself were completely blown out and needed to be recreated from scratch and 2) you don’t have a replicated backup available. At all. For mission-critical systems, that’s just ridiculously poor design.
The 3rd bullet point is, frankly, verbiage from the vendor’s sales brochure. It’s completely irrelevant that the systems in place have a billion hours of history or 5-nines of reliability under normal circumstances. It’s absolutely beside the point that the failure occurred in a completely unprecedented way. The fact of the matter is that the design of this system could not handle the failure of the primary database. A continuance-of-operation plan (COOP), if done properly, takes this kind of event into account and provides for the enterprise’s ability to keep working when a complete failure of the primary system occurs. COOP’s aren’t rocket science. They are well understood endeavors that are – or should be – part of any major enterprise design where mission-critical, it’s-gotta-work-every-time functions are handled.
Gov. McDonnell needs to be putting together an audit of this contractor’s work (NG and EMC both) that makes no assumptions. He should also be auditing VITA and find out why this situation (referring to the lack of a back up that could handle the load) was allowed to exist. Whether this contract should have been extended or not is beside the immediate point and this situation, bad as it is, is not an indictment of the practice of privatizing functions supporting government operations. Northrop Grumman needs to put the design in place to see to it that the government operation doesn’t go down regardless of major crashes like this. Finding out where they fell down is the 1st step. Following through and making them correct this situation – under this existing contract – is step 2.
I’ve never liked the many-splendored labels being put into common use in political discussion these days that end in the term, “-phobia.” Homophobia. Xenophobia (for the illegal “immigration” debate.) Islamophobia. The reason I don’t like them is that they are all pretty much manifestations of a logical/argumentational fallacy called ad hominem. In an ad hominem attack, it is the person making the argument that is attacked rather than the argument itself. In most of the cases I’ve ever seen where these labels are flung about, it is the use of the label that is, itself, the ad hominem attack.
“Phobia” is a word rooted in the Greek word “phobos” and means “fear” or “morbid fear.” In common language, it refers to an irrational fear. While I have an issue with someone considering my dislike of a particular thing a “fear” it’s the “irrational” part that’s the real problem. Look up the definition of “ophidiophobia” – a fear of snakes. Is it irrational for someone coming in contact with a 10-foot long, venom-spitting king cobra to fear that snake? Of course not. It’s irrational not to. In fact, the link I provided to Wikipedia says something about just this thing: “Care must also be taken to differentiate people who do not like snakes or fear them for their venom or the inherent danger involved. An ophidiophobic would not only fear them when in live contact but also dreads to think about them or even see them on TV or in pictures.“
Roger Simon of Pajamas Media addresses the concept in an article on August 27 wherein he challenges the use of the term “Islamophobia.”
With very minor exceptions, I have seen little irrational fear of Islam in our society. What I have seen is a lot of serious and justifiable dislike of the religion for its ideology — notably its heinous treatment of women and homosexuals and its opposition to the separation of church and state, all codified by its all-encompassing Sharia law that seeks to legislate all facets of existence while instituting a global caliphate.
Nevertheless, soi-disant liberals and progressives or whatever they want to call themselves accuse those who dislike Islam for those reasons of irrational fear. That’s like having an irrational fear of totalitarianism.
Indeed. The “-phobia” terms all provide a double-hit: they cast any true concern the target has as irrational on its face and they provide the user with cover to completely dismiss any further conversation, question, or debate. You can’t logically argue with an irrational viewpoint. If the person in question – someone who thinks the building of a mosque near the site of the best-known attack by forces loudly identifying as Islamic against America is a bad idea – is just being irrational, then there’s no point in even addressing his concerns, right? I mean, what a waste of time! It is ad hominem at its finest.
The fact that the majority of these terms have originated with the Left and are utilized largely by so-called “progressives” is just a symptom of a larger problem: they’ve lost the public debate on a variety of issues lately and are looking for any face-saving mechanism they can find.
Well, that’s their problem. I refuse to accept the terms they’ve taken to tossing around. It is they, through their continued usage of ad hominen fallacies in the place of reasoned argument, that have proven themselves irrational.
It is a fact that petroleum leaks out of the seafloor and into the ocean on a constant, naturally-occurring basis. It has for centuries – as best as we can tell, it’s actually been millennia. Oil is present in seawater, in other words, and always has been. When massive quantities of the stuff gets released at once, however, it represents an unsafe condition. We don’t want to be immersed in the stuff nor do we want to be ingesting it, either directly with the water or indirectly by consuming seafood that’s been over contaminated. But if oil is always in the seawater, then the seafood we eat is always in contact with it. It is scientifically impossible to get seafood from the open ocean that hasn’t been in contact with petroleum at some level or another.
Ah, but what level is considered unsafe versus safe? That’s what we’ve done research to figure out and what we developed tests to determine. Given the fact that it’s an absolute certainty that there’s some level of these substances present in any seafood we catch, up to what level can we feel secure in seeing and still consider it safe to eat? Since 1970 we’ve had an agency of the federal government around to look at these matters: the EPA. Along with the Food & Drug Administration (FDA), the EPA has set levels that cannot be exceeded and still have a given food store permitted into the food supply. Operating under those guidelines, officials tasked with the responsibility to monitor and report on the safety of seafood coming out of the gulf after the BP oil spill are testing the gulf catch and either opening or closing areas of the gulf for fishing. They are confident their testing methods and abilities are sufficient to make that determination.
Some of the fisherman are reportedly not in agreement:
“We are not here to listen to your protocols,” [fisherman Danny] Ross said. “We have questions and there has been a breakdown in the pipeline since this whole thing started.
“Everybody that works these waters is seeing strange things out there and you cannot tell me that with that much oil and dispersant something did not get contaminated.”
So, if the testing we’re doing and the fact that they’re coming up with a “safe” determination “can’t tell him” that a given area isn’t showing contamination, what is he looking for? He says he doesn’t want to “listen to your protocols” but it’s those protocols that are describing the testing being done. If no amount of testing will do the trick, then how does he suggest we proceed? Ban seafood from the gulf permanently?
I also note that there were no examples given of the “strange things” “everybody” is seeing out there that’s convincing them the area’s contaminated. And I think it’s not unreasonable for people to ask for him and his “everybody” to be able to provide those examples in hard evidence. Of course, if we’re going to hold him to the same standard that he’s using, I’m unsure how he plans on getting the hard evidence. No testing is good enough, remember?
Perhaps there were other questions raised at the meeting that just didn’t get reported but I certainly hope there’s more to it than asking “who’s responsible if someone gets sick.” That’s not a question about how to determine safety, that’s a question about who’s going to pay the jury-awarded penalty to the people who get sick. That’s a legal question, not a scientific one, and if the fisherman want that one answered they should address it to the right people in the right venue. These guys are scientists and lab geeks; they talk tests and data.
To simply dismiss the possibility that seafood from a given area might be safe in the face of the best scientific evidence we can muster isn’t rational and the people who have concerns – rightly, I might add – need to understand that. If they have an issue with the process of determining safety they need to lay their cards on the table and explain themselves. They also need to be ready to help us figure out what would be required to allay their fears.
A few days ago I caught wind of an attempt by some environmental activists to have the EPA ban lead ammunition under the Toxic Substances Control Act of 1976. I was trying to perform some actual research on the matter (was this for real, was the EPA really considering it, etc.) and kept getting distracted by the day-to-day stuff including – shockingly – work. Bottom line, not only was it for real, the EPA has apparently decided to do the smart thing and cancel the public commentary period they had opened. They have denied the petition.
The Environmental Protection Agency has denied a petition filed by environmental activists seeking to ban lead in ammunition, saying such regulation is beyond the agency’s authority.
The agency’s decision, announced Friday shortly after FoxNews.com published its report on the issue, sided with hunters and fishermen who had argued that the such regulations weren’t allowed under the Toxic Substances Control Act of 1976.
The TSCA specifically excepts a variety of substances and materials from the EPA’s regulatory powers, including ammunition. The TSCA, in other words, explicitly denies the EPA the authority to regulate ammo. The petition was a no-go on its face. The EPA, recognizing that the law explicitly prohibits it from applying the TSCA to ammunition, decided that a public comment period was a waste of its time and resources. Good call. They were also up front with the notification that no one at the EPA is suggesting that such authority be granted.
Now if we can just get them to be as reasonable about carbon dioxide.
Thanks to an e-mail I received from VA State Delegate Tom Rust (VA-86), it’s come to my attention that a major information systems failure is currently affecting Virginia’s government systems. The DMV is affected:
Statewide Computer Issue Halts In-Person Driver’s License Transactions Vehicle Titles, Decals, Etc. Still Being Processed in DMV Offices
RICHMOND – Due to a statewide computer system problem, the Virginia Department of Motor Vehicles is currently unable to process driver’s license transactions inside its 74 customer service centers in Virginia. Customers who can renew licenses through another service option are encouraged to go online at www.dmvNOW.com, or use the automated telephone service at 1-888-337-4782. The Virginia Information Technologies Agency (VITA) is working to resolve the server problem that is affecting 24 different state agencies. VITA has not reported a recovery time.
At DMV, all other types of transactions, including vehicle decals and titles, transcripts, etc. are being processed normally.
That’s an interesting comment – the web site appears to work and it’s handling transactions that the in-person, on-site DMV employees can’t? As a network professional, the situation interests me and suggests quite a number of scenarios. Clearly, if the transaction resources aren’t completely dead (and they’re not, or you wouldn’t be able to do them on the web site) and the interface web site isn’t dead (ditto) then the issue must be one that affects the local sites directly. That’s either a connectivity problem (which wouldn’t affect multiple locations simultaneously) or there’s some issue with the internal systems at the DMV that’s not affecting external systems such as the DMV’s site.
The outage, flaring Wednesday afternoon and expected to disrupt some services through the weekend, is attributed to 228 malfunctioning servers, which supply shared software and applications to clusters of state agency computers.
Twenty-six of more than 80 state agencies were hit by the shutdown, including the office of Gov. Bob McDonnell.
“We’re disappointed to have a failure, an outage of this magnitude,” Samuel A. Nixon Jr., head of the Virginia Information Technologies Agency, said yesterday. “No matter what you do, it’s going to happen on occasion.”
Ah, so what’s happened appears to be that several servers handling internal access to resources have died. The external ones that we all access via the Internet (likely located in what’s called a DMZ) were apparently unaffected. The fact that those servers can still apparently access the data means that the data stores appear to be available and intact, it’s just that the internal servers that handle the government employee’s interfaces are down.
I’m going to have to take some issue with that last comment by Mr. Nixon at VITA. I can assure you that there are companies out there (banks, Google, Amazon) who architect their environments to make it as sure as is humanly possible that such “occasions” do not occur. The story at WSLS indicates that the infamous VITA-Northrop Grumman IT contact is at the heart of this. NG’s subcontractor EMC (a network storage company) was reportedly “checking for faulty equipment” and, in the process, killed the 228 servers. That’s a helluva faulty equipment check that manages to take out a couple of hundred mission-critical systems. And do it badly enough, I might add, that it takes 5 days to recover. Where are the backup systems?
Nixon also said that the interruption was of insufficient magnitude to activate a backup system at a duplicate computer center in Russell County, in Southwest Virginia.
A full-on, deader-than-a-doornail failure is of “insufficient magnitude” to “activate” a backup system? Sorry, again, but I call bullshit on that one. Any backup system that can’t step in and handle the task for which it was designed to back up when the primary drops dead of brain failure isn’t a backup system at all – it’s a completely separate system that’s now sucking up power and providing nothing but space heating. I may not be privy to the internal discussions and the full picture of the situation but that line of crap is a man just throwing chaff into the air hoping to confuse people. If he’s serious, then he either doesn’t know what the hell he’s talking about or he’s being lied to.
Either way, an investigation needs to be performed and the citizens of Virginia need to be apprised of the results, in clear language. Either the government wasn’t clear or knowledgeable enough to articulate proper requirements or the contractor simply isn’t performing according to the contract. Either way, that needs to be fixed and fixed now.
It’s time to face the facts: you don’t boost economic health by creating massive federal debt, taking over significant chunks of the private sector, creating massive new bureaucracies that to hobble what parts of the private sector you’re not putting under direct government control already, and trying to gin up class warfare that results in what success stories still exist in the private sector refusing to risk growing any further. There’s a lot that’s been put in place over the past 19 months that needs to be removed. They were bad ideas when they were proposed, people who know this stuff said they’d result in even further slowing of the economy and lower tax revenues, and the results now in the books have proven those people right.
Of course, this administration will do no such thing. They’ve invested everything into passing this agenda and to pull any of it back would be admitting they set us on the wrong track. It’s up to we, the people, to handle it for them and we can start that process in earnest on November 2nd.
You know, for a group that insists that we on the right wear ideological blinders and live in echo chambers, the Left these days are awfully fast to blame conservatives on the basis of nothing more than the Left’s own biases and prejudices. In just the last week 2 accusations by the Left regarding violence perpetrated by conservatives – most notably Tea Party activists – have been trumpeted far and wide and both times the accusations have been proven false. They were never supported by anything even closely resembling facts.
One of the campaign offices for Russ Carnahan (D-MO) was firebombed. With barely a second’s worth of hesitation, the Left’s allies in the MSM jumped to the conclusion that it must be the work of Tea Partiers. Now, no violent actions have ever been shown to be initiated by or supported by any of the Tea Party movements but a lack of facts doesn’t appear to slow the Left down these days. Moe Lane over at Red State and Jim Hoft at Gateway Pundit wrote on Wednesday that the truth of the matter appeared to be that a Dem operative (one of Carnahan’s own employees, to boot) had done the deed. That evening Carnahan’s office admitted that was the case. There was never any evidence it was an action taken by conservative activists at all.
Now we have the story of a Muslim cabbie up in New York who was attacked by a passenger after (reportedly) he answered the question of whether he was a Muslim affirmatively. Again, the Left loudly accused conservatives for creating the atmosphere of hate, hate, hate over their vicious opposition to the building of the Ground Zero Mosque. If you’d like to see a sample of the litany going on over at the Lefty blogs, you can check Michelle Malkin’s site or Prof. William Jacobson of Cornell Law School at his blog, Legal Insurrection.
The truth of the matter, however, is that the assailant is a lefty himself. Hasn’t stopped people from continuing to push the meme, of course, not even when it’s discovered that the lefty worked for a liberal interfaith organization that supports the building of the mosque.
The Left is very keen on delegitimizing the Tea Party in any way they can. They’ve tried calling them racist – their old standby, of course – and tried painting them as completely unreasonable, frothing-at-the-mouth, dangerous inciters of violence. Time and again it’s turning out that they’re no such things and they are being unjustly accused by the Left when it is their membership who are responsible for the violent behavior. The facts are showing that the Left’s accusations are just not trustworthy and they’ve fallen into the habit of saying whatever they think will cloud the issue and attack the persons rather than the matters at hand. This is only going to get worse over the next 60 days. I suggest you keep that firmly in mind as these next 2 months go by and that you ask yourself whether these people can really be trusted to tell you who to vote for.
Senator John McCain has won the nomination in AZ to run for re-election to his Senate seat as was widely expected these past few weeks. This, along with the Meek’s win in Florida, is part of the primary story from yesterday. AZ Governor Jan Brewer also won her primary and will stand for re-election.
Not every incumbent is faring so well. Alaska’s Senator Lisa Murkowski was facing another Tea Party-backed candidate, Joe Miller, and as of this morning it’s looking like Miller might have pulled off an upset. According to the Alaska State Elections results, Miller is leading Murkowski 51.09% to 48.91% with nearly 98% of all precincts reporting. The vote count difference is showing 1,960 votes in Miller’s favor out of 89,858 cast. It’s 4 in the morning there at the time I’m writing this so we might have to wait a few hours for an official ruling.
Search the web:
RSS Feed
Ric's Twitter