Gov. Bob McDonnell wants an “independent third party” to investigate the collapse of a Virginia government computer system nearly a week ago that continues to paralyze some agencies.
“I am not pleased that our employees and citizens have experienced this disruption in service,” McDonnell said yesterday in ending his public silence on the crisis.
Six of 26 agencies hit by the outage Wednesday still were not fully up and running, despite a promise by state computer chief Samuel A. Nixon Jr. to have their service restored by yesterday at 8 a.m.
The VITA update as of yesterday at 5:00 pm said that 3 agencies – the DMV, the Department of Taxation, and the State Board of Elections – are still offline. While the VITA site hasn’t given much detail, some items are starting to come to light. According to the story in the Richmond Times-Dispatch, the failure in the SAN was specifically in a storage device built by EMC, a DMX-3. Supposedly, a circuit board failure in the unit caused the array to crash. Now, the word is that the load transferred to a backup unit which then failed in precisely the same way. EMC is supposedly adamant that this is unprecedented. Yeah, I’ll bet it is.
The widespread outage of state agency computer systems continues today. The Virginia Department of Motor Vehicles has issued a statement that they will be unable to renew driver’s licenses at their services locations again today. They offered no insight as to when they expect to be able to resume this rather critical service. They did make certain to say that when service is restored, you can expect seriously long lines and major delays. (At least, they’ve implied that so heavily as to be an explicit announcement.)
Over at the Virginia Information Technologies Agency (VITA) they are posting updates on the progress they’re making. As of 10:00pm Sunday, 29 August 2010, they reported that they were “continuing to work to restore services to agencies.” From their update page:
Update on storage outage
10 p.m., Sunday, August 29, 2010
Throughout the weekend, teams have been working steadily and deliberately to ensure the restoration process is complete and that all data is verified following the networked storage system failure experienced last week. We appreciate the cooperation and patience of every agency and citizen affected by this issue. Below is an update regarding the substantial progress that has been made over the weekend and that we expect will continue to be made through the night:
• Successful repair to the storage system hardware is complete, and all but three or possibly four agencies out of the 26 agency systems have been restored. Agencies continue to perform verification testing. • Progress continues, but work is not yet complete for the three or four agencies that have some of the largest and most complex databases. These databases make the restoration process extremely time consuming. The unfortunate result is the agencies will not be able to process some customer transactions until additional testing and validation are complete. • According to the manufacturer of the storage system, the events that led to the outage appear to be unprecedented. The manufacturer reports that the system and its underlying technology have an exemplary history of reliability, industry-leading data availability of more than 99.999% and no similar failure in one billion hours of run time. • While most issues have been resolved over the weekend, some issues may continue as the impacted systems are tested and validated. State agencies should report any issues to the VITA Customer Care Center (VCCC) at (866) 637-8482 or vccc@vita.virginia.gov. Additional staff will be available to handle any increase in call volume. Please note: E-mail should not be used to report critical issues or outages impacting an agency. To report a critical issue, please call the VCCC directly.
The outage, they say, “resulted from a damaged networked storage system” which now directly identifies the SAN part of their data center as the location of the outage. That makes sense, considering the presence of EMC in the troubleshooting team. That second bullet point is especially worrisome, though, from a design standpoint. About the only reason you’d need to take this incredibly long to repair damage to a database is if 1) the indexes of the database itself were completely blown out and needed to be recreated from scratch and 2) you don’t have a replicated backup available. At all. For mission-critical systems, that’s just ridiculously poor design.
The 3rd bullet point is, frankly, verbiage from the vendor’s sales brochure. It’s completely irrelevant that the systems in place have a billion hours of history or 5-nines of reliability under normal circumstances. It’s absolutely beside the point that the failure occurred in a completely unprecedented way. The fact of the matter is that the design of this system could not handle the failure of the primary database. A continuance-of-operation plan (COOP), if done properly, takes this kind of event into account and provides for the enterprise’s ability to keep working when a complete failure of the primary system occurs. COOP’s aren’t rocket science. They are well understood endeavors that are – or should be – part of any major enterprise design where mission-critical, it’s-gotta-work-every-time functions are handled.
Gov. McDonnell needs to be putting together an audit of this contractor’s work (NG and EMC both) that makes no assumptions. He should also be auditing VITA and find out why this situation (referring to the lack of a back up that could handle the load) was allowed to exist. Whether this contract should have been extended or not is beside the immediate point and this situation, bad as it is, is not an indictment of the practice of privatizing functions supporting government operations. Northrop Grumman needs to put the design in place to see to it that the government operation doesn’t go down regardless of major crashes like this. Finding out where they fell down is the 1st step. Following through and making them correct this situation – under this existing contract – is step 2.
Thanks to an e-mail I received from VA State Delegate Tom Rust (VA-86), it’s come to my attention that a major information systems failure is currently affecting Virginia’s government systems. The DMV is affected:
Statewide Computer Issue Halts In-Person Driver’s License Transactions Vehicle Titles, Decals, Etc. Still Being Processed in DMV Offices
RICHMOND – Due to a statewide computer system problem, the Virginia Department of Motor Vehicles is currently unable to process driver’s license transactions inside its 74 customer service centers in Virginia. Customers who can renew licenses through another service option are encouraged to go online at www.dmvNOW.com, or use the automated telephone service at 1-888-337-4782. The Virginia Information Technologies Agency (VITA) is working to resolve the server problem that is affecting 24 different state agencies. VITA has not reported a recovery time.
At DMV, all other types of transactions, including vehicle decals and titles, transcripts, etc. are being processed normally.
That’s an interesting comment – the web site appears to work and it’s handling transactions that the in-person, on-site DMV employees can’t? As a network professional, the situation interests me and suggests quite a number of scenarios. Clearly, if the transaction resources aren’t completely dead (and they’re not, or you wouldn’t be able to do them on the web site) and the interface web site isn’t dead (ditto) then the issue must be one that affects the local sites directly. That’s either a connectivity problem (which wouldn’t affect multiple locations simultaneously) or there’s some issue with the internal systems at the DMV that’s not affecting external systems such as the DMV’s site.
The outage, flaring Wednesday afternoon and expected to disrupt some services through the weekend, is attributed to 228 malfunctioning servers, which supply shared software and applications to clusters of state agency computers.
Twenty-six of more than 80 state agencies were hit by the shutdown, including the office of Gov. Bob McDonnell.
“We’re disappointed to have a failure, an outage of this magnitude,” Samuel A. Nixon Jr., head of the Virginia Information Technologies Agency, said yesterday. “No matter what you do, it’s going to happen on occasion.”
Ah, so what’s happened appears to be that several servers handling internal access to resources have died. The external ones that we all access via the Internet (likely located in what’s called a DMZ) were apparently unaffected. The fact that those servers can still apparently access the data means that the data stores appear to be available and intact, it’s just that the internal servers that handle the government employee’s interfaces are down.
I’m going to have to take some issue with that last comment by Mr. Nixon at VITA. I can assure you that there are companies out there (banks, Google, Amazon) who architect their environments to make it as sure as is humanly possible that such “occasions” do not occur. The story at WSLS indicates that the infamous VITA-Northrop Grumman IT contact is at the heart of this. NG’s subcontractor EMC (a network storage company) was reportedly “checking for faulty equipment” and, in the process, killed the 228 servers. That’s a helluva faulty equipment check that manages to take out a couple of hundred mission-critical systems. And do it badly enough, I might add, that it takes 5 days to recover. Where are the backup systems?
Nixon also said that the interruption was of insufficient magnitude to activate a backup system at a duplicate computer center in Russell County, in Southwest Virginia.
A full-on, deader-than-a-doornail failure is of “insufficient magnitude” to “activate” a backup system? Sorry, again, but I call bullshit on that one. Any backup system that can’t step in and handle the task for which it was designed to back up when the primary drops dead of brain failure isn’t a backup system at all – it’s a completely separate system that’s now sucking up power and providing nothing but space heating. I may not be privy to the internal discussions and the full picture of the situation but that line of crap is a man just throwing chaff into the air hoping to confuse people. If he’s serious, then he either doesn’t know what the hell he’s talking about or he’s being lied to.
Either way, an investigation needs to be performed and the citizens of Virginia need to be apprised of the results, in clear language. Either the government wasn’t clear or knowledgeable enough to articulate proper requirements or the contractor simply isn’t performing according to the contract. Either way, that needs to be fixed and fixed now.
I was told this morning by a very trusted source (who is also on the exploratory committee) that Dick Black is in the running for a State Senate seat in 2011. Given that we still don’t know what the redistricting process is going to look like we don’t know which seat. It’s going to be one of the seats up here, however, so keep your eyes open.
Oh-ho-ho-ho, noooo…. I can not wait to see the reaction of Loudoun Insider over at Too Conservative on this one:
One former Virginia politician is not sure at this point whether his time in the limelight of the commonwealth’s politics has faded. Both parties are speculating about whether this is the return of Richard “Dick” Black, a former member of the House of Delegates who reigned over Loudoun County from 1998 through 2005, when he was ousted by Democrat David Poisson. Black, a far-right conservative, said he is “very seriously considering” a run for state Senate in 2011, potentially in the 33rd Senate District.
Actually, as I was putting this post together, I recalled seeing some small post over there at TC about this and went looking for it. Sure enough, the word’s already made it over there. Loudoun Insider sums up his take on this news with 1 question: “Can any sane Republican really think this is good news?”
One of the commenters on the TC post asks what’s the big deal, highlighting very nicely the point where the general electorate is concerned: Black’s been out of the political eye for about 5 years now. I don’t have the exact figures but I’m guessing there’s been a lot of people move into Loudoun in the last 5 years who have never even heard of Dick Black, let alone have had any dealings with him to form an opinion. I would suggest that simply writing all of those people off as “not sane” (should Black become the Republican nominee and they choose to support Republicans) isn’t a winning approach.
I would also like to point out to those people who are adamantly against Black and who are predicting an election-day disaster should he be the GOP nominee that much the same was said about Ken Cuccinelli during the 2009 elections. He was going to be some kind of social uber-conservative embarrassment that would drive away voters in droves, remember? Check out my writing right here and you’ll see that I was one of those people concerned about that. Didn’t happen, not even close. Dick Black’s got some wince-inducing moments in his history, that’s for certain, and he should be required to answer Loudoun’s Republicans’ questions on whether he’d do it all over again. He’s got a bit higher of a hurdle to jump than other candidates in convincing his fellow Republicans but that’s not the same as saying he’s got no chance. As for those people who remain strenuously opposed to his candidacy, I would suggest they get involved early and remain so in pressing for a candidate they approve of more.
It’s still early in this and, I’ll say again, no decisions have been made. We have far more immediate concerns in supporting our Congressional candidates like Frank Wolf, Keith Fimian, Patrick Murray, and all the rest of Virginia’s GOP candidates. I’m keeping my eyes open, that’s for sure, but we’ve got priorities. Let’s stay focused.
A program run by the US Immigration and Customs Enforcement (ICE) agency within the Department of Homeland Security that trains local law enforcement to begin immigrations proceedings against criminals captured in their localities is called the Section 287(g) program, a reference to its number in the legal code. Under 287(g) a suspect arrested for a crime may, if the local law enforcement officer (LEO) has reasonable doubt as to his legal residency, be checked out under ICE databases to confirm his immigration status. If it’s determined that the suspect is in the US illegally, he may be detained by the local LEO’s and turned over to ICE for deportation proceedings. Loudoun County finally signed onto a 287(g) agreement in June of 2008. While the agreement has, reportedly, been altered somewhat by ICE it remains an effective tool to combat illegal alien presence in Loudoun.
Virginia’s Governor Bob McDonnell is apparently pursuing such an agreement for our state troopers to use all across Virginia:
Gov. Bob McDonnell is pursuing a deal with federal officials that would allow state troopers to enforce immigration law, effectively expanding statewide a program that a handful of Virginia counties have implemented.
The move comes as state Attorney General Ken Cuccinelli issued a legal opinion saying state police officers can check the immigration status of anyone they stop or arrest.
Though the decision prompted a call for the governor to codify that decision in state law, McDonnell’s office said he is already working toward giving state police new enforcement powers.
Good for you, Governor. There’s no reason at all for our state troopers to not be participating in this program and having a uniform ability across the state will only enhance Virginia’s safety in the long run. I’ll post more on this as this situation unfolds.
I had a very interesting conversation this morning with a noted member of the local media who’s preparing a story on the blogosphere (I would guess with an eye toward the local Loudoun ‘sphere, particularly). I’ll wait until I see whether the story is actually going to go anywhere before I mention any specifics but it should be an interesting piece. Stay tuned.
Update: I see that noted Too Conservative contributor Loudoun Insider was also contacted. I’m curious to see the scope of this story, now.
After years of Christmas displays being set up with nary a complaint on the courthouse grounds in Leesburg, VA, last year’s display came under attack by people who apparently can’t abide the thought of someone believing something they don’t strongly enough to want to display something for a nationally-known holiday. In short, some person demanded that the Christmas display be taken down and never return. To those people I’m going to refer this quote from noted philosopher Aristotle:
It is the mark of an educated mind to be able to entertain a thought without accepting it.
How they can consider themselves both more cerebral and advanced than the people who support having a Christmas display – and, oh yes they do – while being apparently unable to remain intellectually intact at the mere sight of a display of other peoples’ religious faith at a time of one of their most revered religious holidays defies rational understanding. If they are that mentally weak, that morally porous, that a traditional holiday display will somehow wreck their worldview, then I would suggest they’re far too fragile to be let out of the house.
What applies to citizens in this country also applies to a government of the people 1. It is absolutely possible for a government to permit a display of something in the public space without endorsing it. The mere mention of a religion does not constitute an establishment of that religion. And, most certainly, neither does the refusal of a government to deny citizen speech of a religious nature constitute an establishment or an endorsement of that religion. There is no right to be insulated from any expression of a fellow citizen while going about in the public space.
The issue was brought up again with the Board of Supervisors and scheduled to be heard in their July meeting, held just this past Monday, July 19th. As with the previous matter of the CBPO, the public turnout was quite strong and, apparently, overwhelmingly against banning displays on the courthouse grounds. Once again (as with the CBPO), the Board voted to delay a decision on this matter until later – in this case until September. However, unlike the decision to kick the can down the road on the matter of the CBPO – which was done in order to give the Board more time to “educate” the public on the matter, since we’re clearly too stupid to understand things as demonstrated by our opposition – this decision was delayed in response to a specific request by Judge Thomas Horne, Chief of the 20th Judicial Circuit Court.
Judge Horne’s request to delay was based on the fact that he was going to be out of town on the night the vote was to take place (Tuesday, July 20). According to the report in the Loudoun Times-Mirror, Horne’s request was for additional time to allow “all our judges to be given the opportunity to consider, should they wish to do so, the impact of the use of the grounds on the operation of their Courts.” So, let me get this straight: a matter of the governance of Loudoun County, one that clearly has great interest to the public, is being delayed because 1 man, a man not part of the County’s elected leadership nor a participant in the legislative process outside of being a fellow citizen, was going to be out of town? Is this a new precedent? That any time a matter comes before the Board that affects a given citizen who’s going to be out of town on the day the Board is to vote on the matter then the matter gets delayed until that guy comes back? And what about his being out of town precluded the other judges from “consider[ing]… the impact of the use of the grounds”? They can’t consider things without his presence?
Which brings me to the other part of Judge Horne’s request that has me bristling: he thinks the judges should be allowed to consider the impact of the use of the grounds on the operation of their Courts? Excuse me, sir, those ain’t “their” courts. They’re our courts. They have been granted the authority to use our courts to adjudicate matters of law and justice by we, the People, in our consent to be governed. No mistake should be made, however, that ownership of those courts – and most certainly of the surrounding terrain – is held firmly in the hands of the citizens, not the individual judges. Their concerns should absolutely be taken into account but that’s what public hearings are for. What, may I ask, kept those judges from coming to the public hearing on Monday and speaking out? I hope that appearances are deceiving in this case but it certainly appears that what kept them from doing so was the assumption of being owed preferential treatment on this matter.
I fail to see how the presence of a static display outside of the courthouse would affect the operation of a courtroom, particularly one secured and monitored by such a capable police force as Loudoun has deployed there in Leesburg.
In the end, the courthouse grounds are the public’s grounds. This is a matter for the public, through their elected representatives and leaders, to decide and that process involves public hearings, debate, and votes. That mechanism is as accessible to members of the judiciary as it is to network engineers and they have the same obligation to make use of it as it is presented, should they choose to do so. The Board, well intentioned as it might have been, made the wrong decision and set the wrong precedent. It’s done, however, and that’s that. I would hope that the Board is explicit that this request should not be viewed as permanent policy and the judges in Leesburg are now expected to be timely in any commentary they may choose to give. Clock’s ticking.
Following a link on Hot Air that asked the question about whether Virginia Congressman Eric Cantor had “gone rogue”, I started reading this story on Politico by John Bresnahan and Jake Sherman titled, “Eric Cantor’s ambition raises concerns, debate.” I didn’t even make it out of the 2nd paragraph before I ran into something so jarring to read from allegedly informed professional journalists that I stopped in my tracks.
House Minority Whip Eric Cantor has asked the ethics committee to greenlight a national book tour this August for a new GOP manifesto he’s co-authoring with two younger members of Congress, according to sources familiar with the situation.
This is classic Cantor: a hyperambitious move to publish and push ideas he thinks will help rebrand the GOP, on his terms — and not necessarily those of his boss, Minority Leader John Boehner.
Emphasis mine. Newsflash to Politico: no elected representative from Ohio holds the title “boss” when referring to an elected official from Virginia. The “bosses” of Virginia Congressman Eric Cantor are the residents of Virginia’s 7th District – and don’t you for one minute forget it!
As Republicans we caucus with Republicans from other states because that’s how you get things done in a legislative body and we acknowledge leaders within that caucus who – with the input of our state’s representatives – formulate strategy to achieve those ends. But a “boss” he ain’t and both journalists and Ohio representatives should keep that firmly in mind. Cantor owes his work and allegiance to the people who elected him as their representative, not to the leadership of a party or a caucus. That’s how we get legislators in power who pass laws the vast majority of our citizens don’t want.
Search the web:
RSS Feed
Ric's Twitter