What happens when important software breaks?

Something happened on 5/3/2010 that isn't being explained. On 5/3 the Dow instantly plunged 1000 points in an unexplained free fall and then magically recovered. Several theories came and went but no specific explanation every stuck. Then, the next day the NASDAQ nullified trades that were "out of bounds". On the 7^th the WSJ reported,

"Meanwhile a heated debate has emerged about the various rules that govern equities trading. The biggest U.S. stock exchanges Friday fired shots at one another, each alleging that the other's market model contributed to the chaos."

Here's specifically what bothers me. In my mind you don't reverse or nullify trades unless you know what happened. If the situation that resulted in this is 'unexplained' then on what grounds do you reverse millions, tens of millions, or hundreds of millions of dollars of trades? Some have pointed out that they simple unwound the trades, no harm no foul. Well, not really. Although the trades are reversed the market can't be dissected like that. People made other investments, based on the investments that came before. It's not like they "rolled back" the stack market as if it were a database restore.

So why did some of the trades get unwound? I think someone does know what happened and the reason that information is not public is it would erode confidence in the system, and probably for a good reason.

I am a reasonably aggressive market investor with the help of a manager that I have used for several years. I mostly invest in ETF's rather that individual stocks. I think you have to be in the market for the long haul, just as everyone says. But I also have a substantive lack of trust in computers and the software that drives them. Not Ted Bundy'esc, but rather a healthy skepticism and understanding that complex software is actually complex. It's hard to write software for every conceivable situation and it's very hard to test software correctly. Ultimately, it's not that difficult for a system to be put into a situation that is beyond is design considerations. Computer driven automated trades that hold stocks for milliseconds anyone?

I suppose this opinion comes from a career designing, writing, testing, debugging, and apologizing for software. At NASA we tested flight control software within an inch of its life and still missed stuff from time-to-time. By comparison all other software, outside that aforementioned "man rated software", is tested about 5% as rigorously in my experience.

I NEVER play any game in Las Vegas that is controlled by software. A game of chance is not a game of chance if it is controlled by someone's code to simulate chance. In fact this came up a week or two ago when someone won $42M on a slot machine the Lady Luck casino in CO. That turned out to be a machine malfunction. Really.. I'm sure it was, but I doubt it as a hardware problem.

It seems to me that software has finally reached that turning point where it can create meaningful and large scale disasters. This of course is inevitable in the evolution of automation and complex systems. It's also inevitable that these things will happen. Was 5/3 one of those situations?

Comments

[scott evans]

You should read The Quants. A book up your alley.

[rando]

Everyone is so focused on the sci-fi scenario of robot computers taking over the world that they seem to have let the computers quietly take over the infrastructure. 5/3 was interesting and you are probably correct: there is more to it than is public. It hit me that humans are serving the machines when, more than once, I stand with a valid credit card in my hand to buy some goods, and the clerk looks at me blankly and says "sorry the computer won't accept it, I don't know what to do..."

[software testing services]

Yeah I am with you rando. You are right. Now a days, Everyone is so focused on the sci-fi scenario of robot computers taking over the world hat they seem to have let the computers quietly take over the infrastructure.