Data can never be perfect...

[This was originally posted on an old blog: Data can never be perfect... ]

This is a re-post of a comment I made on Doug Henschen's article about data governance and the suprime crisis.

Doug's position (quoting from a lot of data governance vendors) is: "A first step toward avoiding such calamities... is an integrated, overarching data governance program that addresses data security, data privacy and data quality so that risks can be better understood and outcomes anticipated."

Basically, if the banks had better data they would have made better decisions and not got themselves into this mess.

The problem is not a lack of governance but an unshakeable belief in the data and risk models. Interested readers should look at "Did Black-Scholes Cause the Housing Bubble?" in Portfolio.

I'm a data guy, but every executive needs to understand that data is merely a map and the map is not the territory. If an explorer has a map that does not match the territory they can see, they would do well to question the map, rather than ask the territory to change.

The credit score is simply another map. There is evidence that they were significantly weakened by new financial products over the last 7 years. Again, see "Credit Scores: Not-So-Magic Numbers" for details.

Data quality, data governance, etc. are all **super** important. However, as data professionals we need to build systems that incorporate common sense, human based checks and balances. Trusting too much in software will eventually get you fired or indicted for criminal negligence.