I've made a couple abortive blog starts in the past so… no promises! I'll also be putting up some essays that I've written in the past, probably reworked to save embarrassment.
Wednesday, 29 April 2009
Let the macro-blogging begin...
I'm setting this blog up as place to put thoughts that don't fit into Twitter's 140 character limit.
Friday, 29 February 2008
Data can never be perfect...
[This was originally posted on an old blog: Data can never be perfect... ]
This is a re-post of a comment I made on Doug Henschen's article about data governance and the suprime crisis.
Doug's position (quoting from a lot of data governance vendors) is: "A first step toward avoiding such calamities... is an integrated, overarching data governance program that addresses data security, data privacy and data quality so that risks can be better understood and outcomes anticipated."
Basically, if the banks had better data they would have made better decisions and not got themselves into this mess.
The problem is not a lack of governance but an unshakeable belief in the data and risk models. Interested readers should look at "Did Black-Scholes Cause the Housing Bubble?" in Portfolio.
I'm a data guy, but every executive needs to understand that data is merely a map and the map is not the territory. If an explorer has a map that does not match the territory they can see, they would do well to question the map, rather than ask the territory to change.
The credit score is simply another map. There is evidence that they were significantly weakened by new financial products over the last 7 years. Again, see "Credit Scores: Not-So-Magic Numbers" for details.
Data quality, data governance, etc. are all **super** important. However, as data professionals we need to build systems that incorporate common sense, human based checks and balances. Trusting too much in software will eventually get you fired or indicted for criminal negligence.
This is a re-post of a comment I made on Doug Henschen's article about data governance and the suprime crisis.
Doug's position (quoting from a lot of data governance vendors) is: "A first step toward avoiding such calamities... is an integrated, overarching data governance program that addresses data security, data privacy and data quality so that risks can be better understood and outcomes anticipated."
Basically, if the banks had better data they would have made better decisions and not got themselves into this mess.
The problem is not a lack of governance but an unshakeable belief in the data and risk models. Interested readers should look at "Did Black-Scholes Cause the Housing Bubble?" in Portfolio.
I'm a data guy, but every executive needs to understand that data is merely a map and the map is not the territory. If an explorer has a map that does not match the territory they can see, they would do well to question the map, rather than ask the territory to change.
The credit score is simply another map. There is evidence that they were significantly weakened by new financial products over the last 7 years. Again, see "Credit Scores: Not-So-Magic Numbers" for details.
Data quality, data governance, etc. are all **super** important. However, as data professionals we need to build systems that incorporate common sense, human based checks and balances. Trusting too much in software will eventually get you fired or indicted for criminal negligence.
Friday, 18 January 2008
Social Networking: The new Rock n' Roll
[ This was originally posted on an old blog: Social Networking: The new Rock n' Roll ]
In the 1950's Rock n' Roll swept the (western) world and created a youth culture impact that still reverberates today. It also created a complete break between the "kids" who loved and the older generation who just didn't get it. Some said Rock n' Roll was undermining the moral fiber of the nation, some just thought it was a bunch of noise. The point is that music and culture where fundamentally changed by Rock n' Roll and everyone over a certain age was simply left behind. All of their objections and concerns simply became irrellevent. The kids just didn't care and they quickly found that they could define the world on their own terms.
Social Networking is the new Rock n' Roll, it creates a complete break between generations.
In our era Social Networking (Facebook, MySpace, Bebo, et al) is going to generate another break. Our culture will be fundamentally changed and a new generation will re-define how the world works. You don't have to look far (or listening long) to hear complaints about lost productivity in the workplace, moral decline, "infantile" behaviour and kids not understaning the "real world". Well, once again, the kids just don't care. They grew up with IM and email. Everyone they know or care to know is online all the time and they expect be there to.
I suppose that some of you who are 30+ (like me) will be thinking "I get it, I'm there". I congratulate you for being so hip, but the truth is that you don't/can't really get it. No matter how much you try to engage with this new paradigm it will always be an effort. Some of you friends just won't participate. Those that do won't be totally open, or totally engaged. You may feel like you're joining in but it will never be the same.
We're like old jazz fans who appreciate Rock n' Roll for it's roots in Jazz but really we long for something more nuanced and sophisticated. Roll on you cool cats.
In the 1950's Rock n' Roll swept the (western) world and created a youth culture impact that still reverberates today. It also created a complete break between the "kids" who loved and the older generation who just didn't get it. Some said Rock n' Roll was undermining the moral fiber of the nation, some just thought it was a bunch of noise. The point is that music and culture where fundamentally changed by Rock n' Roll and everyone over a certain age was simply left behind. All of their objections and concerns simply became irrellevent. The kids just didn't care and they quickly found that they could define the world on their own terms.
Social Networking is the new Rock n' Roll, it creates a complete break between generations.
In our era Social Networking (Facebook, MySpace, Bebo, et al) is going to generate another break. Our culture will be fundamentally changed and a new generation will re-define how the world works. You don't have to look far (or listening long) to hear complaints about lost productivity in the workplace, moral decline, "infantile" behaviour and kids not understaning the "real world". Well, once again, the kids just don't care. They grew up with IM and email. Everyone they know or care to know is online all the time and they expect be there to.
I suppose that some of you who are 30+ (like me) will be thinking "I get it, I'm there". I congratulate you for being so hip, but the truth is that you don't/can't really get it. No matter how much you try to engage with this new paradigm it will always be an effort. Some of you friends just won't participate. Those that do won't be totally open, or totally engaged. You may feel like you're joining in but it will never be the same.
We're like old jazz fans who appreciate Rock n' Roll for it's roots in Jazz but really we long for something more nuanced and sophisticated. Roll on you cool cats.
Blog Interrupted: It's been a while...
[This was originally posted on an old blog: Blog Interrupted: It's been a while... ]
Well it's been almost a year and a half since my last post. You would have been right to assume that this blog was dead but, just like Lazarus, it's alive again. Resurrected and better than ever.
I've been out of the Business Intelligence / Data Warehouse arena working in Database Marketing. It's definitely given me an extra perspective on data warehouse and the value of data for it's own sake. The first lesson was that it's not polite to laugh at people when they call a million row table "big";-). The second lesson was that sub-queries in SQL Server do not work and will never return.
Anyway, I'm back in the industry working as a Data Warehouse Designer / Architect so I'll be using this blog as a place to crystallise my thoughts on how data warehousing should be done. I also love to research new products in this space so I'll be putting company and product summaries here as well with links to loads of useful resources.
Enjoy.
Well it's been almost a year and a half since my last post. You would have been right to assume that this blog was dead but, just like Lazarus, it's alive again. Resurrected and better than ever.
I've been out of the Business Intelligence / Data Warehouse arena working in Database Marketing. It's definitely given me an extra perspective on data warehouse and the value of data for it's own sake. The first lesson was that it's not polite to laugh at people when they call a million row table "big";-). The second lesson was that sub-queries in SQL Server do not work and will never return.
Anyway, I'm back in the industry working as a Data Warehouse Designer / Architect so I'll be using this blog as a place to crystallise my thoughts on how data warehousing should be done. I also love to research new products in this space so I'll be putting company and product summaries here as well with links to loads of useful resources.
Enjoy.
Friday, 25 August 2006
Why is Business Intelligence innovation so slow?
[ This was originally posted on an old blog: Why is Business Intelligence innovation so slow? ]
You might have gathered that I'm a big fan of Business Intelligence. The potential that it has to impact a business is incredible (and largely unrealized - but that's another post).
I'm also a big fan of innovation. I had subscriptions to Popular Science and Omni as a kid (can you tell?) and now I get my fix for innovation thinking from the Killer Innovations podcast.
The point is that I've been wondering why the pace of innovation is slow in Business Intelligence.
One reason could be entrenchment. Classical BI/DW projects have been so costly and so complicated that the solutions, once delivered, have an incredibly high level of entrenchment. After all, the business thinks, now that we've spent so much on this solution we need squeeze every last bit of value out of it. I see this in action across all the elements of the solution.
How many BI "shops" do you know that have changed from, say, Oracle to SQL Server of from Cognos to BusinessObjects. I honestly don't know of any significant examples of an established shop switching (maybe a couple of instances of DW appliances being added on top). It seems that once the vendor is in it takes dynamite to get them out again.
This level of entrenchment is potentially extremely costly. If your vendor wants to double your "suport" fees or gouge you on additional licenses then you're pretty much forced to put up with it because you don't have an easy way out.
Another reason may be time. BI projects take time and lots of it. Kimball has taught us that successful BI projects are incremental and iterative (and big bang projects fail) but this implies a long term vision and the timescale to go with it. My hunch is that most shops aren't that receptive to "new and improved" tech when they're knee deep in the political quagmire half way through a project.
I could probably come up with more reasons... but (at the risk of repeating myself) I think it comes down to open standards and how thin they are in the BI space.
Sure we have ODBC, which is brilliant, but many key technologies don't use it, particularly if you're in a dedicated Oracle or SQL Server environment. For example, SQL Server Integration Services doesn't even give you the option to use ODBC against SQL Server databases (AFAIK). Meaning that you have to keep your warehouse on SQL Server of face up to the nightmare of re-writing all your ETL packages.
My plea to the vendors is this:
[ All of this reminds me that I need to do a post on open source BI. ]
You might have gathered that I'm a big fan of Business Intelligence. The potential that it has to impact a business is incredible (and largely unrealized - but that's another post).
I'm also a big fan of innovation. I had subscriptions to Popular Science and Omni as a kid (can you tell?) and now I get my fix for innovation thinking from the Killer Innovations podcast.
The point is that I've been wondering why the pace of innovation is slow in Business Intelligence.
One reason could be entrenchment. Classical BI/DW projects have been so costly and so complicated that the solutions, once delivered, have an incredibly high level of entrenchment. After all, the business thinks, now that we've spent so much on this solution we need squeeze every last bit of value out of it. I see this in action across all the elements of the solution.
How many BI "shops" do you know that have changed from, say, Oracle to SQL Server of from Cognos to BusinessObjects. I honestly don't know of any significant examples of an established shop switching (maybe a couple of instances of DW appliances being added on top). It seems that once the vendor is in it takes dynamite to get them out again.
This level of entrenchment is potentially extremely costly. If your vendor wants to double your "suport" fees or gouge you on additional licenses then you're pretty much forced to put up with it because you don't have an easy way out.
Another reason may be time. BI projects take time and lots of it. Kimball has taught us that successful BI projects are incremental and iterative (and big bang projects fail) but this implies a long term vision and the timescale to go with it. My hunch is that most shops aren't that receptive to "new and improved" tech when they're knee deep in the political quagmire half way through a project.
I could probably come up with more reasons... but (at the risk of repeating myself) I think it comes down to open standards and how thin they are in the BI space.
Sure we have ODBC, which is brilliant, but many key technologies don't use it, particularly if you're in a dedicated Oracle or SQL Server environment. For example, SQL Server Integration Services doesn't even give you the option to use ODBC against SQL Server databases (AFAIK). Meaning that you have to keep your warehouse on SQL Server of face up to the nightmare of re-writing all your ETL packages.
My plea to the vendors is this:
- implement open standards in your products (this will help me trust you)
- make migration to your product easy, whatever it takes (or else I'll never change).
[ All of this reminds me that I need to do a post on open source BI. ]
Wednesday, 16 August 2006
The enterprise software business model is broken
[This was originally posted on an old blog: The enterprise software business model is broken ]
Andy asks whether enterprise software is finished.
My view from the trenches is that the business model for enterprise software (and hardware…) is broken. They (particularly the BI guys) reportedly spend as much as 90% of their income on sales and marketing.
The number of sales people that I deal with at these companies is mind numbing. The number of reseller layers that get a cut of my license fees is shocking.
These companies could learn a huge lesson from the “web2.0″ upstarts by shortening their development cycles and moving their engineers out towards their customers. How do they know what needs to be improved, what needs to be simplified or what needs to be added? It certainly isn’t because I’ve told them.
What I really want are a set of building blocks that stack together nicely, use industry standards, can be interchanged if I’ not happy, can be managed by fewer (possibly better) people and can be consumedly easily across networks.
Is that too much to ask for?
BTW, my favourite example of getting this right is Netezza. I load my data into their box and my queries instantly run 1000x faster. It uses no proprietary SQL, has no indexes or tuning and I only need 1 DBA. Awesome.
Future post to come on this topic, but I think the market is looking for a complete solution that is relatively hardware and OS agnostic but does everything else. It also needs to come totally pre-configured with best practice set-ups. I'm not holding my breath.
Andy asks whether enterprise software is finished.
My view from the trenches is that the business model for enterprise software (and hardware…) is broken. They (particularly the BI guys) reportedly spend as much as 90% of their income on sales and marketing.
The number of sales people that I deal with at these companies is mind numbing. The number of reseller layers that get a cut of my license fees is shocking.
These companies could learn a huge lesson from the “web2.0″ upstarts by shortening their development cycles and moving their engineers out towards their customers. How do they know what needs to be improved, what needs to be simplified or what needs to be added? It certainly isn’t because I’ve told them.
What I really want are a set of building blocks that stack together nicely, use industry standards, can be interchanged if I’ not happy, can be managed by fewer (possibly better) people and can be consumedly easily across networks.
Is that too much to ask for?
BTW, my favourite example of getting this right is Netezza. I load my data into their box and my queries instantly run 1000x faster. It uses no proprietary SQL, has no indexes or tuning and I only need 1 DBA. Awesome.
Future post to come on this topic, but I think the market is looking for a complete solution that is relatively hardware and OS agnostic but does everything else. It also needs to come totally pre-configured with best practice set-ups. I'm not holding my breath.
Monday, 14 August 2006
The occupation of an Analyst
[ This was originally posted on an old blog: The occupation of an Analyst ]
This is the first instalment of my "book", as explained here. It’s a work in progress and your feedback would be a great help.
What is an Analyst? Recently it seems that every second job includes the term analyst.
When I was (briefly) at college, a friend had a mentor who would go on for hours (literally) about how devalued the term doctor had become. His premise was that there were too many people who were "officially" doctors of this or that but had just "cruised" through grad school and didn't deserve the accolade.
Now, 12 years later, I feel very much the same way about the title of Analyst. Everywhere I look in organisation charts I see people who are ostensibly employed as analysts. Sadly very few of these seem to actually do any analysis. Such as a Network Support Analyst who's job is correctly route the LAN cabling or a Helpdesk Analyst who's job is to deal with support calls to reset user passwords. Important jobs but not really analysts.
The occupation of an Analyst is to analyse. To paraphrase Webster's, analysis is the “resolution of anything into its constituent elements” or the creation of a “brief methodical illustration of the principles” of a subject.
For me, an Analyst seeks to:
If you can twist what you do into this framework then I salute you as a fellow analyst.
Now get back to work.
This is the first instalment of my "book", as explained here. It’s a work in progress and your feedback would be a great help.
What is an Analyst? Recently it seems that every second job includes the term analyst.
When I was (briefly) at college, a friend had a mentor who would go on for hours (literally) about how devalued the term doctor had become. His premise was that there were too many people who were "officially" doctors of this or that but had just "cruised" through grad school and didn't deserve the accolade.
Now, 12 years later, I feel very much the same way about the title of Analyst. Everywhere I look in organisation charts I see people who are ostensibly employed as analysts. Sadly very few of these seem to actually do any analysis. Such as a Network Support Analyst who's job is correctly route the LAN cabling or a Helpdesk Analyst who's job is to deal with support calls to reset user passwords. Important jobs but not really analysts.
The occupation of an Analyst is to analyse. To paraphrase Webster's, analysis is the “resolution of anything into its constituent elements” or the creation of a “brief methodical illustration of the principles” of a subject.
For me, an Analyst seeks to:
- understand the present
- by examining the past
- in order to influence the future.
- by examining the past
If you can twist what you do into this framework then I salute you as a fellow analyst.
Now get back to work.
Subscribe to:
Posts (Atom)