What you can do with large datasets

books:


	• Super Crunchers: Why Thinking-by-Numbers Is the New Way to Be Smart Ian Ayres Bantam, 2007 - 272 pages average customer review:based on 71 reviews view larger image
	for more information click here

Entertaining, but far from super

This is an easy and mostly entertaining read. The author uses many anecdotes to
persuade us that statistics can be a useful tool for decision making. Some of
the described applications use lots of data and multiple regression. Those are
easier to do now than they used to be, because more data is collected and kept.
Some are trivial. If your company hurts a customer, apologize. You might get
some ideas of thing to do that might help your organization. You will not get
any detailed help about how to implement the improvement, but there is a good
chance there is enough information that some systems person can figure out what
other skills are needed to make the idea work.

There is some discussion of limitations on the methods, and some warnings about
potential abuse, but not enough. Ayres seems to confuse correlation with causation.
He also frequently assumes the sample is representative of the population.
Even when trying to make the sample representative, it often is not. He also
assumes the answer is in the data. Sometimes it is not. Ayres reports a study
concluding widespread point shaving in college basketball because a distribution
at game end did not match the distribution five minutes earlier when a highly
favored team was ahead by about the spread. I have no opinion about the conclusion,
but the simpler explanation of the coach thinking it was late enough to safely
let the weaker players participate more was not considered.

Regression is a powerful tool, but it is easy to misuse. For an ongoing
survey of misuses, see junkfoodscience dot com, a blog. Many of the entries show
the flaws in statistical claims of medical trials. Also try stats dot org.

for more information click here

The End of Intuition

The author explains that he originally intended to title the book, "The End of Intuition". I think that would have been the better title. Interestingly, the name change resulted in a clever use of Google's adwords. The problem with the title Super Crunchers is that it makes the impression that the book is about data mining huge data sets. I have read one scathing review in the KDD community that makes this point, but data mining is not the focus of the book. If you are looking for a book about data mining, it is a poor choice.

The theme of the book is how data, and data analysis, is making inroads in industries and firms where it has not been emphasized in the past and how the use of data to make decisions is often fiercely resisted. The techniques will seem mundane to professional analysts. I won't be recommending this to many of my data mining colleagues - there is nothing really new here. However, this book does have its place. I often meet analytically oriented managers that are trying to introduce advanced analytics into departments (or entire firms) that are not accustomed to data based decision making. It doesn't matter that Ayres' examples center on ordinary multiple regression, or standard randomized trials. This is where the battle is being fought and won in many organizations. The reluctance of some doctors to embrace "evidence based medicine" made for compelling reading.

The examples Ayres uses are diverse and unusual: wine rating, medicine, welfare reform, prison sentencing, dating web sites, Capital One and others. Some are much better than others. I didn't learn anything about data mining or analysis while reading this book. But I did learn something about people, and it confirmed some things I have learned about trying to deploy models in organizations. So while this book is not without flaws, if you want to better understand why there is often so much reluctance to use data to drive decisions it is worth a read.

for more information click here

What you can do with large datasets

The answer is of course: a lot.
And Ian Ayres' book will tell you a little about it.

Super crunchers are those who use lage datasets
to find patterns in human behaviour, and
predict the future based on these large datasets.

The book informs us that super crunching is on the verge of being
used all over. E.g.
Chess grandmaster Kasparov was no match
for IBMs Deep Blue chess computer,
that stored some 700.000 grandmaster chess games to help find the
winning move.
The IRS could use its data to tell a small business,
if it is spending too much or too little on advertising.
Indeed, the IRS probably has enough data to
make good estimates on whether business, marriages, etc. etc.
will fail - based only on comparison with its existing dataset.

For the paranoid, it is a horror that supermarkets could map your life cycle and predict your next purchases pretty accurately (based on
what other similar customers did).
For the optimist data mining is a good thing and we'll all lead better lives because of it.

Want to write a bestseller about it? Compare your title and some key words with data from a database of books, titlescore.com, containing millions of bestsellers and flops, and you will get your answer.

It all seems pretty straight forward, and the book has some nice examples of what we can expect in the coming years.

-Simon

for more information click here

reviews: page 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

Why would a casino try and stop you from losing? How can a mathematical formula find your future spouse? Would you know if a statistical analysis blackballed you from a job you wanted?

Today, number crunching affects your life in ways you might never imagine. In this lively and groundbreaking new book, economist Ian Ayres shows how today's best and brightest organizations are analyzing massive databases at lightening speed to provide greater insights into human behavior. They are the Super Crunchers. From internet sites like Google and Amazon that know your tastes better than you do, to a physician's diagnosis and your child's education, to boardrooms and government agencies, this new breed of decision makers are calling the shots. And they are delivering staggeringly accurate results. How can a football coach evaluate a player without ever seeing him play? Want to know whether the price of an airline ticket will go up or down before you buy? How can a formula outpredict wine experts in determining the best vintages? Super crunchers have the answers. In this brave new world of equation versus expertise, Ayres shows us the benefits and risks, who loses and who wins, and how super crunching can be used to help, not manipulate us.

Gone are the days of solely relying on intuition to make decisions. No businessperson, consumer, or student who wants to stay ahead of the curve should make another keystroke without reading Super Crunchers.

for more information click here