By Rob Farber
Technology and computational evangelists quickly learn that human psychology is a key component of any project roadmap. The truth in the quip that it took one social genius plus 500,000 scientists and engineers to put a man on the moon can be appreciated when one observes the social issues that surface in technical discussions within even small groups of people. As the name implies, data-intensive computing requires large amounts of data.

My article “Big Money for Big Data” in the June 2012 issue of HPC Source notes that massive parallelism is the only path forward for organizations wishing to cope with the ever-increasing size of big data sets. Just as technology is changing the meaning of “big” data, so is it increasing what is meant by “massive” parallelism. At the moment, GPU programmers work with tens of thousands of threads, while multi-core programmers utilize tens of threads. This technical dichotomy has created a tension between the CTO (chief technical officer) who is responsible for defining the data processing goals to keep an organization competitive over time, and those within the organization vested with the responsibility to evaluate, integrate and manage the technology used to reach those goals within a production environment.

Full Story