Economics and Big Data – A Match Made In Heaven?

I’ve said many times that I think there is an important role that folks with an economics background can play in the world of Big Data. Maybe it’s wishful thinking on my part to believe that I can find my son Alec (who is an Economics major from Saint Mary’s College in Moraga, CA) a job in Big Data. But he’s got his own, more exciting dreams (heck, what could be more exciting than a world of Hadoop, R, MapReduce, Yarn, Pig, Stinger, Oink, Zookeeper…). But seriously, let’s examine why a team member with an economics background might be a valuable addition to your Big Data team.

What Is Economics?

Let’s start with a definition of “economics[1]” (preferably one that I can understand):

Economics is the social science that studies how individuals and organizations manage scarce resources to achieve desirable ends. The object of economics is to understand the behavior of organizations when they have multiple ends in sight, limited resources to obtain them, a set of preferences, and the capability of making a choice.

An economic problem exists whenever there are scarce resources available that have alternative uses and a decision has to be made to attain the best possible outcome under bounded rational conditions.

In other words, the goal of economics is to maximize the business value subject to the constraints imposed by the available information, the cognitive limitations of the team of people, and the finite amount of time they have in which to make a decision.

Economists have to balance these factors as they evaluate different alternative uses of their people and financial capital—within information, organizational, political, and time constraints—in order to make a financially rational (optimal) decision.

Understanding Economic Utility

Economic Utility is one of the more important economic concepts that decision makers employ when they look to make investment or business decisions.  Let’s start with a definition:

Economic Utility is the ability of a good or service to satisfy a customer’s needs or wants. Utility is an important concept in economics (and game theory as well) because it tries to measure satisfaction experienced by a consumer from goods or services received.

Since one cannot directly measure the benefit, satisfaction, or happiness resulting from received goods or services, economists have devised ways of representing and measuring utility in terms of economic choices that can be evaluated.

Economists consider utility to be revealed in people’s willingness to pay different amounts of money for different, but related goods and services. Economists determine economic utility by using tools such as the indifference curve (see Figure 1) that plots different combinations of goods or services that individuals or organizations would accept to maintain a given level of satisfaction.

Figure 1: Indifference Map with 3 indifference curves

An indifference curve is a graph showing different bundles of goods amongst which a consumer is indifferent to their relative value. That is, at each point on the curve, the consumer has no preference for one bundle over another on that particular curve. One can say that each point on the indifference curve renders or delivers the same level of utility (satisfaction) for the consumer.

An indifference curve concept is a valuable tool in guiding organizations to make the trade-off between different big data opportunities that are competing for the same human and financial resources.

Pareto Efficiency and Resource Allocation

Another useful economic concept is Pareto efficiency[2]. Pareto efficiency, or Pareto optimality, is a state of allocation of resources in which it is impossible to make any one individual better off without making at least one individual worse off.

The concept behind Pareto efficiency is that given an initial allocation of goods among a set of individuals or projects, a change to a different allocation that makes at least one individual or project better off without making any other individual or project worse off is called a Pareto improvement. An allocation is defined as “Pareto efficient” or “Pareto optimal” when no further Pareto improvements can be made.

Pareto efficiency can be applied to the selection between alternative choices, much like we see when deciding where and how to start our Big Data journeys.

Figure 2: Pareto Efficiency

Here’s an example of Pareto efficiency. Let’s say a factory can produce different quantities of two goods, as shown in Figure 2. One point is said to dominate another point on the chart if it is better on the Pareto dimensions. For example, points D and E dominate point K because both D and E have higher production levels of each item than point K. The Pareto Efficient Frontier is the set of non-dominated points, shown here in red (A-H). All of the grey points are said to be Pareto inefficient.

Economic Theory and the Big Data Prioritization Process

Economists bring a discipline for making rational (optimal) financially based decisions subject to the constraints imposed by the available information, the cognitive, organizational, and political limitations of the team, and a finite amount of time in which to make a decision. Economist use tools like the indifference curve and Pareto efficiency to facilitate making the necessary trade-offs.

Deciding where and how to start your Big Data journey involves many of the same challenges and trade-offs. As discussed in a previous blog “Prioritization Matrix: Aligning Business and IT on the Big Data Journey,” the prioritization matrix process enforces a discipline for helping organizations make a choice with respect to where and how to start their Big Data journey.

Much like the indifference curve concept, the prioritization matrix process helps organizations balance and evaluate the different options (or alternative uses) for where they could apply their human and financial capital to start the journey.

What are the “right” use cases that balance business value, the cognitive limitations (or biases) of the team, and a finite amount of time around which to deliver that business value?

The Prioritization Matrix Process

The prioritization matrix facilitates the discussion between the business and IT stakeholders in identifying the “right” use cases to start a big data initiative—use cases with both meaningful business value (from the business stakeholders’ perspective) and feasibility of successful implementation (see Figure 3).

Figure 3: Prioritization Matrix

The prioritization matrix process leverages group dynamics to prioritize the different Big Data use cases. The team (which must include both business and IT stakeholders) decides upon the placement of each use case on the prioritization matrix (weighing business value and implementation feasibility) vis-à-vis the relative placement of the other use cases on the matrix (see Figure 4).

Figure 4: Prioritization Process

The heart of the prioritization process is the discussion that ensues about the relative placement of each of the use cases on the priority matrix, such as:

• Why is use case [B] more or less valuable than use case [A]? What are the specific business drivers or variables that make use case [B] more or less valuable than use case [A] (see Figure 2)?
• Why is use case [B] less or more feasible from an implementation perspective than use case [A]? What are the specific implementation risks that make use case [B] less or more feasible than use case [A]?

Summary

The discipline introduced by economics can play a major role in helping organizations weigh different variables and constraints (around business value, organizational limitations and biases, and finite amounts of time) to identify and prioritize those Big Data use cases and business opportunities. Economics helps when there are scarce resources with alternative uses to be deployed and a decision has to be made to attain the best possible outcome under bounded rational conditions.

Sounds like a Big Data challenge to me.