Big Data Revolution What farmers, doctors and insurance agents teach us about discovering big data patterns

Big Data Revolution is a guide to improving performance, making better decisions, and transforming business through the effective use of Big Data. In this collaborative work by an IBM Vice President of Big Data Products and an Oxford Research Fellow, this book presents inside stories that demonstrate the power and potential of Big Data within the business realm. Readers are guided through tried-and-true methodologies for getting more out of data, and using it to the utmost advantage. This book describes the major trends emerging in the field, the pitfalls and triumphs being experienced, and the many considerations surrounding Big Data, all while guiding readers toward better decision making from the perspective of a data scientist.

Companies are generating data faster than ever before, and managing that data has become a major challenge. With the right strategy, Big Data can be a powerful tool for creating effective business solutions ? but deep understanding is key when applying it to individual business needs. Big Data Revolution provides the insight executives need to incorporate Big Data into a better business strategy, improving outcomes with innovation and efficient use of technology.

• Examine the major emerging patterns in Big Data
• Consider the debate surrounding the ethical use of data
• Recognize patterns and improve personal and organizational performance
• Make more informed decisions with quantifiable results

In an information society, it is becoming increasingly important to make sense of data in an economically viable way. It can drive new revenue streams and give companies a competitive advantage, providing a way forward for businesses navigating an increasingly complex marketplace. Big Data Revolution provides expert insight on the tool that can revolutionize industries.

Prologue 1

Berkeley, 1930s 1

Pattern Recognition 2

Nelson Peltz 3

Committing to One Percent 5

The Big Data Revolution 6

Introduction 7

Storytelling 7

Objective 7

Outline 8

Part I “The Revolution Starts Now: 9 Industries Transforming with Data” 8

Part II “Learning from Patterns in Big Data” 11

Part III “Leading the Revolution” 11

Storytelling (Continued) 13

Part I: the Revolution Starts Now:9 Industries Transforming With Data 15

Chapter 1: Transforming Farms with Data 17

California, 2013 17

Brief History of Farming 18

The Data Era 19

Potato Farming 20

Precision Farming 21

Capturing Farm Data 22

Deere & Company Versus Monsanto 24

Integrated Farming Systems 25

Data Prevails 26

The Climate Corporation 26

Growsafe Systems 27

Farm of the Future 27

California, 2013 (Continued) 29

Chapter 2: Why Doctors Will Have Math Degrees 31

United States, 2014 31

The History of Medical Education 32

Scientific Method 32

Rise of Specialists 33

We Have a Problem 34

Ben Goldacre 35

Vinod Khosla 35

The Data Era 36

Collecting Data 36

Telemedicine 38

Innovating with Data 40

Implications of a Data-Driven Medical World 42

The Future of Medical School 42

A Typical Medical School 42

A Medical School for the Data Era 43

United States, 2030 44

Chapter 3: Revolutionizing Insurance: Why Actuaries Will Become Data Scientists 45

Middle of Somewhere, 2012 45

Short History of Property & Casualty Insurance and Underwriting 46

Actuarial Science In Insurance 47

Pensions, Insurance, Leases 49

Compound Interest 50

Probability 50

Mortality Data 50

Modern-Day Insurance 51

Eight Weeks to Eight Days 51

Online Policies 52

The Data Era 52

Dynamic Risk Management 52

Catastrophe Risk 54

Open Access Modeling 55

Opportunities 56

Middle of Somewhere, 2012 (Continued) 58

Chapter 4: Personalizing Retail and Fashion 59

Karolina 59

A Brief History of Retail 60

Retail Eras 60

Aristide Boucicaut 61

The Shift 62

The Data Era 63

Stitch Fix 63

Keaton Row 65

Zara 66

Karolina (Continued) 67

Chapter 5: Transforming Customer Relationships with Data 69

Buying a House 69

Brief History of Customer Service 70

Customer Service Over Time 70

Boeing 72

Financial Services 74

The Data Era 75

An Automobile Manufacturer 76

Zendesk 76

Buying a House (Continued) 77

Chapter 6: Intelligent Machines 79

Denmark 79

Intelligent Machines 80

Machine Data 81

The Data Era 82

General Electric 82

Drones 84

Tesla 86

Networks of Data 87

Denmark (Continued) 88

Chapter 7: Government and Society 89

Egypt, 2011 89

Social Media 90

Intelligence 90

Snowden Effect 91

Privacy Risk Versus Reward 91

Observation or Surveillance 93

Development Targets 93

Open Data 95

Hackathons 95

Open Access 95

Ensuring Personal Protection 96

Private Clouds 97

Sanitizing Data 97

Evidence-Based Policy 97

Public-Private Partnerships 98

Impact Bonds 101

Social Impact Bond 102

Development Impact Bonds 103

The Role of Big Data 104

Egypt, 2011 (Continued) 105

Chapter 8: Corporate Sustainability 107

City of London 107

Global Megaforces 109

Population 109

Carbon Footprint 110

Water Scarcity 110

Environmental Risk 111

BP and Exxon Mobile 111

Early Warning Systems 112

Social Media 113

Risk and Resilience 114

Measuring Sustainability 115

Long-Term Decision Making 116

Stranded Assets 117

City of London (Continued) 118

Chapter 9: Weather and Energy 119

India, 2012 119

The Weather 120

Forecasting the Weather 120

When are Weather Forecasts Wrong? 121

Chaos 122

Ensemble Forecasts 122

Communication 123

Renewable Energy 124

Solar, Hydro, and Wind Power 124

Volatile or Intermittent Supply 125

Energy Consumption 126

Smart Meters 127

Intelligent Demand-Side Management 128

India, 2012 (Continued) 129

Part II: Learning From Patterns in Big Data 131

Chapter 10: Pattern Recognition 133

Elements of Success Rhyme 133

Pattern Recognition: A Gift or Trap? 134

What Fish Teach Us About Pattern Recognition 135

Bayes’ Theorem 135

Tsukiji Market 135

Pattern Recognition 137

Rochester Institute of Technology 137

A Method for Recognizing Patterns 137

Elements of Success Rhyme (Continued) 140

Chapter 11: Why Patterns in Big Data Have Emerged 141

Meatpacking District 141

Business Models in the Data Era 142

Data as a Competitive Advantage 143

Data Improves Existing Products or Services 145

Data as the Product 145

Dun & Bradstreet 146

CoStar 148

Ihs 149

Meatpacking District (Continued) 151

Chapter 12: Patterns in Big Data 153

The Data Factor 154

Summary of Big Data Patterns 155

Redefining a Skilled Worker 155

Creating and Utilizing New Sources of Data 156

Building New Data Applications 157

Transforming and Creating New Business Processes 157

Data Collection for Competitive Advantage 158

Exposing Opinion-Based Biases 159

Real-Time Monitoring and Decision Making 159

Social Networks Leveraging and Creating Data 160

Deconstructing the Value Chain 161

New Product Offerings 161

Building for Customers Instead of Markets 162

Tradeoff Between Privacy and Insight 163

Changing the Definition of a Product 163

Inverting the Search Paradigm for Data Discovery 164

Data Security 165

New Partnerships Founded on Data 165

Shortening the Innovation Lifecycle 166

Defining New Channels to Market 166

New Economic Models 167

Forecasting and Predicting Future Events 168

Changing Incentives 168

New Partnerships (Public/Private) 169

Real-Time Monitoring and Decision Making (Early Warning Systems) 169

A Framework for Big Data Patterns 170

Part III: Leading the Revolution 171

Chapter 13: The Data Opportunity 173

What Oil Teaches Us About Data 173

Bain Study 175

Seizing the Opportunity 176

Chapter 14: Porsche 177

Rome 177

Ferdinand Porsche 178

The Birth of Porsche 178

The Porsche Sports Car 179

Porsche Today 180

Rome (Continued) 180

Chapter 15: Puma 181

Herzogenaurach 181

Advertising Wars 182

Jochen Zeitz 182

Environmental Profit and Loss 183

Herzogenaurach (Continued) 184

Chapter 16: A Methodology for Applying Big Data Patterns 185

Introduction 185

The Method 186

Step 1: Understand Data Assets 187

The Patterns 188

Step 2: Explore Data 191

Challenges 192

Questions 192

Hypotheses 193

Data 193

Models 193

Statistical Significance 194

Step 3: Design the Future 194

The Patterns 195

Step 4: Design a Data-Driven Business Model 197

The Patterns 197

Step 5: Transform Business Processes for the Data Era 199

The Patterns 199

Step 6: Design for Governance and Security 201

The Patterns 201

Step 7: Share Metrics and Incentives 202

Chapter 17: Big Data Architecture 205

Introduction 205

Architect for the Future 206

Lessons from Stuttgart 207

Big Data Reference Architectures 207

Leveraging Investments in Architecture 208

Big Data Reference Architectures 211

Business View 212

Logical View 213

Chapter 18: Business View Reference Architecture 215

Introduction 215

Men’s Trunk: A Retailer in the Data Era 216

The Business View Reference Architecture 217

Answer Fabric 218

Data Virtualization 219

Data Engines 220

Management 221

Data Governance 221

User Interface, Applications, and Business Processes 222

Summary 222

Chapter 19: Logical View Reference Architecture 223

Introduction 223

Men’s Trunk: A Retailer in the Data Era (Continued) 224

The Logical View Reference Architecture 226

Data Ingest 227

Analytics 227

Discovery 228

Landing 228

Operational Warehouse 229

Information Insight 230

Operational Data 231

Governance 231

Men’s Trunk: A Retailer in the Data Era (Continued) 232

Chapter 20: The Architecture of the Future 233

Men’s Trunk: A Retailer in the Data Era (Continued) 233

Men’s Trunk: Applying the Methodology 235

Step 1: Understand Data Assets 235

Step 2: Explore the Data 236

Step 3: Design the Future 237

Step 4: Design a Data-Driven Business Model 237

Step 5: Transform Business Processes for the Data Era 237

Step 6: Design for Governance and Security 237

Step 7: Share Metrics and Incentives 238

Men’s Trunk: The Business View Reference Architecture 239

Answer Fabric 240

Data Virtualization 241

Data Engines 241

Management 242

Data Governance 242

User Interface, Applications, and Business Processes 243

Men’s Trunk: The Logical View Reference Architecture 244

Approach 244

Men’s Trunk: A Retailer in the Data Era (Continued) 248

Epilogue 249

The Time is Now 249

Taking Action 250

Fear not Usual Competitors 251

The Future 252

Index 255

Primary market:  Business Executive, that wants to transform their business with Big Data

Secondary Market: Any IT Architect, chartered with executing a successful big data project

Rob Thomas is Vice President of Product Development for Big Data and Information Management in IBM Software Group. Previously, he had responsibility for global sales and mergers & acquisitions. Patrick McSharry is a Senior Research Fellow at the Smith School of Enterprise and the Environment, Faculty Member of the Oxford Man Institute of Quantitative Finance at Oxford University and Visiting Professor at the Department of Electrical and Computer Engineering, Carnegie Mellon University.