Big Data, Machine Learning, and Data Mining Explained

Big Data, Machine Learning, and Data Mining Explained

Chitrali Kaul

Big Data, Machine Learning, and Data Mining Explained

Chitrali Kaul

ISBN - 9789361523496

COPYRIGHT © 2025 by Educohack Press. All rights reserved.

This work is protected by copyright, and all rights are reserved by the Publisher. This includes, but is not limited to, the rights to translate, reprint, reproduce, broadcast, electronically store or retrieve, and adapt the work using any methodology, whether currently known or developed in the future.

The use of general descriptive names, registered names, trademarks, service marks, or similar designations in this publication does not imply that such terms are exempt from applicable protective laws and regulations or that they are available for unrestricted use.

The Publisher, authors, and editors have taken great care to ensure the accuracy and reliability of the information presented in this publication at the time of its release. However, no explicit or implied guarantees are provided regarding the accuracy, completeness, or suitability of the content for any particular purpose.

If you identify any errors or omissions, please notify us promptly at “educohackpress@gmail.com” & “sales@educohackpress.com” We deeply value your feedback and will take appropriate corrective actions.

The Publisher remains neutral concerning jurisdictional claims in published maps and institutional affiliations.

Published by Educohack Press, House No. 537, Delhi- 110042, INDIA

Email: educohackpress@gmail.com & sales@educohackpress.com

Cover design by Team EDUCOHACK

Preface

In a world full of human beings and an abundance of information Data is ubiquitous. Managing such vast data becomes a very hectic task with traditional methods. However, it is important to manage the data to target relevant and useful information for speeding and improving our work. Therefore, in current times there is a big demand for database engineers and Data Analysts.

This book will highlight popular advanced methods used for data managing, which successfully makes this hectic task very easy and understandable. The most popular data management techniques are Big Data, Data Mining and Machine Learning. And this book will enlighten you with all these techniques which you can use to enhance your knowledge.

Big Data has been used in the industry for extracting relevant information from large data to gain profit. Companies use data mining with a strong consumer focus — retail, financial, communication, and marketing organizations, to "drill down" into their transactional data and determine price, customer preferences, and product positioning, impacting sales, customer satisfaction, and corporate profits. Whereas, Machine learning mainly focuses on improving and including various features in a website or application like google maps, etc.

This book will contain all the details regarding these techniques and fulfill all your questions regarding the above topics. This book will also give you a chance to test your knowledge as there are questions related to each topic. As a bonus, you will find reference links to dive more into the topic. I hope you will be satisfied and will be able to gain useful knowledge to shape your career.

Happy Reading!!

Content

Part-I

Overview Of Big Data

01. Overview of Big Data

1.1 History of Big Data 3

1.2 Examples of Big Data 4

1.3 Impacts of Big data 5

1.4 Future Expectations 6

1.5 Career Scope 7

1.6 Tools Used 10

1.7 Summary 14

1.9 Exercise 15

02. Types of Big Data

2.1 Structured Data 19

2.2 Unstructured Data 19

2.3 Semi-Structured Data 20

2.4 Difference between types of big data 20

2.5 Summary 21

2.6 Exercise 21

03. Characteristics of Big Data 24

3.1 Volume 25

3.3 Veracity 26

3.4 Value 26

3.5 Velocity 27

3.6 Summary 27

3.8 Exercise 28

04. Working of Big Data

4.1 Integration 30

4.2 Management 31

4.3 Analysis 32

4.4 Big Data practices 33

4.5 Summary 35

4.6 Exercise 35

05. Advantages and Disadvantages of Big Data

5.1 Benefits 38

5.2 Advantages of Big Data 39

5.3 Disadvantages of Big Data 41

5.4 Summary 42

5.5 Exercise 42

06. Real-Time Applications

6.1 Fields where big data is being used 45

6.2 Companies that use big data 54

6.3 Companies that provide big data 54

6.4 Summary 55

6.5 Exercise 55

07. Case Studies

7.1 What are case studies? 58

7.2 Case Study 1 – Walmart 58

7.3 Case Study 2 – Uber 59

7.4 Case Study 3 – Netflix 59

7.5 Case Study 4 – eBay 59

7.6 Case Study 5 – Procter & Gamble 60

7.7 Case Study 6 – LinkedIn 60

7.8 Summary 60

Part-II

DATA MINING

08. Overview of Data Mining

8.1 What do you mean by Data Mining? 65

8.2 History of Data Mining 66

8.3 Tasks Related 67

8.3. Summarization 70

8.4 Career Scope 71

8.5 Future of Data Mining 73

8.6 Data Mining V/S Big Data 74

8.7 Summary 74

8.8 Exercise 75

09. Stages Of Data Mining Process

9.1 Data Purification 78

9.2 Data Integration 78

9.3 Data Selection 79

9.4 Data Transformation 80

9.5 Pattern Evaluation 81

9.6 Knowledge Representation 82

9.7 Summary 82

9.8 Exercise 83

010. Data Mining Techniques

10.1 Tracking patterns 86

10.2 Classification 86

10.3 Clustering 86

10.4 Outlier detection 87

10.5 Association 87

10.6 Regression 88

10.7 Prediction 89

10.8 Sequential patterns 89

10.9 Decision trees 89

10.10 Statistical techniques 90

10.11 Visualization 91

10.12 Neural networks 91

10.13 Data warehousing 92

10.14 Long-term memory processing 93

10.15 Machine learning and AI 93

10.16 Summary 93

10.17 Exercises 94

11. Tools Used In Data Mining

11.1 MonkeyLearn 96

11.2 RapidMiner 97

11.3 Oracle Data Mining 98

11.4 IBM SPSS Modeler 99

11.5 Weka 100

11.6 KNIME 100

11.7 H2O 101

11.8 Orange 102

11.9 Apache Mahout 102

11.10 SAS Enterprise Mining 103

11.11 Rattle 104

11.12 Python 105

11.13 Summary 106

11.14 Exercise 107

12. Data Mining Algorithms

12.1 C4.5 109

12.2 k-means 110

12.3 Support vector machines 112

12.4 Apriori 113

12.5 EM 115

12.6 Page Rank 117

12.7 AdaBoost 117

12.8 kNN 119

12.9 Naive Bayes 120

12.10 CART 121

12.13 Summary 122

12.14 Exercise 122

13. Applications and Drawbacks of Data Mining

13.1 Future Healthcare 126

13.2 Market Based Analysis 126

13.3 Manufacturing Engineering 127

13.4 Education 128

13.5 CRM 129

13.6 Fraud Detection 130

13.7 Intrusion Detection 131

13.8 Lie Detection 132

13.9 Customer Segmentation 133

13.10 Financial Banking 134

13.11 Corporate Surveillance 135

13.12 Research Analysis 136

13.13 Criminal Investigation 136

13.14 BioInformatics 137

13.15 Companies extensively using Data Mining 138

13.16 Drawbacks of Data mining 139

13.17 Summary 140

13.18 Exercise 141

Part-III

MACHINE LEARNING

14. Introduction to Machine Learning

14.1 What is ML? 148

14.2 History of ML 148

14.3 How is it different from AI? 149

14.4 Examples of Machine Learning 150

14.5 Demand Of Machine Learning 158

14.6 Summary 159

14.7 Exercise 159

15. Types of Machine Learning

15.1 Supervised Learning 162

15.2 Examples of Supervised Learning 163

15.3 Unsupervised Learning 164

15.4 Examples of Unsupervised Learning 165

15.5 Reinforcement Learning 165

15.6 Examples of Reinforcement Learning 166

15.7 Summary 167

15.8 Exercise 167

16. Models in Machine Learning

16.1 Classification models 170

16.2 Regression models 177

16.3 Clustering models 182

16.3.4 EM with GMM 185

16.4 Association models 187

16.5 Reinforcement learning models 191

16.6 Difference between ML models 198

16.7 Summary 199

16.8 Exercise 200

17. Emerging Trends in ML

17.1 Intersection Of ML and IoT 204

17.2 Automated Machine Learning 205

17.3 Machine Learning in Cyber Security 206

17.4 Rise of AI Ethics 207

17.5 AI Engineering 207

17.6 AI-driven Biometric Security Solution 208

17.7 AI Analysis for the business forecast 208

17.8 Companies dependent on ML 209

17.9 Summary 210

17.10 Exercise 210

18. Model building in ML

18.1 Defining the problem 213

18.2 Collecting Data 214

18.3 Measure of success 215

18.4 Evaluation Protocol 215

18.5 Preparing the Data 218

18.6 Benchmark model 225

18.7 Tunning the hyperparameters 226

18.8 Summary 231

18.9 Exercise 232

Glossary 235

Index244

Part 1

Part 1

Overview of Big Data

Chapter 1. Exploring Big Data

Abstract

This chapter throws light on Big Data and its importance in our lives. Also, it will enlighten us about how big data can bring improvements and profit if it is managed properly. Here we will see the whole history and how big data has become an essential requirement for today’s generation. This will explain the future needs and increasing demands of big data. The career scope is also covered in this chapter. It will give you an overview and clear the importance of big data with its excellent uses. So keep reading to dive into the world of big data.

1.1 History of Big Data

The term big data was first used in 2005 by O’Reilly Media. Big data is mainly used for tracking down one’s business and making improvements. This concept is not new; it was used 7000 years ago in Mesopotamia when accounting was introduced there. The first major project was carried out in 1937, ordered by Franklin D. Roosevelt’s administration in the USA. The government had to keep track of contributions from 26 million Americans, and more than 2 million employers as the Social Security Act became law in 1937. Hereafter many such projects were conducted. The first data processing machine was developed in 1943 by the British to decipher Nazi codes in World War II.

In 2005, for the first time, the term Big Data was coined by Roger Mougalas from O’Reilly Media. In the same year, Hadoop was created by yahoo. Its goal was to index the entire World Wide Web. Nowadays, Hadoop is used by a lot of organizations to crunch through huge data. Afterward, increasing social media boosted Big Data, and it became the essential demand of every existing and emerging company.

Fig. 1.1 Evolution of Big Data

1.2 Examples of Big Data

Nowadays, Big Data is everywhere; most companies use it for gaining profit and making improvements in their business. Here are some of the examples:

Fig. 1.2 Applications of Big Data

1.2.1 Marketing

Big Data has become an essential tool for marketing. Most companies rely on big data to improve their sales. It saves marketers time as big data allows them to advertise their ads to a targeted audience. It has entered into every sector, resulting in advanced marketing and gaining maximum profits quickly. This can be seen in many ways, like How Amazon shows the correct ad related to your interests? The suggestions of accurate products while shopping, etc., can be achieved using big data.

1.2.2 Transportation

In this modern world, everyone has a smartphone, but what makes it smart is its features. Almost everyone, while traveling, uses maps to see the shortest route or traffic areas, etc., for their convenience. Transportation relies on big data for achieving better and safe transport from one place to another. It allows them to navigate and detect people, cars, traffic expected, etc. Tracking down the parcels is also one of the greatest examples.

1.2.3 Business Insights

Most companies collect large amounts of data and analytics to gain customers’ requirements and fulfill them. Using big data enables them to understand their online audience, their expectations and predict future buying. It leads to an increase in customer satisfaction and helps to build trust among them.

1.3 Impacts of Big data

The impacts of big data are in every sector. They use it for the improvement and benefit of gaining people’s trust and attention. As an audience, we find it quite appealing when companies or organizations bring us news or products as per our needs which helps the company gain a reputation in the market. Here are some impacts of Big data :

•Manufacturing

•Logistics, Media, and Entertainment

•Oil and Gas

•Social Media, etc.

•We will see detailed information further in upcoming chapters.

1.4 Future Expectations

Fig. 1.3 Future of Big Data

1.4.1 Data volumes will continue to increase and migrate to the cloud

The majority of Big Data experts agree that a very large amount of data will be generated in the upcoming years. Companies will continue to analyze these tons of data. They require large data sets and infinite data stacks to avoid this problem; the whole data is stored in the cloud, such as AWS. This creates a challenge for big data processing though it has become easier through open-source ecosystems such as Hadoop and No SQL.

1.4.2 Machine learning will continue to change the landscape

Apart from Big Data, one of the future technologies includes Machine Learning, which will greatly impact our future changing lifestyles. Combining big data with ML will automate big data analysis, where computers will analyze the data by themselves.

1.4.3 Data scientists and CDOs will be in high demand

The positions of Data scientists and CDOs, i.e., Chief Data Officers, will be in high demand. In the current times, the gap between the need and data professionals is high, So in the future, there will be a great demand for data professionals as with increasing data, we need more people to manage it. No doubt that Data Analysts, data engineers are one of the fastest-growing jobs.

1.4.4 Fast data and actionable data will come to the forefront

Big Data future is also related to Fast data and Actionable data. Big Data relies on Hadoop and No SQL database to analyze information, whereas Fast data allows processing in real-time streams. As a result, data can be analyzed in just one millisecond. One of the research predicted that nearly 30% of data would be real-time by 2025.

1.5 Career Scope

Observing the benefits of big data demand of data professionals has been in high demand. Depending on your positions and skills, big data jobs are lucrative. It also secures a nice living for you and your family. The pay range lies between the range of $ 50,000 - $165,000. The topmost jobs in the field of Big Data are engineers, managers, and developers. Also, some highest-paying big data positions include analysts, scientists, statisticians, and specialists. You can easily find a job in big data as there are plenty of jobs available. Most of Big data positions rely on solid programming background. You should at least know the basics of C, Python, Java, and SQL. To make your profile stronger, you may require certifications in Hadoop, Apache Spark, and Machine learning.

Fig. 1.4 Why Big Data is best as a career option

Here is the list of some job positions popular in big data :

1.5.1 Big Data Engineer

Annual Salary Range: $130,000-$222,000

Big data engineers are similar to data analysts in that they turn large volumes of data into insights that organizations can use to make smarter business decisions, but they’re also tasked with retrieving, interpreting, analyzing, and reporting on a business’s data—which they typically have to gather from a variety of different sources. These professionals are also often responsible for creating and maintaining the company’s software and hardware architecture, including the systems and processes users need to work with that data.

1.5.2 Data Architect

Annual Salary Range: $119,750-$193,500

These professionals are tasked with designing the structure of complex data frameworks and building and maintaining these databases. In addition, data architects develop strategies for each subject area of the enterprise data model and communicate plans, status, and issues to their company’s executives.

1.5.3 Data Warehouse Manager

Annual Salary Range: $80,000-$160,250

Data warehouse managers are responsible for the storage and analysis of data in facilities. These professionals use performance and usage metrics to evaluate data, analyze data load, and monitor job usage. In addition, they may be responsible for identifying and mitigating potential risks to data storage and transfer.

1.5.4 Database Manager

Annual Salary Range: $111,250-$186,500

Database managers identify problems in databases, take corrective action to remedy those issues, and assist with the design and physical implementation of storage hardware and maintenance.

These professionals work closely with database developers and often provide guidance and training to lower-level staff.

1.5.5 Business Intelligence Analyst

Annual Salary Range: $87,500-$185,500

Business intelligence analysts turn companies’ data into insights that executives can use to make better business decisions. These professionals often respond to management’s requests for specific information but might also be expected to scrutinize data independently to find patterns and trends.

Business intelligence analysts should have a strong background in analytical and reporting tools, several years of experience with database queries and stored procedure writing, as well as online analytical processing (OLAP) and data cube technology skills.

1.5.6 Data Scientist

Annual Salary Range: $105,750-$180,250

Data scientists design and construct new processes for modeling, data mining, and production. In addition to conducting data studies and product experiments, these professionals develop prototypes, algorithms, predictive models, and custom analysis.

Previous work experience in a similar position is usually required, and data scientists should be skilled in different data mining techniques, such as clustering, regression analysis, and decision trees.

1.5.7 Data Modeler

Annual Salary Range: $80,750-$170,000

These professionals turn large volumes of data into insights, such as micro and macro trends, which are gathered into business reports. Data modelers must be skilled in both information science and statistical analysis and should have proficient programming skills.

Data modelers often specialize in a particular business area, which makes it easier to find useful data trends for their employers.

1.5.8 Database Developer

Annual Salary Range: $101,250-$172,750

Database developers are responsible for analyzing current database processes to modernize, streamline, or eliminate inefficient coding. In addition, these professionals are often charged with monitoring database performance, developing new databases, and troubleshooting issues as they arise.

Database developers work closely with other members of the development team. As a result, they’re often required to have prior experience with database development, data analysis, and unit testing.

1.6 Tools Used

Big Data processing is incomplete without tools. One needs to be an expert in using those tools to become a Big data professional.

Fig. 1.5 Tools Used

Here there is a list of tools you need to know to start a career in big data:

1.6.1 Hadoop

Hadoop is one of the best open-source tools for big data processing. It offers massive storage for all kinds of data. With its amazing processing power and capability to handle innumerable tasks, you never can ponder over hardware failure. However, you need to know JAVA to work with Hadoop. A Hadoop certification can accelerate your resume and open more opportunities for big data.

Fig. 1.6 Key Milestones in Hadoop’s development

1.6.2 Xplenty

This cloud-based Big Data Analytics tool for integrating, analyzing, and preparing data brings all data sources together. Its intuitive graphical interface allows you with ETL, ELT, or replication. Xplenty is a full toolkit to create low-code and no-code data pipelines. It provides solutions for marketing, distribution, and development.

It is elastic and scalable. One can immediately access a range of data stores and a diverse collection of data transformation components. Furthermore, by using the rich expression language of Xplenty, you can incorporate complex data preparation functions.

1.6.3 CDH (Cloudera Distribution For Hadoop)

CDH is a complete open-source Big Data Analytics tool and includes Apache Hadoop, Apache Spark, Apache Impala, and many more on its free distribution site. It enables you to acquire, store, manage, discover, model, and distribute limitless data. It gives complete and accurate distribution. Hadoop can be easily managed by Cloudera Manager. Moreover, it is simple to deploy.

Fig. 1.7 CDH

1.6.4 R

R is one of the most comprehensive Big Data analytics tools for statistical analysis. The software ecosystem is open-source, free, multi-paradigm, and diverse. The programming languages are C, Fortran, and R. Most extensively used by statisticians and data miners; its use cases include data processing, data manipulation, analysis, and visualization.

1.6.5 Cassandra

Apache Cassandra is free of cost Big Data analytics tool designed to handle large quantities of data across many commodity servers, offering high availability. The open-source NoSQL DBMS uses CQL (Cassandra Structure Language) to interact with the database. As a result, it quickly manages a huge amount of data. It also comes with log-structured storage and linear scalability.

Fig. 1.8 Cassandra

1.6.6 KNIME

KNIME is an abbreviation for Konstanz Information Miner, which is an open-source Big Data Analytics tool. It is used for enterprise reporting, integration, data mining, data analytics, and business intelligence. It supports operating systems such as Linux and Windows X. It is very well integrated with other technologies and languages.

Fig. 1.9 KNIME

1.6.7 MongoDB

MongoDB is a contemporary alternative to databases. It’s one of the best Big Data Analytics tools for working on data sets that vary or change frequently or the ones that are semi or unstructured. Some of the best uses of MongoDB include storage of data from mobile apps, content management systems, product catalogs, and more. Like Hadoop, you can’t get started with MongoDB instantly. You need to learn the tool from scratch and be aware of working on queries.

1.6.8 HPCC

HPCC is an abbreviation for High-Performance Computing Cluster. This open-source Big Data Analytics tool is a complete Big Data solution over a highly scalable supercomputing platform. HPCC is also known as DAS (Data Analytics Supercomputer) and was developed by LexisNexis Risk Solutions. Written in C++ and ECL(Enterprise Control Language), it is based on a Thor architecture that enables data parallelism, pipeline parallelism, and system parallelism.

Fig. 1.10 HCPP

1.7 Summary

In the above Chapter, we took an overview of Big Data and explored its benefits, needs, and importance. From the origin to the future, this chapter covered all of the topics. It threw light on necessary details and important aspects of big data. Increasing big data unlocks more employment opportunities and offers a more secure future. By reading this chapter, you get an idea about the skills you require and build your career as per your interests. It is going to change our lives and lead to new technologies and empower each field. Big data is vaster than this; we will slowly unfold all the topics and concepts related to Big Data. Further chapters will take you to details of each idea.

1.9 Exercise

1. What is Big Data?

2. How is it beneficial to us?

3. When was the term Big Data firstly introduced?

4. Is Big Data the future of marketing?

5. List a few examples of Big Data.

6. Explain one of its examples.

7. Can Big data make an impact on the future?

8. What are the future expectations from big data?

9. Will Data increase in the future?

10. What is its importance in the current market?

11. How is big data changing advertising?

12. List the job positions in Big Data.

13. What is a data analyst?

14. How much salary can you aspect by becoming a Data Engineer?

15. What are the various tools used for big data processing

References

Built-In. 2021. 17 Big Data Examples & Applications. Available at: <https://builtin.com/big-data/big-data-examples-applications>.

InData Labs. 2021. Impact of Big Data on Business – InData Labs. Available at: <https://indatalabs.com/blog/impact-of-big-data-on-business>.

Jigsaw Academy. 2021. 20 Big Data Analytics Tools You Need To Know In 2021. Available at: <https://www.jigsawacademy.com/8-big-data-tools-need-know/>.

Karau, H. (2015). Learning Spark. Beijing: OReilly.

Data Science and Big Data Analytics: Discovering, Analyzing, Visualizing and Presenting Data by EMC Education Services

Big Data: Does Size Matter? by Timandra Harkness

Statistics for Big Data for Dummies by Alan Anderson and David Semmelroth

 Big Data, Big Analytics: Emerging Business Intelligence and Analytic Trends for Today’s Businesses by Michael Minelli

Big Data, Data Mining and Machine Learning (WILEY Big Data Series) by Jared Dean

 Analytics: Data Science, Data Analysis and Predictive Analytics for Business” by Daniel Covington

Machine Learning for Big Data: Hands-On for Developers and Technical Professionals by Jason Bell

Big Data Analytics by Parag Kulkarni and Sarang Joshi

Informatica and Capgemini, The Big Data Payoff: Turning Big Data into Business Value, 2016.

V. Kayser, B. Nehrke, and D. Zubovic, Data Science as an Innovation Challenge: From Big Data to Value Proposition, Technology Innovation Management Review, 2018.

M. S. Hopkins and R. Shockley, Big Data, Analytics, and the Path from Insights to Value, MIT Sloan Management Review, 2011.

Harvard Business Review, The Enterprise Lacks a Big Data Strategy for IoT Transformation, 2017, pp.1-12.

S. Lavalle, M. S. Hopkins, E. Lesser, R. Schockley, and N. Krushcwitz, Analytics: The New Path to Value, MIT Sloan Management Rev., 2010.

S. Viaene and A. Van den Bunder, The secrets to managing business analytics projects, MIT Sloan Manag. Rev., 2011, pp. 65–69.

A. Chebotko, A. Kashlev, and S. Lu, “A Big Data Modeling Methodology for Apache Cassandra,” Proc. of IEEE Int. Congress on Big Data, 2015, pp.238- p.245.

A. Fink, R. Guzzo, and S. Roberts, “Big Data at Work: Lessons from the Field,” Society for Industrial and Organizational Psychology, 2017.

Figure Resources

Fig. 1.1. https://betravingknows.com/wp-content/uploads/2018/08/image-wr-180507-big-data-history.jpg

Fig. 1.2. https://www.greycampus.com/userfiles/big%20data%20applications%20-%20Copy.png

Fig. 1.3. https://www.ringcentral.co.uk/gb/en/blog/wp-content/uploads/2020/05/a-day-in-data2.jpg

Fig. 1.4. https://www.edureka.co/blog/wp-content/uploads/2015/01/domain-1.png

Fig. 1.5. https://lh6.googleusercontent.com/Ju4rknKC6KiWmlI2QXTY9Kxv1Qt48dCk6HUnc3nLd2ps-fr4n23gj1wk79OEunP10jXH5hJQ6Yx5AoagTcYA8x8t8eYyyWb7oyR5amO6IS3Hq3VG1tl__Crs31om8_tvhwS01ybI

Fig. 1.6. https://cdn.ttgtmedia.com/rms/onlineimages/data_management-hadoop_milestones_mobile.png

Fig. 1.7. https://www.researchgate.net/profile/Allae-Erraissi/publication/319935616/figure/fig2/AS:570392433049600@1513003561746/Cloudera-Distribution-for-Hadoop-Platform-CDH-12.png

Fig. 1.8. https://cassandra.apache.org/_/_images/diagrams/apache-cassandra-diagrams-01.jpg

Fig. 1.9. https://miro.medium.com/max/720/1*yjTF6IPHqqlnHTvatIUEBg.png

Fig. 1.10. https://risk.lexisnexis.com/global/-/media/images/our%20tech/hpcc%20systems%20diagram_v4_outlined.png?h=1435&iar=0&w=3000&hash=25347B16797776441625B5E7C21A4359

Chapter 2. Types of Big Data

Abstract

In this chapter, you will learn in detail about types of Big Data. The name describes its vastness; obviously, it will consist of various data types. Not all of that data is readily usable in analytics and has to undergo a transformation known as data cleansing to make it understandable. They make it easier to bifurcate a large amount of data to be processed easily and accurately. The structure of the data is the key to how to go about working with it and what insights it can produce. All data goes through a process called extract, transform, load (ETL) before being analyzed. There are three types of big data Structured, Unstructured, and Semi-Structured Data, and we will see them below in detail.

Fig 2.1 Types of Big Data

2.1 Structured Data

Structured Data are easier to handle. As they are highly organized with dimensions defined by set parameters. You can take the example spreadsheets in that every piece of information is grouped into rows and columns. Specific elements defined by certain variables are easily discoverable.

As it is already organized, a program finds it easier to sort and collect data, making it simpler to analyze. In addition, it requires little to no preparation before processing. One of the major perks of using structured data is the streamlined process of merging enterprise data with relational. Because pertinent data dimensions are usually defined, and specific elements are in a uniform format, very little preparation needs to be done to make all sources compatible. The ETL process for structured data stores the finished product in what is called a data warehouse. These databases are highly structured and filtered for the specific analytics purpose the initial data was harvested for.

2.2 Unstructured Data

Unstructured data, as the name suggests it is unorganized. Every data is not as neatly packed and sorted. Only 20% of all data is structured. Here you can get the example of your computer. Almost everything you do with a computer generates unstructured data. No one is transcribing their phone calls or assigning semantic tags to every tweet they send. For structured data, the ETL process is very simple. It is simply cleansed and validated in the transform stage before loading into a database. But for unstructured data, that second step is much more complicated.

The toughest part here is to teach an application to understand the information it is extracting. This means that this data needs to be transformed into some form of structured data. Methods like text parsing, natural language processing, and developing content hierarchies via taxonomy are common. However, it involves a complex algorithm blending the processes of scanning, interpreting, and contextualizing functions.

For Example, Alissa Lorentz, then the vice president of creative, marketing, and design at Augify, explained in a guest article for Wired: a query on an unstructured data set might yield the number 31, but without context, it’s meaningless. For example, it could be “the number of days in a month, the number of dollars a stock increased…, or the number of items sold today.”

This context can be created from unstructured datasets, like NoSQL databases or human dictation. We can tell applications and AI what data means. You have probably been doing it for years whenever Google asks you to prove you are not a robot. The world of machine learning, or AI teaching itself how to improve and discover patterns, is becoming instrumental in the world of big data because of its ability to improve on models autonomously.

2.3 Semi-Structured Data

Semi-Structured data lies between structured and unstructured data. It can be interpreted as unstructured data with metadata attached to it. This can be inherent data collected, such as time, location, device ID stamp, or email address, or a semantic tag attached to the data later.

For Example: Let’s say you take a picture of your cat from your phone. It automatically logs the time the picture was taken, the GPS data at the time of the capture, and your device ID. If you’re using any kind of web service for storage, like iCloud, your account info becomes attached to the file.

If you send an email, the time sent, email addresses to and from, the IP address from the device sent from, and other pieces of information are linked to the actual content of the email.

In both scenarios, the actual content (i.e., the pixels that compose the photo and the characters that make up the email) is not structured, but some components allow the data to be grouped based on certain characteristics.

By using the right datasets, semi-structured data can be a huge asset. It can inform AI training and machine learning by associating patterns with metadata. It has no set schema. This can be both a benefit and a challenge. It can be more difficult to work with because it will require more effort to tell the application what each data point means. But this also means that the limits in structured data ETL in terms of definition don,t exist.

2.4 Difference between types of big data

Fig 2.2 Difference between types of data

2.5 Summary

This chapter contained detailed information on the types of big data. There are three types of big data Structured, Unstructured, and Semi-Structured Data. Structured Data are easier to handle. It is already organized a program finds it easier to sort and collect data, making it simpler to analyze. It requires little to no preparation before processing. Unstructured data, as the name suggests it is unorganized. Every data is not as neatly packed and sorted. Only 20% of all data is structured. Semi-Structured data lies between structured and unstructured data. It can be interpreted as unstructured data with metadata attached to it.

2.6 Exercise

1. What are types of Big Data?

2. What is Structured Data?

3. Give some examples of Structured data.

4. What is Unstructured Data?

5. Give some examples of Unstructured data.

6. What is the toughest part of Unstructured Data?

7. What is Semi-structured data?

8. Semi-structured lies between which data types?

9. Give an example of a Semi-structured data type.

10. Give some differences between all three.

11. Which type of data type is easy to handle?

12. Why is it easy to handle that data type?

References

Selecthub.com. 2021. Types of Big Data | Understanding & Interacting With Key Types. [online] Available at: <https://www.selecthub.com/big-data-analytics/types-of-big-data-analytics/>.

Slideshare.net. 2021. Big data and the different types including unstructured, structured a…. [online] Available at: <https://www.slideshare.net/JanetDorenkott/big-data-whats-the-difference-silos-august-2014>

S. Nalchigar and E. Yu, “Business-driven data analytics: a conceptual modeling framework”, Data & Knowledge Engineering, 2018, pp. 1-14.

Lakster, Laurens L., & Keur, Ron (2012). Big data: Too big to ignore. What organizations can learn from the American presidential elections. Compact, 2: 1–8.

Adami, Christoph (1995). Self-organized criticality in living systems. Physics Letter A, 203(1): 29–32.

Adams, Douglas (2009). The hitchhiker’s guide to the galaxy. London: Macmillan.

Adams, Pauline A., & Adams, Joe K. (1960). Confidence in the recognition and reproduction of words difficult to spell. American Journal of Psychology, 73(4): 544–552.

Adee, Sarah (2008). The data: 37 Years of Moore’s Law. IEEE Spectrum, 45(5): 56–56.

Adler, Mortimer J. (1986). A guidebook to learning: For a lifelong pursuit of wisdom. London: Macmillan.

Agrawal, Divyakant, El Abbadi, Amr, Arora, Vaibhav, Budak, Ceren, Georgiou, Theodore, Mahmoud, Hatem A., Nawab, Faisal, Sahin, Cetin, & Wang, Shiyuan (2015). Mind your Ps and Vs.: A perspective on the challenges of big data management and privacy concerns. International Conference on Big Data and Smart Computing (BigComp), Jeju, South Korea, 1–6.

Ahalt, Stan, & Kelly, Kip (2013). The big data talent gap. UNC Kenan-Flagler Business School White Paper, 1–15.

Aiden, Erez, & Michel, Jean-Baptiste (2013). Uncharted. Big data as a lens on human culture. New York: Riverhead.

Alavi, Maryam, & Leidner, Dorothy E. (2001). Knowledge management and knowledge management systems: Conceptual foundations and research issues. MIS Quarterly, 25(1): 107–136.

Demirkan, H., Delen, D.: Leveraging the capabilities of service-oriented decision support systems: Putting analytics and big data in the cloud, Decision Support Systems 55, 412–421(2013)

Barton, D., Court, D.: Making Advanced Analytics Work For You, Harvard business

review, 90 (10), pp. 79–83 (2012)

McAfee, A., Brynjolfsson, E.: Big data: the management revolution, Harvard business

review, 90 (10), pp. 60–68 (2012)

X. Wu, G. Wu, W. Ding, Data mining with big data, IEEE Trans. Knowl. Data Eng. 28 (2014) 97–106.

Paakkonen, P., Pakkaka, D.: “Reference Architecture and Classification of Technologies,

Products and Services for Big Data Systems”, Big DataResearch2, 166–186 (2015)

Thus, Z. Shao, S. Anthony, D. Borthakur, N. Jain, J. S. Sarma, R. Murthy, H. Liu, Data warehousing and analytics infrastructure at Facebook, in 2010

ACM SIGMOD International Conference on Management of Data, Indianapolis, Indiana, USA, 6–11 June (2010)

Figure Resources

Fig. 2.1. https://www.selecthub.com/wp-content/uploads/2020/03/Types-of-Big-Data-Infographic-768x402.png

Fig. 2.2. https://image.slidesharecdn.com/bigdatawhatsthedifferencesilosaugust2014-140917234427-phpapp01/95/big-data-and-the-different-types-including-unstructured-structured-and-hybrid-1-638.jpg?cb=1410997531

Chapter 3. Characteristics of Big Data

Abstract

In this chapter, you will learn in detail about the characteristics of Big Data. The Five V’s correctly explain the characteristics of big data. Volume, Veracity, Variety, Value, and Velocity are the 5 V’s that will give us the characteristics of big data. This chapter will explain every characteristic. And you will come to know why big data is so popular and worthy. So, let’s learn more about big data. After reading this chapter, you will learn about characteristics that make it stand out in today,s technology.

Fig 3.1 Characteristics of Big Data

3.1 Volume

Big data, the name itself, reveals its vast nature. It is basically the large amount of data generated from business processes, machines, social media platforms, networks, human interactions, and many more. The size of the data plays a very crucial role while determining the value out of data. For Example, Facebook Facebook can generate approximately a billion messages, 4.5 billion times that the “Like” button is recorded, and more than 350 million new posts are uploaded each day. Therefore, big data technologies can handle large amounts of data. Hence, ‘Volume’ is one characteristic that needs to be considered while dealing with Big Data solutions.

Fig 3.2 Volume

3.2 Variety

As we saw in the previous chapter, there are types of big data which give variety to it. It can be structured, unstructured, and semi-structured, which can be obtained from various sources. During earlier days, spreadsheets and databases were the only sources of data considered by most applications. Nowadays, data in emails, photos, videos, monitoring devices, PDFs, audio, etc., are also being considered in the analysis applications. This variety of unstructured data poses certain issues for storage, mining, and analyzing data. Here is a quick review of the varieties available in big data.

•Structured data: In Structured schema, along with all the required columns. It is in a tabular form. Structured data is stored in the relational database management system.

•Semi-structured: In Semi-structured, the schema is not appropriately defined, e.g., JSON, XML, CSV, TSV, and email. OLTP (Online Transaction Processing) systems are built to work with semi-structured data. It is stored in relations, i.e., tables.

•Unstructured Data: All the unstructured files, log files, audio files, and image files are included in the unstructured data. Some organizations have much data available, but they did not know how to derive the value of data since the data is raw.

Fig 3.3 Variety

3.3 Veracity

Veracity refers to the amount of data reliable. There are many ways to filter or translate the data. Veracity is the process of being able to handle and manage data efficiently. Big Data is also essential in business development. For example, Facebook posts with hashtags. Basically, it is about determining how accurate or truthful a data set may be. Especially when it comes to the accuracy of big data, it’s not just the quality of the data itself but also how trustworthy the data source, type, and processing is.

3.4 Value

Value is the most important characteristic of big data. It is often quantified as the potential social or economic value that the data might create. However, the whole concept is weakly defined since high valuable data might sit at your warehouse without any value without proper intention or application. This is often the case when the actors producing the data are not necessarily capable of putting it into value. It is also the use of gathered data for your business. A big volume isn’t always very useful to be valuable. It needs to be converted into insights or information, and that is where data processing steps in.

3.5 Velocity

Velocity plays an important role compared to others. Velocity creates the speed by which the data is created in real-time. It contains the linking of incoming data sets speeds, rate of change, and activity bursts. The primary aspect of Big Data is to provide demanding data rapidly.

Big data velocity deals with the data flow from sources like application logs, business processes, networks, social media sites, sensors, mobile devices, etc.

3.6 Summary

In the above Chapter, we saw various characteristics of big data. These V’s give big data its importance and makes it an important feature for today’s generation. Here we got an idea about the characteristics big data possesses. We need to keep all these characteristics while dealing with data. Volume defines the huge amount of data that is produced each day by companies. Variety refers to the diversity of data types and data sources. Velocity refers to the speed with which the data is generated, analyzed, and reprocessed. Veracity is the authenticity and credibility of the data. Value denotes the added value for companies.

Fig 3.4 5 V’s of Big Data

3.8 Exercise

1. What are the characteristics of Big Data?

2. What are the 5 V’s of Big Data?

3. What is meant by volume? Give an Example of Volume.

4. What is meant by variety? Give an Example of Variety.

5. What is meant by Veracity? Give an Example of Veracity.

6. What is meant by value? Give an example of Value.

7. What is meant by Velocity? Give an Example of Velocity.

References

Salzig, C., 2021. What is Big Data? – A definition with five Vs. [online] Blog.unbelievable-machine.com. Available at: <https://blog.unbelievable-machine.com/en/what-is-big-data-definition-five-vs>.

Guru99.com. 2021. What is BIG DATA? Introduction, Types, Characteristics, and Examples. [online] Available at: <https://www.guru99.com/what-is-big-data.html>

Kreps, J., Narkhede, N., Rao, J.: Kafka: a distributed messaging system for log processing, in The 6th International Workshop on Networking Meets Databases, Athens, Greece, 12 June (2011)

Lin, J., Ryaboy, D.: Scaling big data mining infrastructure: the Twitter experience, ACM SIGKDD Explor. News. 14, 6–19 (2013)

A. Zimmermann, K. Sandkuhl, M. Pretz, M. Falkenthal, D. Jugel, M. Wissotzki, Towards an integrated service-oriented reference enterprise architecture, in International Workshop on Ecosystem Architectures, Saint Petersburg, Russia, 19 August 2013.

J. A. Zachman A framework for information systems architecture, IBM Systems Journal, Volume: 26, Issue: 3, 1987

M. Mior and A. Aboulnaga, “NoSE: Schema Desing for NoSQL Applications,” IEEE Transaction on Knowledge and Data Engineering, Vol. 29, No.10, 2017, pp. 2275 – pp. 2289.

T. M. Mitchell, Machine Learning, McGraw-Hill, 1997.

M. B. Christopher, Pattern Recognition and Machine Learning, Springer New York, 2016.

F. Caro, J. Gallien, M.Miranda., J. Torralbo, J. Corras, M.Vazquez, J. Calamonte, and J.Correa, “Zara uses operations research to reengineer its global distribution process,” Interfaces 40(1), 2010. pp.71-84. <https://sites.google.com/site/irisforbigdata/ >

Unisphere Research, 2016, The 2016 Enterprise Data Management Survey, Unisphere Research, pp.1-2

Alavi, Maryam, & Leidner, Dorothy E. (2001). Knowledge management and knowledge management systems: Conceptual foundations and research issues. MIS Quarterly, 25(1): 107–136.

Demirkan, H., Delen, D.: Leveraging the capabilities of service-oriented decision support systems: Putting analytics and big data in the cloud, Decision Support Systems 55, 412–421(2013)

Barton, D., Court, D.: Making Advanced Analytics Work For You, Harvard business review, 90 (10), pp. 79–83 (2012)

McAfee, A., Brynjolfsson, E.: Big data: the management revolution, Harvard business review, 90 (10), pp. 60–68 (2012)

X. Wu, G. Wu, W. Ding, Data mining with big data, IEEE Trans. Knowl. Data Eng. 28 (2014) 97–106.

Paakkonen, P., Pakkaka, D.: “Reference Architecture and Classification of Technologies,

Products and Services for Big Data Systems”, Big DataResearch2, 166–186 (2015)

Thus, Z. Shao, S. Anthony, D. Borthakur, N. Jain, J. S. Sarma, R. Murthy, H. Liu, Data warehousing and analytics infrastructure at Facebook, in 2010

ACM SIGMOD International Conference on Management of Data, Indianapolis, Indiana, USA, 6–11 June (2010)

Figure Resources

Fig. 3.1. https://static.javatpoint.com/hadooppages/images/big-data-characteristics.png

Fig. 3.2. https://static.javatpoint.com/hadooppages/images/big-data-characteristics2.png

Fig. 3.3. https://crmconsultant.weebly.com/uploads/1/3/5/7/135749203/big-data-types_orig.png

Fig. 3.4. https://www.researchgate.net/profile/Yuri-Demchenko/publication/273945634/figure/fig1/AS:391885618204672@1470444220432/5-Vs-of-Big-Data-and-security-related-properties-of-Veracity-Variety-and-Volume.png

Chapter 4. Working of Big Data

Abstract

Far till now, we have encountered the benefits and importance of Big Data in our lives. Now it’s time to learn how actually Big Data works. This chapter will tell us about how big data becomes so useful. We will see the process of converting this enormous data into valuable information. To make a career in big data, one needs to be an expert in these processes. Here we will understand some of the processes that are carried out while working with big data.

It mainly includes processes like integration, management, analysis. All these processes need to be followed to get desired output.

4.1 Integration

Firstly to process the data, we need to collect it. Data is obtained from various sources to get more information. Therefore, we need to collect it from as many sources as possible. As a result, it gives us an enormous amount of data. In some cases, we are talking about petabytes of information flowing into your system, so it will be a challenge to integrate such a volume of information into your system. In some cases, we are talking about petabytes of information flowing into your system, so it will be a challenge to integrate such a volume of information into your system. Big data integration is very challenging, especially after the traditional data integration techniques failed to handle it.

Fig 4.1 Traditional Data Integration

4.2 Management

Obviously, in this step, the management role comes. Obviously, such enormous data needs to be managed. The data needs to be stored somewhere. For this purpose, one can use cloud, on-premises, or both. You can also choose in what form your data will be stored, so you can have it available in real-time on-demand. This is why more and more people are choosing a cloud solution for storage because it supports your current compute requirements. The cloud is gradually gaining popularity because it supports your current compute requirements and enables you to spin up resources as needed.

Fig 4.2 Data management

4.3 Analysis

Now comes the final part, i.e., analyzing all the data to use it. This will help you to explore your data and make useful conclusions for your purpose. For example, knowing what features are mostly researched by your customers or use them to share research. Also, you can build your data models with machine learning and artificial intelligence. Hence this is how big data becomes useful and productive in our lives.

Fig 4.3 Data Analysis

4.4 Big Data practices

Apart from these steps, here are some of the best principles which can be useful for Big Data. These guidelines can help you to build a perfect big data project.

4.4.1 Align big data with specific business goals

More extensive data sets enable you to make discoveries. To that end, it is important to base new investments in skills, organization, or infrastructure with a strong business-driven context to guarantee ongoing project investments and funding. To determine if you are on the right track, ask how big data supports and enables your top business and IT priorities.

Examples include filtering weblogs to understand eCommerce behavior, deriving sentiment from social media and customer support interactions, understanding statistical correlation methods and their relevance for the customer, product, manufacturing, and engineering data.

4.4.2 Ease skills shortage with standards and governance

One of the biggest obstacles to benefiting from your investment in big data is a skills shortage. You can mitigate this risk by ensuring that big data technologies, considerations, and decisions are added to your IT governance program. Standardizing your approach will allow you to manage costs and leverage resources. Organizations implementing big data solutions and strategies should assess their skill requirements early and often and should proactively identify any potential skill gaps. These can be addressed by training/cross-training existing resources, hiring new resources, and leveraging consulting firms.

4.4.3 Optimize knowledge transfer with a center of excellence

Use a center of excellence approach to sharing knowledge, control oversight, and manage project communications. Whether big data is a new or expanding investment, the soft and hard costs can be shared across the enterprise. Leveraging this approach can help increase big data capabilities and overall information architecture maturity more structured and systematic.

4.4.4 Top payoff is aligning unstructured with structured data

It is certainly valuable to analyze big data on its own. But you can bring even greater business insights by connecting and integrating big low-density data with the structured data you are already using today.

Whether you are capturing customer, product, equipment, or big environmental data, the goal is to add more relevant data points to your core master and analytical summaries, leading to better conclusions. For example, there is a difference in distinguishing all customer sentiment from only your best customers. This is why many see big data as an integral extension of their existing business intelligence capabilities, data warehousing platform, and information architecture.

Keep in mind that the big data analytical processes and models can be both human- and machine-based. Big data analytical capabilities include statistics, spatial analysis, semantics, interactive discovery, and visualization. Using analytical models, you can correlate different types and data sources to make associations and meaningful discoveries.

4.4.5 Plan your discovery lab for performance

Discovering meaning in your data is not always straightforward. Sometimes we don’t even know what we are looking for. That is expected. Management and IT needs to support this “lack of direction” or “lack of clear requirement.”

At the same time, analysts and data scientists need to work closely with the business to understand key business knowledge gaps and requirements. To accommodate the interactive exploration of data and statistical algorithms, you need high-performance work areas. Be sure that sandbox environments have the support they need—and are properly governed.

4.4.6 Align with the cloud operating model

Big data processes and users require access to a broad array of resources for both iterative experimentation and running production jobs. A big data solution includes all data realms, including transactions, master data, reference data, and summarized data. Analytical sandboxes should be created on-demand. Resource management is critical to ensure control of the entire data flow, including pre-and post-processing, integration, in-database summarization, and analytical modeling. A well-planned private and public cloud provisioning and security strategy are integral in supporting these changing requirements.

4.5 Summary

In the above chapter, we came across some steps carried out while working with big data. The three steps include integration, management, and analysis. In short words, we first collect the data, manage the data and then analyze it to make it worthy. Also, this chapter includes some of the practices that help us gain more pace in our process. So this is all about how big data is processed to bring profit and beneficial conclusions to us.

4.6 Exercise

1. What are the steps followed in Big Data?

2. What is integration?

3. Why is it important to integrate?

4. What is management?

5. Why is it important to manage the data?

6. What is analysis?

7. What is the final output after analysis?

8. List down the practices followed.

9. Explain one of the practices.

10. Give a short description of the working of big data.

References

Oracle.com. 2021. What Is Big Data? | Oracle. Available at: <https://www.oracle.com/big-data/what-is-big-data/>

Coursehero.com. 2021. Course Hero. Available at: <https://www.coursehero.com/tutors-problems/Computer-Science/27501807-httpswwwbusiness2communitycombig-data19-data-and-analytics-pred/>

Medium. 2021. 100 open source Big Data and ML architecture papers for data professionals (sequel).Available at: <https://towardsdatascience.com/100-open-source-big-data-and-ml-architecture-papers-for-data-professionals-sequel-53fe690440a8?source=post_internal_links---------6---------------------------->

J. Horkoff, D. Barone, L. Jiang, E. Yu, D. Amyot, A. Borgida, and J. Mylopoulos, “Strategic Business Modeling: Representation and Reasoning,” Software & Systems Modeling (13:3), 2014, pp.1015-1041.

P. Giorgini, S. Rizzi, and M. Garcetti, “Goal-oriented requirement analysis for data warehouse design,” Proc. of the 8th ACM Int. Workshop on Data Warehousing and OLAP, 2005.

P. Chapman, J. Clinton, R. Kerber, T. Khabazat, T. Reinartz, C. Shearer, and R. Wirth, CRISP-DM 1.0 Step-by-step data mining guide.

M. K.Kakhani, S. Kakhani and S. R.Biradar, Research issues in big data analytics, International Journal of Application or Innovation in Engineering & Management, 2(8) (2015), pp.228-232.

A. Gandomi and M. Haider, Beyond the hype: Big data concepts, methods, and analytics, International Journal of Information Management,35(2) (2015), pp.137-144.

C. Lynch, Big data: How do your data grow?, Nature, 455 (2008), pp.28-29.

X. Jin, B. W.Wah, X. Cheng and Y. Wang, Significance and challenges of big data research, Big Data Research, 2(2) (2015), pp.59-64.

R. Kitchin, Big Data, new epistemologies and paradigm shifts, Big Data Society, 1(1) (2014), pp.1-12.

C. L. Philip, Q. Chen and C. Y. Zhang, Data-intensive applications, challenges, techniques, and technologies: A survey on big data, Information Sciences, 275 (2014), pp.314-347.

K. Kambatla, G. Kollias, V. Kumar and A. Gram, Trends in big data analytics, Journal of Parallel and Distributed Computing, 74(7) (2014), pp.2561-2573.

S. Del. Rio, V. Lopez, J. M. Bentez and F. Herrera, On the use of MapReduce for big imbalanced data using random forest, Information Sciences, 285 (2014), pp.112-137.

MH. Kuo, T. Sahama, A. W. Kushniruk, E. M. Borycki and D. K. Grunwell, Health big data analytics: current perspectives, challenges and potential solutions, International Journal of Big Data Intelligence,(2014), pp.114-126.

R. Nambiar, A. Sethi, R. Bhardwaj and R. Vargheese, A look at challenges and opportunities of big data analytics in healthcare, IEEE International Conference on Big Data, 2013, pp.17-22.

Z. Huang, A fast clustering algorithm to cluster very large categorical data sets in data mining, SIGMOD Workshop on Research Issues on Data Mining and Knowledge Discovery, 1997.

X. Wu, G. Wu, W. Ding, Data mining with big data, IEEE Trans. Knowl. Data Eng. 28 (2014) 97–106.

Paakkonen, P., Pakkaka, D.: “Reference Architecture and Classification of Technologies,

Products and Services for Big Data Systems”, Big DataResearch2, 166–186 (2015)

Thus, Z. Shao, S. Anthony, D. Borthakur, N. Jain, J. S. Sarma, R. Murthy, H. Liu, Data warehousing and analytics infrastructure at Facebook, in 2010

ACM SIGMOD International Conference on Management of Data, Indianapolis, Indiana, USA, 6–11 June (2010)

Figure Resources

Fig. 4.1. https://miro.medium.com/max/1232/0*1hP9c7U7KMqnEPdn

Fig. 4.2. https://www.brainwareuniversity.ac.in/blog/wp-content/uploads/2019/03/data_management_chart-620x620.png

Fig. 4.3. https://www.sas.com/en_us/insights/analytics/big-data-analytics/_jcr_content/par/styledcontainer_2f7b/par/textimage_f442/image.img.png/1626454122986.png

Chapter 5. Advantages and Disadvantages of Big Data

Abstract

In the previous chapter, we learned about the processes of Big Data. Beyond the benefits mentioned above, there are numerous advantages of big data. We will see some of them in detail and also how they are achieved. Also, everything has some or other disadvantages. Let’s see what those are in terms of big data. You will learn about various advantages and how big data is being used in today’s day-to-day life.

5.1 Benefits

1. Big data analysis derives innovative solutions.

2. Big data analysis helps in understanding and targeting customers.

3. It helps in optimizing business processes.

4. It helps in improving science and research.

5. It improves healthcare and public health with the availability of records of patients.

6. It helps in financial tradings, sports, polling, security/law enforcement, etc.

7. Anyone can access vast information via surveys and deliver an answer to any query.

8. Every second, an addition is made.

9. One platform carries unlimited information.

Fig 5.1 Benefits of big data

5.2 Advantages of Big Data

Big Data is most beneficial for businesses. It is a major pillar for every business and gives it a chance to improve and gain success. Today, it has become an essential need for the business world. Let us see what makes it so valuable for the market industry.

5.2.1 Better customer insight

Every newly started business needs to build its reputation among its customers, and big data helps them achieve that. Therefore, companies collect huge amounts of data to understand their customers and their needs. They collect data from various sources. Big data sources that shed light on customers include the following:

1. Traditional sources of customer insight such as purchases and support calls

2. External sources such as financial transactions and credit status, if these details are available within a company’s terms of service

3. Social media activity

4. Data from external surveys

5. Computer cookies

Clickstream Analysis of e-commerce activity is especially useful in an increasingly digital marketplace, shedding light on how customers navigate through a company’s various web pages and menus to find products and services. Companies can see which items customers added to their carts but perhaps removed or later abandoned without purchasing; this provides important clues as to what customers might like to buy, even if they don’t make a purchase.

5.2.2 More insightful market intelligence

Big Data helps marketers to understand market dynamics. For example, social media is a common source of market intelligence for product categories ranging from breakfast cereal to vacation packages. In every commercial, you can see people sharing their references, experiences, recommendations ... and selfies! These shared opinions are invaluable for marketers. In addition to competitive analysis, big data can also help in product development: prioritizing different customer preferences. In fact, big data does not just assist with modern market intelligence; in almost any e-commerce or online market, almost all market intelligence is driven by diverse, ever-changing data.

5.2.3 Agile supply chain management

Generally, we don’t notice our supply chains until there is a truly major disruption. However, big data, including predictive analytics, often in near real-time, helps keep our global network of demand, production, and distribution working well for the most part. This is possible because big data analytics can integrate customer trends from e-commerce sites and retail applications with supplier data, real-time pricing, and even shipping and weather information to give a level of business intelligence not seen before.

It’s not just large enterprises that benefit from these insights. Even modestly sized e-commerce businesses can use customer intelligence and real-time pricing to optimize business decisions such as stock levels, risk reduction, or temporary or seasonal staffing.

5.2.4 Smarter recommendations and targeting

This is one of the most noticeable features achieved by big data. Most companies use this technique and give their customers satisfactory results. It builds an image in the eyes of the customers. Companies analyze your data and give you recommendations related to your searched product. Also, they give special offers and many more features to the regular and interested customers. At one time, the predictive analysis for recommendation engines was quite simple: association rules found common items in market baskets.

Streaming content providers use even more sophisticated techniques. For example, they may not even ask customers what they want to see next: even before the current movie, program, or song finishes, the next selection fades in, keeping viewers binge-watching by utilizing their preferences combined with a great deal of big data analysis gleaned from other users and social media.

5.2.5 Data-driven innovation

Big Data also reflects the drawbacks of a product and gives the companies to improve their product. This brings a wave of innovation to the company and makes the company more innovative. Also here, human talent is most required as the best big data tools will not produce new insights. We still need the human element: the understanding and imagination of data scientists and business intelligence analysts. However, the breadth and scope of big data, especially when stored on a single platform such as Hadoop or a cloud data warehouse, can lead teams to a new understanding of trends, which would be difficult to glean in a less integrated environment.

5.2.6 Improved operations

Business activity of all kinds can be improved by the use of big data, but one of the most interesting and rewarding has been the use of big data analytics to improve physical operations.

For example, big data and data science to inform predictive maintenance schedules can reduce costly repairs and downtime for critical systems. You can start by analyzing the age, condition, location, warranty, and service details. However, some of these systems, such as security and HVAC in facilities, are notably affected by other business activities such as staffing and production schedules, which may, in turn, be influenced by sales cycles and, therefore, by customer behavior. Well-integrated big data analytics can pull all this together to help you maintain the right equipment at the optimal time.

5.3 Disadvantages of Big Data

1. Traditional storage can cost a lot of money to store big data.

2. Lots of big data is unstructured.

3. Big data analysis violates principles of privacy.

4. It can be used for the manipulation of customer records.

5. It may increase social stratification.

6. Big data analysis is not useful in the short run. It needs to be analyzed for a longer duration to leverage its benefits.

7. Big data analysis results are sometimes misleading.

8. Speedy updates in big data can mismatch real figures.

Fig 5.2 Disadvantages of big data

5.4 Summary

In the above chapter, we came to know some advantages and disadvantages of using big data. However, disadvantages are merely sincere as most of having solutions which also makes big data wholely beneficial. As a result, its demand is continuously increasing in the entire world. Therefore, we should also explore more ways to gain and kickstart our work by using big data. Now you would have figured out why big data is so trending, and also, making a career in this is beneficial in many ways. In the future, you can rely on more advantages from big data. Further, let’s see more about real applications and current services which are actually in use.

5.5 Exercise

1. List down the benefits of big data.

2. In which field is big data most beneficial?

3. What are the advantages of big data?

4. What do you mean by Better customer insight?

5. What do you mean by More insightful market intelligence?

6. What do you mean by Agile supply chain management?

7. What do you mean by Smarter recommendations and targeting?

8. How do companies predict your future buying?

9. How does big data inspire companies to become innovative?

10. How is human talent being used in big data?

11. List down disadvantages of big data?

References

SearchBusinessAnalytics. 2021. 6 Big Data Benefits for Businesses. Available at: <https://searchbusinessanalytics.techtarget.com/feature/6-big-data-benefits-for-businesses>.

Oracle.com. 2021. What Is Big Data? | Oracle. Available at: <https://www.oracle.com/big-data/what-is-big-data/>.

TechVidvan. 2021. Top 10 Advantages of Big Data You Must Know - TechVidvan. Available at: <https://techvidvan.com/tutorials/advantages-of-big-data/>

Selecthub.com. 2021. Types of Big Data | Understanding & Interacting With Key Types. [online] Available at: <https://www.selecthub.com/big-data-analytics/types-of-big-data-analytics/>

Slideshare.net. 2021. Big data and the different types including unstructured, structured a…. Available at: <https://www.slideshare.net/JanetDorenkott/big-data-whats-the-difference-silos-august-2014>.

S. Nalchigar and E. Yu, “Business-driven data analytics: a conceptual modeling framework,” Data & Knowledge Engineering, 2018, pp. 1-14.

Lakster, Laurens L., & Keur, Ron (2012). Big data: Too big to ignore. What organizations can learn from the American presidential elections. Compact, 2: 1–8.

Adami, Christoph (1995). Self-organized criticality in living systems. Physics Letter A, 203(1): 29–32.

Adams, Douglas (2009). The hitchhiker’s guide to the galaxy. London: Macmillan.

Adams, Pauline A., & Adams, Joe K. (1960). Confidence in the recognition and reproduction of words difficult to spell. American Journal of Psychology, 73(4): 544–552.

Adee, Sarah (2008). The data: 37 Years of Moore’s Law. IEEE Spectrum, 45(5): 56–56.

Adler, Mortimer J. (1986). A guidebook to learning: For a lifelong pursuit of wisdom. London: Macmillan.

Agrawal, Divyakant, El Abbadi, Amr, Arora, Vaibhav, Budak, Ceren, Georgiou, Theodore, Mahmoud, Hatem A., Nawab, Faisal, Sahin, Cetin, & Wang, Shiyuan (2015). Mind your Ps and Vs.: A perspective on the challenges of big data management and privacy concerns. International Conference on Big Data and Smart Computing (BigComp), Jeju, South Korea, 1–6.

Ahalt, Stan, & Kelly, Kip (2013). The big data talent gap. UNC Kenan-Flagler Business School White Paper, 1–15.

Aiden, Erez, & Michel, Jean-Baptiste (2013). Uncharted. Big data as a lens on human culture. New York: Riverhead.

Alavi, Maryam, & Leidner, Dorothy E. (2001). Knowledge management and knowledge management systems: Conceptual foundations and research issues. MIS Quarterly, 25(1): 107–136.

Demirkan, H., Delen, D.: Leveraging the capabilities of service-oriented decision support systems: Putting analytics and big data in the cloud, Decision Support Systems 55, 412–421(2013)

Barton, D., Court, D.: Making Advanced Analytics Work For You, Harvard business review, 90 (10), pp. 79–83 (2012)

McAfee, A., Brynjolfsson, E.: Big data: the management revolution, Harvard business review, 90 (10), pp. 60–68 (2012)

X. Wu, G. Wu, W. Ding, Data mining with big data, IEEE Trans. Knowl. Data Eng. 28 (2014) 97–106.

Paakkonen, P., Pakkaka, D.: “Reference Architecture and Classification of Technologies, Products and Services for Big Data Systems”, Big DataResearch2, 166–186 (2015)

Thusoo, Z. Shao, S. Anthony, D. Borthakur, N. Jain, J. S. Sarma, R. Murthy, H. Liu, Data warehousing and analytics infrastructure at Facebook, in 2010

ACM SIGMOD International Conference on Management of Data, Indianapolis, Indiana, USA, 6–11 June (2010)

Figure Resources

Fig. 5.1. https://i2.wp.com/techvidvan.com/tutorials/wp-content/uploads/sites/2/2020/04/benefits-of-big-data.jpg?w=802&ssl=1

Fig. 5.2. http://ivyproschool.com/blog/wp-content/uploads/2015/08/ML-Disadvantages.png

Chapter 6. Real-Time Applications

Abstract

In the previous chapter, we saw the advantages and disadvantages of big data. Now it is time to see some real-time applications. In today’s world, big data is changing the way of marketing and various industries. We will explore more about the current use of big data and the fields or companies using big data to gain benefits for them. Also, how these make their profit and how this chapter will provide you with all the information.

6.1 Fields where big data is being used

•Banking and Securities

•Communications, Media and Entertainment

•Healthcare Providers

•Education

•Manufacturing and Natural Resources

•Government

•Insurance

•Retail and Wholesale trade

•Transportation

•Energy and Utilities

Fig 6.1 Various fields where big data is used

6.1.1 Banking and Securities

This field uses big data to monitor financial market activity. Also, network analytics and natural language processors are being used to catch illegal trading activities in the financial markets; retail traders, Big banks, hedge funds, and other use big data for analyzing trade used in high-frequency trading, pre-trade decision-support analytics, sentiment measurement, Predictive Analytics, etc. Moreover, it also relies on big data to analyze risks, including anti-money laundering, demand enterprise risk management, “Know Your Customer,” and fraud mitigation.

The sources who provide big data to this industry includes

1. 1010data

2. Panopticon Software

3. Streambase Systems

4. Nice Actimize

5. Quartet FS.

6.1.2 Communications, Media and Entertainment

Everyone wishes to see great content on their televisions or some other devices. Therefore, the industry needs to fulfill consumer needs with respect to media and social content. The organizations in this industry analyze their customer data with behavioral data to depict detailed customer profiles that can be used to create content for different target audiences, predict the most demanded content, and also measure their content performance.

Some very successful companies like Spotify use Hadoop’s big data analytics to collect data from millions of their users worldwide and eventually give personalized recommendations to their customers. Amazon Prime, which serves great customer experience in video, music, and Kindle books, also heavily used big data.

The sources who provide big data to this industry includes

•Infochimps

•Splunk

•Pervasive Software

•Visible Measures.

6.1.3 Healthcare Providers

Some hospitals use big data collected from cell phone apps to allow doctors to give evidence-based medicine; this can avoid several lab tests of all the patients who go to the hospital. However, it can be expensive and usually ineffective.

Free public health data and Google Maps have been used by the University of Florida to create visual data that allows for faster identification and efficient analysis of healthcare information used in tracking the spread of chronic disease.

The sources who provide big data to this industry includes

•Recombinant Data

•Humedica, Explorys

•Cerner.

Fig 6.2 Big Data in health care

6.1.4 Education

Big Data is extensively used for higher education. It is mainly required to track down students, teachers, or staff performance. Some of the uses include measuring teacher’s effectiveness to ensure a pleasant experience for both students and teachers. Teacher’s performance can be fine-tuned and measured against student numbers, subject matter, student demographics, student aspirations, behavioral classification, and several other variables.

As far as the government is concerned, the Office of Educational Technology in the U. S. Department of Education uses Big Data to develop analytics to help correct course students who are going astray while using online Big Data certification courses.

The sources who provide big data to this industry includes

•Knewton and Carnegie Learning

•MyFit/Naviance

Fig 6.3 Big data in education

6.1.5 Manufacturing and Natural Resources

In the natural resource industry, big data helps to predict models that support the decision taken, ingesting and integrating large amounts of data from geospatial data, graphical data, text, and temporal data. Seismic interpretation and reservoir characterization are the areas where this has been included. In manufacturing, big data plays a role in solving today’s manufacturing challenges which consist of manufacturing right and suitable product which is more preferred by the customers. It gives them an idea of what we should manufacture next to meet people’s expectations and profit from it.

Here is the graph that shows the use of supply chain capabilities from Big Data currently in use and their expected use in the future.

Fig 6.4 Supply chain capabilities from Big Data

The sources that provide big data to this industry include:

•CSC

•Aspen Technology

•Invensys

•Pentaho.

6.1.6 Government

In government, the main challenge is to integrate big data across various government departments and affiliated organizations. Energy exploration, financial market analysis, fraud detection, health-related research, and environmental protection are some of the big data applications in public services.

For Example, Big Data is used to analyze large amounts of social disability claims made to the Social Security Administration (SSA) that arrive in unstructured data. This analysis is used to process medical information rapidly and efficiently for faster decision-making and to detect suspicious or fraudulent claims.

Also, Food and Drug Administration (FDA) uses big data to detect and study patterns of food-related illnesses and diseases. This allows for a faster response, which has led to more rapid treatment and less death.

The sources who provide big data to this industry includes

•Digital Reasoning

•Socrata

•HP

Fig 6.5 Big data in government

6.1.7 Insurance

In the insurance industry, big data is used to provide customer insights for transparent and simpler products, for which they analyze and predict customer behavior through data derived from social media, GPS-enabled devices, and CCTV footage. It also allows for better customer retention from insurance companies.

In claims management, by analyzing massive amounts of big data, mainly in the underwriting stage, they offer faster customer service. An improvement in fraud detection has also occurred. By obtaining ma data from digital channels and social media, real-time monitoring of claims throughout the claims cycle has provided insights.

The sources who provide big data to this industry includes

•Sprint

•Qualcomm

•Octo Telematics

•The Climate Corp

Fig 6.6 Big data in insurance

6.1.8 Retail and Wholesale trade

Big Data which comes from loyalty data, POS, store inventory, local demographics data, is continuously being gathered by retail and wholesale stores. Companies like Microsoft, Cisco, and IBM pitched the need for the retail industry to utilize Big Data for analytics and other uses, including optimized staffing through data from shopping patterns, local events, reduction in fraud, timely analysis of inventory. Social media is also involved in gaining customer prospecting, customer retention, promotion of products, and more.

The sources who provide big data to this industry includes

•First Retail

•First Insight

•Fujitsu

•Infor

•Epicor

•Vistex

Fig 6.7 How big data is revolutionizing the retail industry

6.1.9 Transportation

Currently, huge amounts of data from location-based social networks and high-speed data from telecoms have affected travel behavior. However, it is happening at a slower pace. Here applications can be categorized into governments, private organizations, and individuals.

The government utilizes big data to control traffic, route planning, intelligent transport systems, congestion management (by predicting traffic conditions). In comparison, the Private sector uses big data in transport for revenue management, technological enhancements, logistics, and competitive advantage (consolidating shipments and optimizing freight movement). Lastly, individuals use big data to plan their routes to save fuel and time, travel arrangements in tourism, etc.

The sources who provide big data to this industry includes

•Qualcomm

•Manhattan Associates.

Fig 6.8 Big Data in transportation

6.1.10 Energy and Utilities

Here data is collected from the consumers to analyze the consumption of utilities better. As a result, it allows for improved customer feedback and better control of utilities use. In utility companies, big data is used for better asset and workforce management, which is useful to recognize errors and correct them before complete failure.

The sources who provide big data to this industry includes

•Alstom Siemens ABB

•Cloudera

Fig 6.9 Big data in energy and utilities

6.2 Companies that use big data

Here is the list of top companies that use big data:

•Amazon.

•American Express.

•BDO

•Capital One

•General Electric (GE)

•Miniclip.

•Netflix.

•Next Big Sound

•Apple

•Google

•Spotify

•FaceBook

•Instagram

•Starbucks

•Mc Donald’s

6.3 Companies that provide big data

Here is the list of top companies that provide big data:

•iTechArt

•ScienceSoft

•Xplenty

•IBM

•HP Enterprise

•Teradata

•Oracle

•SAP

•EMC

•Amazon

•Microsoft

•Google

•VMware

•Splunk

•Alteryx

•Cogito

6.4 Summary

In the above chapter, we explored the big data industry. This chapter threw light on popular industries or fields that use big data extensively to meet the new advancements and innovations of the upcoming generation. Top fields include. Banking and Securities, Communications, Media and Entertainment, Healthcare Providers, Education, Manufacturing and Natural Resources, Government, Insurance, Retail and Wholesale Trade, Transportation. This chapter explains how these use big data to elevate their performance with their data sources. This also accommodates some of the most popular companies who use big data and who provide big data. You get an idea about the vast spread of big data worldwide. Here we have covered almost 90% of the information on big data.

6.5 Exercise

1. List down the fields that use big data. Then, explain any one of them.

2. Why do these use big data?

3. How is big data useful in education?

4. Who provides data to the Transportation industry?

5. List down the key features of big data.

6. Which fields use big data to get customer insights?

7. What is the role of big data in government?

8. Name the company that uses big data to give recommendations on music?

9. Big Data brings more innovation to which fields?

10. List down the companies that extensively use big data.

11. List the companies that provide a huge amount of big data.

Reference

2021. [online] Available at: <https://www.softwaretestinghelp.com/big-data-companies/> [Accessed 25 July 2021].

O’Neill, E. (2019, October 4). Ten companies are using big data. icas.com. https://www.icas.com/thought-leadership/technology/10-companies-using-big-data.

Simplilearn. (2020, April 28). Top 10 big data applications across industries. Simplilearn.com. https://www.simplilearn.com/tutorials/big-data-tutorial/big-data-applications

E. Yu, P. Giorgini, N. Maiden, and J. Mylopoulos, Social Modeling for Requirements Engineering, The MIT Press, 2011.

D. Barone, E. Yu, J. Won, L. Jiang, and J. Mylopoulos, “Enterprise Modeling for Business Intelligence,” The Practice of Enterprise Modeling. PoEM, 2010, pp. 31 – 45.

G. Park, L. Chung, L. Khan, “A Modeling Framework for Business Process Reengineering Using Big Data Analytics and A Goal-Orientation,” Int. Conf. on RCIS, 2017.

M. Mior and A. Aboulnaga, “NoSE: Schema Desing for NoSQL Applications,” IEEE Transaction on Knowledge and Data Engineering, Vol. 29, No.10, 2017, pp. 2275 – pp. 2289.

T. M. Mitchell, Machine Learning, McGraw-Hill, 1997.

M. B. Christopher, Pattern Recognition and Machine Learning, Springer New York, 2016. [28] F. Caro, J. Gallien, M.Miranda., J. Torralbo, J. Corras, M.Vazquez, J. Calamonte, and J.Correa, “Zara uses operations research to reengineer its global distribution process,” Interfaces 40(1), 2010. pp.71-84.

https://sites.google.com/site/irisforbigdata/

Unisphere Research, 2016, The 2016 Enterprise Data Management Survey, Unisphere Research, pp.1-21.

S. Supakkul, and L. Chung, 2009, “Extending Problem Frames to Deal with Stakeholders Problems,” Proc. of the ACM Symposium on Applied Computing, 2009, pp. l389-394.

G. Park, L. Chung, L. Zhao, and S. Supakkul, “A Goal-Oriented Big Data Analytics Framework for Aligning with Business,” Proc. Of IEEE Int. Conf. on Big Data Computing Service and Applications, 2017, pp. 31- 40

Figure Resources

Fig. 6.1. https://www.digitalvidya.com/wp-content/uploads/2018/06/big-data-applications.jpg

Fig. 6.2. https://www.simplilearn.com/ice9/portal_simplilearn_curatasite_com/media/big-data.jpeg

Fig. 6.3. https://4.bp.blogspot.com/-3HHbBP6fIiw/VvbJaRVsEuI/AAAAAAAACZU/iH88CZV7nGggxm-ZBP1A3Qtycowu05d9Q/s1600/big6.jpg

Fig. 6.4. https://www.simplilearn.com/ice9/portal_simplilearn_curatasite_com/media/use-of-supply-chain-capabilities.jpeg

Fig. 6.5. https://www.tutorialandexample.com/wp-content/uploads/2020/01/Big-Data-Application-in-Government.png

Fig. 6.6. http://www.allerin.com/wp-blog/wp-content/uploads/2018/03/4-Big-Data-use-cases-in-the-insurance-industry-you-should-know-about-1.jpg

Fig. 6.7. https://i2.wp.com/techvidvan.com/tutorials/wp-content/uploads/sites/2/2020/03/how-big-data-is-revolutionizing-retail-industry.jpg?w=802&ssl=1

Fig. 6.8. https://mms.businesswire.com/media/20180914005108/en/678365/5/Analyzing_Benefits_of_Big_Data_-_The_Real_Application_of_Big_Data_in_Transportation.jpg

Fig. 6.9. https://image.slidesharecdn.com/bigdataanalyticsinenergyutilities-140605030431-phpapp01/95/big-data-analytics-in-energy-utilities-3-638.jpg?cb=1401937951

Chapter 7. Case Studies

7.1 What are case studies?

A case study is a research methodology typically seen in social and life sciences. There is no one definition of case study research. However, very simply, a case study can be defined as an intensive study about a person, a group of people or a unit aimed to generalize over several units. A case study has also been described as an intensive, systematic investigation of a single individual, group, community, or other unit. The researcher examines in-depth data relating to several variables.

7.2 Case Study 1 – Walmart

Walmart is considered the largest retailer in the entire world. Also, it has the world’s largest company revenue, which has more than 2 million employees,2000 stores, and spread in 28 countries. This company uses big data even much before the term big data came into existence. It uses data mining to provide product recommendations to the user, based on which products were brought together. With this technique, it has increased its conversion rate of customers. It intelligently uses big data to speed up its chores and provide best-in-class e-commerce technologies with a motive to deliver a superior customer experience. The main intention is to provide the best shopping experience to its customers in Walmart stores. Big data solutions at Walmart are developed to redesign global websites and build innovative applications to customize the shopping experience for customers whilst increasing logistics efficiency. It uses Hadoop and NoSQL to provide internal customers with access to real-time data collected from different sources and centralized for effective use.

7.3 Case Study 2 – Uber

Uber is a very popular company when it comes to traveling and making deliveries. They use the customers’ personal data to closely analyze which features of the service are mostly used, analyze usage patterns, and determine where the services should be more focused. It mainly focuses on the supply and demand of the services, resulting in changes in the services provided. For example, if you are running late for an appointment and book a cab in a crowded place, you must be ready to pay twice the amount.

Also, On New Year’s Eve, the price for driving for one mile can go from 200 to 1000. In the short term, surge pricing affects the rate of demand, while long-term use could be the key to retaining or losing customers. This is because it mainly uses machine learning algorithms to determine where the demand is strong.

7.4 Case Study 3 – Netflix

Netflix is the most loved and on-demand when it comes to entertainment. It is continuously using big data to predict what exactly its customers will enjoy watching. Big data fuels their recommendation engine designed to serve this purpose. Recently Netflix has started to work more on content creation and not just a distribution method. Netflix’s recommendation engines and new content decisions are fed by data points such as what titles customers watch, how often playback stopped, ratings are given, etc. It mainly uses Hadoop, Hive, and Pig with many other traditional business intelligence. Netflix wouldn’t have survived so much without big data. A huge amount of its performance and profit relies on Big data.

7.5 Case Study 4 – eBay

eBay is also another popular one when it comes to shopping. It uses big data to search for information tags that have been associated with the data (metadata) and make it consumable to as many people as possible with the right level of security and permissions (data governance).

It uses tools like Apache Spark, Storm, Kafka. The company accurately uses big data solutions and actively contributes its knowledge back to the open-source community.

7.6 Case Study 5 – Procter & Gamble

Procter & Gamble, popularly known as P&G, is a 179-year-old company whose products we all use 2-3 times a day. This company intelligently recognized the value of big data and put it into use for its business around the globe. It has emphasized more on using big data to make better, smarter, real-time business decisions. The Global Business Services organization has developed tools, systems, and processes to provide managers with direct access to the latest data and advanced analytics. Therefore P&G being the oldest company, still holding a great share in the market despite having many emerging companies.

7.7 Case Study 6 – LinkedIn

LinkedIn is very popular nowadays more when it comes to jobs. However, it is a business-oriented social networking service founded in December 2002, launched in 2003, and is mainly used for professional networking. It utilizes big data to develop product offerings such as people you may know, jobs you may be interested in, who has viewed my profile, and more.

7.8 Summary

With this chapter, we can conclude our Part 1 that is an overview of big data. This chapter included some of the case studies of popular companies and vividly explained where exactly they use big data. Big data is expected to grow more in the future, and many different advantages could come into the light. Till now, we have covered most of the topics and concepts related to big data. This is enough to know and understand big data and its importance in our lives, especially in business. Now you can easily tell the secret behind popularity and accurate predictions made by the companies, especially while you are shopping or watching some social media content. Big data is the main reason why these companies can give you personalized services and gain your confidence. Keep reading more to explore other techniques related to data that play an important role in our lives apart from big data.

References

DataFlair Team. (2019, October 15). Five big data case studies - How big companies use big data - DataFlair. DataFlair

Seventeen important case studies on big data. (2018, November 30). Big Data Made Simple. https://bigdata-madesimple.com/17-important-case-studies-on-big-data/

Amatriain, X., 2013. Beyond data: from user information to business value through personalized recommendations and consumer science. Proceedings of the 22nd ACM international conference on information & knowledge management 2201–2208.

Bärenfänger, R., Otto, B., Österle, H., 2014. The business value of in-memory technology–multiple-case study insights. Industrial Management & Data Systems 114, 1396–1414.

Bekmamedova, N., Shanks, G., 2014. Social Media Analytics and Business Value: A Theoretical Framework and Case Study. System Sciences (HICSS), 2014 47th Hawaii International Conference on 3728–3737.

Berelson, B., 1952. Content analysis in communication research. US Free Press, New York.

Bettencourt-Silva, J.H., Clark, J., Cooper, C.S., Mills, R., Rayward-Smith, V.J., De La Iglesia, B., 2015.

Building Data-Driven Pathways From Routinely Collected Hospital Data: A Case Study on Prostate Cancer. JMIR medical informatics 3, 1–21. Cai, H., Jia, X., Chiu, A.S., Hu, X., Xu, M., 2014.

Siting public electric vehicle charging stations in Beijing using big-data informed travel patterns of the taxi fleet. Transportation Research Part D: Transport and Environment 33, 39–46.

Cheng, Y.-C., Chen, P.-L., 2014. Global social media, local context: A case study of Chinese-language tweets about the 2012 presidential election in Taiwan. Aslib Journal of Information Management 66, 342–356.

Christensen, C., 2013. The innovator’s dilemma: when new technologies cause great firms to fail. Harvard Business Review Press 12 (16) Ciulla, F., Mocanu, D., Baronchelli, A., Gonçalves, B., Perra, N., Vespignani, A., 2012.

Beating the news using social media: the case study of American Idol. EPJ Data Science 1, 1–11.

Clauset, A., Newman, M.E., Moore, C., 2004. Finding community structure in very large networks. Physical Review E 70, 1–6.

Crampton, J.W., Graham, M., Poorthuis, A., Shelton, T., Stephens, M., Wilson, M.W., Zook, M., 2013. Beyond the geotag: situating “big data” and leveraging the potential of the geoweb. Cartography and geographic information science 40, 130–139.

Davenport, T., 2014. Big data at work: dispelling the myths, uncovering the opportunities. Harvard Business Review Press.

Laney, D., 2001. 3D data management: Controlling data volume, velocity and variety. META Group Research Note 6, 70.

Lewis, S.C., Zamith, R., Hermida, A., 2013. Content analysis in an era of big data: A hybrid approach to computational and manual methods. Journal of Broadcasting & Electronic Media 57, 34–52.

Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C., Byers, A.H., 2011. Big data: The next frontier for innovation, competition, and productivity. McKinsey Global Institute.

Marine-Roig, E., Clavé, S.A., 2015. Tourism analytics with massive user-generated content: A case study of Barcelona. Journal of Destination Marketing & Management. doi:http://dx.doi.org/10.1016/j.jdmm.2015.06.004i

Martinez, M.G., Walton, B., 2014. The wisdom of crowds: The potential of online communities as a tool for data analysis. Technovation 34, 203–214.

Mathew, P.A., Dunn, L.N., Sohn, M.D., Mercado, A., Custodio, C., Walter, T., 2015. Big data for building energy performance: Lessons from assembling a very large national database of building energy use. Applied Energy 140, 85–93.

Mayer-Schönberger, V., Cukier, K., 2013. Big data: A revolution that will transform how we live, work, and think. Houghton Mifflin Harcourt.

McAfee, A., Brynjolfsson, E., 2012. Big data: The Management Revolution. Harvard Business Review 90, 61–67

Part 2

Part 2

Data Mining

Chapter 8. Overview of Data Mining

Abstract

Far till now, we have seen all about big data. Now onwards, we will learn more about data mining, an important process that works for hand in hand with big data. In this chapter, you will know what data mining is and how it is useful in our lives. Also, it will shed light on its importance in the industry and exactly for what purpose it is used. Data mining comes after collecting enormous data from various sources. Its main aim is to extract useful information from a large amount of data and then convert it into an understandable structure for future use. So let us learn more about data mining!

8.1 What do you mean by Data Mining?

Data mining is a process used by companies to convert raw data into beneficial information. They use software to look for patterns in large batches of data; businesses can learn more about their customers to develop more effective marketing strategies, increase sales and decrease costs. Data mining solely depends on effective data collection, warehousing, and computer processing.

8.1.1 Working of Data Mining

Data mining involves exploring and analyzing large blocks of information to achieve meaningful patterns and trends. This process is broken down into five steps:

1. Collecting data and loading it into the data warehouses.

2. Storing and managing the data, either on in-house servers or the cloud.

3. Business analysts, management teams, and information technology professionals access the data and determine how they organize it.

4. Sorting of the data based on the user’s results by using application software.

5. The end-user presents the data in an easy-to-share format, such as a graph or table.

The above steps complete the data mining process and give us a well-organized form of data that can then be used to make valuable decisions. Further, we will see detailed information on all of these processes.

8.1.2 Example of Data Mining

Grocery stores are well-known users of data mining techniques. For example, most supermarkets offer free loyalty cards to customers that give them access to reduced prices not available to non-members. The cards make it easy for stores to track who is buying what, when, and at what price. After analyzing the data, stores can then use this data to offer customers coupons targeted to their buying habits and decide when to put items on sale or sell them at full price.

8.2 History of Data Mining

The term “Data Mining” was introduced in the 1990s, but still, data mining was begun a lot before that. With respect to previous topics, data mining means converting a large amount of mixed-up data into beneficial information. To achieve these, we use some kind of techniques. Earlier, the techniques for identifying patterns in data include Bayes theorem (the 1700s) and the evolution of regression(1800s). As time passed and technology was increasing, it also gave a boost to data collection, storage, and manipulation as data sets have broad in size and complexity level. Later many data investigations were carried out, which led to the improvement of indirect, automatic data processing and other computer science discoveries such as neural networks, clustering, genetic algorithms (the 1950s), decision trees(1960s), and supporting vector machines (1990s).

Classical statistics, Artificial intelligence, and Machine learning are families on which was originated.

8.2.1 Classical statistics

Statistics is the base for most of the technologies used in data mining, such as:

•Regression Analysis

•Standard Deviation

•Standard Distribution

•Standard Variance

•Discriminatory Analysis

•Cluster Analysis

•Confidence Intervals

All of these are used to analyze data and data connections.

8.2.2 Artificial Intelligence

AI or Artificial intelligence somehow opposes statistics. Instead, it tries to apply human-thought-like processing to statistical problems. Some high-end commercial products adopted a specific AI concept, such as query optimization modules for Relational Database Management System(RDBMS). It is used to connect and understand the statistical problems more with real life.

8.2.3 Machine Learning

Whereas, Machine learning is a combination of statistics and AI. It might be considered as an evolution of AI because it mixes AI heuristics with complex statistical analysis. Machine learning tries to enable computer programs to know about the data they are studying so that programs make a distinct decision based on the characteristics of the data examined. It uses statistics for basic concepts and adding more AI heuristics and algorithms to accomplish its target.

8.3 Tasks Related

The tasks related to data mining can be classified into two types based on what a specific task tries to achieve. The two categories include descriptive tasks and predictive tasks. The descriptive tasks characterize the general properties of data, whereas predictive tasks perform inference on the available data set to predict the behavior of the new data set.

Fig 8.1 Data Mining tasks

8.3.1 Classification

Classification as the name suggests here, objects will be classified according to their attributes. The existing collection of records where each record has a set of attributes is USD for this purpose. Among these attributes, one will be the class attribute. The goal of the classification task is to assign a class attribute to a new set of records as accurately as possible.

The use of classification can be seen in direct marketing, where marketing costs are reduced by targeting customers who are genuinely interested in buying their products. It can also be used to predict the future buying of a customer and recommend the products accordingly.

Fig 8.2 Classification Vs. Clustering

8.3.2 Prediction

This task predicts the values of missing or future data. Here, a model is developed based on available data, then used to predict future values of a new data set. For example, a model can be developed to predict an employee’s income based on education, experience, and other demographic factors like place of stay, gender, etc. It can also be used to predict various areas, including medical diagnosis, fraud detection, etc.

8.3.3 Time - Series Analysis

Time series can be taken as a sequence of events in which the next event is determined by one or more of the preceding events. It basically measures a process also; certain components affect the behavior of a process. To extract useful patterns, trend rules, and statistics, it uses some methods to analyze time-series data. An application of time series analysis can be seen in Stock marker prediction.

Fig 8.3 Example of time series analysis

8.3.4 Association

Association reveals the connection among a set of items. It derives the relationship between various objects. Commodity Management, Advertising, Catalog Design, Direct Marketing, etc., are some of the applications of association. For example, A etailer can identify the products that normally customers purchase together or even find the customers who respond to the promotion of the same kind of products. If a retailer finds that beer and nappy are mostly bought together, he can put nappies on sale to promote beer.

Fig 8.4 Association

8.3.5 Clustering

Clustering identifies data objects which are similar to each other. Similarity can be decided on a number of factors like purchase behavior, responsiveness to certain actions, geographical locations, etc. For example, an insurance company can cluster its customers based on age, residence, income, etc. This information will be helpful to understand the customers better and hence provide better-customized services.

8.3. Summarization

Here generalization of data takes place. A set of data is summarized to get a smaller set that gives aggregated information of the data. For example, the shopping done by a customer can be summarized into total products, total spending, offers used, etc. This summarized information can be useful for the sales or customer relationship team for analyzing detailed customer and purchase behavior. Data can be summarized at different abstraction levels and from different angles.

Fig. 8.5 Data Summarization

8.4 Career Scope

Data also has a promising career scope as it is also related to data that is growing at a faster rate. Therefore, it needs data professionals to carry out various techniques to carry out data mining by using complex algorithms and accurate techniques. Therefore, the major career or job you can expect is data mining specialist. Below you will find all the information related to data mining specialists.

8.4.1 Data Mining Specialist

The main job of a data mining specialist is to find hidden information from vast data stores, interpret the value and meaning of the information and understand how it will be beneficial to the organization. They use some of the software to analyze data and get solutions to business problems. To do these things, one must have a mastery of technological skills (especially programming software) and business intelligence.

A data mining specialist should find patterns and relationships that exist among large piles of data. This information can be used to make future predictions and improve business strategies. The main challenge for a data mining specialist is to create useful insights from the gained information. These insights can be useful for :

•Minimizing Risk And Costs

•Increasing Revenue

•Discovering New Markets

•Understanding consumer behavior better to produce tailored marketing campaigns.

The data mining specialist uses data analysis programs to research, mine data, model relationships, and then report these findings to the client using data visualization techniques, such as graphs, bar charts, scatterplots, etc.

Data mining specialists work with three types of data which are:

1. Transactional

2. Non-Operational

3. Metadata.

Transactional data documents an exchange of some sort, generally an agreement or monetary transfer between organizations like a company and/or individuals. This is the kind produced daily in sales, most obviously, but may also cover inventory, operating costs, and client visits. A data mining specialist for a business is often looking at who is buying what and why this is obviously a key data set.

Non-operational data includes data produced by industries that can be used to create insights for an organization or a company. In contrast, metadata gives you more information about the existing data.

Here are the skills needed to become a data mining specialist:

•SQL

•NoSQL

•SAS

•Hadoop

•Java

•Python

•Perl

•Linux

Also, one needs to have business sense. A data mining specialist must have a good idea of business to make beneficial decisions for the company. At last, the insights or conclusions made by he/she should be converted into a simple or understandable form so that non-technical staff can understand it. Hence a data mining specialist should have strong public-speaking skills and communicate results to internal and external shareholders.

The average salary of a data mining specialist is around $67,407 per year. Although if you have experience, great skills, and have a high level of data knowledge, you can reach up to $100,00 annually. Far till now, you can be the best-paid data mining specialist if you have strong skills in SAS and SQL.

8.5 Future of Data Mining

8.5.1 Future trends in data mining

1. Multimedia Data Mining

This is one of the latest methods catching up because of the growing ability to capture useful data accurately. It involves data extraction from different kinds of multimedia sources such as audio, text, hypertext, video, images, etc. The data is converted into a numerical representation in different formats. This method can be used in clustering and classifications, performing similarity checks, and identifying associations.

2. Ubiquitous Data Mining

This method involves the mining of data from mobile devices to get information about individuals. Despite having several challenges in this type, such as complexity, privacy, cost, etc., this method has a lot of opportunities to be enormous in various industries, especially in studying human-computer interactions.

3. Distributed Data Mining

This type of data mining is gaining popularity as it involves mining huge amounts of information stored in different company locations or at different organizations. Highly sophisticated algorithms extract data from different locations and provide proper insights and reports based upon them.

4. Spatial and Geographic Data Mining

This is a new trending type of data mining that includes extracting information from environmental, astronomical, and geographical data, including images taken from outer space. This type of data mining can reveal various aspects such as distance and topology, mainly used in geographic information systems and other navigation applications.

5. Time Series and Sequence Data Mining

The primary application of this type of data mining is the study of cyclical and seasonal trends. However, this practice is also helpful in analyzing even random events which occur outside the normal series of events. Retail companies are mainly using this method to access customer’s buying patterns and their behaviors.

The above are some of the emerging data mining trends that can improve the field and bring more profit and accurate information.

8.6 Data Mining V/S Big Data

Data Mining

Big Data

It is one of the methods in the pipeline of Big Data.

Big Data is a technique to collect, maintain and process huge information. It explains the data relationship

Data mining is a part of the Knowledge Discovery of the Data. It is a close view of the data.

It is about extracting vital and valuable information from a huge amount of data. Second, it is a technique of tracking and discovering trends of complex data sets. Finally, it is a large or overall view of the data.

The goal is the same as Big Data as it is one of the tools of Big Data.

The goal is to make data more vital and usable, i.e., by extracting only important information from the huge data within existing traditional aspects.

It is manual as well as automated in nature.

It is only automated as computing huge data is difficult.

It only focuses on only one form of data. i.e., structured.

It focuses and works with all forms of data, i.e., structured, unstructured, or semi-structured.

It is used to create certain business insights. Data mining is a manager of the mine.

It is mainly used for business purposes and customer satisfaction. Big Data is a mine.

It is a subset of Big Data. i.e., one of the tools.

It is a superset of Data Mining.

It is a tool to dig up vital information from large data. Data can be large as well as small.

It is more involved with the processes of handling voluminous data. This is because data can only be large.

8.7 Summary

In the above chapter, we took an overview of data mining and understood how this process occurs. There are total five steps in which data mining is generally carried out. The history of data mining is also quite interesting and concludes that data mining is successful today only because of researches and studies carried out. It has been observed that mathematics plays a very important role when it comes to data mining. Data mining is originated from Classical Statistics, Artificial Intelligence, and Machine learning. We also saw tasks that are related to data mining. Also, how these small tasks help us or organizations to gain more knowledge about their business and customers. These tasks help you to make organized and well-understood data. This chapter also enlightens us with a vast career scope available in data mining with future expectations and predictions made for Data mining. It is a vast field, and there is a lot more to study.

8.8 Exercise

1. What is Data Mining? How is it beneficial to us?

2. When was the term Data Mining firstly introduced?

3. What are the five steps of data mining?

4. Name three families from which data mining was originated?

5. What is Classical Statistics?

6. What is Artificial Intelligence?

7. What is Machine Learning?

8. What are the two categories of tasks?

9. List down the tasks related to data mining.

10. What is the career scope of data mining?

11. Explain Prediction.

12. What are the future trends of data mining? Give an example of Association.

13. Stock market prediction is done by using _______________.

References

History of data mining. (n.d.). www.javatpoint.com.

What are transactional data? - Definition from Techopedia. (2014, July 11). Techopedia.com.

Data Mining and Statistics: What’s the Connection?

Data Mining: Statistics and More?, D. Hand, American Statistician, 52(2):112-118.

Data Mining, G. Weiss and B. Davison, in Handbook of Technology Management, John Wiley and Sons, expected 2010.

From Data Mining to Knowledge Discovery in Databases, U. Fayyad, G. Piatesky-Shapiro & P. Smyth, AI Magazine, 17(3):37-54, Fall 1996.

Mining Business Databases, Communications of the ACM, 39(11): 42-48.

10 Challenging Problems in Data Mining Research, Q. Yiang and X. Wu, International Journal of Information Technology & Decision Making, Vol. 5, No. 4, 2006, 597-604. (slides)

Top 10 Algorithms in Data Mining, X. Wu, V. Kumar, J.R. Quinlan, J. Ghosh, Q. Yang, H. motoda, G.J. MClachlan, A. Ng, B. Liu, P.S. Yu, Z. Zhou, M. Steinbach, D. J. Hand, D. Steinberg, Knowl Inf Syst (2008) 141-37.

Induction of Decision Trees, R. Quinlan, Machine Learning, 1(1):81-106, 1986.

The Pagerank Citation Ranking: Bringing Order to the Web, L. Page, S. Brin, R. Motwani, T. Winograd, Technical Report, Stanford University, 1999.

The Structure and Function of Complex Networks, M. E. J. Newman, SIAM Review, 2003, 45, 167-256.

Link Mining: A New Data Mining Challenge, L. Getoor, SIGKDD Explorations, 2003, 5(1), 84-89.

Link Mining: A Survey, L. Getoor, SIGKDD Explorations, 2005, 7(2), 3-12.

Introduction to Semi-Supervised Learning, in Semi-Supervised Learning (Chapter 1) O. Chapelle, B. Scholkopf, A. Zien (eds.), MIT Press, 2006. (Fordham’s library has online access to the entire text)

Learning with Labeled and Unlabeled Data, M. Seeger, University of Edinburgh (unpublished), 2002.

Person Identification in Webcam Images: An Application of Semi-Supervised Learning, M. Balcan, A. Blum, P. Choi, J. Lafferty, B. Pantano, M. Rwebangira, X. Zhu, Proceedings of the 22nd ICML Workshop on Learning with Partially Classified Training Data, 2005.

Learning from Labeled and Unlabeled Data: An Empirical Study across Techniques and Domains, N. Chawla, G. Karakoulas, Journal of Artificial Intelligence Research, 23:331-366, 2005.

Text Classification from Labeled and Unlabeled Documents using EM, K. Nigam, A. McCallum, S. Thrun, T. Mitchell, Machine Learning, 39, 103-134, 2000.

An iterative algorithm for extending learners to a semisupervised setting, M. Culp, G. Michailidis, 2007 Joint Statistical Meetings (JSM), 2007

Figure Resources

Fig. 8.1. https://www.wideskills.com/sites/default/files/subjects/Data%20Mining%20Tutorial/05/image1.jpeg

Fig. 8.2. https://cdn.educba.com/academy/wp-content/uploads/2019/02/vs.jpg

Fig. 8.3. https://algorithmia.com/blog/wp-content/uploads/2018/02/blue_decomposition.jpeg

Fig. 8.4. https://miro.medium.com/max/839/1*xu3-x32Vep_NooObcL_y3g.png

Fig. 8.5. https://media.springernature.com/lw685/springer-static/image/art%3A10.1007%2Fs10115-018-1183-0/MediaObjects/10115_2018_1183_Fig2_HTML.gif

Chapter 9. Stages Of Data Mining Process

Abstract

After taking an overview of data mining, let’s check out the stages of data mining processes. In data mining, every stage brings you near to your final insight or decision. These stages need to be followed to work with data. They help you to make data mining an easier process and not as tedious as it looks. Once you understand all these stages, you will be able to do data mining smoothly. In this chapter, let’s look at what those stages are and study each of them.

9.1 Data Purification

Data purification is the initial stage where you clean a large amount of data gathered. In this, you should complete missing values, maintain consistency, and then combine the data for further process. It also includes organizing and extracting only valuable information by eliminating unnecessary data. This stage requires a lot of labor work and IT resources. Ultimately it is too time-consuming but again necessary to gain the best quality data.

9.2 Data Integration

In the second step, data from various sources are combined together and dumped into one single source. This can be a very tough process if you have messed up or lost data. It prevents data redundancy without affecting the reliability of data. There are two approaches for data integration:

9.2.1 Tight Coupling

In this data, the warehouse is considered an information retrieval component. Data is collected from different sources and combined into a single physical location. For this, we use ETL, i.e., Extraction, Transformation, and Loading. To perform data mining tasks, all the features of the data warehouse are utilized. This gives us scalability, high performance, and integrated information.

9.2.2 Loose Coupling

Here instead of a data warehouse, a database is used for retrieving data. The result is also stored in those systems. It is mainly for memory-based data mining systems. This does not give us high scalability and high performance.

Fig 9.1 Data integration

9.3 Data Selection

In the third stage, relevant data is filtered from the database. Here you need to select the data which is actually beneficial for your purpose. In some cases, data selection is carried out before the two processes. This can help you focus on only relevant and useful data among unuseful data, which can make your work faster and smarter.

9.4 Data Transformation

In this stage, data is transformed into a more clear and accurate form. For example, figures like “-4, 101 and 65 ” can be transformed into “-0.04, 1.01 and 0.65”. Here are some ways in which data transformation can be done:

9.4.1 Smoothing

Smoothing is a process used to remove noise from the dataset using some algorithms; as a result, important features present in the dataset get highlighted. It also helps in predicting the patterns. The main intention behind data smoothing is to identify simple changes to help predict different trends and patterns. This helps the analysts or traders who need to look at a lot of data that can often be difficult to digest to find patterns they wouldn’t see otherwise.

9.4.2 Aggregation

After smoothing, you can aggregate your data which means you can store and present the data in a summary format. This is a very important step as the accuracy of your data insights is highly dependent on the quality and quantity of the data used. This is then further used to make accurate and profitable decisions for a company or an organization, build business strategies, pricing, marketing strategies, etc.

Fig 9.2 Aggregation

9.4.3 Discretization

Discretization is a process of transforming data into a set of small intervals. Most Data Mining activities in the real world require continuous attributes. Still, many of the existing data mining frameworks are unable to handle these attributes. Even if a data mining task can manage a continuous attribute, it can significantly improve efficiency by replacing a constant quality attribute with its discrete values.

For example, (1-10, 11-20) (age:- young, middle age, senior).

9.4.4 Generalization

Generalization is a process of converting low-level data attributes to high-level data attributes by using concept hierarchy. For example, you can convert age in numerical form into a categorical value such as young, old.

9.3 Example of Data transformation

9.5 Pattern Evaluation

In this stage, patterns between the data are analyzed. These patterns give us a chance to know more about our data. Also, it can make it easier for non-technical people to understand while making decisions based on the insights. It is also a new form of data with a degree of certainty. Here one can easily see the increasing or decreasing patterns or new trends running.

9.6 Knowledge Representation

In the final step, the data processed is being converted into an engaging and organized form. Basically, you design your data and make it more interesting. Further, this data will be presented to make final outcomes, and righteous decisions will be made.

Fig 9.4 Knowledge Representation Techniques

9.7 Summary

In this chapter, we explored six essential stages of the data mining process. These stages are very important when you are doing data mining. It helps to create good quality data, which can be super beneficial for a company or organization. The six stages include Data Purification, Data Integration, Data Selection, Data Transformation, Data transformation, Pattern Evaluation, Knowledge Representation. Completing each stage gives you more filtered and valuable data so that the best insights can be created through that, which will then be analyzed and utilized to make important decisions and improve the company.

9.8 Exercise

1. How many stages does data mining have?

2. List down all the stages.

3. What is 4 stage in data mining?

4. Why does data need to be cleansed?

5. What do you mean by data integration?

6. Data integration can be done in how many ways?

7. What is loose and tight coupling?

8. What do you mean by aggregation?

9. List down the ways of data transformation.

10. Explain smoothing.

References

Porter, M. (2019, February 14). 6 stages of data mining process in the wisdom of business. Medium.

What is data transformation? (2020, November 3). Trifacta.

Data mining - Quick guide. (n.d.). RxJS, ggplot2, Python Data Persistence, Caffe2, PyBrain, Python Data Access, H2O, Colab, Theano, Flutter, KNime, Mean.js, Weka, Solidity.

Fayyad, Usama, Gregory Piatetsky-Shapiro, Padhraic Smyth, From Data Mining to Knowledge Discovery in Databases, 1996.

Jiawei Han, Micheline Kamber, Data Mining: Concepts and Techniques, London: Academic Press, 5, 2001.

Kantardzic, Mehmed, Data Mining: Concepts, Models, Methods, and Algorithms, New York: John Wiley & Sons Inc publishes, 2003.

Michael Chau, Reynold Cheng, Ben Pao, Uncertain Data Mining: A New Research Direction, Introduction, 2005.

Mika Sato, Yoshiharu Sato, L.C.Jain, J.Kacprzyk, Fuzzy Clustering Models and Applications. UK: Physica-Verlag Heidelberg, 1999.

Ouri Wolfson, A. Prasad Sistla, Sam Chamberlain, and Yelena Yesha, Updating and Querying Databases that Track Mobile Units, MA: Kluwer Academic Publishers, 1999.

Reynold Cheng, Dmitri V.Kalashnikov, Sunil Prabhakar, Evaluating Probabilistic Queries over Imprecise Data, UK: Elsevier Science Ltd, 2007.

Sapphire, Large Scale Data Mining and Pattern Recognition, https://computation.llnl.gov/casc/sapphire/overview/data_mining_steps.gif, 1999.

Introduction to Semi-Supervised Learning, in Semi-Supervised Learning (Chapter 1) O. Chapelle, B. Scholkopf, A. Zien (eds.), MIT Press, 2006. (Fordham’s library has online access to the entire text)

Learning with Labeled and Unlabeled Data, M. Seeger, University of Edinburgh (unpublished), 2002.

Person Identification in Webcam Images: An Application of Semi-Supervised Learning, M. Balcan, A. Blum, P. Choi, J. Lafferty, B. Pantano, M. Rwebangira, X. Zhu, Proceedings of the 22nd ICML Workshop on Learning with Partially Classified Training Data, 2005.

Learning from Labeled and Unlabeled Data: An Empirical Study across Techniques and Domains, N. Chawla, G. Karakoulas, Journal of Artificial Intelligence Research, 23:331-366, 2005.

Text Classification from Labeled and Unlabeled Documents using EM, K. Nigam, A. McCallum, S. Thrun, T. Mitchell, Machine Learning, 39, 103-134, 2000.

An iterative algorithm for extending learners to a semisupervised setting, M. Culp, G. Michailidis, 2007 Joint Statistical Meetings (JSM), 2007

Figure Resources

Fig. 9.1. https://media.geeksforgeeks.org/wp-content/uploads/ 20190627172215/ 661.png

Fig. 9.2. https://www.google.com/url?sa=i&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DbDwF-TOMtWw&psig=AOvVaw06v6fOozQbNcfQyMVpl-Lf&ust=1626787082488000&source=images&cd=vfe&ved=0CAsQjRxqFwoTCKjC4fyH7_ECFQAAAAAdAAAAABAD

Fig. 9.3. https://www.researchgate.net/profile/Ian-Foster-3/publication/221909965/figure/fig3/AS:667708106498061@1536205426477/An-example-of-data-transformation-I-Hard-coded-data-elements-II-Semantically-unified.png

Fig. 9.4. https://static.javatpoint.com/tutorial/ai/images/ai-techniques-of-knowledge-representation.png

Chapter 10 .Data Mining Techniques

Abstract

In the previous chapter, we learned stages of data mining now; let’s see which techniques are used to achieve and carry out those stages. As a data mining scientist, one needs to know different types of techniques and should be able to select the best breed for his/her data. Using a suitable technique plays a very important role and can bring a serious problem if the technique is wrong. Suitable techniques help you to reduce time and work so that the data mining process will become faster. Therefore knowledge of various techniques is a must in data mining. Let’s see some of the best techniques used currently and are successfully giving valuable output.

Fig 10.1 Data mining techniques

10.1 Tracking patterns

In this technique, patterns between the data are tracked down to recognize trends that are consistently happening. This will help to recognize various trends or increasing demands so that the organization or the company can make improvements and manufacture projects according to that. For Example, if you are analyzing data that includes your company sales, you can track down at which season what type of products are being purchased and what do people prefer more. So according to that you can manufacture more products of that kind and can increase your sales.

10.2 Classification

Classification is a bit complex process in data mining. First, you need to collect data of various attributes and club them into suitable attributes that can be used to derive further conclusions. This helps you to learn more about a particular field of data. For Example, if you are analyzing data that consists of a list of customers with their no. of purchases, you can further classify them into high, medium, and low purchases.

10.3 Clustering

In clustering, data is grouped based on similarities that exist between them. For Example, while analyzing various biscuits, you find that many biscuits are of chocolate flavor to group them. So later, if you want to analyze things related to chocolate, this can be very useful.

10.4 Outlier detection

In most cases, simply identifying patterns won’t help derive a better conclusion; you need to investigate the reason or outliers of the data. For example, while analyzing the purchase of chocolates, you found that particular chocolate is being purchased more in the past few days. You can find the reason behind that and replicate that situation if possible to increase purchase.

Fig 10.2 Outliers

10.5 Association

Association, as we discussed earlier, you need to find the links between various data sets and explore how they are linked. These links can be useful to predict the future buying of the customers. For Example, if a customer buys a desktop, he\she will definitely need a mouse so that you can add that suggestion for them. You can see these types of suggestions while you are shopping from Amazon or Flipkart. They show the “frequently bought together section,” which gives you an idea of the suitable product to buy further.

Fig 10.3 Association

10.6 Regression

Regression is used to plan, and a model is used to identify a certain variable’s likelihood given other variables. For Example, you can use it to determine the price based on availability, consumer demand, and competition. Its main focus is to help you uncover the exact relationship between two or more variables in a data set.

Fig 10.4 Regression

10.7 Prediction

Prediction is most valued in data mining. In this, you predict the future of your product. If accurate predictions are not made, then patterns recognized earlier will be of no use. Here you can predict the future buying of your customer and suggest the same which will increase the sales.

10.8 Sequential patterns

Sequential patterns the name itself reveals what it is. Here the main focus is on finding series of events that take place in a sequence. This can reveal the sequence of products being purchased. For Example, if a customer is buying a laptop, then there are some of the products which they can buy like speakers, mouse, extra keyboard, pen drive, etc., which you can suggest to your customers.

Fig 10.5 Sequence in database

10.9 Decision trees

Decision trees are more part of machine learning. It is considered as a specific type of predictive model for effectively mining data. A decision tree enables users to clearly understand how the data inputs affect the outputs. When various decision tree models are combined, they create predictive analytics models known as a random forest.

Fig 10.6 Decision trees

10.10 Statistical techniques

The statistical technique is the core of the data mining process. Different types of analytics models are based on this technique in which numeric values apply to specific business objectives. For example, neural networks use complex statistics based on different weights and measures to determine if a picture is a dog or a cat in image recognition systems. It also represents one of the two main branches of artificial intelligence.

Fig 10.7 Statistics

10.11 Visualization

Visualization is another important element of data mining. Data visualizations are another important element of data mining. They grant users insight into data based on sensory perceptions that people can see. In current times data visualizations are dynamic, useful for streaming data in real-time, and characterized by different colors that reveal different trends and patterns in data. Dashboards are a powerful way to use data visualizations to uncover data mining insights. Dashboards can be based on different metrics and use visualizations to visually highlight patterns in data instead of simply using numerical outputs of statistical models.

Fig 10.8 Data Mining Visualization Techniques

10.12 Neural networks

The neural network comes under the machine learning model that is often used with AI and deep learning. They consist of different layers which depict the way neurons work in the human brain.

It is one of the more accurate machine learning models used today. It is also a most powerful tool when it comes to data mining. Though organizations should take caution when using it. Some of these neural networks models are incredibly complex, which can be difficult to understand how a neural network determined an output.

Fig 10.9 Neural network

10.13 Data warehousing

Data warehousing is one of the important parts of the data mining process. Previously, data warehousing involved storing structured data in relational database management systems to be analyzed for business intelligence, reporting, and basic dashboarding capabilities. Currently, there are cloud data warehouses and data warehouses in semi-structured and unstructured data stores like Hadoop. While data warehouses were traditionally used for historical data, many modern approaches can provide an in-depth, real-time data analysis.

Fig 10.10 Data warehousing

10.14 Long-term memory processing

Long-term memory processing refers to the ability to analyze data over extended periods. The historical data stored in data warehouses is useful for this purpose. Once an organization can perform analytics for an extended period, it can identify patterns that otherwise might be too subtle to detect. For example, by analyzing attrition over several years, an organization may find subtle clues that could reduce churn in finance.

10.15 Machine learning and AI

Machine learning and artificial intelligence (AI) are some of the most advanced developments in data mining. Advanced forms of machine learning like deep learning offer highly accurate predictions when working with data at scale. Consequently, they are useful for processing data in AI deployments like computer vision, speech recognition, or sophisticated text analytics using Natural Language Processing. In addition, these data mining techniques are good for determining value from semi-structured and unstructured data.

Fig 10.11 ML and AI

10.16 Summary

In this chapter, we saw some of the extensively used techniques in the data mining process. Each technique has its own importance and is used for gaining multiple outputs. These are very useful for data analysts to give better outputs and make the process easier. Currently, these techniques are proving a big help to the data professionals in retrieving suitable insights from a large set of data.

10.17 Exercises

1. What are the various techniques in data mining?

2. Explain the below terms:

3. Regression

4. Sequential Patterns

5. Clustering

6. Warehousing

7. Visualization

8. Neural Network

9. Decision trees

10. How do these techniques help us?

11. What are the different data warehouses available?

References

History of data mining. (n.d.). www.javatpoint.com.

Data mining tasks | Data mining tutorial by Wideskills. (n.d.). wideskills.com | Develop new skills and find better jobs

What are transactional data? - Definition from Techopedia. (2014, July 11). Techopedia.com.

M.S. Mythili and A.R. Mohamed Shanavas, “An analysis of students” performance using classification algorithms,” IOSR, Journal of Computing Engineering, volume 16, Issue 1, January 2014.

S.Lakshmi Prabha, A.R.Mohamed Shanavas, “Educational data mining applications”, Operations Re-search and Applications: An International Journal (ORAJ), volume 1, No. 1, August 2014.

Davinder Kaur, Rajeev Bedi and S.K Gupta, “Review of decision tree data mining algorithms: ID3 and C4.5”, pp. 5-8, July 2015

S.Ayesha, T.Musafa, A.Sattar, and M.khan, “Data mining model for the higher education system,” European Journal of Scientific Research, vol. 43, no 1, 2010. pp. 24-29.

Open for Innovation KNIME Software, URL: https://www.knime.com/knime-software.

Electronic URL Source- https://slideplayer.com/slide/13868930/

URL Source -https://bigdatanerd.wordpress.com/2011/06/25/introduction-to-data-mining-types-of-data-mining-techniques/

.R Kabra and R.S Bichkar, “Performance prediction of engineering students using a decision tree,” International Journal of Computer Applications. Vol.36. No 11, December 2011, pp. 8-12.

Classification and Prediction, Electronic URL source- http://web.ydu.tw/~alan9956/doc101/datamining/dm06.pdf

S.Neelamegam and E. Ramaraj, “Classification algorithm in Data mining: An Overview,” volume 3, issue 5, September 2013, pp. 1-5.

Figure Resources

Fig. 10.1. https://miro.medium.com/max/1400/0*V1zUdsmZhx0E7Fh_.png

Fig. 10.2. https://media.geeksforgeeks.org/wp-content/uploads/outlier.jpg

Fig. 10.3. https://www.researchgate.net/profile/Pavithra-A/publication/329144978/figure/fig1/AS:699718669725696@1543837339012/The-Association-Rules-Data-Mining-Architecture.jpg

Fig. 10.4. https://www.mssqltips.com/tipimages2/3471_Example.JPG

Fig. 10.5. https://data-mining.philippe-fournier-viger.com/wp-content/uploads/2013/10/seq_rules.png

Fig. 10.6. https://www.saedsayad.com/images/Decision_Tree_1.png

Fig. 10.7. https://cdn.scribbr.com/wp-content/uploads/2020/10/standard-normal-distribution-768x475.png

Fig. 10.8. https://study.com/cimages/multimages/16/data-mining-visualization.jpg

Fig. 10.9. https://www.oracle.com/a/tech/img/art-neural-network-image001.png

Fig. 10.10. https://upload.wikimedia.org/wikipedia/commons/thumb/8/8d/Data_warehouse_architecture.jpg/330px-Data_warehouse_architecture.jpg

Fig. 10.11. http://mse238blog.stanford.edu/wp-content/uploads/2017/07/AI_ML_DL.png

Chapter 11. Tools Used In Data Mining

Abstract

In this chapter, we will see some of the tools which are very popular in data mining. These tools make the data mining process successful and easier, which saves time. For a data mining specialist, these tools are a blessing. For creating a successful career in data, mining one needs to be familiar with the best tools and expertise. This chapter will enlighten you with some of the best tools you can use to elevate your skills and increase your caliber.

11.1 MonkeyLearn

MonkeyLearn is mostly preferred when it comes to text mining. It has a user-friendly interface which makes it easier to work with. Furthermore, one can easily integrate MonkeyLearn with existing tools for performing data mining in real-time. It supports data mining tasks like detecting topics, sentiments, intent also extracting keywords and named entities. Currently, MonkeyLearn mining tools are used to automate ticket tagging and routing in customer support, automatically detect negative feedback in social media, and deliver fine-grained insights that lead to better decision-making.

With this, one can also connect analyzed data to MonkeyLearn Studio, a customizable data visualization dashboard that makes it even easier to detect trends and patterns in the data.

For example, you can see the picture below:

Fig 11.1 MonkeyLearn

11.2 RapidMiner

Rapid Miner is a free, open-source platform with hundreds of algorithms for data preparation, deep learning, machine learning, predictive analysis, and text mining. It consists of a drag-and-drop interface and pre-built models that allow non-programmers to intuitively create predictive workflows for specific use cases, like fraud detection and customer churn. So it is beneficial for non-programmers also.

As for programmers, they can look at R and Python extensions and work with them for their data mining process. It will also help you spot patterns, outliers, and trends in your data after creating your workflows and analyzing your data.

Fig. 11.2 RapidMiner Overview

11.3 Oracle Data Mining

Oracle data mining is a component of advanced oracle analytics that enables data analysts to build and implement predictive models. It consists of data mining algorithms for performing tasks like classification, regression, anomaly detection, prediction, and more. It also helps you predict customer behavior, segment customer profiles, detect fraud, and identify the best prospects to target.

Fig. 11.3 Oracle data mining

11.4 IBM SPSS Modeler

IBM SPSS Modeler is a solution to data mining that enables data scientists to speed up and visualize the data mining process. Even when it comes to non-programmers, this is best as they can use advanced algorithms to build predictive models in a drag-drop interface. Data science teams can also import vast amounts of data from various sources and rearrange it to get trends and patterns.

It also has a standard version that works with numerical data coming from relational databases or spreadsheets. Though it is not freely available, certain things can be done only with the premium version.

Fig. 11.4 IBM SPSS Modeler

11.5 Weka

Weka is again an open-source machine learning software that comes with a massive collection of algorithms. This was developed by the University of Waikato, situated in New Zealand. Data mining tasks like classification, clustering, regression, and visualization, in a graphical interface that makes it easy to use. It gives built-in machine learning algorithms for each task that can save your time and deploy models without code.

One with sound knowledge of the different algorithms can take full advantage of this. It was originally designed for analyzing data for the field of agriculture. But now its main use is done by researchers, industrial scientists and also for educational purposes.

Fig. 11.5 Weka Overview

11.6 KNIME

KNIME is also an open-source platform. Its interface enables you to create end-to-end data science workflows, which can be used right from modeling to production. It is also best for non-programmers as it consists of inbuilt components, which result in fast modeling without code. Furthermore, it is a versatile and scalable platform for processing complex types of data and advanced algorithms as it contains a set of powerful extensions and integrations. It also allows data scientists to create applications and services for analytics and business intelligence.

Fig 11.6 KNIME Overview

11.7 H2O

Here comes another open-source platform, which aims to make AI accessible to everyone. It supports machine learning algorithms that are most common and can offer auto ML functions that can be used to build and deploy ML models in a fast and simple way.

H2O is available in all major programming languages and also uses distributed in-memory computing. As a result, it is most suitable when it comes to analyzing huge data sets.

Fig. 11.7 H2O Overview

11.8 Orange

Orange is considered a data science toolbox for developing, testing, and visualizing data mining workflows. It is a component-based software that supports a large collection of inbuilt machine learning algorithms and text mining add-ons. Also, it has extended functionalities for bioinformaticians and molecular biologists. It offers interactive data visualization, numerous graphics like silhouette plots and sieve diagrams, and for non-programmers, it has data mining tasks through visual programming in a drag-and-drop interface. In contrast, programmers can do their tasks in Python.

Fig. 11.8 Orange Overview

11.9 Apache Mahout

Apache mahout is also an open-source platform used for creating scalable applications through machine learning. Its main motto is to help data scientists and researchers implementing their own algorithms. It is written in Javascript and implemented on apache Hadoop; this framework focuses on recommender engines, clustering, and classification. Thus, it is a perfect tool for complex, large-scale data mining projects involving huge amounts of data.

Fig. 11.9 Apache Mahout with Hadoop

LinkedIn, Yahoo are some of the leading web companies which use Apache mahout.

11.10 SAS Enterprise Mining

SAS Enterprise Mining is considered an analytics and data management platform. The goal is to simplify the data mining process to help analytics professionals to turn large amounts of data into insights. It consists of an interactive graphical user interface (GUI) by which users can generate data mining models and use them in order to solve critical business issues.

It also provides a rich set of algorithms by which you can prepare and explore data. Also, you can use it for building advanced predictive and descriptive models. Various companies can utilize SAS for fraud detection, resource planning, and increasing response rates on marketing campaigns, among other applications.

Fig. 11.10 SAS Enterprise Mining Overview

11.11 Rattle

Rattle is also an open-source platform. Also, it is a GUI for data mining that uses R statistical programming language provided by Togaware. It provides data mining functionality by exposing the power of the R through a GUI. Rattle is also utilized for teaching purposes or a facility to learn the R.

It also has an option called as Log Code tab, Which can replicate the R code for any activity undertaken in the GUI that can be copied and pasted. Statistical analysis or Model generation can also be performed through rattle. It also allows dataset partitioning into training, validation, and testing, which can be viewed and edited.

Fig. 11.11 Rattle - A data mining GUI for R

11.12 Python

Python is most popular when it comes to programming language. It is a free, open-source language. Python and R are often compared in terms of ease of use. Unlike R, Python’s learning curve tends to be so short that it becomes easy to use. Most users found that they can start building data sets and do extremely complex affinity analysis in minutes. The most common business-use case-data visualizations are straightforward if you are comfortable with basic programming concepts like variables, data types, functions, conditionals, and loops.

Fig. 11.12 Python overview

11.13 Summary

Fig. 11.13 US-Mining Automation Market size, 2014-2025 (USD Million)

In this chapter, we saw some of the tools extensively used for data mining and are very useful. It makes the work easier for data miners and saves their time. Also, many students can approach it as most of it is free and can be used for their learning. Let’s take the short description of some tools :

•MonkeyLearn - No-code text mining tools

•RapidMiner - Drag and drop workflows or data mining in Python

•Oracle Data Mining - Predictive data mining models

•IBM SPSS Modeler - A predictive analytics platform for data scientists

•Weka - Open-source software for data mining

•Knime - Pre-built components for data mining projects

•H2O - Open-source library offering data mining in Python

•Orange - Open-source data mining toolbox

•Apache Mahout - Ideal for complex and large-scale data mining

•SAS Enterprise Miner - Solve business problems with data mining

11.14 Exercise

1. Why are data mining tools used? Explain various mining tools with examples.

2. Explain:

(a) MonkeyLearn

(b) H20

(c) RapidMiner

(d) KNIME

(e) Orange

(f) Weka

3. For what python is popular?

References

Ten best data mining tools in 2021. (2020, December 22). MonkeyLearn Blog.

Adam Baba, Gouse Pasha, Shaik Althaf, Ahammed, S. Nasira Tabassum,” Introduction to Neural Networks Design Architecture,” International Journal of Scientific & Engineering Research Volume 4, Issue 2, February 2013, ISSN 2229-5518.

Arun K Pujari, Data Mining Techniques, University Press, 2013.

Christos N. Moridis and Anastasios A.Economides “Mood Recognition during Online Self- Assessment Tests” IEEE transactions on learning technologies, vol. 2, no. 1, January March 2009

Eric Hsueh-Chan Lu, Wang-Chien Lee, Member, IEEE, and Vincent S. Tseng, Member, IEEE,” A Framework for Personal Mobile Commerce Pattern Mining and Prediction,” IEEE transactions on knowledge and data engineering, vol. 24, no. 5, May 2012.

H. Kargupta and A. Joshi, “Data Mining to Go: Ubiquitous KDD for Mobile and Distributed Environments”, KDD-2001, San Francisco, August 2001. 6. J. Han, V.S. Lakshmanan, and RTNg, “Constraint-based, Multidimensional Data Mining,” COMPUTER (Special issue on Data Mining), 32(8): 45-50, 1999.

R. Agrawal, T. Imielinski, and A. Swami (1993). “Mining associations between sets of items in massive databases,” in Proceedings of the 1993 ACM-SIGMOD International Conference on Management of Data (pp. 207–216), New York: ACM Press.

M. J. A. Berry, and G. S. Linoff (1997). Data Mining Techniques. New York: Wiley

M. J. A. Berry, and G. S. Linoff (2000). Mastering Data Mining. New York: Wiley.

L. Breiman, J. Friedman, R. Olshen, and C. Stone (1984). Classification and Regression Trees. Boca Raton, FL: Chapman & Hall/CRC (orig. published by Wadsworth).

C. Chatfield (2003). The Analysis of Time Series: An Introduction, 6th ed. Chapman & Hall/CRC.

Figure Resources

Fig. 11.1. https://d33wubrfki0l68.cloudfront.net/892c58ef039ca68490882f10cf3c73aa41f065c6/6b92a/static/aa6e95739ff4a0dc1437ed292e36ca53/b4ad3/studio.png

Fig. 11.2. https://i.pinimg.com/originals/50/e8/f0/50e8f0cac77f92675c3eeec365b288ed.jpg

Fig. 11.3. https://docs.oracle.com/database/121/DMCON/img/GUID-F2A42A65-C615-4AC3-871F-DD8F15EC1714-default.png

Fig. 11.4. https://sv-europe.com/wp-content/uploads/SPSS-modeler-screengrab-1.png

Fig. 11.5. https://www.opensourceforu.com/wp-content/uploads/2016/11/Figure-2-Weka%E2%80%99s-application-interfaces.jpg

Fig. 11.6. https://image.slidesharecdn.com/knimewebinar20180616slideshare-180718134113/95/knime-software-overview-4-638.jpg?cb=1531921413

Fig. 11.7. https://miro.medium.com/max/1400/1*CvTFpjTxJNQIuEvmXVxeww.png

Fig. 11.8. https://upload.wikimedia.org/wikipedia/commons/thumb/3/33/Workflow-Orange3.png/220px-Workflow-Orange3.png

Fig. 11.9. https://static.packt-cdn.com/products/9781783554997/graphics/B03506_05_03.jpg

Fig. 11.10. http://documentation.sas.com/api/docsets/emag/14.3/content/images/iparchitecture.png?locale=en

Fig. 11.11. https://i0.wp.com/r4stats.com/wp-content/uploads/2018/06/Rattle-Main-Screen.png?resize=640%2C455

Fig. 11.12. http://www.sahosofttutorials.com/ContentPageImages/Python/sahosoft-tutorials-python-features.JPG

Fig. 11.13. https://www.grandviewresearch.com/static/img/research/us-mining-automation-market.png

Chapter 12. Data Mining Algorithms

Abstract

In this chapter, we will learn some of the algorithms used in data mining to execute the processes more simply and systematically. It guides data mining specialists to carry out various processes in data mining. These algorithms were built by researchers to ensure accuracy and convenience in data mining. These algorithms are the soul of data mining. Without these, large amounts of data would have been impossible to manage. Algorithms are guides to the best path or procedure for a data mining process.

12.1 C4.5

C4.5 is related to decision trees; it constructs a classifier in the form of a decision tree. For this purpose, it is provided with a set of data that is already classified. The classifier tool in data mining takes a bunch of data with the things we want to classify and predict the class for the new data. Now, this is converted into a decision tree, i.e., similar to a flowchart to classify new data. This makes it easier to understand relationships between data and how it can be useful.

C4.5 is different from other decision tree systems in the following ways:

1. It uses information gain while generating the decision tree.

2. C4.5 uses a single pass purring process to mitigate over-fitting.

3. It can work with both continuous and discrete data.

C4.5 provides the ease of interpretation and explanation for a decision tree. It is quite fast, quite popular, and the output is human-readable. For example, orange, an open-source data visualization and analysis tool for data mining implements C4.5 in their decision tree classifier.

Example: Take a data set of a bunch of patients, which contains various attributes such as age, pulse, blood pressure, VO2max, family history, etc. Now from the given attributes, we want to predict whether a patient will get cancer. So two classes are possible, i.e., will get cancer or won’t get cancer. Here we use C4.5 to class for each patient. Now it will use a set of patient attributes and the patient’s corresponding class; it will construct a decision tree that can predict the class for new patients based on their attributes. Looking at the flowchart, there could be one particular path among:

•Patient has a history of cancer.

•Patient is expressing a gene highly correlated with cancer patients.

•Patient has tumors.

•Patient’s tumor size is greater than 5cm.

At each point in the flowchart is a question about the value of some attribute, and depending on those values, they get classified.

Fig. 12.1 Example of C4.5

12.2 k-means

k-means create k groups from a set of objects to make the members of a group are similar. It is a cluster analysis technique for processing and exploring a data set. Cluster analysis is a family of algorithms designed to form groups such that the group members are more similar versus non-group members. Clusters and groups are synonymous in the world of cluster analysis. K-means consist of lots of variations to optimize certain types of data. Suppose we take the example of patients. It performs tasks like:

1. K-means will pick points in multi-dimensional space to represent each of the k clusters. These are called centroids.

2. Every patient will be closest to 1 of these k centroids. However, they hopefully won’t all be closest to the same one to form a cluster around their nearest centroid.

3. Now each patient is a member of a cluster.

4. Then it finds the center for each of the k clusters based on its cluster members.

5. This center becomes the new centroid for the cluster.

6. Since the centroid is in a different place now, patients might now be closer to other centroids. In other words, they may change cluster membership.

7. Steps 2-6 are repeated until the centroids no longer change, and the cluster memberships stabilize. This is called convergence.

k-means is very simple, which makes it faster and more efficient than other algorithms. It can be used to pre-cluster a massive dataset followed by more expensive cluster analysis on the sub-clusters. k-means can also be used to rapidly play with k and explore overlooked patterns or relationships in the dataset. However, k means is sensitive when it comes to outliers ad the initial choice of centroids. It is designed for continuous data, and for discrete data, one needs to perform some tricks.

k-means has been implemented in tools like Apache mahout, Julia, R, SciPy, Weka, MATLAB, SAS.

Fig. 12.2 Example of K-means clustering algorithm

Fig. 12.3 K-means clustering algorithm

12.3 Support vector machines

Support vector machines, popularly known as SVM, uses a hyperplane to classify data into 2 classes. It performs tasks similar to C4.5 but doesn’t use decision trees. We need to take an example in order to understand it. Suppose there are a bunch of red and blue balls on a table. If the balls aren’t too mixed together, you could take a stick and, without moving the balls, separate them with the stick. You will see that when a new ball is added to the table, by knowing which side of the stick the ball is on, you can predict its color. Here the balls represent data points, and the red and blue colors represent 2 classes. The stick represents the simplest hyperplane which is a line. SVM acts on a hyperplane; it figures out the function of a hyperplane.

Fig. 12.4 SVM algorithm

In the second condition, where the balls are mixed, a stick wouldn’t give us the best results. Now what we will do is quickly lift the table throwing the balls in the air. While the balls are in the air and thrown up in just the right way, you use a large sheet of paper to divide the balls in the air. Lifting the table is the equivalent of mapping your data into higher dimensions. In this case, we go from the 2-dimensional table surface to the 3 dimensional balls in the air.

The SVM performs this task by using kernel as it gives a way to operate in higher dimensions. The large sheet of paper is still called a hyperplane, but it is now a function for a plane rather than a line. Note from Yuval that once we are in 3 dimensions, the hyperplane must be a plane rather than a line.

In real life, as for a patient data set, each patient could be described by various measurements like pulse, cholesterol level, blood pressure, etc. Each of these measurements is a dimension. So SVM does its thing, maps them into a higher dimension, and then finds the hyperplane to separate the classes.

SVM and C4.5 are the 2 best classifiers which one can try: Scikit-learn, MATLAB, and libsvm implement SVM in them.

12.4 Apriori

The Apriori algorithm is related to an association; it learns association rules and applies to a database containing many transactions. For example, you can take supermart transactions. It consists of a giant spreadsheet where each row is a customer transaction, and every column represents a different grocery item. Here you can apply the apriori algorithm to learn the grocery items purchased together, i.e., association rules. With the help of this, You can find those items that tend to be purchased together more frequently than other items — the ultimate goal being to get shoppers to buy more. Together, these items are called itemsets.

Fig. 12.5 Apriori Algorithm

For example, You observe that chips + dip and chips + soda seem to frequently occur together. These are called 2-itemsets. It will be much harder to see the relationships with a large enough dataset, especially when you are dealing with 3-itemsets or more. Here Apriori helps you.

Before implementing Apriori, you need to specify three things:

1. The first is the size of your itemset.

2. The second is your support or the number of transactions containing the itemset divided by the total number of transactions. An itemset that meets the support is called a frequent itemset.

3. The third is your confidence or the conditional probability of some item given you have certain other items in your itemset. A good example is given chips in your itemset; there is a 67% confidence of having soda also in the itemset.

Apriori works in 3 steps:

1. Join. Scan the whole database for how frequent 1-itemsets are.

2. Prune. Those itemsets that satisfy the support and confidence move onto the next round for 2-itemsets.

3. Repeat. This is repeated for each item set level until we reach our previously defined size.

Fig. 12.6 Example of Apriori

Apriori is well understood, easy to implement, and consists of various derivatives. But it utilizes more memory, space, and time while generating itemsets. AR tool, Weka, Orange are some of the tools which implement Apriori though there are plenty of implementations.

12.5 EM

EM in data mining stands for Expectation-Maximization that is used as a clustering algorithm. In statistics, the EM algorithm iterates and optimizes the likelihood of seeing observed data while estimating the parameters of a statistical model with unobserved variables. By optimizing the likelihood, EM generates an awesome model that assigns class labels to data points. Initially, it guesses model parameters. Then it follows three steps:

1. E-step: Based on the model parameters, it calculates the probabilities for assignments of each data point to a cluster.

2. M-step: Update the model parameters based on the cluster assignments from the E-step.

3. Repeat until the model parameters and cluster assignments stabilize (a.k.a. convergence).

Fig. 12.7 EM algorithm

EM is simple and straightforward to implement. It can optimize model parameters also can iteratively make guesses about missing data. It is best for clustering and generating a model with parameters. Knowing the clusters and model parameters, it is possible to reason about what the clusters have in common and which cluster new data belongs to.

Fig. 12.8 Example of EM

There are two drawbacks of EM :

EM is fast in the early iterations but slow in the later iterations.

Second, EM does not always find the optimal parameters and gets stuck in local optima rather than global optima.

The EM algorithm is available in Weka. R has an implementation in the mclust package. scikit-learn also has an implementation in its gmm module.

12.6 Page Rank

Page Rank is a link analysis algorithm designed to determine the relative importance of some object linked within a network of objects. The most popular real-time application is Google’s search engine. Web pages on the World Wide Web link to each other. If rayli.net links to a web page on CNN, a vote is added for the CNN page, indicating rayli.net finds the CNN web page relevant. rayli.net’s votes are in turn weighted by rayli.net’s importance and relevance.

In other words, any web page that’s voted for rayli.net increases rayli.net’s relevance. This concept of voting and relevance is PageRank. rayli.net’s vote for CNN increases CNN’s PageRank, and the strength of rayli.net’s PageRank influences how much its vote affects CNN’s PageRank.

Page Rank is robust as it is difficult to get a relevant incoming link. The PageRank trademark is owned by Google. However, the PageRank algorithm is actually patented by Stanford University.

Fig. 12.9 Example of Page Rank algorithm

12.7 AdaBoost

AdaBoost is again related to classification. It is a boosting algorithm that constructs a classifier. Boosting is an ensemble learning algorithm that takes multiple learning algorithms (e.g., decision trees) and combines them. The goal is to take an ensemble or group of weak learners and combine them to create a single strong learner.

AdaBoost is simple and relatively straightforward to program. It is a super elegant way to auto-tune a classifier since each successive AdaBoost round refines the weights for each of the best learners. All you need to specify is the number of rounds. Finally, it’s flexible and versatile. AdaBoost can incorporate any learning algorithm, and it can work with a large variety of data.

Fig. 12.10 AdaBoost algorithm

This also has tons of implementation and variants such as scikit-learn, ICSIBoost, Generalised Boosted Regression Models.

Fig. 12.11 Example of AdaBoost

12.8 kNN

kNN means k-Nearest Neighbors, also a classification algorithm. It differs from the previous classifiers as it is a lazy learner. The lazy learner is those that don’t do much during the training process other than storing the data. If you input new unlabeled data, then only it will look for classification.

kNN does not build a classification model like c4.5 or SVM; instead, it just stores the labeled training data.

Fig. 12.12 KNN filtering algorithm

kNN works in two steps:

1. It looks at the k closest labeled training data points — in other words, the k-nearest neighbors.

2. Second, using the neighbor’s classes, kNN gets a better idea of how the new data should be classified.

To find the closest data in continuous data, kNN uses distance metrics. The choice of distance metric largely depends on the data. Some even suggest learning a distance metric based on the training data. However, when it comes to discrete data, it converts it into continuous data.

kNN provides ease of understanding and implementation. It can be very accurate when dependent on distance metrics.

There are 5 important things one need to take care of while using kNN:

1. kNN can get very computationally expensive when trying to determine the nearest neighbors on a large dataset.

2. Noisy data can throw off kNN classifications.

3. Features with a larger range of values can dominate the distance metric relative to features with a smaller range, so feature scaling is important.

4. Since data processing is deferred, kNN generally requires greater storage requirements than eager classifiers.

5. Selecting a good distance metric is crucial to kNN’s accuracy.

MATLAB, scikit-learn, R are some who use kNN.

12.9 Naive Bayes

Naive Bayes is a family of classification algorithms. Every feature of the data being classified is independent of all other features given the class. The assumption that all features of a dataset are independent is called naive; it’s generally not the case that all features are independent. Where Bayes comes from Thomas Bayes, an English statistician who generated Baye’s theorem. In short, the theorem allows us to predict that class given a set of features using probability.

For Example, you have a training dataset of 1,000 fruits. The fruit can be a Banana, Orange, or Other (these are the classes). The fruit can be Long, Sweet, or Yellow (these are the features). And you see conditions like:

1. Out of 500 bananas, 400 are long, 350 are sweet, and 450 are yellow.

2. Out of 300 oranges, none are long, 150 are sweet, and 300 are yellow.

3. Out of the remaining 200 fruit, 100 are long, 150 are sweet, and 50 are yellow.

If we are given the length, sweetness, and color of fruit (without knowing its class), we can now calculate the probability of it being a banana, orange or other fruit.

Suppose we are told the unknown fruit is long, sweet, and yellow. Further, we can calculate probability using Baye’s theorem.

Fig. 12.13 Naive Bayes Example

Fig. 12.14 Working of Naive Bayes Algorithm

Implementation of Naive Bayes can be found in Orange, scikit-learn, Weka, R.

12.10 CART

CART stands for Classification And Regression Trees, which generates classification and regression trees using Gini impurity. For pruning, it uses cost-complexity. Starting at the bottom of the tree, CART evaluates the misclassification cost with the node vs. without the node. If the cost doesn’t meet a threshold, it is pruned away.

CART is a decision tree learning technique. Also, ease of interpretation and explanation applies to CART as well. It is very similar to C4.5. scikit-learn implements CART in their decision tree classifier. R’s tree package has an implementation of CART. Weka and MATLAB also have implementations.

Fig 12.1 CART with Decision tree

12.13 Summary

In this chapter, we learned some data mining algorithms used in various data mining processes like association, classification, clustering, etc. These algorithms make it easier for data mining specialists to perform various processes for data analysis. Algorithms include C4.5, k-means, Support vector machines, Apriori, EM, PageRank, AdaBoost, kNN, Naive Bayes, CART. This chapter explains each algorithm with its use. Further, let’s see some of the applications of data mining where these algorithms are implemented.

12.14 Exercise

1. List down the algorithms used in data mining.

2. What do you mean by

(a) C4.5

(b) CART

(c) Apriori

(d) AdaBoost

(e) Naive Bayes

3. Give an example of k-means.

4. Where can we find the implementation of Naive Bayes?

5. Give an example of Naive Bayes.

6. What is the full form of EM?

References

Top 10 data mining algorithms, explained. (n.d.). KDnuggets.

A step-by-step C4.5 decision tree example. (2021, March 6). Sefik Ilkin Serengil.

10 best data mining tools in 2021. (2020, December 22). MonkeyLearn Blog.

Adam Baba, Gouse Pasha, Shaik Althaf Ahammed, S. Nasira Tabassum, “Introduction to Neural Networks Design Architecture”, International Journal of Scientific & Engineering Research Volume 4, Issue 2, February 2013, ISSN 2229-5518.

Arun K Pujari, Data Mining Techniques, University Press, 2013.

Christos N. Moridis and Anastasios A.Economides “Mood Recognition during Online Self- Assessment Tests” IEEE transactions on learning technologies, vol. 2, no. 1, January March 2009

Eric Hsueh-Chan Lu, Wang-Chien Lee, Member, IEEE, and Vincent S. Tseng, Member, IEEE,” A Framework for Personal Mobile Commerce Pattern Mining and Prediction,” IEEE transactions on knowledge and data engineering, vol. 24, no. 5, May 2012.

H. Kargupta and A. Joshi, “Data Mining to Go: Ubiquitous KDD for Mobile and Distributed Environments”, KDD-2001, San Francisco, August 2001. 6. J. Han, V.S. Lakshmanan, and R T Ng, “Constraint-based, Multidimensional Data Mining,” COMPUTER (Special issue on Data Mining), 32(8): 45-50, 1999.

R. Agrawal, T. Imielinski, and A. Swami (1993). “Mining associations between sets of items in massive databases,” in Proceedings of the 1993 ACM-SIGMOD International Conference on Management of Data (pp. 207–216), New York: ACM Press.

M. J. A. Berry, and G. S. Linoff (1997). Data Mining Techniques. New York: Wiley

M. J. A. Berry, and G. S. Linoff (2000). Mastering Data Mining. New York: Wiley.

L. Breiman, J. Friedman, R. Olshen, and C. Stone (1984). Classification and Regression Trees. Boca Raton, FL: Chapman & Hall/CRC (orig. published by Wadsworth).

C. Chatfield (2003). The Analysis of Time Series: An Introduction, 6th ed. Chapman & Hall/CRC.

M.S. Mythili and A.R. Mohamed Shanavas, “An analysis of students” performance using classification algorithms,” IOSR, Journal of Computing Engineering, volume 16, Issue 1, January 2014.

S.Lakshmi Prabha, A.R.Mohamed Shanavas, “Educational data mining applications,” Operations Research and Applications: An International Journal (ORAJ), volume 1, No. 1, August 2014.

Davinder Kaur, Rajeev Bedi and S.K Gupta, “Review of decision tree data mining algorithms: ID3 and C4.5”, pp. 5-8, July 2015

S.Ayesha, T.Musafa, A.Sattar, and M.khan, “Data mining model for the higher education system,” European Journal of Scientific Research, vol. 43, no 1, 2010. pp. 24-29.

Open for Innovation KNIME Software, URL: https://www.knime.com/knime-software.

Figure Resources

Fig. 12.1. https://miro.medium.com/max/766/1*BVlXBsUgnzRmkE4EgwBb8Q.gif

Fig. 12.2. https://i.stack.imgur.com/ibYKU.png

Fig. 12.3. https://www.researchgate.net/profile/Zhang-Qiu-Yu/publication/286480276/figure/fig1/AS:338438109843456@1457701341733/K-means-Clustering-Algorithm-K-means-algorithm-is-the-clustering-algorithm-based-on.png

Fig. 12.4. https://static.javatpoint.com/tutorial/machine-learning/images/support-vector-machine-algorithm.png

Fig. 12.5. https://www.softwaretestinghelp.com/wp-content/qa/uploads/2019/09/AprioriSteps.png

Fig. 12.6. https://lh3.googleusercontent.com/proxy/VPJV8tysOlv2W67hwmfvcnBgCMtZxCZpT_LxVbv8ilNUfg8PsaqEK8K5qK719LYLhl1gPOVYkCbQxsit8h-9TGKU1m1hBQ

Fig. 12.7. https://i.stack.imgur.com/TVluI.png

Fig. 12.8. https://i.stack.imgur.com/mj0nb.gif

Fig. 12.9. https://image.slidesharecdn.com/pagerankalgortihm-170520030901/95/page-rank-algortihm-11-638.jpg?cb=1495249908

Fig. 12.10. https://www.researchgate.net/profile/Zhuo-Wang-36/publication/288699540/figure/fig9/AS:668373486686246@1536364065786/Illustration-of-AdaBoost-algorithm-for-creating-a-strong-classifier-based-on-multiple.png

Fig. 12.11. https://slideplayer.com/slide/9092209/27/images/20/Algorithm+Adaboost+-+Example.jpg

Fig. 12.12. https://www.researchgate.net/publication/326472969/figure/fig5/AS:649910470844421@1531962139798/Flow-diagram-of-the-serial-implementation-of-the-KNN-filtering-algorithm.png

Fig. 12.13. https://www.elegantjbi.com/blog/wp-content/uploads/2018/10/naive-bayes-classification-example.jpg

Fig. 12.14. https://www.machinelearningplus.com/wp-content/uploads/2018/11/06_Naive_bayes_example_answer_new.png

Fig. 12.15. https://miro.medium.com/max/1400/1*WerHJ14JQAd3j8ASaVjAhw.jpeg

Chapter 13. Applications and Drawbacks of Data Mining

Abstract

In this chapter, we will see real-time applications of data mining along with the drawbacks. Let us explore data mining in real life how companies and which companies use it for what purpose. This chapter sheds light on the real-world usage of data mining. In today’s life, data mining has affected the industry a lot and extensively uses data mining to gain improvements and taking marketing techniques to the next level. Although there are drawbacks that challenge the reliability of data mining, let’s see what those are further in the chapter.

Fig 13.1 Applications of data mining

13.1 Future Healthcare

Data mining plays an important role in improving health systems all over the globe. Data and analytics are utilized to uncover best practices that can improve care and reduce costs. Data mining techniques like multi-dimensional databases, machine learning, soft computing, data visualization, and statistics are utilized for research. In hospitals, mining techniques are used to predict the volumes of patients in every category. Data mining processes are developed to ensure that the patients receive appropriate care at the right place and time. Also, when it comes to the security of health care ensures data mining can also be used to detect fraud and abuse.

Fig 13.2 Data mining in healthcare

13.2 Market Based Analysis

Market-Based Analysis is considered a technique based upon a theory that if you buy a particular group of items, you are more likely to buy another group of items. By implementing this technique through data mining, the retailers understand the purchase behavior of their customers and utilize it for their improvement and profit. Also, this information can help the retailers to gain the current demand of products and manufacture those accordingly. Using a technique like differential analysis, comparing results different stores between customers in different demographic groups can be made.

Fig. 13.3 Market-based analysis and recommendation engine

13.3 Manufacturing Engineering

Various data mining tools can be used to gain more knowledge on complex manufacturing processes. Therefore, data mining plays a very important role as gaining knowledge about manufacturing is very useful. Also, it can be used for system-level designing to extract relationships between product architecture, product portfolio, customer needs, etc. Furthermore, prediction of the product’s development span time, cost, and dependencies, among other tasks, can also be done using the data mining technique. Hence data mining is very useful for manufacturing engineering.

Fig. 13.4 Data mining in manufacturing

13.4 Education

Data mining in education has recently come up, which is also known as educational data mining. This mainly focuses on developing methods that can help in discovering knowledge of data originating from educational environments. In the education industry, data mining can predict students’ future learning behavior, studying the effects of educational support, and advancing scientific knowledge about learning. Also, it can be used by institutions to make accurate decisions and predict the students’ results, which has got a boost since Covid19. Moreover, it can also help institutions improve themselves and find new teaching methods by analyzing students’ behavior and capabilities.

Fig. 13.5 Educational Data mining

13.5 CRM

CRM stands for Customer Relationship Management which mainly focuses on acquiring as much data related to customers. By which companies can build a better relationship among customers. They use customer-focused strategies. The collected data is processed using data mining techniques which helps to understand the customers in a better manner and take accurate decisions that will bring profit to the company and customer satisfaction. Hence data mining techniques help to eliminate confusion among customers and companies.

Fig. 13.6 Data mining in CRM

13.6 Fraud Detection

Nowadays, with increasing technology, fraud also has increased, and billions of dollars have been lost. It is very important to detect such frauds to stop them in the future. Unfortunately, the traditional methods are very time-consuming and complex, which slows down the whole investigation process failing.

That’s why data mining has been introduced, which helps provide meaningful patterns and turn data into information. In this field, every piece of data is useful, and one can’t say which data will provide valuable information. Data mining techniques help in understanding this data so that investigators get a link and can gain progress. A system should protect all the user’s information then only it’s a perfect system. A supervised method should include a collection of sample records. These records are then classified as fraudulent or non-fraudulent with the help of a model built using this data, and the algorithm is made to identify whether the record is fraudulent or not.

Fig. 13.7 DM based Financial fraud detection

13.7 Intrusion Detection

An action that comprises the integrity and confidentiality of a resource can be called an intrusion. User authentication, avoiding programming errors, and information protection are defensive measures to avoid an intrusion. Here, data mining can improve intrusion detection by adding a level of focus to anomaly detection. It helps an analyst distinguish activity from common everyday network activity. Data mining also helps extract more relevant data to the problem.

Fig. 13.8 Data mining approaches for intrusion detection

13.8 Lie Detection

Catching a criminal is easy but again, bringing out the truth is a very difficult task. Law enforcement makes use of data mining techniques to investigate crimes, monitor the communication of suspected terrorists. It also includes text mining. Data mining here is done to find meaningful patterns in data that are usually unstructured. Model for lie detection is created by comparing data samples collected from previous investigations. Hence model processes can be created according to the necessity and help to uncover the truth.

Fig. 13.9 Lie detection based on F-score and extreme learning machine

13.9 Customer Segmentation

Customer segmentation can be done by traditional market research, but it cannot give accurate results as it doesn’t go in deep. Therefore data mining is used here to go deeper and do a detailed analysis to increase market effectiveness. It helps to align the customers into a segment and can easily fulfill the needs of customers. Customer satisfaction is everything when it comes to market. By using data mining, companies can offer the best products and offers suitable for the customer resulting in increased sales. It helps to segment customers based on vulnerability and business. Hence data mining is extensively being used for customer segmentation.

Fig. 13.10 Clustering algorithm for customer segmentation

13.10 Financial Banking

In current times, most transactions have been made online, where a huge amount of data is generated with every new transaction. Data mining helps solve business problems in banking and finance by finding patterns, causalities, and correlations in business information and market prices that cannot be given to managers as the volume of data are too large or is generated too quickly to screen by experts. Managers can use this kind of data for better segmenting, targeting, acquiring, retaining, and maintaining a profitable customer.

Fig. 13.11 Data mining framework for banking

13.11 Corporate Surveillance

Monitoring a person’s or group’s behavior by a corporation is termed corporate surveillance. Data collected can be very useful for marketing purposes or sold to other corporations though government agencies regularly share it. Also can be used by the business to customize products according to the demands of customers. Direct marketing can also be done using this; for example, you can take Google or Yahoo ads targeted to the user by analyzing their search history, emails, and actions.

Fig. 13.12 Corporate surveillance in everyday life

13.12 Research Analysis

Research is an everlasting field; it doesn’t end as it gives us valuable information and helps humans explore new things. In the past times, we have witnessed remarkable results due to research. As you know, data mining is very helpful for data cleaning, data pre-processing, and the integration of databases. This can be a great help for researchers to find any similar data from their database that can bring change in the research. Also, the identification of any co-occurring sequences and the correlation between any activities can be known. Moreover, data techniques like data visualization and visual data mining can provide a clear view of data.

13.13 Criminal Investigation

Crime has kept on increasing day by day. It is necessary to control them and give justice to every citizen. Criminal Investigation is one of the parts of this process where the main focus is on understanding crime characteristics. It involves exploring and detecting crimes and their relationships with criminals. During this process, a high volume of datasets and complex relationships between them are generated. Data mining plays its role in sorting things out and making it easier for investigators. For example, Text-based crime reports can be converted into word processing files. This information can be used to perform the crime matching process.

Fig. 13.13 Criminal investigations based on data mining

13.14 BioInformatics

Bioinformatics generates an extensive amount of data that needs to be processed. Therefore, data mining plays a very important role in bioinformatics. Biological data mining helps extract useful knowledge from massive datasets gathered in biology and other related life sciences areas such as medicine and neuroscience. Data mining helps bioinformatics in the field of gene finding, protein function inference, disease diagnosis, disease prognosis, disease treatment optimization, protein and gene interaction network reconstruction, data cleansing, and protein subcellular location prediction.

Fig. 13.14 Data mining and Gene expression analysis in bioinformatics

13.15 Companies extensively using Data Mining

•Amazon.

•American Express.

•BDO.

•Capital One.

•General Electric (GE)

•Miniclip.

•Netflix.

•Next Big Sound.

•Starbucks

•T mobile

•Delta

•Duetto

•Express Scripts

•Mc Donald’s

•PredPol

•Thomson Reuters

•Walmart

•Kreditech

•Sisense

•Palantir Technologies

These companies use data mining to increase their sales, performance, and development. That, in return, brings them customer satisfaction and helps to gain reputation in the market by constantly improving and making advancements. A big part of these companies is dependent on data mining. All of them use data mining techniques that are useful for them and their product to make accurate decisions.

13.16 Drawbacks of Data mining

Though data mining is very useful, everything has its drawbacks, and this too. So let us see what those are:

13.16.1 Cost

To conduct data mining, one needs a strong amount of money, as data mining involves a lot of technology for the data collection process. Also, these data need to be stored as well, so it will require high maintenance. This can increase the cost. Moreover, for the tool selection and other operations, a specialist must be hired, contributing to the overall expenses.

13.16.2 Security

Identity theft is a big issue when it comes to data mining. Adequate security needs to be provided to prevent vulnerabilities in the security. Various information of the customers is collected in data mining that needs to be kept secure. It consists of very personal information, and due to its vastness, hackers can find it easy to access and can misuse the information.

13.16.3 Privacy

Since minor to minor details of the customers are collected here, it leads to exploitation of privacy and can sometimes be a big issue. Also, some information could be leaked unknowingly, or else they would be sold to others, intentionally violating user privacy. If this data goes to the wrong hands, it can be very dangerous.

13.16.4 Accuracy

Although data mining has given way for easy data collection and various processes that can help us, it still possesses some limitations when it comes to accuracy. Sometimes the information which has been gathered can be inaccurate and can cause problems in decision making.

13.16.5 Technical Skills

There are n number of tools available with their advantages. Each one has its algorithm and design. To work with them, one needs to be specialized and needs to study each of them. Without proper technical knowledge, tool selection will be a difficult task. Hence it is important to have a skilled technician for the tool selection and for working on that.

13.16.6 Information Misuse

The weak security present in data mining can also lead to the misuse of information. People can use this information from data mining for their gain. Moreover, a group of people can be targeted to be harmed by using this information. Hence, Companies using data mining must keep tight security in data mining.

13.16.7 Additional Information

During the data collection step, a vast amount of data is being collected. However, everyone can’t deal with every data, and when some additional data gets added, it wastes time and money required for its maintenance. Hence it is necessary to maintain a limit of data collected through data mining techniques.

13.17 Summary

In this chapter, we explored some of the real-time data mining applications, including Future healthcare, Market Basket Analysis, Manufacturing engineering, CRM, Fraud Detection, Intrusion Detection, Customer Segmentation, Financial Banking, etc. These use data mining extensively to gain better results and also bring improvement in each field. This is how data mining plays a very important part in our lives and opens new ways to think and explore the world. However, there are issues related to data mining, it is very useful, but it still has some drawbacks that can become a major aspect and should be taken into consideration.

13.18 Exercise

1. List down the applications of data mining.

2. How is data mining used for educational purposes?

3. What does CRM stand for?

4. What do you mean by corporate surveillance?

5. What role does data mining play in the field of Healthcare?

6. Explain data mining in research analysis.

7. What is an intrusion?

8. Which system is called a perfect system in terms of fraud detection?

9. List 5 companies that extensively use data mining.

10. How does data mining help those companies?

11. List down the drawbacks of data mining.

References

Fourteen useful applications of data mining. (2019, September 11). Big Data Made Simple.

Twelve most useful data mining applications of 2021. (2021, April 3). upGrad blog.

R. Agrawal and R. Srikant, Fast Algorithms for Mining Association Rules (1994) Proc. 20th Int. Conf. Very Large Data Bases, VLDB-94.

Bayardo, R. and R. Srikant, Technological Solutions for Protecting Privacy, IEEE Computer, Sep 2003

Chernoff, H. (1973). Using faces to represent points in k-dimensional space graphically. Journal of American Statistical Association, 68, 361-368.

P. Domingos, MetaCost: a general method for making classifiers cost-sensitive, KDD-99, Proceedings of the 5th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM Press, 1999.

Fayyad, Piatetsky-Shapiro, Smyth, and Uthurusamy, Advances in Knowledge Discovery and Data Mining, (Chapter 1), AAAI/MIT Press 1996.

Golub, T. et al., Science (v. 286), Oct 1999.

Inselberg, A., and Dimsdale, B. 1987. Parallel Coordinates for Visualizing Multi-Dimensional Geometry. In Proceedings of the Computer Graphics Intl. Conf.

C. Matheus, G. Piatetsky-Shapiro, and D. McNeill Selecting and Reporting What is Interesting: The KEFIR Application to Healthcare Data, in Advances in Knowledge Discovery and Data Mining, AAAI/MIT Press, 1996

R. Simon, Supervised Analysis When the Number of Candidate Features (p) Greatly Exceeds The Number of Cases (n), SIGKDD Explorations Special Issue on Microarray Data Mining, Dec 2003,

E. R. Tufte, The Visual Display of Quantitative Information”, 2nd Edition, Graphics Press, 2001.

Quinlan, J. Ross, C4.5: Programs for Machine Learning, Morgan Kaufmann, 1993.

Witten, Ian, and Eibe Frank, Data Mining, Practical Machine Learning Tools and Techniques with Java Implementations, Morgan Kaufmann, 1999.

Weng-Keen Wong, Andrew Moore, Gregory Cooper, Michael Wagner, Rule-based Anomaly Pattern Detection for Detecting Disease Outbreaks, Proc. of the 18th National Conference on Artificial Intelligence, 2002.

Figure Resources

Fig. 13.1. https://media.geeksforgeeks.org/wp-content/uploads/20210603232955/ApplicationsofDataMining.png

Fig. 13.2. https://ars.els-cdn.com/content/image/1-s2.0-S1532046416300296-gr1.jpg

Fig. 13.3. http://dataminingreporting.weebly.com/uploads/4/0/9/7/4097240/ 7452694_orig.png

Fig. 13.4. https://ars.els-cdn.com/content/image/1-s2.0-S095741742030823X-gr4.jpg

Fig. 13.5. https://www.researchgate.net/profile/Mykola-Pechenizkiy/publication/254008370/figure/fig1/AS:669059674808330@1536527665896/Educational-data-mining-in-a-nutshell.png

Fig. 13.6. https://www.researchgate.net/publication/297898260/figure/fig3/AS:341617891397634@1458459460173/ntegrated-model-of-Data-Mining-for-CRM-DM-processes-CRM-stages-Techniques-Applications.png

Fig. 13.7. https://d3i71xaburhd42.cloudfront.net/780382bc1f2d9d28c58ba5a29f0dfcd9d3e176aa/2-Figure1-1.png

Fig. 13.8. https://wenke.gtisc.gatech.edu/papers/usenix/architecture.gif

Fig. 13.9. https://journals.plos.org/plosone/article/figure/image?size=inline&id=10.1371/journal.pone.0064704.g003

Fig. 13.10. https://miro.medium.com/max/1400/1*xCKGM7tUzoR53oFv4yTPWA.png

Fig. 13.11. https://www.researchgate.net/profile/Hong-Ning-Dai/publication/333883837/figure/fig1/AS:772148578770946@1561105975516/Banking-big-data-mining-framework.png

Fig. 13.12. https://crackedlabs.org/en/corporate-surveillance/img/infographic_datacollection1300.jpg

Fig. 13.13. https://d3i71xaburhd42.cloudfront.net/ed8f508f063beb1f8e2ca0899be3b15ea221e984/3-Figure5.1-1.png

Fig. 13.14. https://d3i71xaburhd42.cloudfront.net/cbd7cb810179f1c5d036a56c369775076b3a267b/3-Figure1-1.png

Part 3

Part 3

Machine Learning

Chapter 14. Introduction to Machine Learning

Abstract

Till now, we have seen all about big data and data mining. Now let’s learn a newly emerging technique in this field, i.e., Machine Learning. This technique has brought a new evolution in the field of data mining. Also, it is helping humankind to explore a new world of machines and technology. In the future, it can invent such things, which are now fiction. By this, you can predict the power of Machine learning, and one who desires to go into data mining or making changes for the future should study it. In this chapter, let’s learn what exactly machine learning is and explore the whole new world of Machine learning and its awful advantages.

Fig. 14.1 Machine learning overview

14.1 What is ML?

Machine learning is a branch of AI and Computer science that uses data algorithms to imitate the Human’s way of learning and improving its accuracy. It is designed to study how a human learns things, and this can be implemented in a machine or used to study humans and improve them. This process powers many of the services we use today.

14.2 History of ML

Machine learning is a branch, or you can say a part of Artificial Intelligence. Here computer algorithms are used to learn from data and information independently. Machine learning gives computers the ability to change and improve their algorithms by themselves. They don’t need to be explicitly programmed.

The first action which led to the origin of Machine learning was done in 1950 by Alan Turing to test whether computers have real intelligence and was given the task to fool a human into believing that it is also a human. Since then, many scientists have started developing machines or algorithms that can study or work with humans. Further, in 1990 machine learning shifts from a knowledge-driven approach to a data-driven approach. Scientists began to create computer programs to analyze large amounts of data, draw conclusions, and learn from that data.

Fig. 14.2 History and evolution of machine learning

Later, in 1997, IBM developed Blue beat the human and became the world chess champion, which brought a huge revolution. And since then, IBM has to continue to set its record in machine learning. After that, many companies like google, amazon, and Facebook started to get interest in ML, developed their platforms, and recognized the advantages ML can provide them.

The power of the machine and its way to becoming a human or act like them have always fascinated us. Still, many scientists believe that machines or computers can never think like an exact human brain. Comparing the computational analysis and algorithms to the machinations of the human mind is like comparing apples and oranges. Although computers have made a lot of improvements and have come closer to humans, they still lack somewhere that makes them different from humans.

14.3 How is it different from AI?

•Artificial intelligence is a technology that enables a machine to simulate human behavior. Machine learning is a subset of AI which allows a machine to automatically learn from past data without programming explicitly.

•The goal of AI is to make a smart computer system like humans to solve complex problems. Whereas ML allows machines to learn from data so that they can give accurate output.

•In AI, we make intelligent systems to perform any task like a human. In ML, we teach machines with data to perform a particular task and give an accurate result.

•Machine learning and deep learning are the two main subsets of AI, and Deep Learning is the main subset of machine learning.

•AI has a very wide range of scope, whereas Machine learning has a limited scope.

•AI is working to create an intelligent system that can perform various complex tasks. Machine learning is working to create machines that can perform only those specific tasks they are trained in.

•AI system is concerned about maximizing the chances of success. On the other hand, machine learning is mainly concerned with accuracy and patterns.

•The main applications of AI are Siri, customer support using catboats, Expert System, Online game playing, an intelligent humanoid robot, etc. The main applications of machine learning are the Online recommender system, Google search algorithms, Facebook auto friend tagging suggestions, etc.

•AI can be divided into three types based on capabilities: Weak AI, General AI, and Strong AI. Likewise, machine learning can also be divided into mainly three types: supervised learning, Unsupervised Learning, and Reinforcement Learning.

•It includes learning, reasoning, and self-correction. It includes learning and self-correction when introduced with new data.

•AI completely deals with Structured, semi-structured, and unstructured data. On the other hand, machine learning deals with Structured and semi-structured data.

14.4 Examples of Machine Learning

There are n number of examples of machine learning that we use in our day to day life but are unaware that they are driven by machine learning lets see what are those:

•Virtual Personal Assistants

•Predictions while Commuting.

•Videos Surveillance.

•Social Media Services.

•Email Spam and Malware Filtering.

•Online Customer Support

•Search Engine Result Refining

•Product Recommendations

•Online Fraud Detection

•Intelligent Gaming

•Self-Driving Cars and Automated Transportation

•AI for dangerous jobs

•Environment Protection

•Improved ElderCare

•Home Security and Smart Homes

Let’s see how ML plays an important role in each of these.

14.4.1 Virtual Personal Assistants

Alexa, Google Now, Siri are popular examples of Virtual Personal Assistants. Their work is to assist in finding information when asked. One needs to activate it and ask questions like “How is the weather today?” or “ Give me today’s news,” etc. To answer your questions, it looks for information, recalls your related queries, or sends a command to other resources (like phone apps) to collect information.

Fig. 14.3 History of voice assistance

Machine learning plays its part by helping assistants collect and refine the information based on your previous involvement with them. Later, this set of data is utilized to render results that are tailored to your preferences.

Virtual assistants are integrated into various platforms that include:

•Smart Speakers – Amazon Echo, Google home, etc.

•Smartphones – Samsung Bixby on Samsung S8

•Mobile apps – Google Allo

Fig 14.4 Asia Pacific Intelligent virtual assistant market size, 2017-2028 (USD million)

14.4.2 Predictions while Commuting

Nowadays, everyone has a smartphone and uses GPS navigation services. Our current location is being saved at a central server and by which we can predict or manage traffic. This data is then used to build a map of the current traffic. It helps us analyze high and low traffic areas and save time by suggesting an alternative route when the traffic is high. In such scenarios, machine learning models help estimate the regions where congestion can be found based on daily experiences. This is one example the another can be seen while booking a cab.

While you book a cab, it estimates the price of the ride. Here machine learning plays its part in defining price surge hours by predicting the rider demand. So in the entire cycle of the services, ML is playing a major role. Jeff Schneider, the engineering lead at Uber ATC, reveals this in an interview.

14.4.3 Video Surveillance

It is necessary to maintain security, and for this, there are CCTV cameras everywhere. But it is difficult to manage and observe every camera at a time and more when there are several cameras. Hence to do this job, computers are trained using AI and ML.

Nowadays video surveillance system is powered by AI that makes it possible to detect crime before they happen. By tracking unusual behavior of people like standing motionless for a long time, stumbling, or napping on benches, etc. The system can thus give an alert to human attendants, which can ultimately help to avoid mishaps. Such predictions can help to improve and reduce crime rates. All this happens due to Machine learning models playing their parts in the background.

Fig. 14.5 Video Surveillance

14.4.4 Social Media Services

Social media services are also the result of machine learning. Companies like Google, Amazon, Flipkart, etc., can provide us the best services due to machine learning. From personalizing your news feed to better ad targeting, social media platforms utilize machine learning for their own and user benefits.

Here are few examples which you see in your day to day life:

•The service of People you may know on Facebook or Instagram is a result of machine learning. Machine learning works on a simple concept: understanding with experiences. It continuously notices the friends you connect with, the profiles you visit very often, your interests, workplace, or a group you share with someone, etc. Based on this continuous learning, a list of Facebook or Instagram users is suggested that you can become friends with.

•Most mobile phones now come with an AI camera that focuses on face recognition and makes it easier to find a person in our gallery. Again you can take the example of Facebook here, checks the poses and projections in the picture, notice the unique features, and then match them with the people in your friend list. The entire process at the backend is complicated and takes care of the precision factor but seems to be a simple application of ML at the front end.

•Machine learning is the core element of Computer Vision, a technique to extract useful information from images and videos. For example, Pinterest uses computer vision to identify the objects (or pins) in the images and recommend similar pins accordingly.

ML powers all these services and constantly work to give us the best services and profit the social media or companies involved.

14.4.5 Email Spam and Malware Filtering

Everyone uses email, and every day, we get a number of emails, and half of it is spam. Many spam filtering approaches are available which are used by people. These spam filters are continuously powered by ML for their updation. However, when rule-based spam filtering is done, it fails to track the latest tricks adopted by spammers. Multi-Layer Perceptron, C 4.5 Decision Tree Induction are some of the spam filtering techniques powered by ML.

Fig. 14.6 ML in email spam and malware filtering

Thousands of malware are detected every day, and each code is 90% similar to its previous versions. These coding patterns are understood by machine learning which is used in system security programs. Therefore, they can easily detect new malware with 2-10% variation and offer protection against them.

14.4.6 Online Customer Support

Customer support is the best facility a company can provide to its customers, which can help its customers and build a nice relationship. By using online customer support, companies can provide 24/7 customer support.

Most of the websites offer the option to chat with customer support representatives while navigating within the site. However, not every website has a live executive to answer your queries. In most cases, you talk to a chatbot—ML powers these bots to extract information from the websites and present it to the customers. Also, machine learning algorithms understand the user queries better and serve them with better answers.

Fig. 14.7 ML and customer support

14.4.7 Search Engine Result Refining

The most popular search engine is Google. Google and other search engines use machine learning to improve the search results for you. Every time you execute a search, the algorithms at the backend watch how you respond to the results.

If you open the top results and stay on the web page for long, the search engine assumes that the results it displayed were in accordance with the query. Similarly, if you reach the second or third page of the search results but do not open any of the results, the search engine estimates that the results served did not match the requirement. This way, the algorithms working at the backend improve the search results.

14.4.8 Product Recommendations

Product recommendations are seen by everyone when you shop products from websites like Amazon, Flipkart, etc. They use ML algorithms to track your product orders and give you suggestions according to that. The product recommendations are made based on your behavior with the website/app, past purchases, items liked or added to the cart, brand preferences, etc. It focuses on giving you the best shopping experience.

Fig. 14.8 ML and product recommendations

14.4.9 Online Fraud Detection

Nowadays, everything is made online. So when you do things online, there is always a chance of fraud. To detect these kinds of frauds, ML is used as it has the potential to make cyberspace a secure place, and tracking monetary frauds online is one of its examples.

Paypal can be taken as an example. It uses ML for protection against money laundering. The company uses a set of tools that helps them to compare millions of transactions taking place and distinguish between legitimate or illegitimate transactions between the buyers and sellers.

14.4.10 Intelligent Gaming

The best example of this is the chess match between Gary Kasparov and IBM’s Deep Blue, where Deep Blue came out victorious. Or a couple of years back in 2016 when Google DeepMind’s AlphaGod defeated Lee Dedol, the Go world champion.

By using ML and AI, machines or computers have been made to set new challenges in gaming. For example, the ancient Chinese game of Go is considered much more difficult for computers to learn and then to master than chess. However, the AI of AlphaGo was specifically trained to play Go and not by simply analyzing the moves of the world’s best players but by practicing against itself millions of times.

14.4.11 Self-Driving Cars and Automated Transportation

Self-driving cars are the future of cars and one of the fiction which has turned into reality. Auto-pilot is most used in airplanes and is a great help for pilots. Flight Management System combines GPS, motion sensors, and computer systems to track its position during flight. However, we cannot apply the same concept to cars as it’s very different and can make drastic changes.

There are other cars on the road, obstacles to be avoided, and limitations subject to the traffic rules. Even so, self-driving cars are a reality. These AI-powered cars can have better records than their human counterparts, according to a study with 55 Google vehicles that the driven more than 1.3 million miles altogether. Moreover, the navigation issues have already been solved by using Google Maps, which sources location data from drivers’ smartphones.

Fig 14.9 Self-driving cars and transportation

14.4.12 AI for dangerous jobs

Bomb disposal can be an example of this. It is one of the most dangerous jobs. Here AI can save lives. Robots and drones integrated with AI and ML algorithms can take this risk and help detect bombs or dispose of them. Presently, drones require human control, but as ML evolves, these can also be controlled by AI in the future.

14.4.13 Environment Protection

Machines can access and store huge data. By which identification of trends and use the information to determine solutions for the untenable problems. For example, IBM’s Green Horizon Project analyzes environmental data from multiple sensors and sources to produce accurate, evolving weather and pollution forecast. It helps city planners to understand the impact of the environment in their planning. Amazing environment-oriented innovations are emerging in the market regularly, from self-adjusting smart thermostats to distributed energy grids.

14.4.14 Improved ElderCare

Daily chores can be hectic for most of the elders. Due to work, many people rely on help from outside for their elderly family members. It has become a growing concern around the globe.

The solution to this can be AI-powered in-home robots. These robots can help the elderly with everyday tasks, keeping them independent and in their home, thus improving their overall well-being. Also, Medical and AI researchers have even piloted systems based on infrared cameras that can detect when the elderly fall, monitor food and alcohol consumption, restlessness, fevers, urinary frequency, chair and bed comfort, fluid intake, sleeping, eating, declining mobility, and more. This is great for elders and can become their support.

14.4.15 Home Security and Smart Homes

AI can also for utilized for home safety. Alarms and cameras powered with AI and ML algorithms are used to ensure security in houses. They use facial recognition software and machine learning models to build a catalog of your home’s frequent visitors. This allows the system to detect uninvited guests. Other intriguing features include tracking when you last walked your dog or notifying when your kids are back home from school. In addition, some of the latest systems can automatically call emergency services, making it a beneficial alternative to subscription bases services of the same category.

These advancements in human life have been possible only because of AI and Machine learning, where machines can be a great help to humans.

14.5 Demand Of Machine Learning

The demand for Machine Learning has been very high since the beginning due to its vast nature and numerous advantages which can change the world. And in the present, due to Covid 19, it has been in great demand and helped humans in this pandemic situation, increasing its importance. Most of the crowd is driven by machine learning due to its live applications and the power to solve impossible problems with traditional methods. It helps us in collecting, analyzing, and understanding a vast quantity of data. Combined with the power of Artificial Intelligence, Machine Learning can help with an early understanding of problems and quick resolutions. Machine learning is implemented in every field; it can bring benefits to everyone.

Even during this pandemic which generated a huge economic crisis all over the globe, the jobs in data science and machine learning have been least affected. However, the world is slowly moving towards advanced technologies, and AI and ML have increased rapidly. In the future, the impacts of ML can be seen everywhere. Hence it is important to know ML and AI. Everyone should at least have basic knowledge.

The demand for ML specialists will never end; moreover, it is likely to increase and can be a promising career. According to a research report released in February, demand for data engineers was up 50%, and demand for data scientists was up 32% in 2019 compared to the prior year.

14.6 Summary

In this chapter, we took an overview of machine learning and its tremendous advantages. By this, you can decide at what rate ML is changing the whole world. In a few years, everything can be made automatic and intelligent to bring a significant change in human lives. Hence, it is necessary to learn and know these technologies as they rapidly emerge and interfere with our day-to-day lives. Also, making a career in ML or AI can be very beneficial and bring a prestigious life. Though it also requires extensive study and hard work. Here in this field, learning never ends. Machine learning has a rich history, and its future can be more impressive. Let’s learn more about this technology in the upcoming chapters.

14.7 Exercise

1. What is Machine Learning?

2. Is machine learning different from AI?

3. Who is the subset of Machine Learning?

4. Explain any two differences between ML and AI?

5. List down advantages of ML?

6. Deep Blue was invented by?

7. How does a Chatbot work?

8. What is the future of transportation?

9. How can ML increase home safety?

References

The rise in the demand for machine learning & AI skills in the post-COVID world - Times of India. (2020, April 30). The Times of India.

Oberoi, A. (2017, June 16). 9 machine learning examples from day-to-day life. Your Technology Partner for Custom Software Development & Innovation | Daffodil Software.

Hastie, Friedman, and Tibshirani, The Elements of Statistical Learning, 2001

Bishop, Pattern Recognition and Machine Learning, 2006

Ripley, Pattern Recognition and Neural Networks, 1996

Duda, Hart, and Stork, Pattern Classification, 2nd Ed., 2002

Tan, Steinbach, and Kumar, Introduction to Data Mining, Addison-Wesley, 2005.

Scholkopf and Smola, Learning with Kernels, 2002

Mardia, Kent, and Bibby, Multivariate Analysis, 1979

Computational Statistics (online book)

Sutton and Barto, Reinforcement Learning: An Introduction, MIT Press, 1998.

Bertsekas and Tsitsiklis, Neuro-Dynamic Programming, Athena Scientific, 1996

The first comprehensive treatment and still a standard reference: Brieman, Friedman, Olshen and Stone, Classification and Regression Trees, 1984

The other standard reference is Quinlan, J. R. C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, 1993.

A somewhat recent survey of research on decision trees: Sreerama K. Murthy: Automatic Construction of Decision Trees from Data: A Multi-Disciplinary Survey. Data Min. Knowl. Discov. 2(4): 345-389 (1998)

Ripley has a nice chapter on decision trees -- probably the best place to start.

The primary research area relating to nearest neighbor methods is the problem of storage, data reduction, and rapid calculation of nearest neighbors. A search on “nearest neighbor search” or “condensed nearest neighbors” or “edited nearest neighbors” will return a number of references.

Theory: Devroye, Gyorfi and Lugosi, A Probabilistic Theory of Pattern Recognition, 1996

Figure Resources

Fig. 14.1. https://www.cygnismedia.com/images/post-images/machine-learning-in-ecommerce/types-of-machine-learning.jpg

Fig. 14.2. https://idapgroup.com/blog/blog/wp-content/uploads/2021/02/business_14-14-1024x640.png

Fig. 14.3. https://mobcoder.com/blog/wp-content/uploads/1_History-of-Voice-Assistance.png

Fig. 14.4. https://www.grandviewresearch.com/static/img/research/asia-pacific-intelligent-virtual-assistant-market.png

Fig. 14.5. https://miro.medium.com/max/641/1*zutTOZnISc_P7QGZ6wBurA.png

Fig. 14.6. https://media.springernature.com/lw685/springer-static/image/art%3A10.1007%2Fs10462-020-09814-9/MediaObjects/10462_2020_9814_Fig1_HTML.png

Fig. 14.7. https://miro.medium.com/max/1039/1*Ks2RHyn70qr8A2MlR8KcMA.png

Fig. 14.8. https://www.researchgate.net/profile/Marwa-Mohamed-54/publication/331063850/figure/fig1/AS:729493727617027@1550936266588/General-Concept-of-Recommender-Systems-8.jpg

Fig. 14.9. https://storage.ning.com/topology/rest/1.0/file/get/3823609685?profile=original

Chapter 15. Types of Machine Learning

Abstract

In this chapter, we will type machine learning. For better understanding and ease of work, it is mainly divided into 3 types. It can be a very difficult and tedious task to perform machine learning, so one should look for keeping things simple and not complicated, which can cause problems in results. Machine learning is capable of taking on important responsibilities, and it depends on how it is designed. It is necessary to design or choose the correct and suitable model or method. This chapter will shed light on types of machine learning which are used to identify or class data to give suitable treatment.

15.1 Supervised Learning

This is the most basic type of machine learning. The ML algorithm is trained on labeled data. Although the data needs to be labeled accurately for proper working, it can be very powerful if used in the right circumstances.

Here an ML algorithm is provided with a small training data set to work with. This data set is a smaller part of a bigger dataset, which gives the algorithm a basic idea of the problem, solution, and data points to be dealt with. This training data set is almost similar to the final dataset in its characteristics and provides the algorithm with the labeled parameters required for the problem. After this, the algorithm looks for the relationships between the given parameters; it first identifies the cause and effect relationship between the variables in the dataset.

By running such processes, the algorithm studies the data set and, in the end, gets an idea of how the data works and the relationship between the input and the output. Later once the algorithm is ready, it is deployed for use with the final dataset. Supervised machine learning algorithms will continue to improve even after being deployed, discovering new patterns and relationships as it trains itself on new data.

Fig 15.1 Supervised Learning

The whole process revolves around two concepts, i.e., Classification and Regression.

For this, it uses algorithms like:

1. Linear Classifiers

2. Support Vector Machines (SVM)

3. Decision Trees

4. K-Nearest Neighbor

5. Random Forest

6. Linear Regression

7. Logistical Regression

8. Polynomial Regression

15.2 Examples of Supervised Learning

Supervised learning models can be utilized to build and advance business applications which include the following:

•Image and object recognition: Algorithms in supervised learning can locate and categorize objects from videos or images, making them useful when applied to various computer vision techniques and imagery analysis.

•Predictive Analysis: Supervised learning models can also provide deep insights into various business data points. This allows enterprises to anticipate certain results based on a given output variable, helping business leaders justify decisions or pivot for the organization’s benefit.

•Customer sentiment analysis: Using supervised algorithms, organizations can extract and classify important information from large volumes of data, including context, emotion, and intent, with very little human intervention. This can be incredibly useful when gaining a better understanding of customer interactions and can be used to improve brand engagement efforts.

•Spam detection: Organizations use supervised algorithms to train databases to recognize patterns or anomalies in new data to effectively organize spam and non-spam-related correspondences.

15.3 Unsupervised Learning

Unsupervised learning overcomes supervised learning by possessing the ability to work with unlabeled data. This avoids the human labor required for making the dataset machine-readable, allowing much larger datasets to be worked on by the program.

The labels allow the algorithm to find the exact nature of the relationship between any two data points when it comes to supervised learning. Whereas here in unsupervised learning does not have labels to work off, resulting in the creation of hidden structures. Thus, the algorithm perceives relationships between data points abstractly, with no input required from human beings.

The creation of hidden structures makes unsupervised learning versatile. Instead of a defined and set problem statement, unsupervised learning algorithms adapt to the data by dynamically changing hidden structures. This offers more post-deployment development than supervised learning algorithms. The main goal is to find the underlying structure of the dataset, group that data according to similarities, and represent that dataset in a compressed format.

Fig 15.2 Unsupervised learning

Un supervised learning uses the concept of Clustering and Association

For performing these tasks, it uses algorithms like:

1. K-means clustering

2. KNN (k-nearest neighbors)

3. Hierarchal clustering

4. Anomaly detection

5. Neural Networks

6. Principle Component Analysis

7. Independent Component Analysis

8. Apriori algorithm

9. Singular value decomposition

15.4 Examples of Unsupervised Learning

Unsupervised learning can help us to gain answers to questions such as:

•Finding customer segments: Clustering is commonly used for determining customer segments in marketing data. Being able to determine different segments of customers helps marketing teams approach these customer segments in unique ways.

•Reducing the complexity of a problem: Dimensionality reduction is a commonly used unsupervised learning technique where the goal is to reduce the number of random variables under consideration.

•Feature selection: Feature selection and dimensionality reduction aim towards reducing the number of features in the original set of features.

15.5 Reinforcement Learning

Reinforcement Learning focuses on how human beings learn from data in their lives. It generates an algorithm that improves upon itself and learns from new situations using a trial-and-error method. Favorable outputs are encouraged, and non-favorable outputs are discouraged.

It uses the psychological concept of conditioning and works by putting the algorithm in a work environment with an interpreter and a reward system. In every iteration of the algorithm, the output result is given to the interpreter, which decides whether the outcome is favorable or not. If the program finds the correct solution, the interpreter reinforces the solution by providing a reward to the algorithm. On the other hand, if the outcome is not favorable, the algorithm is forced to reiterate until it finds a better result. In most cases, the reward system is directly tied to the effectiveness of the result.

Fig. 15.3 Reinforcement learning

For performing these tasks, it uses algorithms like:

1. Q-learning

2. Sarsa

3. Deep Q network

There are two types of reinforcement learning:

1. Positive: When an event occurs due to a particular behavior that becomes beneficial by increasing the strength and frequency of the behavior. As a result, it can maximize performance and sustain change for a long period.

2. Negative: It can be explained as strengthening of behavior because a negative condition is stopped or avoided. Hence it increases behavior and also provides defiance to a minimum standard of performance.

15.6 Examples of Reinforcement Learning

•It can be used in robotics for industrial automation.

•Used in machine learning and data processing

•To create training systems that provide custom instruction and materials according to the requirement of students.

Fig. 15.4 Difference between supervised, unsupervised, and reinforcement learning

15.7 Summary

In this chapter, we learned about types of Machine learning, i.e., Supervised, Unsupervised, and Reinforcement. These methods are designed to work with different data types, including labeled, unlabeled, and mixed-up data. Each of them is important and can be utilized in various ways. Many companies use ML to automate or bring innovations in them, which involves working with various kinds of datasets. We also saw how it works and algorithms or models used in that it uses algorithms same as data mining. Still, there are some which need to be covered in further chapters.

15.8 Exercise

1. What are the types of ML?

2. What do you mean by Supervised learning? Give an example of Supervised Learning?

3. Supervised learning deals with which kind of data?

4. List down the algorithms used by Supervised learning

5. What do you mean by Unsupervised learning? Give an example of Unsupervised vLearning?

6. Unsupervised learning deals with which kind of data?

7. List down the algorithms used by Unsupervised learning

8. What do you mean by Reinforcement learning? Give an example of Reinforcement Learning?

9. Reinforcement learning deals with which kind of data?

10. List down the algorithms used by Reinforcement learning.

References

What is supervised learning? (2020, August 19). IBM - United States.

Reinforcement learning. (2020, May 17). GeeksforGeeks.

Unsupervised machine learning - Javatpoint. (n.d.).

Aamodt, Agnar, and Enric Plaza. “Case-based reasoning: Foundational issues, methodological variations, and system approach.” AI Communications 7.1 (1994): 39-59.

Alain, Guillaume, et al. “Understanding intermediate layers using linear classifier probes.” arXiv preprint arXiv:1610.01644 (2018).

Alberto, Túlio C, Johannes V Lochter, and Tiago A Almeida. “Tubespam: comment spam filtering on YouTube.” In Machine Learning and Applications (Icmla), Ieee 14th International Conference on, 138–43. IEEE. (2015).

Alvarez-Melis, David, and Tommi S. Jaakkola. “On the robustness of interpretability methods.” arXiv preprint arXiv:1806.08049 (2018).

Apley, Daniel W. “Visualizing the effects of predictor variables in black box supervised learning models.” arXiv preprint arXiv:1612.08468 (2016).

Athalye, Anish, and Ilya Sutskever. “Synthesizing robust adversarial examples.” arXiv preprint arXiv:1707.07397 (2017).

Bach, Sebastian, et al. “On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.” PloS one 10.7 (2015).

Breiman, Leo.“Random Forests.” Machine Learning 45 (1). Springer: 5-32 (2001).

Brown, Tom B., et al. “Adversarial patch.” arXiv preprint arXiv:1712.09665 (2017).

Chen, Zhi, et al. “Concept Whitening for Interpretable Image Recognition.” Nature Machine Intelligence 2, 772–782 (2020).

Cohen, William W. “Fast, effective rule induction.” Machine Learning Proceedings (1995). 115-123.

Friedman, Jerome H, and Bogdan E Popescu. “Predictive learning via rule ensembles.” The Annals of Applied Statistics. JSTOR, 916–54. (2008).

Friedman, Jerome H. “Greedy function approximation: A gradient boosting machine.” Annals of statistics (2001): 1189-1232.

Friedman, Jerome, Trevor Hastie, and Robert Tibshirani. “The elements of statistical learning”. www.web.stanford.edu/~hastie/ElemStatLearn/ (2009).

Figure Resources

Fig. 15.1. https://static.javatpoint.com/tutorial/machine-learning/images/supervised-machine-learning.png

Fig. 15.2. https://miro.medium.com/max/1954/1*iNIixlUFwhJvLQjlilI7eQ.jpeg

Fig. 15.3. https://dynagrace.com/wp-content/uploads/2021/04/Reinforcement-Learning-in-ML-TV.jpg

Fig. 15.4. https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3492504%2F6cceb78efdaac8aad244dcde1d18dec7%2Fd.png?generation=1598256475310490&alt=media

Chapter 16. Models in Machine Learning

Abstract

In this chapter, we will models and also techniques of machine learning. These are very similar to data mining algorithms. Let’s revise them here also with some new algorithms. These algorithms are used to make things better and easy. The types of ML that are supervised, unsupervised, and reinforcement use these algorithms to perform their tasks and work with data. To perform those processes on data, one needs to study these algorithms. Let’s see which algorithms are used in which processes.

16.1 Classification models

This model is used in supervised learning, as we saw in previous chapters. Let’s see what the various algorithm it uses to perform classification are.

16.1.1 Logistic Regression

Logistic Regression comes under classification. The probabilities describing the possible outcomes of a single trial are modeled using a logistic function. It is based on maximum likelihood estimation, which states that coefficients should be chosen to maximize the probability of Y given X (likelihood). Along with maximum likelihood, the computer uses various iterations, and it keeps on trying various solutions until it gets the maximum likelihood estimates. For iteration, Fisher Scoring is the most popular method.

This algorithm is mainly designed for classification, which is most useful for understanding the influence of several independent variables on a single outcome variable. However, it works only when the predicted variable is binary and assumes that the data is free of missing values which can be a problem if the data has missing values.

The best application of logistic regression can be seen in Credit Card Fraud, where the bank notes several factors. For instance, the date of the transaction, amount, place, type of purchase, etc. Based on these factors, they develop a Logistic Regression model of whether the transaction is a fraud or not. If the amount is too high from regular or any such abnormal transactions have been seen, it will be treated as a fraud transaction and can go for investigation.

Fig. 16.1 Logistic Regression Model

Fig. 16.2 Linear vs. logistic regression

16.1.2 Naïve Bayes

Naive Bayes is a family of classification algorithms. Every feature of the data being classified is independent of all other features given the class. The assumption that all dataset features are independent is called naive; it’s generally not the case that all features are independent. Whereas Bayes comes from Thomas Bayes, an English statistician who generated Baye’s theorem. In short, the theorem allows us to predict that class given a set of features using probability. For Example, you have a training dataset of 1,000 fruits. The fruit can be a Banana, Orange, or Other (these are the classes). The fruit can be Long, Sweet, or Yellow (these are the features). And you see conditions like:

•Out of 500 bananas, 400 are long, 350 are sweet, and 450 are yellow.

•Out of 300 oranges, none are long, 150 are sweet, and 300 are yellow.

•Out of the remaining 200 fruit, 100 are long, 150 are sweet, and 50 are yellow.

If we are given the length, sweetness, and color of fruit (without knowing its class), we can now calculate the probability of it being a banana, orange or other fruit.

This algorithm needs a small amount of training data to estimate the essential parameters. It is also well known for being extremely fast as compared to other sophisticated methods. However, it can also be a bad estimator.

Fig 16.3 Example of Naïve Bayes

16.1.3 Stochastic Gradient Descent

This algorithm is most common when it comes to optimization strategy. It is simple and very efficient to fit linear models. Mainly used when the number of samples is very large. It also supports different loss functions and penalties for classification. It is used to minimize the error by adjusting weights after passing through all the samples in the training set. If the weights are updated after a specified subset of training samples or after each sample in the training set, this is called Stochastic Gradient Descent.

But it consists of various limitations which have affected its usability. For example, limitations like the error signal diminish as the gradient is back-propagated (i.e., the gradient vanishes), and SGD is sensitive to poor conditioning, which means a small input can change the gradient dramatically.

Fig 16.4 Example of gradient

16.1.4 K-NN

kNN means k-Nearest Neighbors, also a classification algorithm. It differs from the previous classifiers as it is a lazy learner. The lazy learner doesn’t do much during the training process other than storing the data. If you input new unlabeled data, then only it will look for classification. kNN does not build a classification model like c4.5 or SVM; instead, it stores the labeled training data.

kNN works in two steps:

1. It looks at the k closest labeled training data points — in other words, the k-nearest neighbors.

2. Second, using the neighbor’s classes, kNN gets a better idea of how the new data should be classified.

To find the closest data in continuous data, kNN uses distance metrics. The choice of distance metric largely depends on the data. Some even suggest learning a distance metric based on the training data. However, when it comes to discrete data, it converts it into continuous data.

Here classification is computed from a simple majority vote of the k nearest neighbors of each point. It is very simple to implement, robust to noisy training, and effective for large data.

Fig. 16.5 Flowchart for KNN modeling

16.1.5 Decision trees

A decision tree works on given data attributes, and with respective classes, it will produce a sequence of rules that can be used to classify the data. They are very simple to understand, and one can easily visualize them. It requires less data preparation and can handle both numerical and categorical data. Yet sometimes, it can also create complex decision trees, which can be unstable if any variation occurs; it can largely affect the decision tree.

Fig 16.6 Decision tree

16.1.6 Random Forest

Random Forest is a classifier, basically, a meta-estimator that fits a number of decision trees on various sub-samples of datasets and uses an average to improve the predictive accuracy of the model and controls over-fitting. The sub-sample size is always the same as the original input sample size, but the samples are drawn with replacement.

Random Forest works in the following steps :

•Selection of random k data points from the training set.

•Building the decision trees associated with the selected data points (Subsets).

•Selecting the number N for decision trees to be built.

•Repeat step 1 & 2.

•Finding the predictions of each decision tree for every new data point and assign them to the category that wins the majority votes.

Over-fitting and random forest classifiers in more accurate than decision trees in most cases.

Examples or applications of Random forest can be seen in Banking, Medicine, Land use, Marketing.

Fig 16.7 Random Forest

16.1.7 SVM

SVM stands for Support vector machines that use a hyperplane to classify data into 2 classes. It performs tasks similar to C4.5 but doesn’t use decision trees. We need to take an example to understand it. Suppose there are a bunch of red and blue balls on a table. If the balls aren’t too mixed together, you could take a stick and, without moving the balls, separate them with the stick. You will see that when a new ball is added to the table, you can predict its color by knowing which side of the stick the ball is on. Here the balls represent data points, and the red and blue colors represent 2 classes. The stick represents the simplest hyperplane which is a line. SVM acts on a hyperplane; it figures out the function of a hyperplane.

In the second condition, where the balls are mixed, a stick here wouldn’t give us the best results. Now what we will do is quickly lift the table throwing the balls in the air. While the balls are in the air and thrown up in just the right way, you use a large sheet of paper to divide the balls in the air. Lifting the table is the equivalent of mapping your data into higher dimensions. In this case, we go from the 2-dimensional table surface to the 3 dimensional balls in the air.

The SVM performs this task by using kernel as it gives a way to operate in higher dimensions. The large sheet of paper is still called a hyperplane, but it is now a function for a plane rather than a line. Note from Yuval that once we are in 3 dimensions, the hyperplane must be a plane rather than a line.

In real life, as for a patient data set, each patient could be described by various measurements like pulse, cholesterol level, blood pressure, etc. Each of these measurements is a dimension. SVM does its thing, maps them into a higher dimension, and then finds the hyperplane to separate the classes.

Fig 16.8 SVM terms

16.2 Regression models

Regression is a predictive model technique used to analyze the relationship between the dependent and independent variable data set. There are various types of regression algorithms that are used according to the situation or dataset.

16.2.1 Linear Regression

Linear regression is the basic type of regression in machine learning. It involves a predictor variable and a dependent variable related linearly to each other. If the data consist of more than one independent variable, it is called the multiple linear regression model.

Linear regression is denoted by:

Y = mx + c + e

Where,

m = Slope of the line,

c = Intercept,

e = Error in model

m and c are used to determine the best fit line by varying their values. The difference between the observed values and the predicted value is called predictor error. The values of m and c need to be selected so that they give a minimum predictive error. It is mainly designed to suspect the outliers and can not be used for large data.

Fig 16.9 Linear Regression

16.2.2 Logistic Regression

It is a type of regression that is used in the case when the dependent variable is discrete. This means that the target variable can have only two values, such as 0 or 1, true or false, etc. The relation between the target variable and the independent variable is denoted by a sigmoid curve. And to measure the relationship between them logit function is used.

The equation of logistic regression is given as:

logit(p) = ln(p/(1-p)) = b0 + b1X1 + b2X2 + b3X3…. + bkXk

where p is referred to as the probability of occurrence of the feature. This technique is designed for large data with equal values to come in the target variable. While working with the data using this technique, one should ensure no correlation between independent variables.

Fig 16.10 Logistic regression

16.2.3 Ridge Regression

This technique overcomes the limitation of the logistic technique as it can be used even when there is a high correlation between the independent variable. This is because, in the case of multi collinear data, the least square estimates give unbiased values. Although if the collinearity is very high, there can be some biased values too. Hence a biased matrix was introduced in the equation of ridge regression. The equation for ridge regression looks like this:

β = (X^{T}X + λ*I)^{-1}X^{T}y

Where λ (lambda) solves the problem of multicollinearity.

Fig 16.11 Ridge Regression

16.2.4 Lasso Regression

It is a type of regression that performs regularization along with feature selection. It prohibits the absolute size of the regression coefficient. As a result, the coefficient value gets nearer to zero, which does not happen in Ridge Regression.

This enables feature selection and eventually allows the selection of a set of features from the dataset to build a model. Lasso only makes use of required features, and the rest are made zero. This helps in avoiding overfitting in the model. In case the independent variables are highly collinear, then Lasso regression picks only one variable and makes other variables shrink to zero.

Here is the equation for Lasso Regression

N^{-1}Σ^{N}_{i=1}f(x_{i}, y_{I}, α, β)

Fig 16.12 Lasso Regression

16.2.5 Polynomial Regression

The polynomial regression technique is similar to multiple linear regression. The relationship between independent and dependent variables; that is, X and Y are denoted by the n-th degree. As an estimator, it can be considered as a linear model. The best fit line in this is the line that passes through all the data points but is a curved line that depends upon the power of X or the value of n. It is also important to analyze the curve towards the end as the higher Polynomials can give strange results on extrapolation.

Equation for polynomial regression is:

l = β0 + β0x1 + ε

Fig 16.13 Polynomial Regression

16.2.6 Bayesian Linear Regression

This type uses the Bayes Theorem to find out the value of regression coefficients. Here, the posterior distribution of the features is determined instead of finding the least-squares. Thus, Bayesian Linear Regression is like both Linear Regression and Ridge Regression but is more stable than simple Linear Regression.

Fig 16.14 Bayesian Linear Regression

16.3 Clustering models

The clustering technique involves the grouping of data points. These models are used to classify each data into a specific group. Data points that belong to the same group should have similar properties or features, whereas data points in different groups should have highly dissimilar properties or features. It is an unsupervised learning method and is a common technique for statistical data analysis used in many fields. Below are the algorithms used for clustering.

16.3.1 K-means clustering

k-means create k groups from a set of objects to make the members of a group more similar. It is a cluster analysis technique for processing and exploring a data set. Cluster analysis is a family of algorithms designed to form groups. The group members are more similar than non-group members. Clusters and groups are synonymous in the world of cluster analysis. K-means consist of lots of variations to optimize certain types of data. Suppose we take the example of patients. It performs tasks like:

•K-means will pick points in multi-dimensional space to represent each of the k clusters. These are called centroids.

•Every patient will be closest to 1 of these k centroids. However, they hopefully won’t all be closest to the same one so that they will form a cluster around their nearest centroid.

•Now each patient is a member of a cluster.

•Then it finds the center for each of the k clusters based on its cluster members.

•This center becomes the new centroid for the cluster.

•Since the centroid is in a different place now, patients might now be closer to other centroids. In other words, they may change cluster membership.

•Steps 2-6 are repeated until the centroids no longer change, and the cluster memberships stabilize. This is called convergence.

k-means is very simple, which makes it faster and more efficient than other algorithms. It can be used to pre-cluster a massive dataset followed by more expensive cluster analysis on the sub-clusters. k-means can also be used to rapidly play with k and explore overlooked patterns or relationships in the dataset. However, k means is sensitive when it comes to outliers ad the initial choice of centroids. It is designed for continuous data, and for discrete data, one needs to perform some tricks.

Fig 16.15 K-means

16.3.2 Mean-Shift clustering

Mean–shift clustering is based on a sliding window where it attempts to find dense areas of data points. Its main goal is to locate the center points of each group/class, which works by updating candidates for center points to be the mean of the points within the sliding window. Finally, these windows are further filtered post-processing to eliminate near-duplicates, forming the final set of center points and their corresponding groups.

Let’s take an example to understand the mean shift algorithm:

Consider a set of points in 2D with a circular sliding window centered at a point C (randomly selected) and having radius r as the kernel. Mean shift is a hill-climbing algorithm that involves shifting this kernel iteratively to a higher density region on each step until convergence. In each iteration, the sliding window is shifted towards regions of higher density by shifting the center point to the mean of the points within the window. Further, it continues shifting the sliding window according to the mean until there is no direction at which a shift can accommodate more points inside the kernel. This process of steps 1 to 3 is done with many sliding windows until all points lie within a window. In the end, the data points are then clustered according to the sliding window in which they reside.

The biggest advantage of mean shift is that it doesn’t require selecting the number of clusters as this is automatically done.

Fig 16.16 Mean Shift

16.3.3 DBSCAN

DBSCAN stands for Density-Based Spatial Clustering of Applications with Noise which is somewhat similar to mean shift. The main difference is it has more advantages as compared to the mean shift.

It starts with an arbitrary starting data point that has not been visited. All the points which are at the same distance are termed neighborhood points. If there are a sufficient number of points within this neighborhood, then the clustering process starts, and the current data point becomes the first point in the new cluster. Else the point is labeled as a noisy point. Later the points within an e distance also become a part of the same cluster. This process continues until all the points are visited and labeled.

DBSCAN does not require a preset of a number of clusters; also, it identifies outliers as noises, unlike mean-shift, which simply throws them into a cluster even if the data point is very different. Although it has these advantages, it still lacks in performance. Also, it is very challenging when it comes to very high-dimensional data.

Fig 16.17 DBSCAN

There are three types of points after the DBSCAN clustering is complete:

Core — This is a point that has at least m points within distance n from itself.

Border — This is a point that has at least one Core point at a distance n.

Noise — This is a point that is neither a Core nor a Border. And it has less than m points within distance n from itself.

16.3.4 EM with GMM

EM stands for Expectation-Maximization, whereas GMM means Gaussian Mixture Models they work together for best results. GMM gives more flexibility than K-means. By using GMM, we can assume the data points are Gaussian distributed. Hence now we have two parameters that describe the shape of the clusters: the mean and the standard deviation.

The algorithm starts with selecting the number of clusters and randomly initializing the Gaussian distribution parameters for each cluster. Gaussian distributions for each cluster compute the probability that each data point belongs to a particular cluster. The closest data point belongs to the cluster. Based on these probabilities, we compute a new set of parameters for the Gaussian distributions to maximize the probabilities of data points within the clusters. We compute these new parameters using a weighted sum of the data point positions, where the weights are the probabilities of the data point belonging to that particular cluster. Naturally, the distribution’s mean is shifted closer to those set of points. This process is repeated until the distributions are stable in each iteration.

GMM is more flexible and can have multiple clusters per data point.

Fig 16.18 EM for GMM

Fig. 16.19 Illustrating EM algorithm

16.3.5 Agglomerative Hierarchical Clustering

Agglomerative hierarchical clustering is a bottom-up algorithm that treats each data point as a single cluster at the outset and then successively merges (agglomerate) pairs of clusters until all clusters have been merged into a single cluster that contains all data points.

Fig. 16.20 Agglomerative Clustering algorithm

It begins by treating each data point as a single cluster. Then a distance metrics are being selected to measure the distance between two clusters. With each iteration, it combines two clusters into one. The two clusters to be combined are selected as those with the smallest average linkage, i.e., according to our selected distance metric, these two clusters have the smallest distance between each other and therefore are the most similar and should be combined. Until we reach the root of the tree, this process is repeated.

Fig 16.21 Example of Agglomerative Clustering

16.4 Association models

Association is rule-based learning for unleashing the relations between variables in large databases. Below are some of the algorithms that are used for machine learning.

16.4.1 Apriori

We have seen this algorithm in previous chapters; let’s revise it here once more. The Apriori algorithm is the related association; it learns association rules and applies to a database containing many transactions. For example, you can take supermart transactions. It consists of a giant spreadsheet where each row is a customer transaction, and every column represents a different grocery item. Here you can apply the apriori algorithm to learn the grocery items purchased together, i.e., association rules. With the help of this, You can find those items that tend to be purchased together more frequently than other items — the ultimate goal being to get shoppers to buy more. Together, these items are called itemsets.

For example, You observe that chips + dip and chips + soda seem to frequently occur together. These are called 2-itemsets. With a large enough dataset, it will be much harder to see the relationships, especially when you are dealing with 3-itemsets or more. Here Apriori helps you.

Before implementing Apriori, you need to specify three things:

1. The first is the size of your itemset.

2. The second is your support or the number of transactions containing the itemset divided by the total number of transactions. An itemset that meets the support is called a frequent itemset.

3. The third is your confidence or the conditional probability of some item given you have certain other items in your itemset. A good example is given chips in your itemset; there is a 67% confidence of having soda also in the itemset.

Apriori works in 3 steps:

Join. Scan the whole database for how frequent 1-itemsets are.

Prune. Those itemsets that satisfy the support and confidence move onto the next round for 2-itemsets.

Repeat. This is repeated for each item set level until we reach our previously defined size.

Fig 16.22 Operation of Apriori Algorithm

Fig. 16.23 Example of Apriori Algorithm

16.4.2 Eclat Algorithm

ECLAT stands for Equivalence Class Transformation, which is also referred to as bottom-up lattice traversal. It is an upgraded version of Apriori, which is more efficient and scalable. Moreover, it uses a vertical approach, which makes it faster than Apriori.

It uses transactional ID Sets to calculate a candidate’s support value and avoid the generation of subsets that do not exist in the prefix tree. Initially, all single items are used along with their tidsets. In the next step, the function is called recursively, and in each cell, each item tidest pair is verified and combined with other item-tidset pairs. This process is continued until no candidate item-tidset pairs can be combined.

It uses less memory than apriori as it uses the Depth-First Search approach. As a result, it is also faster and avoids repeated data scanning to compute the individual support values.

Fig 16.24 Eclat Example

16.4.3 FP growth Algorithm

FP stands for the frequent pattern. It counts the occurrence of items in the dataset of transactions and stores these counts in the header table. Secondly, it builds an FP tree structure by inserting transactions into a tree. Before inserting the dataset into the tree, it needs to be sorted in descending order with respect to frequency. Items that do not meet the minimum support requirement are discarded. Recursive processing of this compressed version of the main dataset grows frequent itemsets directly instead of generating candidate items and testing them against the entire database. Once this is complete, and every frequent item set is found, the association rule begins.

Fig 16.25 FP Example

16.5 Reinforcement learning models

These algorithms are used in reinforcement learning which includes Q-learning, Sarsa, and Deep Q network.

16.5.1 Q learning

Q learning refers to model-free reinforcement learning. It is a value-based algorithm that updates the values function based on an equation called the Bellman equation. Also, it has other types based on policy-based that estimates the value function with a greedy policy obtained from the last policy improvement.

Q-learning is an off-policy learner as it learns the value of the optimal policy independently of the agent’s actions. The Q here stands for quality which represents the usefulness of a given action in gaining future rewards.

Let’s understand this process by taking an example. Suppose an agent wants to move from starting point to the ending point along a path consisting of n number of obstacles. First, he needs to determine the shortest path and reach the ending point.

Now let’s learn some prerequisites before starting with the Q algorithm

First comes Q table is a kind of data structure used to calculate maximum expected future rewards for action at each state. Basically, this table will guide us to the best action in each state. Now comes the Q function that uses the Bellman equation.

Fig 16.26 Q-learning

Ok, now let’s start with the algorithm process.

The first one needs to initialize the Q table. Which contains n columns as a number of actions and m rows as a number of states. In the next step, according to our example, the agent needs to choose an action. Now the agent needs to act for an undefined amount of time. These steps are repeated until the training is stopped, or in this case, the agent reaches the endpoint with the shortest route. Also, the agent here is rewarded and becomes more confident in estimating the Q values. At last, the Q function is updated, and evaluation takes place.

Fig 16.27 Agent example

16.5.2 SARSA

SARSA can be interpreted as State–action–reward–state–action, an algorithm for learning a Markov decision process policy. It varies slightly from the Q learning technique; first, it needs to learn Q and then start with SARSA. It is an on-policy technique that uses the action performed by the current policy to learn the Q-value. The equation for this technique is as follows:

Fig 16.28 SARSA Learning

SARSA equation depends on the current state, current action, reward obtained, next state, and next action. Hence it is named SARSA, which works on the following steps: (https://www.geeksforgeeks.org/sarsa-reinforcement-learning/)

1. Importing the required libraries

In python code, it can be written as

import numpy as np

import gym

2. Building the environment

For example, one can use Frozen Lake – v0 environment. Python code:

env = gym.make(‘FrozenLake-v0’)

3. Initializing different parameters

//Python Code

#Defining the different parameters

epsilon = 0.9

total_episodes = 10000

max_steps = 100

alpha = 0.85

gamma = 0.95

#Initializing the Q-matrix

Q = np.zeros((env.observation_space.n, env.action_space.n))

4. Defining utility functions to be used in the learning process.

//Python Code

#Function to choose the next action

def choose_action(state):

action=0

if np.random.uniform(0, 1) < epsilon:

action = env.action_space.sample()

else:

action = np.argmax(Q[state, :])

return action

#Function to learn the Q-value

def update(state, state2, reward, action, action2):

predict = Q[state, action]

target = reward + gamma * Q[state2, action2]

Q[state, action] = Q[state, action] + alpha * (target – predict)

5. Training the learning agent

#Initializing the reward

reward=0

# Starting the SARSA learning

for episode in range(total_episodes):

t = 0

state1 = env.reset()

action1 = choose_action(state1)

while t < max_steps:

#Visualizing the training

env.render()

#Getting the next state

state2, reward, done, info = env.step(action1)

#Choosing the next action

action2 = choose_action(state2)

#Learning the Q-value

update(state1, state2, reward, action1, action2)

state1 = state2

action1 = action2

#Updating the respective vaLues

t += 1

reward += 1

#If at the end of the learning process

if done:

break

6. Evaluating the performance.

#Evaluating the performance

print (“Performance: “, reward/total_episodes)

#Visualizing the Q-matrix

print(Q)

Fig 16.29 Output of SARSA

This will be the final output that will determine the performance of the agent in SARSA.

16.5.3 Deep Q learning

Deep Q learning was brought up to overcome the problem of Q learning. It creates an exact matrix for the working agent, which it can refer to to maximize its reward in the long run. This is only practical for very small environments and quickly loses its feasibility when the number of states and actions in the environment increases.

The matrix values only hold relative importance. And hence Deep Q learning was introduced. It utilizes a deep neural network to approximate the values. The basic step involves feeding the initial state to a neural network, which returns the Q value of all possible actions.

Here the target for the neural network is variable, unlike other typical Deep Learning processes where the target is stationary. It consists of two neural networks; one is used to adjust the parameters of the network, and the other is used for computing the target, which has the same architecture as the first network but has frozen parameters. After an x number of iterations in the primary network, the parameters are copied to the target network.

The below image will help you to understand it more clearly:

Fig 16.30 Deep Q learning Process

Let’s see the difference between Q matrix and deep neural network with the help of the following image:

Fig 16.31 Q matrix

Fig 16.32 Deep Q Network

16.6 Difference between ML models

Let’s see how these models differ from each other you will have a quick overview of all the models and how they differ.

Fig. 16.33 Difference between classification and clustering

Fig. 16.34 Difference between Classification and Regression

16.7 Summary

In this chapter, we have covered all the models in machine learning with the algorithms used in them lets have a quick recap of all the models with their algorithms:

1. Classification model

(i) Logistic Regression

(ii) Naïve Bayes

(iii) Stochastic Gradient Descent

(iv) KNN

(v) Decision Tree

(vi) Random Forest

(vii) SVM

2. Regression model

(i) Linear Regression

(ii) Logistic Regression

(iii) Ridge Regression

(iv) Lasso Regression

(v) Polynomial Regression

(vi) Bayesian Linear Regression

3. Clustering model

(i) K means Clustering

(ii) Mean shift Clustering

(iii) DBSCAN

(iv) EM with GMM

(v) Agglomerative Hierarchical Clustering

4. Association model

(i) Apriori

(ii) Eclat

(iii) FP growth

5. Reinforcement model

(i) Q learning

(ii) SARSA

(iii) Deep Q learning

These models help in various ways and are the backbone of Machine Learning. Also, with these, we have also seen the difference between the models and how they can be used for different requirements and data sets.

16.8 Exercise

1. What are the models of ML?

2. Write down the algorithms used in

(a) Classification model

(b) Regression model

(c) Clustering model

(d) Association model

(e) Reinforcement model

3. Explain the following

(a) k means

(b) KNN

(c) SVM

(d) Decision tree

(e) Logistic Regression

(f) Ridge Regression

(g) SARSA

4. Give an example of Q learning.

5. What does DBSCAN stand for?

References

H. Nguyen and H. La, “Review of Deep Reinforcement Learning for Robot Manipulation,” in 2019 Third IEEE International Conference on Robotic Computing (IRC), Naples, Italy, 2019, pp. 590–595.

The primary research area relating to nearest neighbor methods is the problem of storage, data reduction, and rapid calculation of nearest neighbors. A search on “nearest neighbor search” or “condensed nearest neighbors” or “edited nearest neighbors” will return a number of references.

Theory: Devroye, Gyorfi and Lugosi, A Probabilistic Theory of Pattern Recognition, 1996

David Scott, Multivariate Density Estimation, 1992

Theory: Devroye and Lugosi, Combinatorial Methods in Density Estimation, 2001

Hastie et al., Bishop, and Duda et al. all have chapters on LDA, logistic regression, and other linear classifiers.

The first comprehensive treatment and still a standard reference: Breiman, Friedman, Olshen and Stone, Classification and Regression Trees, 1984

The other standard reference is Quinlan, J. R. C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, 1993.

A somewhat recent survey of research on decision trees: Sreerama K. Murthy: Automatic Construction of Decision Trees from Data: A Multi-Disciplinary Survey. Data Min. Knowl. Discov. 2(4): 345-389 (1998)

Ripley has a nice chapter on decision trees -- probably the best place to start.

On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation, Gavin C. Cawley, Nicola L. C. Talbot; JMLR 11(Jul):2079-2107, 2010. Discussion of how certain model selection strategies are more biased than others; essential reading if you are doing comparative studies of different machine learning methods.

Descent Methods for Tuning Parameter Refinement, Alexander Lorbert, Peter Ramadge; AISTATS 2010. A natural idea.

An entry-level discussion of the bootstrap, cross-validation and other error estimates is given in Efron and Tibshirani, An Introduction to the Bootstrap, 1993.

Adaboost was first developed in Freund and Schapire, A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119-139, 1997.

A simpler proof the Adaboost’s weak learning property is given in Robert E. Schapire and Yoram Singer. Improved boosting algorithms using confidence-rated predictions. Machine Learning, 37(3):297-336, 1999.

The view of Adaboost as performing functional gradient descent was observed by a number of researchers in the late ‘90s and early 00’s. A representative work is L. Mason, J. Baxter, P. L. Bartlett, and M. Frean. Functional gradient techniques for combining hypotheses. In A. J. Smola, P. L. Bartlett, B. SchKopf, and D. Schuurmans, editors, Advances in Large Margin Classifiers, pages 221-246. MIT Press, 2000.

Logitboost was introduced in Friedman, J.H., Hastie, T., Tibshirani, R.: Additive logistic regression: a statistical view of boosting. Annals of Statistics 28, 337-407 (with discussion) (2000).

Empirical Bernstein Boosting, Pannagadatta Shivaswamy, Tony Jebara; AISTATS 2010.

Figure Resources

Fig. 16.1. https://i.pinimg.com/originals/4e/90/49/4e904958f535db5c88837a4f90aaf7ee.jpg

Fig. 16.2. https://static.javatpoint.com/tutorial/machine-learning/images/linear-regression-vs-logistic-regression.png

Fig. 16.3. https://media.geeksforgeeks.org/wp-content/uploads/naive-bayes-classification.png

Fig. 16.4. https://www.analyticsindiamag.com/wp-content/uploads/2019/09/sgd.png

Fig. 16.5. https://www.researchgate.net/profile/Abdolhossein-Hemmati-Sarapardeh-bdalhsyn-hmty-sraprdh/publication/346429285/figure/fig1/AS:963317585690624@1606684218911/A-simple-flowchart-for-the-k-nearest-neighbor-modeling.png

Fig. 16.6. https://www.tutorialspoint.com/data_mining/images/dm_decision_tree.jpg

Fig. 16.7. https://static.javatpoint.com/tutorial/machine-learning/images/random-forest-algorithm.png

Fig. 16.8. https://static.javatpoint.com/tutorial/machine-learning/images/support-vector-machine-algorithm.png

Fig. 16.9. https://www.jmp.com/en_hk/statistics-knowledge-portal/what-is-multiple-regression/fitting-multiple-regression-model/_jcr_content/par/styledcontainer_2069/par/lightbox_4130/lightboxImage.img.png/1548704005203.png

Fig. 16.10. https://static.javatpoint.com/tutorial/machine-learning/images/logistic-regression-in-machine-learning.png

Fig. 16.11. https://miro.medium.com/max/962/1*cB0ESE9z3rB3-rpXPhwgWw.png

Fig. 16.12. https://www.upgrad.com/blog/wp-content/uploads/2020/07/lasso-reg-coef-1.png

Fig. 16.13. https://miro.medium.com/max/1400/1*zOl_ztYqnzyWRkBffeOsRQ.png

Fig. 16.14. https://slidetodoc.com/presentation_image_h/a4ce30e9ea8c25ff1e2db03ba873fa89/image-34.jpg

Fig. 16.15. https://miro.medium.com/max/1122/0*mQCGBdYhzZ8YMZPv.png

Fig. 16.16. https://media.geeksforgeeks.org/wp-content/uploads/20190429213154/1354.png

Fig. 16.17. https://miro.medium.com/max/627/1*yT96veo7Zb5QeswV7Vr7YQ.png

Fig. 16.18. https://miro.medium.com/max/1838/1*qvvT52JVR9wf4MV9FDXC2w.png

Fig. 16.20. https://www.researchgate.net/profile/Heekwan-Lee/publication/323292636/figure/fig1/AS:699421205479426@1543766418688/Agglomerative-hierarchical-clustering-algorithm.ppm

Fig. 16.21. https://miro.medium.com/max/1254/0*E-qictlO_9isi0Dl.png

Fig. 16.22. https://miro.medium.com/max/569/1*qYnO1S5eFQwaCGtS45Vv1Q.png

Fig. 16.23. https://www.researchgate.net/profile/Samuel-Robertson-2/publication/335323554/figure/fig3/AS:798437830303744@1567373821315/Tabular-example-of-the-Apriori-algorithm-process-Processing-example-of-the-Apriori.png

Fig. 16.24. https://www.researchgate.net/profile/Man-Kwan-Shan/publication/220802280/figure/fig3/AS:305532888993795@1449856125704/An-example-for-Eclat-C.png

Fig. 16.25. https://www.kopo.com/ws/ib.php?s=gfl&fileid=62

Fig. 16.26. https://miro.medium.com/max/1196/1*eyvquWnldzyJtyCT5cbljA.png

Fig. 16.27. https://miro.medium.com/max/1330/1*3CsV0MUbcAI-6Xc-29txow.png

Fig. 16.28. https://i.ytimg.com/vi/FhSaHuC0u2M/maxresdefault.jpg

Fig. 16.29. https://media.geeksforgeeks.org/wp-content/uploads/20190613152145/performance1.png

Fig. 16.30. https://cdn.analyticsvidhya.com/wp-content/uploads/2019/04/1_C1AzGsl8OwoeAwoOFwAd9A.png

Fig. 16.31. https://media.geeksforgeeks.org/wp-content/uploads/20190613102817/q-learning.png

Fig. 16.32. https://media.geeksforgeeks.org/wp-content/uploads/20190613104538/Deep-Q-Learning.png

Fig. 16.33. https://www.dotactiv.com/hubfs/4.Classification%20vs%20Clustering.png

Fig. 16.34. https://i1.wp.com/www.protechskills.com/wp-content/uploads/2020/05/Classification-Regression.png?resize=591%2C231

Chapter 17. Emerging Trends in ML

Abstract

In this chapter, we will see the emerging trends in machine learning. Machine learning along with AI is increasing and unleashing new technologies. These technologies are at their peak in today’s world and are increasing day by day. Every day a new technology or updates are added to AI. Moreover, they are used in every field today; most companies use these technologies and have experienced great results. In the future, this technology will acquire each and every field with its amazing features and advantages. AI is touching new heights and helping humans in a very great and advanced way. Let’s see the future trends in AI and ML that can change the future of humans and their lifestyles.

17.1 Intersection Of ML and IoT

IoT, i.e., the Internet of things, is also in competition with ML and AI. In IoT, multiple devices are connected across a network and communicate with each other. Looking at the increasing demand, there can be more than 64 billion IoT devices by 2025. All these devices collect data that can be analyzed and studied to obtain useful insights. Machine Learning algorithms can be used to convert the data collected by IoT devices into useful, actionable results.

An example of this can be seen in Green Horizons, created by IBM’s China research lab. The main focus of this project is to control the pollution levels and make the air breathable. It can achieve this by making the use of IoT where sensors collect emissions from vehicles, pollen levels, airflow direction, weather, traffic levels, etc., and then ML algorithms are used to find the best way to reduce these emissions.

Smart vehicles where self-driving cars need to be extremely accurate and all the parts need to communicate with each other in milliseconds on the road are an example of the intersection of ML with IoT. This shows how important the combination of these technologies is. Gartner even predicts that more than 80% of enterprise IoT projects will use Artificial Intelligence and Machine Learning in some form by 2022, which is much higher than the 10% of projects using it currently.

Using ML and AI in IoT helps us to make our things or devices smart and save time. They help us in our day-to-day life and make an easy living. Most people are making IoT their choice and experiencing comfort.

Fig 17.1 AIoT: Intersection of AI and IoT for better and faster decision making

17.2 Automated Machine Learning

Automated Machine Learning is a new emerging technology that is a great invention for people who are not experts in the complicated world of Machine Learning and experienced data scientists and analysts. It helps in reducing complexity in machine learning and makes it fully automated. This makes it easier for people who wish to perform machine learning but are not well trained or experienced.

Also, it allows data scientists to create Machine Learning models with higher efficiency and productivity while having top-notch quality. So AutoML can be a great option for training high-quality custom machine learning models for classification, regression, and clustering without much programming knowledge, which can eventually save time and resources and reduce costs.

Microsoft Azure currently provides this tool or service; by using this, you can build and deploy predictive models.

Fig 17.2 Azure Automated ML

17.3 Machine Learning in Cyber Security

Cybersecurity is a must nowadays due to our increasing dependency on the internet, where there is n number of threats, and maintaining security is necessary to avoid dangerous problems. Machine learning also has a hands-on field due to its amazing advantages.

Here it is used to create Smart antivirus software that can identify any virus or malware by its abnormal behavior rather than just using its signature like normal Antivirus. This smart antivirus can easily identify older threats and also new threats. Companies like Chronicle, a cybersecurity company handled by Alphabet (Google’s parent company), Sqrrl, founded by ex-National Security Agency employees (Scary!), etc., have already started using these techniques.

17.3 AI and ML in Cybersecurity

17.4 Rise of AI Ethics

Today AI and ML have become so advanced that it has become necessary to discuss the ethics of these technologies. These technologies are interfering and challenging humans. Moreover, it is trying to become somewhat like humans, which is fascinating, but at the same time, it is dangerous for us if it becomes excessive. Questions like What if a self-driving car kills a person? What if a Machine Learning algorithm is biased towards certain people because the data is biased? Should robots have any rights under the law or not? What will happen to people’s jobs if Machine Learning becomes more widespread? Arise in people’s minds, and thus, it is necessary to control AI.

There have already been many controversies relating to AI Ethics. The most popular one was when Amazon found out that their Machine Learning based recruiting algorithm was biased against women in 2018. This is because it was trained on data where most candidates were men, so the algorithm also favored men over women.

17.5 AI Engineering

AI is now entering the field of engineering and creating its own profession. This is very important for the development as integration of Artificial Intelligence and Machine Learning in the industry has been very ad-hoc and haphazard without any regulations of best practices. That’s why Gartner even predicts that only 53% of AI and ML projects complete the journey from a prototype to full production in a company while the rest, 47%, face failure.

Applying AI engineering in a company can bring great performance, reliability, and scalability from a machine learning algorithm which ensures a return on the investment in AI. Also, this can create a focus on concepts like DataOps, ModelOps, DevOps, etc.

17.6 AI-driven Biometric Security Solution

AI is highly preferred when it comes to security. One of the things like Bio ID, which was science fiction back then, has come to reality. This has taken security to an advanced level. Machine Learning’s efficient approach to gathering, processing, and analyzing large data sets can improve the performance of your biometric systems. Biometrics largely depends on quickly matching tasks and accuracy; that’s why ML is integrated here as it is perfect for these tasks.

Nowadays, every phone has a fingerprint lock and face lock; these things result from the ML algorithm that allows us to ensure our security at a personal level. Also, another great application is Amazon’s Alexa, which can now tell who is speaking by comparing the speaker to a predetermined voice profile. No extra hardware is necessary to help a properly trained neural network to accurately identify the speaker.

In the future, much such biometrics will combine with ML to give us more advanced features, and we will be able to build a great and more advanced security system.

Fig 17.4 AI Biometrics

17.7 AI Analysis for the business forecast

The buzzed feature here is ML-based time series analysis. This is a technique that quickly analyses a sequence of data. When used correctly, it aggregates data and analyzes it to allow managers to easily make decisions based on their data. By using this in business, companies can make the best decisions in time. It gives high accuracy forecasts; approximately 90-95% are accurate. It captures features such as seasonality and cross-correlation in demand forecasting for retail.

Apart from forecasting the future, another technology could be widely applied: anomaly detection based on autoencoders that run artificial neural networks using unsupervised learning algorithms. These systems are capable of capturing common patterns while ignoring “noise.”

Fig 17.5 AI and Business

17.8 Companies dependent on ML

•Pinterest.

•Twitter.

•Facebook.

•Google.

•Baidu.

•IBM.

•InData Labs

•Indium Software

•MobiDev

•Fayrix

•Netguru

•iTechArt Group

•DogTown Media

•Xicom Technologies

•Altoros

•Neoteric

•Infopulse

•AntWorks

•AlescoData

•Iflexion

•HiddenBrains

These companies rely on machine learning to gain various business benefits, manufactures, sales, etc. Due to ML’s advantages, it has been rapidly increasing, especially in business or in the marketing field.

17.9 Summary

In this chapter, we learned about some of the emerging trends of machine learning. This is how machine learning is changing the world and making it more advanced. Leaving traditional methods behind, the world has gained new heights of technology with the help of AI, Machine learning, Big data, Data mining, and many more. Even we have turned our fiction or imaginary ideas into reality that makes it quite fascinating.

17.10 Exercise

1. List down the emerging trends in machine learning

2. How is ML used in IoT?

3. What do you mean by IoT?

4. Biometrics include body parts such as?

5. Why is ML used in these?

6. What is AI engineering?

7. What are the roles of an AI engineer?

8. Why is AI ethics introduced?

9. How is ML contributing to Cybersecurity?

10. How does time series analysis help in business?

References

H. Nguyen and H. La, “Review of Deep Reinforcement Learning for Robot Manipulation,” in 2019 Third IEEE International Conference on Robotic Computing (IRC), Naples, Italy, 2019, pp. 590–595.

The primary research area relating to nearest neighbor methods is the problem of storage, data reduction, and rapid calculation of nearest neighbors. A search on “nearest neighbor search” or “condensed nearest neighbors” or “edited nearest neighbors” will return a number of references.

Theory: Devroye, Gyorfi and Lugosi, A Probabilistic Theory of Pattern Recognition, 1996

David Scott, Multivariate Density Estimation, 1992

Theory: Devroye and Lugosi, Combinatorial Methods in Density Estimation, 2001

Hastie et al., Bishop, and Duda et al. all have chapters on LDA, logistic regression, and other linear classifiers.

The first comprehensive treatment and still a standard reference: Breiman, Friedman, Olshen and Stone, Classification and Regression Trees, 1984

The other standard reference is Quinlan, J. R. C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, 1993.

A somewhat recent survey of research on decision trees: Sreerama K. Murthy: Automatic Construction of Decision Trees from Data: A Multi-Disciplinary Survey. Data Min. Knowl. Discov. 2(4): 345-389 (1998)

Ripley has a nice chapter on decision trees -- probably the best place to start.

On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation, Gavin C. Cawley, Nicola L. C. Talbot; JMLR 11(Jul):2079-2107, 2010. Discussion of how certain model selection strategies are more biased than others; essential reading if you are doing comparative studies of different machine learning methods.

Descent Methods for Tuning Parameter Refinement, Alexander Lorbert, Peter Ramadge; AISTATS 2010. A natural idea.

An entry-level discussion of the bootstrap, cross-validation and other error estimates is given in Efron and Tibshirani, An Introduction to the Bootstrap, 1993.

Adaboost was first developed in Freund and Schapire, A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119-139, 1997.

A simpler proof the Adaboost’s weak learning property is given in Robert E. Schapire and Yoram Singer. Improved boosting algorithms using confidence-rated predictions. Machine Learning, 37(3):297-336, 1999.

The view of Adaboost as performing functional gradient descent was observed by a number of researchers in the late ‘90s and early 00’s. A representative work is L. Mason, J. Baxter, P. L. Bartlett, and M. Frean. Functional gradient techniques for combining hypotheses. In A. J. Smola, P. L. Bartlett, B. SchKopf, and D. Schuurmans, editors, Advances in Large Margin Classifiers, pages 221-246. MIT Press, 2000.

Logitboost was introduced in Friedman, J.H., Hastie, T., Tibshirani, R.: Additive logistic regression: a statistical view of boosting. Annals of Statistics 28, 337-407 (with discussion) (2000).

Empirical Bernstein Boosting, Pannagadatta Shivaswamy, Tony Jebara; AISTATS 2010.

Figure Resources

Fig. 17.1. https://data.embeddedcomputing.com/uploads/articles/wp/9145/files-aHViPTYzODY3JmNtZD1pdGVtZWRpdG9yaW1hZ2UmZmlsZW5hbWU9aXRlbWVkaXRvcmltYWdlXzVkNDFhZDRjMDBkNDkucG5nJnZlcnNpb249MDAwMCZzaWc9NjUyYTNiMGZjYjQyNzM3ZDM5YmZmMzFiODc4Njk5MTQ253D

Fig. 17.2. https://docs.microsoft.com/en-us/azure/machine-learning/media/tutorial-auto-train-models/flow2.png

Fig. 17.3. https://www.usmsystems.com/wp-content/uploads/2020/06/Share-of-organizations-that-rely-on-AI-for-cybersecurity-in-selected-countries-by-industries.jpg

Fig. 17.4. https://fcw.com/-/media/GIG/EDIT_SHARED/Cyber/Biometrics/biometrics_shutterstock_788001742.jpg

Fig. 17.5. https://www.datamation.com/wp-content/uploads/2020/12/artificial-intelligence-in-business-using-ai-in-your-company_5fce99dd0551c.png

Chapter 18. Model building in ML

Abstract

In this chapter, we will learn to build a Machine learning model. In the previous chapters, we learned various algorithms and models which carry out the machine learning process. In this chapter, we will build an ML model and see how the process is successfully carried out. Here you will come to know exactly how an ML model is built and what is it processes. The process requires patience and hard work, but indeed it is worth it. One can easily build an ML model following this process. So, let’s learn and build an ML model.

18.1 Defining the problem

This is the foremost step where you need to define your problem. It is a very important step where you need to figure out your problem exactly and accordingly find out the expected input and output for cross-checking your model. You should be clear with the following questions:

What is the main objective? What are we trying to predict?

What are the target features?

What is the input data? Is it available?

What kind of problem are we facing? Binary classification? Clustering?

What is the expected improvement?

What is the current status of the target feature?

How is going to be measured the target feature?

Also, the fact that we are predicting certain things here cannot always be true or correct as first, we need to run the model, then only we can give exact predictions or decisions. Keep in mind that machine learning can only be used to memorize patterns that are present in the training data, so we can only recognize what we have seen before.

Moreover, you may not find a solution to every problem as certain problems can be solved only when the models work. We should have data that will satisfy all the needs required for learning various relationships between the inputs and outputs.

Hence in this step, you need to be clear about things and understand the main motto of building this model. So that you can plan your next step accordingly, which involves collecting data; based on this step’s information, further, the data needs to be collected; that’s why one needs to be confident and clear with the requirements.

18.2 Collecting Data

Here the main work starts, and we take our first step towards building a model. Our model is dependent on the data we put. A rich data set is the key to a better and efficient model. If you provide very good and suitable data, it will build an efficient model. This is just like building a strong foundation; if the foundation is strong, half of the work finishes itself.

Techniques that can be used to collect data include Web scraping, where bots are used to extract content and data from a website. Unlike screen scraping, which only copies pixels displayed onscreen, web scraping extracts underlying HTML code and, with it, data stored in a database.

After the collection, you need to build a database or a table. The ideal format followed for generating a table looks like this:

Fig 18.1 Data table format

Here, each row represents a different Boston neighborhood, and each column indicates some characteristic of the neighborhood (criminality rate, average, etc.). The last column represents the median house price of the neighborhood, and it is the target, the one that will be predicted, taking into account the other.

After this step, make sure your data is ready and useful, consisting of all the attributes needed. Before inputting this data into the model, you need to first determine the success. Let’s see below what does it mean.

18.3 Measure of success

In this step, you need to measure or determine what success means to you, i.e., which result is good or useful. It is said that “If you can’t measure it, you can’t improve it.” As you know, a model in its training period learns and improves itself. But for doing that, it should be made clear that which result means success and which does not. Hence it is important to figure out what exactly is your required output. To improve a model, you need to measure it in terms of a pass and fail. This is why you need to determine the measure of success.

In some cases, success can mean precision, accuracy, Customer-retention rate, etc. You need to figure what it means to your requirement. Then only can you measure the improvement of your model? This measure is also related to the kind of problems you are dealing with, for example:

1. Regression problems use certain evaluation metrics such as mean squared error (MSE).

2. Classification problems use evaluation metrics as precision, accuracy, and recall.

Hence, you need to first figure out your success as there are models that work on a reward basis, which will require this for taking the training in the right direction.

18.4 Evaluation Protocol

After your goal is clear, i.e., you have figured out your success, it is now time to work on how the measuring will be done. When a model achieves a result or is in the process, you will have to measure it and accordingly give the result and improve it. For this purpose, you need to follow some protocols mentioned below:

•Hold Out Validation Set

•K-fold Validation

•Iterated K-Fold Validation

18.4.1 Hold out Validation Set

Here in this protocol, you need to maintain a hold-out validation set. That means you need to put a part of the data aside for the final testing or validation. The process is to train the model with the remaining fraction of the data, tunning its parameters with the validation set, and evaluate its performance on the test set.

This is just like feeding the data part by part. In the initial training, if a large amount of data is given to a model, it can bring various problems like loss or data leak. As it is not yet trained, you cannot expect it to work with a large data set.

Although this protocol sounds perfect, it can be useless for a small data set as first only the data will be small, and dividing it into two parts will make it smaller, i.e., it will contain so few samples that tuning and evaluation processes of the model will not be effective. Below you can see how the data is divided into two parts and evaluated according.

Fig 18.2 Hold out validation set

18.4.2 K-Fold Validation

This is another type of protocol wherein the data is split, but instead of two partitions here, it is split into k partitions of equal size. Accordingly, the training of the model is conducted where for each partition i, it is trained with the remaining K-1 partitions, and it is evaluated on partition i. The average score of the K score is considered to be the final score.

This technique is used when the performance of the model is significantly different from the train-test split—also used to estimate the skill of a machine learning model on unseen data. This is a popular method because it is simple to understand as it gives results that are a less biased or less optimistic estimate of the model skill than other methods.

1. This method can be done by the following procedure :

2. First, randomly shuffle your dataset

3. Then split it into k groups

•Each group is unique and for each group, do the following things:

•Take the group as a hold-out data set.

•Take the remaining groups as a training data set

•Fit a model on the training set and evaluate it on the test set

4. Retain the evaluation score and discard the model

Now you can summarize the skill of your model by using the sample of model evaluation scores.

The most important thing to remember is that each observation in a data set is assigned to an individual group. It stays in that group only till the duration of the procedure. This can be interpreted as each sample is used in the hold-out set for one time but is used to train the model k-1 times.

Here is the picture which helps you understand how the data set is categorized in k fold.

Fig 18.3 K-Fold dataset

18.4.3 Iterated K-Fold Validation

This technique can overcome the drawback of the Holdout validation set, which was not suitable for a small amount of data. Iterated K-Fold validation is most appropriate when you have a small amount of data available for evaluating the model as precisely as possible.

From the name itself, you can guess that it is just like K-Fold, but here the K-fold is applied several times. Also, before the data is split into K partitions, it is shuffled every time. The average of the scores obtained at the end of each K-Fold validation run is considered the final score.

Though it can solve small data, it can be very expensive as the number of trained and evaluating models would be I x K times, where I is the number of iterations and K is the number of partitions.

Below is what the structure of a data set in Iterative K fold looks like:

Fig 18.4 Iterative K Fold data set

18.4.4 Choosing Evaluation Protocol

Above are the few evaluation protocols, and we need to choose one out of them. However, while choosing any of the protocols, there are few things necessary to keep in mind.

1. When it comes to classification problems, both training and testing data should represent the data; hence, we should shuffle our data before splitting it to make sure that it covered the whole dataset.

2. When you want to predict the future given the past, for example, weather prediction, stock price prediction, etc., the data should not be shuffled, as the data sequence is a crucial feature, and doing so would create a temporal leak.

3. Always check if there are duplicates in your data to remove them. Otherwise, vthe redundant data may appear both in the training and testing sets and cause inaccurate learning on our model.

18.5 Preparing the Data

Now it’s time to prepare our data for the model. It should be converted in a format or language our model will understand and is suitable for the process. There are few techniques to do this :

•Missing data

•Categorial Data

•Feature Scaling

•Meaningful Features

•Subsets

Let’s explore each and every technique which will make our data most suitable in other words, it will refine our data.

18.5.1 Missing data

Managing a large amount of data is indeed a hectic task. Moreover, there are chances of data loss. Errors in the data collection, blank spaces on surveys, measurements not applicable, etc., can cause data loss. Every large dataset definitely consists of such missing values.

The missing data in a database is valued as NaN or null. The problem is that every algorithm cannot handle these values. To avoid the problems while the model is in the training period, we need to take care of these missing values and feed them into the model.

To do this, first, you need to identify the missing values from your dataset. Once this is done now, it’s time to handle them either you can eliminate the features with missing values, but if there are too many, it will affect your data and can lead to more data loss. So there is a second way, i.e., to input the missing values. For this, you can use some pre-built estimators, such as the Imputer class from scikit learn. Or else you can set the missing values as the mean value of the rest of the samples.

By these techniques, you can solve your problem of missing data and make your data more suitable for feeding it to the model so that the model doesn’t face any difficulty in training which it cannot solve and can become a failure.

Fig 18.5 Missing data

18.5.2 Categorical Data

Categorical Data is also a thing you should handle with your data. While we deal with categorical data, there are two types of features we focus on, i.e., Ordinal and Nominal features. Ordinal features are categorical features that can be sorted just like cloth’s size L<M<S. In comparison, Nominal features don’t have any order, just like clothes color, red, yellow, etc.

These features need to be made clear. There are two methods you can apply to deal with both features.

1. Mapping of ordinal features: Here, we need to convert the categorical string values into integers to ensure that the algorithm interprets the original features correctly. This making is generally done manually. To understand this, let’s say there are values like L, M, S. You can convert these values into 2, 1, 0.

2. Enoding nominal class labels: This is the most common method to perform one-hot encoding that involves creating a new dummy feature for each unique value in the nominal feature column. Let’s take an example to understand this. Suppose you have a color column with three classes Yellow, Red, Green. Performing one–hot encoding will give us three new columns for each class. Then if we have two red shirts, the two rows can be valued as 1 while others can be 0. This will ensure the good performance of the algorithm. Below is its pictorial representation.

Fig 18.6 Example of categorical data

18.5.3 Feature Scaling

This is the most important step when we are in preprocessing phase. It is observed that most of the machine learning algorithms work best when they are dealing with features that are on the same scale. Hence it is better to perform feature scaling in your data to do these; there are few techniques :

• Normalization

In this, the features are rescaled to a range of 0,1. This is called min-max scaling. To do this, we need to just apply the min-max scaling method to every feature column.

Fig 18.7 Min Max scaling formula

Below is the image of data how it looks after normalization:

Fig 18.8 Data after normalization

• Standardization

In this, the feature columns are centered at mean 0 with a standard deviation 1. This will ensure that the feature column has the same parameters as a standard normal distribution. By doing this, it will be easier for the algorithm to learn the weights of the parameters. Also, it will keep useful information regarding the outliers and make it less sensitive towards them. Here is the formula by which you can standardize your data:

Fig 18.9 Standardization Formula

18.5.4 Meaningful Features

Most of the time, the data overfits. The main reason behind this is data redundancy. Redundant data makes the model more complex for the given training data and makes it unable to generalize unseen data.

To avoid overfitting, the most common solution is to the dimensions of the data. This can be done by reducing the number of features of our dataset via Principal Component Analysis (PCA), a type of Unsupervised Machine Learning algorithm. It focuses on identifying patterns in our data based on the correlations between the features. This correlation tells us that some parts of data can be explained with other parts of it.

This correlation can be removed later as it has nothing to do with the model’s way of learning the weights. It can be removed by directly eliminating certain columns (features) or combining a number of them and getting new ones that hold the most part of the information.

Below is the example of PCA how its graphical representation looks like after reducing the dimensions:

Fig 18.10 PCA

18.5.5 Subsets

Here we split our data into subsets. Generally, as seen above, it is split into three parts: Training set, Testing set, and validation sets. We split our data to train our model in three phases, i.e., training it on training data, then in the validation set, it gets evaluated, and at last, testing is done on testing data.

Though it sounds that there is no need for three sets, but it is necessary to have three sets instead of two as developing a model involves tunning its configuration, in other words, choosing certain values for their hyperparameters which are different from the parameters of the model that is called network’s weights. This tunning is done with the feedback received from the validation set, and is in essence, a form of learning. Our goal is to generalize the model well on unseen data and predict accurate results from new data based on training and validation parameters.

To carry out these tasks, we need to understand some concepts, i.e., we need to learn a process and solve the problem of Overfitting and Underfitting.

Let’s understand how the learning process is done by studying one of the simplest algorithms: Linear Regression.

In linear regression, here, we are given a number of predictor (explanatory) variables and a continuous response variable (outcome), and we have to find a relationship between those variables that allows us to predict a continuous outcome. For example, let’s say we are given X and Y. We will fit a straight line that will minimize the distance using some methods to estimate the coefficients like Ordinary Least Squares and Gradient Descent between the sample points and the fitted line. Then, we will use the intercept and slope learned that form the fitted line to predict the outcome of new data.

Here is the formula for calculating straight lines:

y = B0 + B1x + u

Where,

x = input

B1 = slope

B0 = y-intercept

u = the residual

y = value of the line at the position x.

The values of B0 and B1 will be trained and can affect the position of the line as the only other variables are x the input and y the output (the residual is not considered). These values (B0 and B1) are the weights of the predicting function. These are the values that are called biases which will be later arranged together as matrixes (W for the weights and b for the biases).

The training consists of initializing some random values for each of the training matrixes and attempting to predict the input data’s output using the initial random values. Initially, the errors can be large. Gradually when you compare the model’s prediction with the correct output, the model can adjust the weights and biases values until having a good predicting model. This process is iterated as one iteration at a time. In each iteration, the initial random line moves closer to the ideal and more accurate one.

Fig 18.11 Process of linear regression

After learning the process, let’s solve the problem of overfitting and underfitting. The tension between optimization and generalization is also a problem that generally comes. Here optimization is the process of adjusting a model to get the best performance possible on training data, whereas generalization is how well the model performs on unseen data.

In the initial phase of training, these issues are correlated; the lower the loss on training data, the lower the loss on test data. This happens while the model is still under-fitted: there is still learning to be done; it hasn’t been modeled all the relevant parameters of the model yet. Still, after a number of iterations on the training data, generalization stops to improve, and the validation metrics freezes and then start to degrade. Now in this phase model starts to overfit, i.e., it has learned so well the training data that has learned patterns that are too specific to training data and irrelevant to new data.

To avoid these problems in our model, we can perform two techniques, i.e., getting more data and regularization.

Collecting more data is considered the best option as it is observed that a model trained on large data will automatically generalize better from that of smaller data. But if this is not possible, you can go for regularization where modulation of the quantity of information that the model can store or add constraints on what information it is allowed to keep. Here the optimization is dependent on memorizing the power of the model. If the model can only memorize a small number of patterns, the optimization will make it focus on the most relevant ones, improving the chance of generalizing well.

Regularization can be done by following techniques:

1. Reducing the size of the model: Reducing the number of learnable parameters in the model and its learning capacity. The goal is to get to a sweet spot between too much and not enough learning capacity. Unfortunately, there aren’t any magical formulas to determine this balance; it must be tested and evaluated by setting a different number of parameters and observing its performance.

2. Adding weight regularization: It is better to keep the model simple. As long it can learn well, a simpler model is much less likely to overfit. A common way to achieve this is to constraint the complexity of the network by forcing its weights to only take small values, regularizing the distribution of weight values. This is done by adding to the loss function of the network a cost associated with having large weights. The cost comes in two ways:

The first one is called L1 regularization, where the cost is proportional to the absolute value of the weights coefficients, and the second one is called L2 regularization, where the cost is proportional to the square of the value of the weight coefficients.

In the case of L1 regularization, the computational is inefficient in non-sparse cases. It gives parts outputs and has a built-in feature selection. Whereas L2 gives efficient computational as it has analytical solutions, doesn’t give sparse outputs, and has no built-in feature selection.

Fig 18.12 L1 and L2 regularization formula

By keeping all the above things in mind, you can choose to apply one of them to your model. First, determine your problem and then choose what will be suitable for the betterment of your model. These problems are very crucial and are necessary to solve them. Make sure you take all the considerations and then decide which technique can be best, as the success of your model will totally depend on the techniques and algorithms you select.

18.6 Benchmark model

From the beginning, our goal is to build a perfect and well-functioning model. In this step, you need to focus on building a model that will set a benchmark. This model will help us to measure the performance of a better and more attuned algorithm. This will work like a baseline on which we can compare our model.

Benchmarking focuses on experiments to be comparable, measurable, and reproducible. It is important to emphasize the reproducible part of the last statement. Data science libraries perform random splits of data; this randomness must be consistent through all runs. Most random generators support setting seeds for this purpose. In Python, we will use the random Seed method from the random package. It is beneficial to compare model improvement over a simplified baseline model such as a kNN or Naive Bayes for categorical data or the EWMA of a value in time series data. These baselines provide an understanding of the possible predictive power of a dataset.

These models are perfect for cross-checking our model as it requires less time and computes power to train and predict. Models like KNN or Naïve Bayes cannot capture complex interactions but can provide us reasonable estimate of the minimum bound of predictive capabilities of a benchmarked model. This technique allows us to test the benchmarking pipeline. Benchmark pipeline must be able to provide stable results for a model with understood performance characteristics. Models like kNN or Naive Bayes on a raw dataset generally provides a weak but adequate learner. Also, it will have characteristics that are useful for comparison. The characteristics of more complex models may be less understood and prove challenging.

Hence it is important to build a perfect benchmark model for great results, just like a person’s characteristics depend more on its mentoring from parents and teachers.

18.7 Tunning the hyperparameters

In building an ML model, this will be the last step where you will give a final finish to your model, and the coding part comes here. Let’s see the further procedure:

18.7.1 Finding a good model

It is very important to choose the right model. For doing this, the most common method is cross-validation. There are few things we need to set in our cross-validation data set:

1. A number of folds in which we will split our data.

2. A scoring method (that will vary depending on the problem’s nature — regression, classification).

3. Some appropriate algorithms that we want to check.

After all these things are done, we will pass our data set cross-validation score function and get the model that yielded the best score. That will be the dataset we will optimize, tune its hyperparameters accordingly.

The following is the coding you need to follow in scratch for the above process: (https://towardsdatascience.com/machine-learning-general-process-8f1b510bd8af)

# Test Options and Evaluation Metrics

num_folds = 10

scoring = “neg_mean_squared_error”

# Spot Check Algorithms

models = []

models.append((‘LR’, LinearRegression()))

models.append((‘LASSO’, Lasso()))

models.append((‘EN’, ElasticNet()))

models.append((‘KNN’, KNeighborsRegressor()))

models.append((‘CART’, DecisionTreeRegressor()))

models.append((‘SVR’, SVR()))

results = []

names = []

for name, model in models:

kfold = KFold(n_splits=num_folds, random_state=seed)

cv_results = cross_val_score(model, X_train, y_train, cv=kfold, scoring=scoring)

results.append(cv_results)

names.append(name)

msg = “%s: %f (%f)” % (name, cv_results.mean(), cv_results.std())

print(msg)

The output of the following looks like this:

Fig 18.13 Output

In order to compare the algorithms we need to run a piece of code which is :

# Compare Algorithms

fig = pyplot.figure()

fig.suptitle(‘Algorithm Comparison’)

ax = fig.add_subplot(111)

pyplot.boxplot(results)

ax.set_xticklabels(names)

pyplot.show()

This is how an algorithm comparison looks like

Fig 18.14 Algorithm Comparison

After selecting a suitable model for your data set, it is time to tune its hyperparameters. Let’s learn how it is done in the next step:

18.7.2 Tunning hyperparameters

Generally, a machine learning model has two types of parameters:

1. The parameters are learned throughout the training phase.

2. Hyperparameters are those parameters that we pass to our ML model.

After successfully identifying the correct model you will use, the next step is to tune the hyperparameters that we will pass to our models. One of the common ways to find the best combination of hyperparameters is called Grid Search Cross-Validation.

For tuning your hyperparameters, you should follow the following procedure:

1. Set the parameter grid that we will evaluate. We will create a dictionary of all the parameters and their corresponding set of values that you want to test for best performance.

2. Set the number of folds and the random state, and a scoring method.

3. Build a K-Fold object with the selected number of folds.

4. Build a Grid Search Object with the selected model and fit it.

# Build a scaler

scaler = StandardScaler().fit(X_train)

rescaledX = scaler.transform(X_train)

# Build parameter grid

c_values = [0.1, 0.3, 0.5, 0.7, 0.9, 1.0, 1.3, 1.5, 1.7, 2.0]

kernel_values = [‘linear’, ‘poly’, ‘rbf’, ‘sigmoid’]

param_grid = dict(C=c_values, kernel=kernel_values)

# Build the model

model = SVC()

kfold = KFold(n_splits=num_folds, random_state=seed)

grid = GridSearchCV(estimator=model, param_grid=param_grid, scoring=scoring, cv=kfold)

grid_result = grid.fit(rescaledX, y_train)

# Show the results

print(“Best: %f using %s” % (grid_result.best_score_, grid_result.best_params_))

means = grid_result.cv_results_[‘mean_test_score’]

stds = grid_result.cv_results_[‘std_test_score’]

params = grid_result.cv_results_[‘params’]

for mean, stdev, param in zip(means, stds, params):

print(“%f (%f) with: %r” % (mean, stdev, param))

The output will look like this:

Fig 18.15 Output

After executing the above program, it will return a set of hyperparameters that fit best with the problem at hand. Once they are determined, our model is ready to be used. So we will make the appropriate predictions on the validation dataset and save the model for later use.

This is how a model is generated and deployed. And give us outputs by which we can predict the results required for our business or any of the purposes. Let’s see few real-life examples of a model:

1. For example, you have to decide whether you should wear a jacket or not according to the weather. So here, we will check the temperature outside. And below is the table for the same.

Outside Temperature

Wear a Jacket

30°C

No

25°C

No

20°C

No

15°C

Yes

10°C

Yes

1. Here you can see the connection between the input (temperature) and the output (decision to wear a jacket).

2. Accordingly, if the temperature is 12°C, you would still wear a jacket, although you were never told the outcome for that particular temperature.

3. For solving this problem, we will build a model.

4. For saving time, we will install scikit learn as it provides easy to use functions and predefined models, which saves a lot of time

5. To generate the training set, we will follow the code: (https://towardsdatascience.com/simple-machine-learning-model-in-python-in-5-lines-of-code-fe03d72e78c6)

from random import randint

TRAIN_SET_LIMIT = 1000

TRAIN_SET_COUNT = 100

TRAIN_INPUT = list()

TRAIN_OUTPUT = list()

for i in range(TRAIN_SET_COUNT):

a = randint(0, TRAIN_SET_LIMIT)

b = randint(0, TRAIN_SET_LIMIT)

c = randint(0, TRAIN_SET_LIMIT)

op = a + (2*b) + (3*c)

TRAIN_INPUT.append([a, b, c])

TRAIN_OUTPUT.append(op)

7. Our training data set will look like below after converting it to an algebraic problem

Fig 18.16 Training Data set

8. Here it is clearly visible that Output Y is equal to (x1 + 2*x2 + 3*x3).

9. Now its time to train our model using Linear Regression model and pass it the training data.

10. Here is the code for that

from sklearn.linear_model import LinearRegression

predictor = LinearRegression(n_jobs=-1)

predictor.fit(X=TRAIN_INPUT, y=TRAIN_OUTPUT)

11. After your model is successfully trained to give it test data, for example, here, the X = [[10, 20, 30]], the output should be 140 as per the training data set.

12. Here is the code for that :

X_TEST = [[10, 20, 30]]

outcome = predictor.predict(X=X_TEST)

coefficients = predictor.coef_

print(‘Outcome : {}\nCoefficients : {}’.format(outcome, coefficients))

This is what the output will look like :

Outcome : [ 140.]

Coefficients : [ 1. 2. 3.]

Here the model had access to the training data, through which it calculated the weights to assign to the inputs to arrive at the desired output. When you gave the test data, it accordingly calculated the output.

18.8 Summary

In this chapter, we learned how to successfully build a machine learning model using scratch. Scratch can be the best for beginners who want to build ML models. This is how a model is built and trained on a training data set and tested before actually deploying it for future use. Let’s summarize the whole process once again:

1. Define our problem adequately

2. Gather data.

3. Choose a measure of success.

4. Set an evaluation protocol and the different protocols available.

5. Prepare the data (dealing with missing values, with categorical values)

6. Split the data correctly.

7. Differentiate between over and underfitting

8. Develop a benchmark model.

9. Choose an adequate model and tune it to get the best performance possible.

These are the steps you need to follow when you are building an ML model. This chapter also gave us some examples of ML models with their coding by which one can practice an ML model.

Hence, we have learned everything about big data, data mining, and Machine learning and conclude the book.

18.9 Exercise

1. What do you mean by the ML model?

2. What is the foremost step while building an ML model?

3. For training, the data set should be large or small?

4. Why is the data divided into parts?

5. What do you mean by subset?

6. Explain K-fold

7. Explain iterative K-fold

8. Which protocol is suitable for small data set?

9. What do you mean by the Benchmark model?

10. Why is the Benchmark model build?

11. What do you mean by the measure of success?

12. What do you mean by ordinal features?

13. Explain Feature Scaling.

References

H. Nguyen and H. La, “Review of Deep Reinforcement Learning for Robot Manipulation,” in 2019 Third IEEE International Conference on Robotic Computing (IRC), Naples, Italy, 2019, pp. 590–595.

The primary research area relating to nearest neighbor methods is the problem of storage, data reduction, and rapid calculation of nearest neighbors. A search on “nearest neighbor search” or “condensed nearest neighbors” or “edited nearest neighbors” will return a number of references.

Theory: Devroye, Gyorfi and Lugosi, A Probabilistic Theory of Pattern Recognition, 1996

David Scott, Multivariate Density Estimation, 1992

Theory: Devroye and Lugosi, Combinatorial Methods in Density Estimation, 2001

Hastie et al., Bishop, and Duda et al. all have chapters on LDA, logistic regression, and other linear classifiers.

The first comprehensive treatment and still a standard reference: Breiman, Friedman, Olshen and Stone, Classification and Regression Trees, 1984

The other standard reference is Quinlan, J. R. C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, 1993.

A somewhat recent survey of research on decision trees: Sreerama K. Murthy: Automatic Construction of Decision Trees from Data: A Multi-Disciplinary Survey. Data Min. Knowl. Discov. 2(4): 345-389 (1998)

Ripley has a nice chapter on decision trees -- probably the best place to start.

On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation, Gavin C. Cawley, Nicola L. C. Talbot; JMLR 11(Jul):2079-2107, 2010. Discussion of how certain model selection strategies are more biased than others; essential reading if you are doing comparative studies of different machine learning methods.

Descent Methods for Tuning Parameter Refinement, Alexander Lorbert, Peter Ramadge; AISTATS 2010. A natural idea.

An entry-level discussion of the bootstrap, cross-validation and other error estimates is given in Efron and Tibshirani, An Introduction to the Bootstrap, 1993.

Adaboost was first developed in Freund and Schapire, A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119-139, 1997.

A simpler proof the Adaboost’s weak learning property is given in Robert E. Schapire and Yoram Singer. Improved boosting algorithms using confidence-rated predictions. Machine Learning, 37(3):297-336, 1999.

The view of Adaboost as performing functional gradient descent was observed by a number of researchers in the late 90’s and early ‘00s. A representative work is L. Mason, J. Baxter, P. L. Bartlett, and M. Frean. Functional gradient techniques for combining hypotheses. In A. J. Smola, P. L. Bartlett, B. SchKopf, and D. Schuurmans, editors, Advances in Large Margin Classifiers, pages 221-246. MIT Press, 2000.

Logitboost was introduced in Friedman, J.H., Hastie, T., Tibshirani, R.: Additive logistic regression: a statistical view of boosting. Annals of Statistics 28, 337-407 (with discussion) (2000).

Empirical Bernstein Boosting, Pannagadatta Shivaswamy, Tony Jebara; AISTATS 2010.

Figure Resources

Fig. 18.1. https://miro.medium.com/max/875/0*mudXP5rCV2dph8gF

Fig. 18.2. https://www.datavedas.com/wp-content/uploads/2018/04/image001.jpg

Fig. 18.3. https://miro.medium.com/max/803/0*IgD7gdJ9BdQ-QjjS

Fig. 18.4. https://raw.githubusercontent.com/satishgunjal/images/master/KFold_Cross_Validation.png

Fig. 18.5. https://miro.medium.com/max/700/1*MiJ_HpTbZECYjjF1qepNNQ.png

Fig. 18.6. https://i.imgur.com/TW5m0aJ.png

Fig. 18.7. https://miro.medium.com/max/381/0*pVqd_DjfuvdUN5mc

Fig. 18.8. https://kharshit.github.io/img/scaling.png

Fig. 18.9. https://miro.medium.com/max/413/0*zNH1OCCJKF6u_57Y

Fig. 18.10. https://dezyre.gumlet.net/files.dezyre.com/images/Tutorials/Principal+Component+Analysis.jpg?w=750&dpr=1.0

Fig. 18.11. https://miro.medium.com/max/875/0*7V9HGbEPbEKmHO7J

Fig. 18.12. https://miro.medium.com/max/688/0*ITv81egIxcJdJEbk

Fig. 18.13. https://miro.medium.com/max/299/1*mEFFP1HAUFAKmJuwBZSFJg.png

Fig. 18.14. https://miro.medium.com/max/501/0*Fb5SRh8CpUXAuCqe

Fig. 18.15. https://miro.medium.com/max/565/1*prb3pErQaAd6QmbpFRUWwA.png

Fig. 18.16. https://miro.medium.com/max/520/1*JJGA5nSu0L4Am65LDpjxCQ.png

Glossary

\Aggregation: A process of searching, gathering, and presenting data

Algorithms: A mathematical formula that can perform certain analyses on data

Analytics: The discovery of insights in data

Anonymization: Making data anonymous; removing all data points that could lead to identifying a person

Application: Computer software that enables a computer to perform a certain task

Artificial Intelligence: Developing intelligent machines and software capable of perceiving the environment, taking the corresponding action when required, and even learning from those actions.

Bayes theorem: It is used to calculate conditional probability. Conditional probability is the probability of an event ‘B’ occurring given the related event ‘A’ has already occurred.

For example, Let’s say a clinic wants to cure the cancer of the patients visiting the clinic.

A represents an event “Person has cancer.”

B represents an event “Person is a smoker.”

The clinic wishes to calculate the proportion of smokers from the ones diagnosed with cancer.

Bias: In a neural network, bias refers to the constant terms in the model. (Note that bias has a different meaning to most data analysts.) Also, see precision.

Behavioral Analytics: Analytics that informs about the how, why and what instead of just the who and when. It looks at humanized patterns in the da

Big Data: Big data is an all-encompassing term for any collection of data sets so large or complex that it becomes difficult to process them using traditional data-processing applications.

Biometrics: The use of technology to identify people by one or more of their physical traits.

Bayesian Statistics: Bayesian statistics is a mathematical procedure that applies probabilities to statistical problems. It provides people the tools to update their beliefs in the evidence of new data. It differs from the classical frequentist approach and is based on Bayesian probabilities to summarize the evidence.

CART - Classification And Regression Trees.

CART is a method of splitting the independent variables into small groups and fitting a constant function to the small data sets. In categorical trees, the constant function takes on a small finite set of values (e.g., Y or N, low or medium or high). In regression trees, the mean value of the response is fit to small connected data sets.

Classification analysis: A systematic process for obtaining important and relevant information about data, also metadata called; data about data.

Cloud computing: A distributed computing system over a network used for storing data off-premises

Clustering analysis: The process of identifying objects that are similar to each other and cluster them in order to understand the differences as well as the similarities within the data.

Complex structured data: Data that are composed of two or more complex, complicated, and interrelated parts cannot be easily interpreted by structured query languages and tools.

Computer-generated data: Data generated by computers such as log files, categorical data

Categorical data fits into a small number of discrete categories (as opposed to continuous). Categorical data is either non-ordered (nominal) such as gender or city or ordered (ordinal) such as high, medium, or low temperatures.

CHAID: An algorithm for fitting categorical trees. It relies on the chi-squared statistic to split the data into small connected data sets.

Chi-squared: A statistic that assesses how well a model fits the data. In data mining, it is most commonly used to find homogeneous subsets for fitting categorical trees as in CHAID.

Classification: Refers to the data mining problem of attempting to predict the category of categorical data by building a model based on some predictor variables.

Classification tree: A decision tree that places categorical variables into classes.

Cleaning (Cleansing): This refers to a step in preparing data for a data mining activity. Obvious data errors are detected and corrected (e.g., improbable dates), and missing data is replaced.

Clustering algorithms: It finds groups of items that are similar. For example, clustering could be used by an insurance company to group customers according to income, age, types of policies purchased, and prior claims experience. It divides a data set so that records with similar content are in the same group, and groups are as different as possible from each other. Since the categories are unspecified, this is sometimes referred to as unsupervised learning

Convergence: A state reached during the training of a model when the loss changes very little between each iteration.

Dashboard: Dashboard is an information management tool used to visually track, analyze and display key performance indicators, metrics, and key data points. Dashboards can be customized to fulfill the requirements of a project. It can be used to connect files, attachments, services, and APIs displayed in the form of tables, line charts, bar charts, and gauges. Popular tools for building dashboards include Excel and Tableau.

Data aggregation tools: The process of transforming scattered data from numerous sources into a single new one.

Data analyst: Someone analyzing, modeling, cleaning, or processing data

Database: A digital collection of data stored via a certain technique

Database Management System: Collecting, storing and providing access to data

Datacenter: A physical location that houses the servers for storing data

Data cleansing: The process of reviewing and revising data to delete duplicates, correct errors, and provide consistency

Data Transformation: Data transformation is the process of converting data from one form to the other. This is usually done at a pre-processing step.

Data mining: The process of finding certain patterns or information from data sets

Data modeling: The analysis of data objects using data modeling techniques to create insights from the data

Data set: A collection of data

Data virtualization: A data integration process to gain more insights. Usually, it involves databases, applications, file systems, websites, big data techniques, etc.)

De-identification: Same as anonymization; ensuring a person cannot be identified through the data

Data Mart: Part of data is stored in an organization that is focused on a single subject or department, such as Sales, Finance, or Marketing.

Data Warehouse: Central repository of data collected and/or stored by an enterprise’s various business systems.

Data Quality: The processes and techniques involved in ensuring the reliability and application efficiency of data. Data is of high quality if it reliably reflects underlying processes and fits the intended uses in operations, decision making, and planning.

Data frame: Data Frame is a 2-dimensional labeled data structure with columns of potentially different types. You can think of it as a spreadsheet or SQL table or a duct of Series objects. Data Frame accepts many different kinds of input

ETL - Extract, Transform, and Load: A process in a database and data warehousing meaning extracting the data from various sources, transforming it to fit operational needs, and loading it into the database

Feature: A piece of measurable information about something, for example, features you might store about a set of people, are age, gender, and income.

Fraud Detection: Identifying characteristics of suspicious fraudulent transfers, orders, and other illegal activities against an organization or company and designing relevant triggers in IT systems that raise warnings whenever such transactions are attempted or made.

Graph Databases: They use graph structures (a finite set of ordered pairs or certain entities) with edges, properties, and nodes for data storage. It provides index-free adjacency, meaning that every element is directly linked to its neighbor element.

GUI: Graphical User Interface.

Hadoop: An open-source framework that is built to enable the process and storage of big data across a distributed file system

HBase: An open-source, non-relational, distributed database running in conjunction with Hadoop

In-memory: A database management system stores data on the main memory instead of the disk, resulting in very fast processing, storing, and loading of the data

IoT - Internet of Things

The notion of a wide-ranging network of electronic devices of various kinds (personal, household-level, industrial) and purposes (healthcare, leisure, media consumption, shopping, manufacturing, environment control, etc.) exchanging data over the Internet, coordinating its activities with one another

JavaScript: A scripting language designed in the mid-1990s for embedding logic in web pages, but which later evolved into a more general-purpose development language.

Key-Value Databases: They store data with a primary key, a uniquely identifiable record, making it easy and fast to look up. The data stored in a Key-Value is normally some kind of primitive programming language.

Labeled data: It is a group of samples that have been tagged with one or more labels. Labeling typically takes a set of unlabeled data and augments each piece of it with informative tags.

Latency: A measure of time-delayed in a system

Logfile: A file automatically created by a computer to record events that occur while operational

Machine data: Data created by machines via sensors or algorithms

Machine learning: Part of artificial intelligence where machines learn from what they are doing and become better over time

Metadata: Data about data; gives information about what the data is about.

Multivalued Databases: They are a type of NoSQL and multidimensional databases that understand 3-dimensional data directly. They are primarily giant strings that are perfect for manipulating HTML and XML strings directly.

Market Basket Analysis: Identifying combinations of products or services that frequently co-occur in transactions, for example, products that are often purchased together. Results of such analysis are used to recommend additional purchases, inform decisions on placing products in relation to one another, etc.

Natural Language Processing: A field of computer science involved with interactions between computers and human languages

Network analysis: Viewing relationships among the nodes in terms of the network or graph theory, meaning analyzing connections between nodes in a network and the strength of the ties.

Neural networks: Structured to be similar to the human brain, this AI model incorporates natural language processing and deep learning to identify faces in photos and analyze handwriting.

NoSQL: NoSQL means Not only SQL. A NoSQL database provides a mechanism for storage and retrieval of data that is modeled in means other than the tabular relations used in relational databases. It can accommodate a wide variety of data models, including key-value, document, columnar, and graph.

Object Databases: They store data in the form of objects, as used by object-oriented programming. They are different from relational or graph databases, and most of them offer a query language that allows an object to be found with a declarative programming approach.

Object-based Image Analysis: Analysing digital images can be performed with data from individual pixels, whereas object-based image analysis uses data from a selection of related pixels, called objects or image objects.

Operational Databases: They carry out regular operations of an organization and are generally very important to a business. They generally use online transaction processing that allows them to enter, collect and retrieve specific information about the company

One hot encoding: It is a process by which categorical variables are converted into a form that could be provided to ML algorithms to do a better job in prediction.

OLAP - On-Line Analytical Processing: Tools enabling a user to easily make and browse reports summarizing relevant data, analyzing them from various perspectives.

Predictive Analytics: The practice of extracting information from existing data sets in order to determine patterns and predict future outcomes and trends. Applied to business, predictive models and analysis are used to analyze current data and historical facts in order to better understand customers, products, and partners and to identify potential risks and opportunities for a company.

Pattern Recognition: Identifying patterns in data via algorithms to make predictions of new data coming from the same source

Predictive analysis: Analysis within big data to help predict how someone will behave in the (near) future. It uses a variety of different data sets such as historical, transactional, or social profile data to identify risks and opportunities.

Privacy: To seclude certain data/information about oneself that is deemed personal.

Public data: Public information or data sets that were created with public funding

Query: Asking for information to answer a certain question

Re-identification: Combining several data sets to find a certain person within anonymized data.

Regression analysis: To define the dependency between variables. It assumes a one-way causal effect from one variable to the response of another variable.

Real-time data: Data that is created, processed, stored, analyzed, and visualized within milliseconds.

Scripting: The use of a computer language where your program, or script, can be run directly with no need to first compile it to binary code. Semi-structured data - a form of structured data that does not have a formal structure like structured data. It does, however, have tags or other markers to enforce the hierarchy of records.

Sentiment Analysis: Using algorithms to find out how people feel about certain topics

Semantic analysis: A more sophisticated form of Natural Language Processing, this concept is focused on the process of stringing words together as well as the way that language is understood through cultural context.

Similarity searches: Finding the closest object to a query in a database, where the data object can be of any type of data.

SQL: A programming language for retrieving data from a relational database.

Structured data: Data that is identifiable as it is organized in a structure like rows and columns.

Supervised learning: The collection of techniques where analysis uses a well-defined (known) dependent variable. All regression and classification techniques are supervised.

Statistics: It is the study of the collection, analysis, interpretation, presentation, and organization of data.

Stochastic Gradient Descent: Stochastic Gradient Descent is a gradient descent algorithm where we take a sample of data while computing the gradient. The update to the coefficients is performed for each training instance rather than at the end of the batch of instances.

Sparse data: A variable with sparse data is one in which a relatively high percentage of the variable’s cells do not contain actual data.

Support: The measure of how often the collection of items in an association occur together as a percentage of all the transactions. For example, “In 2% of the purchases at the hardware store, both a pick and a shovel were bought.”

Text Mining: The analysis of data contained in natural language text. It works by computing statistics for words and phrases in source data, thus expressing text structure in numerical terms and then analyzing it with traditional data mining techniques.

Tableau: A commercial data visualization package is often used in data science projects.

Terabytes: Approximately 1000 gigabytes.

Time series analysis: Analysing well-defined data obtained through repeated measurements of time. The data has to be well defined and measured at successive points in time spaced at identical time intervals.

Test Data: A data set independent of the training data set, used to fine-tune the estimates of the model parameters (i.e., weights).

Test Error: The estimate of error is based on the difference between the predictions of a model on a test data set and the observed values in the test data set when the test data set was not used to train the model.

Time Series: A series of measurements are taken at consecutive points in time. Data mining products that handle time series incorporate time-related operators such as moving average.

Time Series Model: A model that forecasts future values of a time series based on past values. The model form and training of the model usually take into consideration the correlation between values as a function of their separation in time.

Topology: For a neural net, topology refers to the number of layers and the number of nodes in each layer.

Training: Another term for estimating a model’s parameters based on the data set at hand.

Training Data: A data set used to estimate or train a model.

Transformation: A re-expression of the data such as aggregating it, normalizing it, changing its unit of measure, or taking the logarithm of each data item.

ubiquitous: existing or being everywhere at the same time: constantly encountered

Unlabeled data: It is a designation for pieces of data that have not been tagged with labels identifying characteristics, properties, or classifications. Unlabeled data is typically used in various forms of machine learning.

Un-structured data: Unstructured data is regarded as data that is in general text-heavy but may also contain dates, numbers, and facts.

Unsupervised Learning: This term refers to the collection of techniques where groupings of the data are defined without using a dependent variable. Cluster analysis is an example.

User Interface: This is the interface part between the user and the data mining system. It helps in fulfilling the requirements of the user from the data mining system. It helps in providing information relevant to the search pattern, helps in visualizing patterns in various forms, sorting out data based on need from different databases, and integrating it as per desired formats, etc.

Variability: It means that the meaning of the data can change (rapidly). In (almost) the same tweets, for example, a word can have a totally different meaning

Variety: Data today comes in many different formats: structured data, semi-structured data, unstructured data, and even complex structured data

Velocity: The speed at which the data is created, stored, analyzed, and visualized

Veracity: Ensuring that the data is correct as well as the analyses performed on the data are correct.

Visualization: Complex graphs that can include many variables of data while remaining understandable and readable

Volume: The amount of data, ranging from megabytes to brontobytes

Validation: The process of testing the models with a data set is different from the training data set.

Variance: The most commonly used statistical measure of dispersion. The first step is to square the deviations of a data item from its average value. Then the average of the squared deviations is calculated to obtain an overall measure of variability.

Visualization: Visualization tools graphically display data to facilitate a better understanding of its meaning. Graphical capabilities range from simple scatter plots to complex multi-dimensional representations.

Web Mining (Web Data Mining): The use of data mining techniques to automatically discover and extract information from Websites, documents and services.

Windowing: Used when training a model with time-series data. A window is a period used for each training case. For example, if we have weekly stock price data that covers fifty weeks, and we set the window to five weeks, then the first training case uses weeks one through five and compares its prediction to week six. The second case uses weeks two through six to predict week seven, and so on.

XML Databases: XML Databases allow data to be stored in XML format. The data stored in an XML database can be queried, exported, and serialized into any format needed.

Index

A

Abstract 3

Adaboost 117

Agglomerative Hierarchical Clustering 186

Aggregation 80

AI Engineering 207

AI Ethics 207

Ai-Driven Biometric 208

Aligning Unstructured 34

Apache Mahout 102

Artificial Intelligence 67

B

Bayesian Linear Regression 181

Benchmark Model 225

Big Data 3

Big Data Engineer 8

Bioinformatics 137

Business Insights 5

Business Intelligence Analyst 9

C

Career Scope 7

Cassandra 12

CDH (Cloudera Distribution For Hadoop) 12

Center Of Excellence 33

Classical Statistics 67

Classification 68

Classification Models 170

Cloud 6

Cloud Operating Model 34

Clustering 86, 70

Clustering Models 182

Corporate Surveillance 135

Customer Segmentation 133

D

Data Architect 8

Data Mining 66

Data Modeler 10

Data Scientist 9, 7

Data Warehouse Manager 9

Database Developer 10

Data-Driven Innovation 41

Decision Trees 89

Discretization 81

Distributed Data Mining 73

E

Ease Skills Shortage 33

Eclat Algorithm 189

Evaluation Protocol 215

F

Fp Growth Algorithm 190

Fraud Detection 130

Future Expectations 6

G

Generalization 81

Geographic Data Mining 73

Governance 33

H

Hadoop 11

Healthcare Providers 47

Hyperparameters 226, 228

I

Ibm Spss Modeler 99

Improved Eldercare 158

Improved Operations 41

Intrusion Detection 131

K

KNIME 13

Knowledge Representation 82

L

Landscape 6

Lasso Regression 180

Lie Detection 132

Linear Regression 177, 178, 170

M

Machine Learning 67

Malware Filtering 154

Manufacturing 48

Market Intelligence 40

Marketing 5

MongoDB 13

Monkeylearn 96

Multimedia Data Mining 73

N

Naïve Bayes 171

Natural Resources 48

Neural Networks 91

Normalization 220

O

Oracle Data Mining 98

Outlier Detection 87

P

Pattern Evaluation 81

Polynomial Regression 180

Prediction 69, 89

R

Rapidminer 97

Regression 88

Regression Models 177

Ridge Regression 179

S

Semi-Structured Data 20

Sequence Data Mining 73

Sequential Patterns 89

Specific Business Goals 33

Standardization 221

Statistical Techniques 90

Stochastic Gradient Descent 173

Structured Data 19, 34

Subsets 222

Supply Chain Management 40

T

Time Series 73

Time Series Analysis 69

Tracking Patterns 86

Transportation 5, 52

U

Ubiquitous Data Mining 73

Utilities 53

V

Veracity 26

W

Wholesale Trade 51

X

Xplenty 11