Information: produced by processing data, reveals meaning, enables decision making
Database: shared, integrated computer structure that stores end-user data and metadata
DBMS: collection of programs that manages the database structure and controls access to data
Advantages of DBMS: better data integration, increased productivity, improved data sharing, security, access, decision making, and quality
Types of Databases
Based on user count: single-user, multiuser (workgroup, enterprise)
Based on location: centralized, distributed, cloud
Based on content: general-purpose, discipline-specific
Based on data currency: operational, analytical (data warehouse, OLAP)
Based on structure: unstructured, structured, semi-structured (XML)
File system problems
Structural dependence, data dependence
Data redundancy, data anomalies (update, insertion, deletion)
DBMS functions: data dictionary, storage, transformation, security, multiuser access, backup and recovery, integrity management, interfaces and communication
Disadvantages of DB systems: increased costs, management complexity, maintaining currency, vendor dependence, frequent upgrade/replacement cycles
Data Modeling
Data modeling: iterative process of creating a specific data model for a problem domain
Data models: simple representations of complex real-world data structures
Importance of data models: communication tool, overall view of the database
Data Model Building Blocks
Entity: a unique and distinct object used to collect and store data
Attribute: characteristic of an entity
Relationship: describes an association among entities (1, M, 1)
Constraint: set of rules to ensure data integrity
Business Rules
Brief, precise, and unambiguous description of a policy, procedure, or principle
Enable defining basic building blocks and describing characteristics of the data
Sources: company policy, department managers, written documentation, interviews
Help standardize the company's view of data and serve as a communication tool
Translate into data model components (nouns → entities, verbs → relationships)
Naming Conventions
Entity names: descriptive of objects in the business environment, familiar to users
Attribute names: descriptive of the data represented
Proper naming facilitates communication and promotes self-documentation
Hierarchical and Network Models
Hierarchical: manages large amounts of data, represents 1 relationships
Network: represents complex data relationships, depicts 1 and M relationships
Relational Model
Based on a relation (table) composed of tuples (rows) and attributes (columns)
Advantages: structural independence, conceptual simplicity, ad hoc query capability
Relational Database Management System (RDBMS): performs basic functions, hides complexities
Entity-Relationship (ER) Model
Graphical representation of entities and their relationships in a database structure
Entity Relationship Diagram (ERD): uses graphic representations to model components
Advantages: visual modeling yields conceptual simplicity, effective communication tool
Disadvantages: limited constraint and relationship representation, no data manipulation language
Data Abstraction Levels
External Model: end users' view of the data environment, represented by ER diagrams
Conceptual Model: global view of the entire database, software and hardware independent
Internal Model: database as seen by the DBMS, software-dependent and hardware-independent
Physical Model: describes how data are saved on storage media, requires definition of physical storage and access methods
Entity Relationship Model (ERM)
Basis of an entity relationship diagram (ERD)
ERD depicts conceptual database as viewed by end users, including entities, attributes, and relationships
Attributes
Characteristics of entities
Required attribute: must have a value
Optional attribute: can be left empty
Domain: set of possible values for an attribute
Identifiers: one or more attributes that uniquely identify each entity instance
Composite identifier: primary key composed of more than one attribute
Attribute types
Composite attribute: can be subdivided into additional attributes
Simple attribute: cannot be subdivided
Single-valued attribute: has only a single value
Multivalued attribute: has many values, may require creating new attributes or entities
Derived attribute: value calculated from other attributes
Relationships
Association between entities that operates in both directions
Participants: entities that participate in a relationship
Connectivity: describes the relationship classification
Cardinality: expresses the minimum and maximum number of entity occurrences associated with one occurrence of a related entity
Existence dependence
Existence dependent: entity exists only when associated with another related entity occurrence
Existence independent: entity exists apart from related entities (strong or regular entity)
Relationship strength
Weak (non-identifying) relationship: primary key of related entity does not contain primary key component of parent entity
Strong (identifying) relationship: primary key of related entity contains primary key component of parent entity
Weak entity: existence-dependent and has a primary key partially or totally derived from parent entity
Relationship participation
Optional participation: entity occurrence does not require corresponding entity occurrence in a relationship
Mandatory participation: entity occurrence requires corresponding entity occurrence in a relationship
Relationship degree: number of entities or participants associated with a relationship
Unary relationship: association maintained within a single entity
Recursive relationship: relationship exists between occurrences of the same entity set
Binary relationship: two entities are associated
Ternary relationship: three entities are associated
Associative Entities
Also known as composite or bridge entities
Represent M relationships between two or more entities
In a 1 relationship with parent entities
Composed of primary key attributes of each parent entity
May contain additional attributes that play no role in the connective process
Developing an ER Diagram
Create a detailed narrative of the organization's description of operations
Identify business rules based on the descriptions
Identify main entities and relationships from the business rules
Develop the initial ERD
Identify attributes and primary keys that adequately describe entities
Revise and review ERD
Extended Entity Relationship Model (EERM)
Result of adding more semantic constructs to the original entity relationship (ER) model
EER diagram (EERD): uses the EER model
Entity Supertypes and Subtypes
Entity supertype: generic entity type related to one or more entity subtypes, contains common characteristics
Entity subtype: contains unique characteristics of each entity subtype
Criteria for usage:
Different, identifiable kinds of the entity in the user's environment
Different kinds of instances should have one or more unique attributes
Specialization Hierarchy
Depicts arrangement of higher-level entity supertypes and lower-level entity subtypes
Relationships described in terms of "is-a" relationships
Subtype exists within the context of a supertype
Every subtype has one directly related supertype, while a supertype can have many subtypes
Provides means to support attribute inheritance, define subtype discriminator, and define disjoint/overlapping and complete/partial constraints
Inheritance
Enables an entity subtype to inherit attributes and relationships of the supertype
All entity subtypes inherit primary key attribute from their supertype
At implementation level, supertype and subtype(s) maintain a 1 relationship
Entity subtypes inherit all relationships in which supertype entity participates
Lower-level subtypes inherit all attributes and relationships from upper-level supertypes
Subtype Discriminator
Attribute in the supertype entity that determines to which entity subtype the supertype occurrence is related
Default comparison condition is the equality comparison
Disjoint and Overlapping Constraints
Disjoint subtypes: contain a unique subset of the supertype entity set (nonoverlapping subtypes)
Implementation based on the value of the subtype discriminator attribute in the supertype
Overlapping subtypes: contain nonunique subsets of the supertype entity set
Implementation requires the use of one discriminator attribute for each subtype
Completeness Constraint
Specifies whether each supertype occurrence must also be a member of at least one subtype
Types:
Partial completeness: not every supertype occurrence is a member of a subtype
Total completeness: every supertype occurrence must be a member of any subtype
Superkey: attribute(s) that uniquely identify a row
Candidate key: irreducible superkey
Foreign key: attribute(s) in one table that match the PK in another table
Secondary key: attribute(s) used for data retrieval purposes
Functional Dependency
Value of one or more attributes determines the value of other attributes
Determinant: attribute whose value determines another
Dependent: attribute whose value is determined by another attribute
Full functional dependence: entire collection of attributes in the determinant is necessary for the relationship
Nulls and Referential Integrity
Null: absence of a data value
Referential integrity: every reference to an entity instance by another entity instance is valid
Handling nulls: flags, NOT NULL constraint, UNIQUE constraint
Relational Algebra
Theoretical way of manipulating table contents using relational operators
Relvar: variable that holds a relation
Closure: use of relational algebra operators on existing relations produces new relations
Relational set operators:
SELECT: outputs a subset of rows
PROJECT: outputs a subset of columns
UNION: combines all rows from two tables, excluding duplicates
INTERSECT: yields only rows that appear in both tables
DIFFERENCE: yields all rows in one table that are not found in the other
PRODUCT: yields all possible pairs of rows from two tables
Joins
Combine information from two or more tables
Types of joins:
Natural join: links tables by selecting rows with common values in common attributes
Equijoin: links tables based on an equality condition comparing specified columns
Theta join: extension of natural join with a theta subscript
Inner join: returns only matched records from joined tables
Outer join: retains matched pairs and leaves unmatched values as null
Left outer join: yields all rows in the first table, including unmatched ones
Right outer join: yields all rows in the second table, including unmatched ones
Data Dictionary and System Catalog
Data dictionary: description of all user-created tables in the database
System catalog: system data dictionary describing all objects within the database
Homonyms (same name for different attributes) and synonyms (different names for the same attribute) should be avoided
Database Normalization
Normalization: evaluating and correcting table structures to minimize data redundancies
Reduces data anomalies and assigns attributes to tables based on determination
Normal forms: 1NF, 2NF, 3NF, BCNF, 4NF
Higher normal forms are better than lower normal forms
Denormalization: produces a lower normal form, increases performance, and introduces greater data redundancy
The Need for Normalization
Used while designing a new database structure or improving an existing one
Analyzes relationships among attributes within each entity
Improves the existing data structure and creates an appropriate database design
The Normalization Process
Objective: ensure each table conforms to well-formed relations
Each table represents a single subject
No data item is unnecessarily stored in more than one table
All nonprime attributes are dependent on the primary key
Each table is void of insertion, update, and deletion anomalies
Ensures all tables are in at least 3NF
Works one relation at a time by identifying dependencies and progressively breaking the relation into new relations
Functional Dependence Concepts
Functional dependence: attribute B is fully functionally dependent on attribute A if each value of A determines one and only one value of B
Fully functional dependence (composite key): attribute B is fully functionally dependent on a composite key A if it is functionally dependent on A but not on any subset of A
Types of Functional Dependencies
Partial dependency: functional dependence in which the determinant is only part of the primary key
Transitive dependency: an attribute functionally depends on another nonkey attribute
Conversion to Normal Forms
First Normal Form (1NF):
Eliminate repeating groups, identify the primary key, and identify all dependencies
All relational tables satisfy 1NF requirements
Second Normal Form (2NF):
Make new tables to eliminate partial dependencies and reassign corresponding dependent attributes
Table is in 2NF when it is in 1NF and includes no partial dependencies
Third Normal Form (3NF):
Make new tables to eliminate transitive dependencies and reassign corresponding dependent attributes
Table is in 3NF when it is in 2NF and contains no transitive dependencies
Requirements for a Good Normalized Set of Tables
Evaluate PK assignments and naming conventions
Refine attribute atomicity (atomic attribute: cannot be further subdivided)
Identify new attributes and relationships
Refine primary keys as required for data granularity (level of detail represented by the values stored in a table's row)
Maintain historical accuracy and evaluate using derived attributes
Denormalization
Design goals: creation of normalized relations and processing requirements and speed
Joining a larger number of tables reduces system speed
Defects in unnormalized tables:
Data updates are less efficient because tables are larger
Indexing is more cumbersome
No simple strategies for creating virtual tables known as views
Structured Query Language (SQL)
SQL is a nonprocedural language with a basic command vocabulary set of less than 100 words
Differences in SQL dialects are minor
SQL is an implementation of Ed Codd's relational set operators (SELECT, PROJECT, JOIN, UNION, INTERSECT, DIFFERENCE, PRODUCT, DIVIDE)
SQL Data Types
Numeric: NUMBER(L,D) or NUMERIC(L,D)
Character: CHAR(L), VARCHAR(L) or VARCHAR2(L)
Date: DATE
Data Definition Language (DDL) Commands
CREATE SCHEMA, CREATE TABLE, NOT NULL, UNIQUE, PRIMARY KEY, FOREIGN KEY, DEFAULT, CHECK, CREATE INDEX, CREATE VIEW, ALTER TABLE, CREATE TABLE AS, DROP TABLE, DROP INDEX, DROP VIEW
Data Manipulation Language (DML) Commands
INSERT, SELECT, COMMIT, UPDATE, ROLLBACK, DELETE
Comparison operators, logical operators, and special operators (BETWEEN, IS NULL, LIKE, IN, EXISTS)
SELECT with ORDER BY, DISTINCT, aggregate functions (MIN, MAX, SUM, AVG), and GROUP BY
Creating Table Structures
Use one line per column definition and spaces to line up attribute characteristics and constraints
Table and attribute names are capitalized
Syntax: CREATE TABLE tablename(...);
Primary Key and Foreign Key
Primary key attributes contain both NOT NULL and UNIQUE specifications
RDBMS enforces referential integrity for foreign keys
Use ON DELETE and ON UPDATE clauses to specify actions on foreign key changes
SQL Constraints
NOT NULL, UNIQUE, DEFAULT, CHECK
Data Manipulation Commands
INSERT: add rows to a table
SELECT: list table contents
UPDATE: modify data
DELETE: delete rows from a table
COMMIT: save changes
ROLLBACK: restore the database
Additional SELECT Query Keywords
ORDER BY: specify listing order
DISTINCT: produce list of unique values
Aggregate functions: MIN, MAX, SUM, AVG
GROUP BY: create frequency distributions
HAVING: extension of GROUP BY, applied to GROUP BY output
Advanced Data Definition Commands
ALTER TABLE: make changes to table structure (ADD, MODIFY, DROP)
DROP TABLE: delete a table from the database
Joining Database Tables
Retrieve data from multiple tables using equality comparisons between foreign and primary keys
Use table aliases to identify source tables
Recursive joins: join a table to itself using aliases
SQL statements for transaction support: COMMIT, ROLLBACK
Transaction sequence continues until COMMIT, ROLLBACK, end of program, or abnormal termination
Transaction Log
Keeps track of all transactions that update the database
Used by DBMS for recovery and rollback
Concurrency Control
Coordination of simultaneous transaction execution in a multiuser database system
Ensures serializability of transactions
Problems in Concurrency Control
Lost update: Same data element updated in two concurrent transactions, one update is lost
Uncommitted data: Transaction is rolled back after a second transaction has accessed its uncommitted data
Inconsistent retrievals: Transaction accesses data before and after other transactions finish working with the data
The Scheduler
Establishes the order of operations within concurrent transactions
Creates serialization schedule to ensure serializability and isolation
Concurrency Control with Locking Methods
Locking methods facilitate isolation of data items used in concurrently executing transactions
Pessimistic locking assumes conflict between transactions is likely
Lock granularity: database, table, page, row, or field level
Lock Types
Binary lock: locked (1) or unlocked (0)
Shared lock: read access granted to concurrent transactions
Exclusive lock: write access reserved for the transaction that locked the object
Two-Phase Locking (2PL)
Guarantees serializability but does not prevent deadlocks
Growing phase: transaction acquires all required locks without unlocking any data
Shrinking phase: transaction releases all locks and cannot obtain new locks
Deadlocks
Occur when two transactions wait indefinitely for each other to unlock data
Control techniques: prevention, detection, avoidance
Time Stamping
Assigns global, unique time stamp to each transaction
Produces explicit order in which transactions are submitted to DBMS
Disadvantages: increased memory needs and processing overhead
Wait/Die and Wound/Wait Concurrency Control Schemes
Two different schemes for requesting access based on transaction age
Phases of Optimistic Approach
Read: transaction reads database, executes computations, and makes updates to a private copy
Validation: transaction is validated to ensure changes will not affect database integrity and consistency
Write: changes are permanently applied to the database
Distributed Database Management Systems (DDBMS)
Centralized databases no longer popular or useful due to globalization and the need for rapid, ad-hoc data access
Distributed databases are now the norm, especially for web-based applications
Distributed Processing vs. Distributed Databases
Distributed processing: database's logical processing shared among physically independent sites via a network
Distributed database: logically related database stored over physically independent sites via a computer network
DDBMS Components and Functions
Components: computer workstations, network hardware and software, communications media, transaction processors (TP), and data processors (DP)
Functions: receives application requests, validates and analyzes them, maps them, decomposes into I/O operations, searches and validates data, ensures consistency and security, and presents data in the required format
Data and Processing Distribution
Single-site processing, single-site data (SPSD): processing and data storage on a single host computer
Multiple-site processing, single-site data (MPSD): multiple processes on different computers sharing a single data repository
Multiple-site processing, multiple-site data (MPMD): fully distributed DBMS supporting multiple data and transaction processors at multiple sites
Distributed Concurrency Control
Important in distributed databases due to multi-site, multiple-process operations that can create inconsistencies and deadlocks
Two-phase commit protocol (2PC): guarantees that if a portion of a transaction cannot be committed, all changes at other sites will be undone
Distributed Database Transparency
Distribution transparency: allows management of physically dispersed databases as if centralized
Performance transparency: allows a DDBMS to perform as if it were a centralized database
Failure transparency: ensures the system will operate in case of network failure
Distributed Database Design
Data fragmentation: breaks a single object into many segments (horizontal, vertical, or mixed)
Data replication: stores data copies at multiple sites served by a computer network
Data allocation: determines where to locate fragments and replicas (centralized, partitioned, or replicated)
The CAP Theorem
Consistency, Availability, Partition tolerance
In a distributed system, only two of the three can be fully achieved
BASE (Basically Available, Soft state, Eventually consistent) model: data changes propagate slowly through the system until all replicas are consistent
Date's 12 Commandments for DDBMS
Local site autonomy, no central site dependence, failure independence, location transparency, fragmentation transparency, replication transparency, distributed query processing, distributed transaction processing, hardware independence, operating system independence, network independence, and database independence
Database Connectivity and Web Technologies
Database Connectivity
Database middleware: provides an interface between the application program and the database
Data repository/source: data management application used to store data generated by an application program
Collection of technologies used to access any type of data source and manage data through a common interface
ODBC, OLE-DB, and ADO.NET form the backbone of the MS UDA architecture
Native SQL Connectivity
Connection interface provided by database vendors, unique to each vendor
Optimized for particular vendor's DBMS, but maintenance is a burden for programmers
ODBC, DAO+Jet, RDO
Open Database Connectivity (ODBC): Microsoft's implementation of SQL Access Group Call Level Interface (CLI) standard
Data Access Objects (DAO): object-oriented API used to access MS Access, FoxPro, and dBase databases from Visual Basic
Remote Data Objects (RDO): higher-level object-oriented application interface to access remote database servers
Object Linking and Embedding for Database (OLE-DB)
Database middleware that adds object-oriented functionality for access to data
Series of COM objects provides low-level database connectivity for applications
ActiveX Data Objects (ADO): provides a high-level application-oriented interface to interact with OLE-DB, DAO, and RDO
ADO.NET
Data access component of Microsoft's .NET application development framework
Manipulates any type of data using any combination of network, operating system, and programming language
Key features: DataSet (disconnected memory-resident representation of database) and XML support
Java Database Connectivity (JDBC)
Application programming interface that allows a Java program to interact with a wide range of data sources
Advantages: leverages existing technology and personnel training, allows direct access to database server or via middleware, and provides a way to connect to databases through an ODBC driver
Database Internet Connectivity
Allows innovative services for rapid response, increased customer satisfaction, anywhere/anytime data access, and effective information dissemination
Web-to-Database Middleware
Server-side extension (program) that interacts directly with the web server
Provides services to the web server transparently to the client browser
Client-Side Extensions
Add functionality to web browsers
Types: plug-ins, Java, JavaScript, ActiveX, and VBScript
Web Application Servers
Middleware application that expands the functionality of web servers by linking them to a wide range of services
Uses: connect to and query databases from web pages, create dynamic web search pages, enforce referential integrity
Features: security, user authentication, access to multiple services, integrated development environment, computational languages, HTML page generation, performance, fault tolerance, and database access with transaction management
Modern Architecture Styles
SOAP: XML-based, for enterprise applications
RESTful: resource-based, for web servers
GraphQL: query language, reduces network load
gRPC: high performance, for microservices
WebSocket: bi-directional, for low-latency data exchange
Webhook: asynchronous, for event-driven applications
Business Intelligence (BI)
Comprehensive, cohesive, integrated set of tools and processes
Captures, collects, integrates, stores, and analyzes data
Transforms data into information, information into knowledge, and knowledge into wisdom
BI Benefits
Improved decision making
Integrating architecture
Common user interface for data reporting and analysis
Common data repository fosters a single version of company data
Improved organizational performance
Decision Support Data
Differ from operational data in time span, granularity, and dimensionality
Drill down: decomposing data to a lower level
Roll up: aggregating data into a higher level
Decision Support Database Requirements
Complex, non-normalized data representations
Aggregated and summarized data
Queries extracting multidimensional time slices
Batch and scheduled data extraction
Support for different data sources and data validation rules
Advanced integration, aggregation, and classification
Support for very large databases (VLDBs) and multiple-processor technologies
Data Warehouses
Characteristics: integrated, subject-oriented, time-variant, and nonvolatile
ETL (Extract, Transform, Load) process: filters, transforms, integrates, classifies, aggregates, and summarizes operational data
Data Marts
Small, single-subject data warehouse subset
Provide decision support to a small group of people
Benefits: lower cost, shorter implementation time, technologically advanced
Star Schema
Data-modeling technique mapping multidimensional decision support data into a relational database
map() applies a function to a sequence (e.g., list).
filter() filters elements of a list based on a Boolean-valued function.
reduce() applies a function repeatedly to elements of a list to generate a single value.
MapReduce Programming Paradigm
Designed for processing Big Data in a parallel and distributed fashion.
Consists of map and reduce phases, with an optional shuffle phase.
Users supply a mapper task and a reducer task; the rest is handled automatically.
Widely used for processing data in NoSQL databases.
Hadoop Ecosystem
Core Components
HDFS -- Hadoop Distributed File System for storing data across clusters.
YARN -- Yet Another Resource Negotiator, a resource management and job scheduling technology.
Data Processing Engines
Hive -- SQL-like scripting language (HiveQL) for querying data stored in HDFS.
Pig -- Data flow language (Pig Latin) for parallel data processing on Hadoop.
Spark -- In-memory data processing engine for iterative and real-time processing.
Storm -- Distributed real-time computation system for processing unbounded streams of data.
Flink -- Parallel data processing platform, generalizing the MapReduce model.
Samza -- Processing engine for handling batch data and streaming data in a unified manner.
Graph Processing
Giraph -- Open-source implementation of Google's Pregel, specifically designed for iterative graph computations.
HAMA -- General-purpose Bulk Synchronous Parallel (BSP) computing engine on top of Hadoop.
Cloud Computing and Virtual Machine Setups
Cloud platforms like AWS, Google Cloud, and Microsoft Azure offer computing resources and services for building Big Data applications.
Virtual machines (VMs) can be used to experiment with Hadoop implementations like Hortonworks Sandbox, MapR Sandbox, and others.
Bulk Synchronous Parallel (BSP) Model
An alternative to MapReduce for parallel and distributed computing.
Computation is divided into supersteps, with phases for local computation, communication, and synchronization barrier.
Suitable for iterative computations, as input data can be saved in memory and transferred between supersteps.
Data Visualization
Visualization of data for understanding, communication, decision-making, and gaining insights.
Types of visualizations: pie charts, bar charts, histograms, scatter plots, maps, network diagrams, word clouds, and more.
Purpose: to make data more intuitive, revealing patterns, trends, and relationships.
Simple Visualizations
Pie charts for expressing relative fractions of a quantity.
Bar charts and histograms for visualizing distributions and comparing values.
Word clouds for indicating relative strengths of keywords or topics.
Bubble plots for displaying multivariate data.
Spatial Data Visualization
Maps for plotting spatial data like locations, incidence, and geographic distributions.
Choropleth maps for displaying aggregated data over regions (unclassed or classed).
Superimposing data on maps for visualizing spatio-temporal data.
Multivariate and Complex Data Visualization
Charles Minard's chart on Napoleon's Russian campaign, considered one of the greatest statistical graphics ever drawn, showing multiple variables.
Network visualizations for displaying relationships between entities, with node and edge attributes.
Interactivity and Animation
Interactive visualizations for exploration, drill-down, and selective viewing.
Animations for presenting changes over time and providing fresh perspectives.
Real-time Visualization
Real-time visualizations for immediate and relevant data representations.
Examples: traffic conditions, stock prices, earthquake monitoring, and population growth.
Data Visualization Tools and Platforms
Spreadsheet software like Excel.
Mathematical and statistical packages: MATLAB, Mathematica, R, Python (matplotlib), and JavaScript (D3.js).
Dedicated data visualization software and platforms: Tableau, Qlik, SiSense, and more.
Online tools and services for creating visualizations.
Principles and Guidelines
Data visualization as an art and science, involving principles from various disciplines like visual perception, color theory, composition, and design.
Guidelines for choosing appropriate visualizations based on data types and analysis goals.
Emphasizing substance over methodology and graphic design (Edward Tufte's principles).
Examples and Resources
A collection of examples showcasing different visualization types and techniques.
Resources for learning and practicing data visualization, including courses, whitepapers, and online galleries.
Privacy, Security, and Data Governance
Privacy
General Data Protection Regulation (GDPR)
The European Union's GDPR requires businesses to protect the personal data and privacy of EU citizens for transactions occurring within the EU.
It provides a single set of privacy guidelines for EU member countries and citizens, effective from May 25, 2018.
Privacy Challenges
Websites and e-commerce companies collect vast amounts of customer data for sales and analytics, making data vulnerable to theft and leakage.
Search data and online activities are tracked, enabling data inference and connecting of dots about individuals.
Surveillance and monitoring of individuals, particularly in countries like China and Japan.
Location tracking through mobile devices, security cameras, and other means.
Fair Information Practices (FIP)
The FIP efforts in organizations followed five tenets:
Openness
Disclosure
Secondary usage limits
Correctability
Security
Redefining Privacy
The notion of "privacy" needs to be redefined in the digital age, where personal data is widely collected and shared.
Security
Data Breaches
Data breaches involving theft or exposure of valuable data, identity information, and customer details are becoming commonplace (e.g., Yahoo, Home Depot, Facebook, Uber, Equifax).
Risks and Costs
Privacy and security breaches can result in significant costs, including lost revenue, customer attrition, fines, and lawsuits. Investing in security technologies and policies is crucial to mitigate these risks.
Internet of Things (IoT) Security
Novel security threats include attacks on IoT devices and data, including connected cars and other smart devices.
Ethics
Ethical Use of Data
Ethical use of data involves multiple aspects, such as:
Potential for biased or unethical behavior in AI systems (e.g., autonomous drones, robot soldiers).
Fairness issues, such as bias in face recognition systems.
Deepfakes and information warfare (fake videos, audio, images, and text).
Interpretability and transparency of machine learning models.
Data Governance
Data governance is the exercise of authority and control over the management of data assets. It involves planning, monitoring, and enforcing policies, procedures, rules, and guidelines for data collection, storage, and usage.
Master Data Management (MDM)
MDM is the management of "master data," providing a single point of reference for critical, shared data within an organization. It ensures data consistency, accuracy, and completeness.
Data Provenance
Data provenance documents the inputs, entities, systems, and processes that influence data, providing a historical record of its origins and lineage. Provenance helps establish trust in data and is crucial for scientific research.
Data Curation
Data curation involves actively managing data throughout its lifecycle to maintain quality, add value, and enable reuse over time. It includes processes and technologies for organizing, preserving, and facilitating access to data.
Compliance
Compliance refers to adhering to laws, regulations, and policies related to data handling and privacy. It involves understanding the differences between laws, regulations, and policies, and ensuring that organizations comply with relevant requirements.
Trust
Trust in organizations handling personal data is proportional to transparency, value delivery, and acceptance of consequences. Individuals need assurance that their data is being used responsibly by businesses, governments, and non-profit organizations.
Generative AI
Generative AI runs neural networks backwards, generating new data instead of classifying existing data.
Generative Adversarial Networks (GANs)
Two neural networks (generator and discriminator) are paired in opposing order, with the generator creating new data and the discriminator evaluating its authenticity.
Style Transfer
One of the early uses of GenAI was to add artistic styles to images.
Autoencoders and Variational Autoencoders (VAEs)
Autoencoders consist of an encoder that maps input to a latent space and a decoder that reconstructs the input from the latent representation.
VAEs produce a distribution in the latent space instead of a single point, allowing for the generation of new data by sampling from the distribution.
Transformers
Transformers, initially introduced for machine translation, can be adapted to various domains, including vision (ViT), time-series prediction, and music generation.
The self-attention mechanism in Transformers addresses the quadratic bottleneck of attention computation, enabling better handling of context.
Retrieval-Augmented Generation (RAG)
RAG models augment language models with external memory (knowledge bases, databases, or text) to enhance their ability to understand and generate relevant content.
Fine-tuning and Representation Fine-Tuning (ReFT)
Fine-tuning is a way to adapt a pre-trained language model to a specialized domain by further training on domain-specific data.
ReFT is a new family of fine-tuning techniques that modify the relevant hidden representations of the language model for the target task, providing a more efficient alternative to traditional fine-tuning methods.
Directions and Challenges
New directions in GenAI include multimodal models, "embodied" language models, sparse and modular architectures, and specialized hardware.
Challenges and issues include context mismatch, input chunking, hallucinations, and the need for computational resources to scale these models.
The Centrality of Data
Data is the driving force behind all these advancements and applications, emphasizing the importance of data organization, governance, curation, and responsible usage.
Continuous learning and exploration of new data-driven technologies and techniques are essential for staying up-to-date and contributing to the field.
Comments0
No comments yet. Be the first to share your thoughts.
