Chapter 44: Data Privacy & Compliance

1. Why Privacy Matters for System Design

Data privacy is no longer an afterthought—it fundamentally shapes system architecture. Regulations directly drive technical decisions about data storage, processing, transfer, and deletion. The stakes are existential:

• GDPR penalties: Up to 4% of global annual revenue or €20 million, whichever is higher
• CCPA fines: $2,500 per violation ($7,500 for intentional violations)
• HIPAA violations: Up to $1.5 million per violation category per year
• Reputational damage: Customer trust erosion, brand impact, market value loss

Regulations Drive Architecture

Privacy requirements manifest as concrete technical constraints:

Trust as a Competitive Advantage

Privacy is not just about avoiding fines—it builds customer trust. Companies with robust privacy practices attract privacy-conscious customers, command premium pricing, and reduce churn. Transparency reports, privacy certifications (SOC 2, ISO 27001), and proactive security measures differentiate brands.

2. GDPR Key Requirements

The General Data Protection Regulation (GDPR) is the gold standard for privacy legislation. Even non-EU companies must comply if they process EU residents' data. Key obligations:

Core Principles

• Lawfulness, fairness, transparency: Clear communication about data use
• Purpose limitation: Collect data only for specified purposes
• Data minimization: Collect only necessary data
• Accuracy: Keep personal data accurate and up-to-date
• Storage limitation: Retain data only as long as necessary
• Integrity and confidentiality: Secure data against unauthorized access
• Accountability: Demonstrate compliance through documentation

Key Rights

Organizational Requirements

• Data Protection Officer (DPO): Required for public authorities or large-scale processing of sensitive data. DPO monitors compliance, advises on data protection impact assessments (DPIAs), coordinates with supervisory authorities.
• Breach Notification: Notify supervisory authority within 72 hours of breach discovery. If high risk to user rights, notify affected individuals "without undue delay."
• Data Protection Impact Assessment (DPIA): Required for high-risk processing (e.g., large-scale profiling, automated decision-making). Assesses necessity, proportionality, risks, and mitigation measures.
• Consent Management: Consent must be freely given, specific, informed, and unambiguous. Pre-ticked boxes are invalid. Withdrawal must be as easy as giving consent.

3. Right to Deletion (RTBF)

The Right to Be Forgotten (RTBF) is architecturally challenging. Deleting a user's data across distributed systems, backups, caches, and third-party processors requires careful orchestration.

Hard Delete vs. Soft Delete

Cascading Deletion Architecture

Key Design Considerations

• Service Discovery: Maintain a registry of all services that process user data. Each service exposes a DELETE /users/:userId endpoint.
• Orchestration Pattern: Saga pattern for coordinated deletion. If one service fails, retry logic with exponential backoff. Log partial failures for manual intervention.
• Audit Trails: Paradox—must delete user data but prove deletion occurred. Solution: Log deletion events with anonymized identifiers (hashed user ID + salt), retain only metadata (timestamp, services affected).
• Backup Handling: Backups are the hardest part. Options:
  • Backup exclusion lists: Add deleted user IDs to exclusion list, filter during restore
  • Re-backup: Trigger new backup after deletion (expensive)
  • Backup expiry: Short retention (e.g., 30 days), accept that old backups contain deleted data but never restore to production
• Legal Holds: Cannot delete data under litigation hold or regulatory investigation. Implement hold flags that prevent deletion until cleared.

Example Deletion Workflow

// Deletion Orchestrator pseudocode
async function deleteUser(userId) {
    const services = await serviceRegistry.getServicesWithUserData();
    const results = [];

    for (const service of services) {
        try {
            await service.deleteUser(userId);
            results.push({ service: service.name, status: 'success' });
        } catch (error) {
            results.push({ service: service.name, status: 'failed', error });
        }
    }

    // Queue backup purge
    await backupPurgeQueue.enqueue({ userId, timestamp: Date.now() });

    // Notify third-party processors
    await thirdPartyAPI.deleteUser(userId);

    // Audit log (anonymized)
    await auditLog.record({
        event: 'user_deletion',
        userId: hash(userId + SALT),
        timestamp: Date.now(),
        services: results
    });

    // Send confirmation email (before email deletion)
    await emailService.send(userId, 'Your data has been deleted');

    // Delete email last
    await emailService.deleteUser(userId);

    return results;
}

4. PII Handling

Personally Identifiable Information (PII) requires special handling. The first step is identifying what constitutes PII, then applying appropriate protection techniques.

PII Classification

Protection Techniques

Tokenization

Replace sensitive data with non-sensitive tokens. Token maps to original value in secure vault.

// Tokenization example
Original: SSN = "123-45-6789"
Tokenized: SSN = "TOK_a8f3b2c1d9e4f5a6"

// Vault stores mapping
Vault: { "TOK_a8f3b2c1d9e4f5a6" => "123-45-6789" }

// Application uses token
SELECT * FROM orders WHERE customer_ssn_token = 'TOK_a8f3b2c1d9e4f5a6';

Pros: Reversible, format-preserving options, reduces PCI DSS scope. Cons: Vault becomes single point of failure, network latency for detokenization.

Pseudonymization

Replace identifying fields with pseudonyms (e.g., hashed or encrypted values). Still allows linking records for same user.

// Pseudonymization
Original: email = "alice@example.com"
Pseudonymized: email_hash = HMAC-SHA256("alice@example.com", secret_key)
              = "a1b2c3d4e5f6..."

Pros: GDPR-compliant with additional safeguards, allows analytics. Cons: Not reversible without key, vulnerable to rainbow table attacks if weak hashing.

Anonymization

Irreversibly remove identifying information. User cannot be re-identified.

• k-anonymity: Each record is indistinguishable from at least k-1 other records. Example: Generalize ZIP codes (94102 → 941**), age ranges (28 → 25-30).
• l-diversity: Each equivalence class has at least l diverse values for sensitive attributes (prevents homogeneity attacks).
• t-closeness: Distribution of sensitive attributes in each group is close to overall distribution (prevents skewness attacks).
• Differential Privacy: Add calibrated noise to query results. Guarantees that adding/removing one individual's data doesn't significantly change output.

PII Processing Pipeline

PII Detection Automation

Manual PII tagging doesn't scale. Use automated tools:

• AWS Macie: ML-based PII detection in S3, scans for SSN, credit cards, addresses
• Google DLP API: Detects 150+ infoTypes (PII categories), applies de-identification transformations
• Microsoft Presidio: Open-source PII detection and anonymization
• Custom regex + NER: Named Entity Recognition models (SpaCy, Stanford NER) to identify PERSON, ORG, LOCATION entities

5. Encryption

Encryption protects data confidentiality. Three layers: at rest, in transit, and application-level.

Encryption at Rest

Protects data stored on disk from physical theft or unauthorized access.

Algorithm: AES-256

Industry standard: AES-256-GCM (Galois/Counter Mode). Provides confidentiality and authenticity. Symmetric key encryption—same key encrypts and decrypts.

Envelope Encryption

Don't encrypt all data with a single master key (key rotation nightmare). Instead, use envelope encryption:

Advantages: Rotate master key without re-encrypting all data, limit blast radius (each dataset has unique DEK), offload key management to KMS.

Encryption in Transit

Protects data moving between client and server, or between services.

• TLS 1.3: Minimum acceptable version. TLS 1.2 acceptable with strong cipher suites (no RC4, no MD5). Enforce HTTPS everywhere.
• Certificate Management: Use Let's Encrypt for auto-renewal, or AWS ACM. Short-lived certificates (90 days) reduce exposure from compromised certs.
• Mutual TLS (mTLS): Both client and server authenticate with certificates. Common in service-to-service communication (Istio, Linkerd enforce mTLS).
• Perfect Forward Secrecy (PFS): Use ephemeral keys (ECDHE) so that past sessions can't be decrypted if long-term key is compromised.

Application-Level Encryption (Field-Level)

Encrypt sensitive fields before writing to database. Useful when you don't trust database administrators or want defense in depth.

// Field-level encryption example (Node.js)
const crypto = require('crypto');

function encrypt(plaintext, key) {
    const iv = crypto.randomBytes(16);
    const cipher = crypto.createCipheriv('aes-256-gcm', key, iv);
    let encrypted = cipher.update(plaintext, 'utf8', 'hex');
    encrypted += cipher.final('hex');
    const authTag = cipher.getAuthTag().toString('hex');
    return { iv: iv.toString('hex'), encrypted, authTag };
}

// Store encrypted SSN
const ssn = '123-45-6789';
const encryptedSSN = encrypt(ssn, masterKey);
db.query('INSERT INTO users (ssn_encrypted, ssn_iv, ssn_auth_tag) VALUES (?, ?, ?)',
         [encryptedSSN.encrypted, encryptedSSN.iv, encryptedSSN.authTag]);

Pros: Protects against DB admin access, SQL injection (attacker gets ciphertext), backup theft. Cons: Cannot query encrypted fields (no WHERE ssn = '123-45-6789'), key management complexity.

Searchable Encryption

Enable queries on encrypted data without decrypting. Techniques:

• Deterministic Encryption: Same plaintext always produces same ciphertext. Allows equality queries. Weakness: Frequency analysis reveals patterns.
• Order-Preserving Encryption (OPE): If plaintext1 < plaintext2, then ciphertext1 < ciphertext2. Allows range queries. Weakness: Reveals order, vulnerable to attacks.
• Homomorphic Encryption: Perform computations on ciphertext (e.g., SUM, COUNT). Result decrypts to same value as if computed on plaintext. Weakness: Slow, limited operations.
• Secure Enclaves (Intel SGX): Decrypt data inside hardware-protected enclave, query plaintext in isolated memory. Weakness: Side-channel attacks, limited availability.

6. Key Management

Encryption is only as strong as key management. Poor key management negates encryption benefits.

Key Management Systems (KMS)

Key Hierarchy

Root Key: Stored in Hardware Security Module (HSM), never leaves device. Used to encrypt master keys.

Master Key: Managed by KMS, used to encrypt Data Encryption Keys (DEKs). Rotated annually.

Data Encryption Key (DEK): Per-dataset or per-user key. Encrypted with master key. Rotated frequently (e.g., monthly).

Key Rotation

Regular key rotation limits exposure from compromised keys.

• Master Key Rotation: KMS supports automatic rotation (AWS KMS: annual). Old versions retained for decryption, new encryptions use new key.
• DEK Rotation: More complex—requires re-encrypting data. Strategies:
  • Lazy rotation: Re-encrypt on access (gradual migration)
  • Background jobs: Batch re-encryption during off-peak hours
  • Envelope rotation: With envelope encryption, only re-encrypt the (small) DEK, not the (large) dataset

Key Storage Best Practices

• Never hardcode keys: No keys in source code, config files, or environment variables visible to developers.
• Use KMS or Vault: Centralized, audited, access-controlled key storage.
• Least privilege: Grant key access only to services that need it. AWS KMS key policies and IAM roles.
• Audit key usage: Log all key operations (encrypt, decrypt, rotate). AWS CloudTrail for KMS.
• Backup keys securely: Encrypt key backups, store in separate physical location.

7. Access Control for Data

Encryption protects data from external threats; access control protects from internal misuse.

Role-Based Access Control (RBAC) on PII

Grant access based on job role, not individual identity. Principle of least privilege.

// RBAC policy example
Roles:
  - Customer Support: Read access to name, email, order history (NOT SSN, payment info)
  - Finance Team: Read access to payment info, billing address
  - Developers: No access to production PII (use anonymized test data)
  - Auditors: Read-only access to all data + audit logs

// Enforcement in application
if (user.role === 'Customer Support') {
    return db.query('SELECT name, email, orders FROM users WHERE id = ?', [userId]);
} else if (user.role === 'Finance Team') {
    return db.query('SELECT payment_info, billing_address FROM users WHERE id = ?', [userId]);
}

Row-Level Security (RLS)

Database-enforced access control. Each user sees only authorized rows.

-- PostgreSQL Row-Level Security
CREATE POLICY customer_support_policy ON users
    USING (current_user_role() = 'Customer Support' AND consent_for_support = true);

-- Automatically filters queries
SELECT * FROM users;  -- Customer Support only sees users who consented

Pros: Enforced at DB level (can't bypass in application code), centralized policy management. Cons: Performance overhead, complex policies are hard to debug.

Column-Level Access Control

Restrict access to specific columns based on role.

-- SQL Server Column-Level Security
GRANT SELECT ON users(name, email) TO CustomerSupportRole;
DENY SELECT ON users(ssn, payment_info) TO CustomerSupportRole;

Dynamic Data Masking

Show masked data to unauthorized users.

-- SQL Server Dynamic Data Masking
ALTER TABLE users
ALTER COLUMN ssn ADD MASKED WITH (FUNCTION = 'partial(0,"XXX-XX-",4)');

-- Customer Support sees: XXX-XX-6789
-- Finance Team sees: 123-45-6789

Types of masking:

• Partial: Show first/last N characters (credit card: XXXX-XXXX-XXXX-1234)
• Email: aXXX@XXXX.com
• Random: Replace with random value (preserves data type for testing)
• Redaction: Replace with NULL or fixed string

Attribute-Based Access Control (ABAC)

Access decisions based on attributes (user role, data classification, time of day, location).

// ABAC policy example
ALLOW access IF:
  user.role = "Finance" AND
  data.classification = "Financial" AND
  time_of_day BETWEEN 08:00 AND 18:00 AND
  user.location = "US"

Pros: Fine-grained, flexible, scales to complex rules. Cons: Complex to implement, requires attribute management infrastructure.

8. Data Residency

Data residency requirements mandate where data is physically stored. Regulations like GDPR (EU), PIPL (China), and data localization laws (Russia, India) require user data to stay within specific borders.

Why Data Residency Matters

• GDPR: Restricts data transfers outside EU/EEA unless adequate safeguards (SCCs, adequacy decisions)
• China PIPL: Personal data of Chinese citizens must be stored in China
• Russia: Personal data of Russian citizens must be stored on servers in Russia
• Data sovereignty: Governments want data under their jurisdiction for law enforcement, national security

Cross-Border Transfer Mechanisms (GDPR)

Multi-Region Architecture with Residency

Implementation Strategies

• Region-pinned storage: AWS S3 bucket in eu-west-1 for EU users, us-east-1 for US users. Enforce with S3 Bucket Policies (deny outside region).
• Geo-routing: Route users to nearest region based on IP/account settings. CloudFront with geo-restriction.
• Data replication: Replicate within same legal jurisdiction (e.g., EU multi-region: eu-west-1 + eu-central-1). Never replicate EU data to US without safeguards.
• Metadata separation: Store PII in region-specific databases, non-PII metadata (analytics) in central location.

Challenges

• Cross-region transactions: User in EU orders product from US warehouse. Split transaction: EU database for user PII, US database for inventory.
• Global search: User searches for content across regions. Solution: Anonymized global search index, retrieve full details from region-specific storage.
• Disaster recovery: Backups must stay within region. EU data cannot fail over to US.
• Cost: Multi-region deployments are expensive (duplicate infrastructure, data transfer costs).

9. Audit Logging

Audit logs prove compliance, detect breaches, and support incident response. Logs must be immutable, comprehensive, and securely stored.

What to Log

Log Entry Structure

{
  "timestamp": "2026-05-12T14:35:22Z",
  "event_type": "pii_access",
  "user_id": "user_12345",
  "user_role": "customer_support",
  "action": "read",
  "resource": "users/user_67890",
  "fields_accessed": ["name", "email", "phone"],
  "ip_address": "192.168.1.100",
  "user_agent": "Mozilla/5.0...",
  "session_id": "sess_abc123",
  "justification": "Support ticket #45678",
  "result": "success"
}

Immutable Audit Trails

Logs must be tamper-proof. If attacker gains access, they shouldn't be able to delete evidence.

• Write-once storage: S3 Object Lock (WORM), AWS S3 Glacier Vault Lock
• Append-only logs: CloudWatch Logs with retention policies, Splunk, Elasticsearch with immutable indices
• Cryptographic integrity: Sign each log entry with HMAC. Hash chain: each entry includes hash of previous entry (blockchain-style).
• Separate accounts: Store logs in dedicated AWS account, no access from production account. Break-glass process for emergencies.

Retention Requirements

Alerting on Suspicious Activity

Logs are useless if no one monitors them. Set up alerts for:

• Mass PII access (e.g., >100 records in 1 hour)
• Access from unusual locations (geofencing)
• Access outside business hours (for roles that shouldn't need it)
• Failed authentication attempts (brute force detection)
• Admin role escalation
• Data export operations (right to access requests, but also potential data exfiltration)

10. Compliance Frameworks

Different regulations apply to different industries and regions. Understanding scope and requirements prevents over-engineering (and under-compliance).

Comparison Table

GDPR vs. CCPA

HIPAA Safeguards

HIPAA requires covered entities to implement safeguards for Protected Health Information (PHI):

• Administrative: Security officer designation, workforce training, contingency plans, business associate agreements
• Physical: Facility access controls, workstation security, device/media controls
• Technical: Access controls (unique user IDs), audit logging, encryption, transmission security

PCI DSS Requirements

12 high-level requirements for cardholder data environment (CDE):

1. Install and maintain firewall configuration
2. Don't use vendor-supplied defaults for passwords
3. Protect stored cardholder data (encryption)
4. Encrypt transmission of cardholder data
5. Use and regularly update anti-virus software
6. Develop and maintain secure systems
7. Restrict access to cardholder data by business need-to-know
8. Assign unique ID to each person with computer access
9. Restrict physical access to cardholder data
10. Track and monitor all access to network resources
11. Regularly test security systems and processes
12. Maintain information security policy

Best practice: Reduce PCI DSS scope by using tokenization. Store tokens (not card numbers) in your database, delegate actual card storage to PCI-compliant payment processor (Stripe, Adyen).

11. Privacy by Design

Privacy by Design (PbD) is a proactive approach: embed privacy into system architecture from the start, not as an afterthought.

Core Principles

1. Proactive not reactive: Anticipate privacy risks before they materialize. Conduct Privacy Impact Assessments (PIAs) during design phase.
2. Privacy as default: Strongest privacy settings by default. User must opt-in to data sharing, not opt-out.
3. Privacy embedded into design: Not bolted on. Example: ephemeral messaging (Signal) where messages auto-delete.
4. Full functionality (positive-sum): Privacy doesn't mean sacrificing features. Example: differential privacy allows analytics without compromising individual privacy.
5. End-to-end security: Protect data throughout lifecycle (collection, storage, processing, deletion).
6. Visibility and transparency: Clear privacy policies, user dashboards showing what data is collected.
7. Respect for user privacy: User-centric design. Easy consent withdrawal, clear data usage explanations.

Data Minimization

Collect only data necessary for the stated purpose. Less data = less risk.

Purpose Limitation

Use data only for the purpose communicated to the user. Can't collect data for "account creation" then repurpose for "marketing" without new consent.

// Purpose tagging example
{
  "user_email": "alice@example.com",
  "consent": {
    "account_creation": true,
    "marketing": false,
    "analytics": true
  }
}

// Application logic
if (user.consent.marketing) {
    sendMarketingEmail(user.email);
} else {
    // Cannot use email for marketing
}

Storage Limitation

Retain data only as long as necessary. Implement automatic deletion policies.

// Automated deletion policy
DELETE FROM analytics_events
WHERE created_at < NOW() - INTERVAL '30 days';

DELETE FROM audit_logs
WHERE created_at < NOW() - INTERVAL '1 year';

DELETE FROM user_sessions
WHERE last_active < NOW() - INTERVAL '90 days';

Privacy Impact Assessment (PIA)

Systematic evaluation of privacy risks. Required by GDPR (Data Protection Impact Assessment) for high-risk processing.

When required: Large-scale profiling, automated decision-making with legal effects, processing of sensitive data, systematic monitoring of public areas.

PIA Steps:

1. Describe processing: What data, why collected, how used, who has access, retention period
2. Assess necessity: Is data collection proportionate to purpose? Can you minimize data?
3. Identify risks: Unauthorized access, data breach, re-identification, function creep
4. Mitigation measures: Encryption, access control, anonymization, audit logging
5. Stakeholder consultation: Involve DPO, security team, legal, user representatives
6. Document: Maintain PIA records for compliance audits

Privacy-Preserving Techniques

• Differential privacy: Add noise to query results (Apple uses for iOS analytics, Google for Chrome telemetry)
• Federated learning: Train ML models on user devices, aggregate model updates (not raw data). Google uses for Gboard keyboard predictions.
• Homomorphic encryption: Compute on encrypted data (e.g., cloud processes encrypted medical images without decrypting)
• Zero-knowledge proofs: Prove you know something without revealing it (e.g., prove age >18 without revealing birthdate)