Project Type: B2B Resource
Project Timeline: September 2024 – December 2024
Client Region: APAC
Industry: Retail

Context & Objectives

The client operated multiple Databricks workspaces with over 3,000 Hive Metastore (HMS) objects spread across them. This fragmentation created challenges in governance, permissions, and lineage visibility, making it difficult to manage access, audit changes, or share data securely across teams.

Our objective was to standardize governance by migrating all existing HMS objects to Unity Catalog (UC). The migration needed to be staged, low-risk, and minimally disruptive to ongoing analytics operations, while establishing a UC-first foundation for governance, lineage, and sharing across workspaces.

Project Goals

Inventory all HMS objects across three workspaces and map their dependencies

The client had thousands of Hive Metastore (HMS) objects spread across multiple Databricks workspaces. These included tables, views, functions, and more. Our first goal was to create a complete inventory of all these objects. This wasn’t just about listing names, we also needed to understand how they were connected.

Translate existing permissions, tags, and policies to Unity Catalog (UC) in controlled waves

Each HMS object had permissions and policy tags that needed to be maintained during the migration. Simply moving objects without translating these controls could cause data exposure or broken access. To mitigate risk, we decided on a wave-based migration, where objects were moved in batches.

Implement UC-first governance with consistent grants, lineage tracking, and sharable data products

Beyond moving objects, the goal was to modernize governance. Unity Catalog provides a centralized and uniform way to manage access, track lineage, and share data. We wanted every migrated object to have consistent grants (who can read/write), lineage visibility (where data comes from and where it is used), and be ready for cross-workspace sharing if needed.

Minimize downtime and operational risk during migration

Migrating thousands of objects in live environments carries inherent risks. Even small errors could disrupt reporting, dashboards, or ML pipelines. Our goal was to design a migration approach that minimized impact on business operations.

Challenges

Migrating thousands of Hive Metastore objects to Unity Catalog was a complex undertaking, involving both technical and operational hurdles. We had to address inconsistencies across multiple workspaces, enforce governance, maintain data lineage, and minimize risk to business-critical pipelines. At the same time, the sheer scale of the migration required careful planning, validation, and traceability to ensure a smooth transition. The following key challenges guided our approach :

• Fragmented HMS Environment
  We faced a client environment spread across three separate Databricks workspaces, each running its own Hive Metastore. Collectively, these workspaces contained thousands of objects, including tables, views, and functions. Naming conventions, permissions, and policy tags were inconsistent across environments. .

• Governance Complexity
  We observed that permissions and policy tags were scattered and inconsistently applied, making it challenging to enforce least-privilege access across the organization. Limited visibility into object lineage meant that any misconfiguration could easily break downstream analytics, dashboards, or ML pipelines.

• Operational Risk
  Many of the objects we were migrating were business-critical or contained sensitive data. Any downtime or errors during migration could disrupt reporting and decision-making.

• Scale and Traceability
  With over 3,000 objects and complex interdependencies, we knew manual validation of permissions, policies, and lineage would be extremely slow and error-prone. We tracked each object to ensure nothing was lost, misconfigured, or broken during migration.

Solution Overview

To tackle the challenges of migrating thousands of Hive Metastore objects across multiple workspaces, we designed a structured, wave-based Unity Catalog migration framework. Our approach focused on minimizing operational risk, enforcing consistent governance, and maintaining full lineage visibility. By combining automated inventory, policy translation, and staged migrations, we ensured a safe, repeatable, and auditable transition to Unity Catalog.

• Discovery & Inventory
  We started by scanning all three Hive Metastore workspaces to capture every object, including tables, views, and functions, along with their dependencies, permissions, and policy tags. Using this data, we built a centralized inventory and dependency graph, which allowed us to plan migration waves safely.

• Policy & Tag Translation
  We mapped existing HMS permissions and policy tags, such as PII and PCI classifications, to their Unity Catalog equivalents. By automating much of this translation, we ensured consistent governance across all environments.

• Wave-Based Migration
  To reduce operational risk, we migrated objects in carefully planned, staged waves. We prepared rollback procedures so that any failures could be safely reverted without affecting business-critical workflows.

• Unified Governance & Lineage
  After migration, all Unity Catalog objects including catalogs, schemas, tables, views, and functions had consistent grants and policy tags applied. We made lineage and access patterns fully queryable and auditable, giving teams visibility in the migrated environment.

We used change calendars and approval processes to handle sensitive objects safely and prevent conflicts. Throughout the migration, we tracked progress, validation results, and compliance using clear reports and dashboards, keeping the process transparent and secure.

Architecture Overview

Constraints & Non-Functional Requirements

In addition to successfully migrating objects, we had to ensure the project met several critical non-functional requirements.
Governance was a top priority, we needed to accurately translate all existing grants, policy tags, and maintain lineage continuity so that access controls and data traceability remained intact across workspaces. Reliability was also essential; each migration wave included dry-run validations, diff comparisons, and smoke tests to catch issues before they could affect production, and rollback procedures were in place in case of failure.

Change control was strictly enforced through CAB approvals and designated maintenance windows for sensitive objects, ensuring minimal disruption to business operations.
Finally, we focused on operational efficiency and cost management by minimizing unnecessary recompute, enforcing cluster policies, and continuously monitoring the migration process through dashboards and reports.

Data Model & Semantics

Naming & Pathing
We established stable and consistent naming conventions for all Unity Catalog objects, including catalogs, schemas, tables, views, and functions. By making these names environment-aware, we ensured that users could quickly understand the context of each object and avoid confusion between similar datasets across workspaces.

Policy Tags
Sensitive data, such as PII (Personally Identifiable Information) or PCI (Payment Card Industry) data, was tagged consistently during the migration. These policy tags were applied in a way that they could be queried for audits, compliance checks, and automated access controls. This approach allowed us to enforce least-privilege access rules while reducing the risk of data exposure.

Shares & Access Patterns
We documented all external and cross-workspace sharing patterns as part of the migration. By capturing how data would be shared across teams, business units, or external partners, we ensured that governance policies were preserved and that users could safely consume the data without breaking lineage or violating compliance rules.

Ops, Security, Quality & Performance

To ensure the migration was smooth and reliable, we implemented strong operational, security, quality, and performance practices.
On the operations side, we followed wave-based migration plans, prioritizing objects based on dependencies and usage, and conducted thorough post-migration validations to confirm everything was functioning correctly. From a security perspective, we mapped existing roles, applied least-privilege access, rotated secrets, and enforced cluster policies to protect sensitive data throughout the process. To maintain quality, we used different reports, smoke tests on row counts and critical views, and verified lineage to ensure the migrated data matched the original objects accurately.
Finally, we optimized performance and cost by limiting unnecessary computation, reusing existing objects where possible, and ensuring an efficient and cost-effective migration.

Tech Stack

Data Sources:

• Existing Hive Metastore objects across three workspaces.

Migration & Discovery:

• Scanners, lineage crawlers, and wave executor

Storage / Lakehouse:

• Databricks Unity Catalog as the target, including catalogs

• schemas, tables, views, and functions.

Governance & Security:

• Unity Catalog grants and shares

• policy tags

• cluster policies to enforce consistent access control.

Observability & Quality:

• Migration reports

• lineage verification

• post-wave validation dashboards for tracking progress and ensuring accuracy.

DevOps / CI-CD:

• Databricks Asset Bundles (DAB)

• SDK automation

• versioned mapping rules for consistent migrations.

FinOps / Cost Management:

• Minimized recompute

• optimized migration windows

• enforced cluster policies to reduce cost and resource usage.

Outcomes & Business Impact

By implementing the Unity Catalog migration framework, we achieved unified governance across all workspaces. UC catalogs, schemas, and grants were standardized, significantly reducing access errors and making lineage fully auditable. We improved operational efficiency by automating provisioning and reducing manual ACL management. The wave-based migration approach minimized human errors and allowed the team to handle thousands of objects reliably.

The migration also reduced risk, as every wave included dry-run validations, diff checks, smoke tests, and predefined rollback procedures. This ensured that production systems remained unaffected, and critical reporting or ML pipelines continued to run without disruption.
Finally, the project created a future-ready platform. With UC-first governance in place and cross-workspace sharing enabled, the organization can onboard new sources more easily, scale analytics efficiently, and maintain consistent, secure data access across all environments.

Deliverables

Workspace Inventory & Dependency Map: A complete catalog of all HMS objects across three workspaces, including their relationships and dependencies.

Migration Playbook: Detailed mapping rules, wave-based migration plan, and rollback procedures to ensure safe and controlled execution.

UC Object Catalog: All catalogs, schemas, tables, views, and functions successfully migrated to Unity Catalog.

Grants & Policy Tags: Permissions and policy tags applied consistently across all objects, with lineage tracking for auditability.

Post-Migration Validation & Audit Report: Comprehensive verification to ensure completeness, accuracy, and compliance of the migration process.

Conclusion: Enabling Controlled, Governance-First Migration

With the Xponent Unity Catalog Migration, we turned a fragmented Hive Metastore setup into a unified and well-governed Databricks Unity Catalog environment. By migrating objects in carefully planned waves and using metadata-driven mappings, we reduced operational risk, ensured zero downtime, and made governance consistent across workspaces.
The migration improved auditability, gave teams full visibility into data lineage, and created a foundation for cross-workspace sharing and future growth. Engineers can now onboard new objects quickly, enforce consistent policies, and trust the data they use, all while keeping operations smooth, secure, and compliant.

This project not only solved the immediate challenges of fragmented permissions and inconsistent governance but also set the stage for scalable, reliable, and repeatable data operations across the organization.