Azure Databricks Reference Architecture

Prévia do material em texto

ServeIngest Store Prep and Train
Azure Data Lake Storage
Logs (unstructured)
Azure Data Factory 
Azure Databricks
Media (unstructured)
Files (unstructured)
Business/custom apps
(structured)
Power BI
Sensors and IoT
(unstructured)
Azure Event Hub
Azure IoT Hub
Kafka
+
Apps
Azure Kubernetes 
Service
Model Serving
Cosmos DB, 
SQL DB
Operational Databases
Ad-hoc 
Analysis
Delta FormatRaw Format
Azure Synapse 
Data Warehouse
Polybase
Step 2
Use Azure Databricks to 
1. Combine streaming and 
batch
2. Save data as Delta 
format
Delta Format
(Bronze Table)
Cosmos DB Apps
Step 3
Use Azure Databricks to 
1. Join, enrich, clean, transform data
2. Develop, train, and score ML 
models with Azure ML + MLFlow
Azure Data Lake Storage
Delta Format
(Silver Table)
+
Delta Format
(Gold Table)
Step 4
Load data into serving layers like 
1. SQL Data Warehouse for 
enterprise BI scenarios.
2. Cosmos DB for real-time Apps
Logs (unstructured)
Azure Data Factory 
Media (unstructured)
Files (unstructured)
Business/custom apps
(structured)
Sensors and IoT
(unstructured)
Azure Event Hub
Azure IoT Hub
Kafka
Raw 
Format
Step 1
Load raw data to Azure Data 
Lake Storage
Power BIAzure Synapse 
Data Warehouse
ServeIngest Store Prep and Train
Azure Data Lake Storage
Logs (unstructured)
Azure Data Factory 
Azure Databricks
Media (unstructured)
Files (unstructured)
Business/custom apps
(structured)
Power BI
Sensors and IoT
(unstructured)
Azure Event Hub
Azure IoT Hub
Kafka
+
Apps
Azure Kubernetes 
Service
Model Serving
Cosmos DB, 
SQL DB
Operational Databases
Ad-hoc 
Analysis
Delta FormatRaw Format
Azure Synapse 
Data Warehouse
ServeIngest Store Prep and Train
Azure Data Lake Storage
Logs (unstructured)
Azure Data Factory 
Azure Databricks
Media (unstructured)
Files (unstructured)
Business/custom apps
(structured)
Power BI
Sensors and IoT
(unstructured)
Azure Event Hubs
Azure IoT Hub
Kafka
+
Apps
Azure Kubernetes 
Service
Model Serving
Ad-hoc 
Analysis
Delta FormatRaw Format
Azure Synapse 
Data Warehouse
Cosmos DB, 
SQL DB
Operational Databases
ServeIngest Store Prep and Train
Azure Data Lake Storage
Logs (unstructured)
Azure Data Factory 
Azure Databricks
Media (unstructured)
Files (unstructured)
Business/custom apps
(structured)
Power BI
Sensors and IoT
(unstructured)
Azure Event Hub
Azure IoT Hub
Kafka
+
Apps
Azure Kubernetes 
Service
Model Serving
Cosmos DB, 
SQL DB
Operational Databases
Ad-hoc 
Analysis
Delta FormatRaw Format
Azure Synapse 
Data Warehouse
ServeIngest Store Prep and Train
Azure Data Lake Storage
Logs (unstructured)
Azure Data Factory 
Azure Databricks
Media (unstructured)
Files (unstructured)
Business/custom apps
(structured)
Power BI
Sensors and IoT
(unstructured)
Azure Event Hub
Azure IoT Hub
Kafka
+
Apps
Azure Kubernetes 
Service
Model Serving
Cosmos DB, 
SQL DB
Operational Databases
Ad-hoc 
Analysis
Delta FormatRaw Format
Azure Synapse 
Data Warehouse