Important
這項功能處於個人預覽版狀態。 若要試用,請洽詢您的 Azure Databricks 連絡人。
宣告式自動化套件(前稱 Databricks 資產套件)讓您能以程式方式驗證、部署並執行 Azure Databricks 資源,例如功能檢視及其實現的管線。 請參閱「什麼是宣告式自動化套件?」。
本教學建立一套宣告四個 功能檢視 的套件,並以三種方式將其物化:兩個批次特徵同時物化至離線與線上商店、一個連續物化至線上商店的串流特徵,以及一個帶有鋸齒視窗的串流特徵,該特徵以批次模式物化至離線與線上商店,並以串流模式物化至線上商店。 接著你要更改設定、重新部署,然後移除所有資源。
在套件中宣告功能,會在你部署到的每個工作區中獲得相同的定義,並作為原始碼與專案其他部分一併檢視。 它是從筆記本中呼叫 materialize_features 和 create_feature 的宣告式替代方案。 請參閱 特色檢視。
每個步驟都會以 YAML 和 Python 兩種方式顯示該設定。 選擇一種語言,頁面就會依照你的選擇切換。
Requirements
- Databricks CLI 版本 1.24.0 或更高。 執行
databricks --version來檢查。 請參閱安裝或更新 Databricks CLI。 - 該直接部署引擎。
features和materialization_sets資源不受 Terraform 部署引擎支援。 參見 資源相容性。 - 一個支援 Unity Catalog 的工作區。 請參閱 需求。
- 授權建立功能並將其具體化。 請參閱 權限。
- 現有的線上商店。 請參閱 「建立線上商店」。
- 一個現有的 Delta 資料表,包含實體欄位與時間戳記欄位,用於批次實體化。 請參閱 Azure Databricks 數據表。
- 一個現有的串流,用於串流實體化。 請參閱 「建立串流」。
來源資料表與串流各自有其生命週期,且兩者都不是套件資源。 先建立它們,並在套件中依名稱參考,就像 步驟 3 對資料表、步驟 4 對串流所做的一樣。
套件中的 Feature Store 資源
本教學宣告兩種功能庫資源類型,兩者之間的差異決定部署的成本是多少。
- 特徵檢視:特徵的定義,涵蓋其來源、彙總及鍵值。 這是輕量級的元資料。 請參閱 功能檢視,以及設定參考,請參閱 功能。
- 實體化集合:一組 實體化特徵。 實體化特徵是透過管線或作業計算成表格的一個特徵視圖,因此這是重量級的部分。 將實體化特徵分組到同一組中,會告訴 Azure Databricks 它們可以一起計算,因此會將它們打包到容納它們的最少資料表和 Lakeflow 管線中。 請參閱 「實體化特徵檢視」,以及設定參考,請參考 materialization_set 和 materialization_set.materialized_features。
接下來的步驟會宣告批次特徵、串流特徵,以及帶有鋸齒視窗的串流特徵。 只宣告專案所需的項目。 關於實體化的運作方式,請參閱 「實體化特徵視圖」。
步驟 1:設定驗證
首先,在你的開發機器上設定 Databricks CLI 與 Azure Databricks 工作區之間的驗證。 此頁面假設您希望使用 OAuth 使用者到機器 (U2M) 驗證,以及名為 DEFAULT 的對應 Azure Databricks 組態設定檔進行驗證。
Note
U2M 驗證適用於即時試用這些步驟。 針對完全自動化的工作流程,Databricks 建議您改用 OAuth 機器對機器 (M2M) 驗證。 請參閱 使用 OAuth 授權服務主體存取 Azure Databricks 中的 M2M 驗證設定指示。
使用 Databricks CLI 在本地端啟動 OAuth 權杖管理,針對每個目標工作區執行以下命令。
在下列命令中,將
<workspace-url>替換成 您的 Azure Databricks 個別工作區的 URL,例如https://adb-1234567890123456.7.azuredatabricks.net。databricks auth login --host <workspace-url>Databricks CLI 會提示你將輸入的資訊儲存為 Azure Databricks configuration profile。 按
Enter以接受建議的設定檔名稱,或輸入新或現有設定檔的名稱。 任何具有相同名稱的現有設定檔,會被您輸入的資訊覆蓋。 您可以使用個人檔案,在多個工作區之間快速切換您的驗證環境。若要取得任何現有設定檔的清單,請在個別的終端機或命令提示字元中,使用 Databricks CLI 來執行
databricks auth profiles命令。 若要檢視特定設定檔的現有設定,請執行 命令databricks auth env --profile <profile-name>。在網頁瀏覽器中,完成畫面上的指示,登入 Azure Databricks 工作區。
若要檢視配置檔目前的 OAuth 令牌值和令牌即將到期的時間戳,請執行下列其中一個命令:
databricks auth token --host <workspace-url>databricks auth token -p <profile-name>databricks auth token --host <workspace-url> -p <profile-name>
如果您有多個具有相同
--host值的設定檔,您可能需要一起指定--host和-p選項,以協助 Databricks CLI 尋找正確的相符 OAuth 權杖資訊。
步驟 2:建立套件組合
為該套件建立一個空白目錄, databricks.yml 根目錄有一個檔案,旁邊則有一個 resources 目錄。
YAML
以下每個步驟都會在 resources/ 之下新增一個檔案,以宣告套件資源:
databricks.yml
resources/
batch_feature.yml
streaming_feature.yml
streaming_feature_sawtooth.yml
databricks.yml 包含三個項目:
- 套件名稱。
- 該
include條目決定該套件從哪些檔案載入其資源。 - 這些變數,用來記錄資源可以參考的值,例如目錄、結構與來源名稱。
bundle:
name: feature-store-quickstart
include:
- resources/*.yml
variables:
catalog:
description: The catalog that features are written to.
default: main
schema:
description: The schema that features are written to.
default: features_demo
source_table:
description: The three-level name of the source Delta table.
default: main.features_demo.transactions
stream:
description: The three-level name of a Stream registered in Unity Catalog.
default: main.features_demo.events
online_store:
description: The online store that serves the features.
default: my_online_store
把所有東西都宣告進 databricks.yml 去也行。 省略 include 並集中於單一 resources 映射下,將每個步驟的宣告彙整:
bundle:
name: feature-store-quickstart
variables:
# The variables above.
resources:
features:
# The Feature Views from the following steps.
materialization_sets:
# The materialization sets from the following steps.
Python
以 Python 宣告的資源僅能透過位於 databricks.yml 中的 python 對應找到。 沒有它,套件會成功驗證並部署,但不含任何資源,因此請立即新增:
bundle:
name: feature-store-quickstart
python:
venv_path: .venv
resources:
- 'resources:load_resources'
variables:
catalog:
description: The catalog that features are written to.
default: main
schema:
description: The schema that features are written to.
default: features_demo
source_table:
description: The three-level name of the source Delta table.
default: main.features_demo.transactions
stream:
description: The three-level name of a Stream registered in Unity Catalog.
default: main.features_demo.events
online_store:
description: The online store that serves the features.
default: my_online_store
venv_path 是 Databricks CLI 執行此程式碼的虛擬環境。
resources 底下的每個項目都會將函數命名為 <module>:<function>,Databricks CLI 會呼叫該函數以收集該套件的資源。 如果在 下resources宣告多個<module>:<function>項目,Databricks CLI 會呼叫每個函式並收集每個函式所回傳的資源。
接著建立專案及其虛擬環境,並新增套件:
uv init --bare
uv venv .venv
uv add databricks-bundles
建立resources/__init__.py,其函式名稱為databricks.yml:
from databricks.bundles.core import Bundle, Resources
def load_resources(bundle: Bundle) -> Resources:
resources = Resources()
# The following steps add resources here.
return resources
如需更多關於在 Python 中撰寫套件,包括專案範本與變異器,請參閱〈Python 中的套件配置〉。
步驟 3:新增批次功能
這兩個 Feature Views 會計算每位用戶在過去七天與過去 30 天的交易,兩者皆每日重新計算,並共同實體化到離線與線上儲存庫。
單純的特徵視圖只是定義而已。 實體化集是從中產生資料的內容。
每種資源都有兩個名稱。
transaction_count 是 資源金鑰:您選擇的名稱,在套件中唯一,用於 databricks bundle 指向該資源及其他設定所參考的指令。 例如,集合transactions中的兩個實體化特徵transaction_count會參考特徵視圖的名稱,${resources.features.transaction_count.full_name}以 表示該集合依賴於特徵視圖transaction_count。
由於此依賴性,當兩者同時建立時,transaction_count 會先於集合 transactions 建立;在兩者同時銷毀時,會在集合之後被刪除。 像 main.features_demo.transaction_count_7d 這樣的硬式編碼名稱,若未寫出 ${},則會省略該依賴性,並以不可預測的順序部署。
full_name 是您為 Unity 目錄下註冊的功能檢視所選擇的三層級名稱,也是稍後查詢時所使用的名稱。 這兩個名稱是獨立的。
持續時間是以秒為單位,並帶有 s 後綴,例如 604800s,表示七天。 請參見 「指定視窗、來源或函式的持續時間」。
在 materialized_features 之下的每個項目都是一個實體化特徵成員:一個特徵檢視實體化至一個目的地。 每個功能檢視同時具現化於離線與線上目的地,因此每個功能視圖貢獻兩個實體化功能成員,且每個成員都有自己的觸發條件。 兩個功能檢視都會寫入同一個離線與線上目的地,因為一個集合只有一個離線目的地,且最多只有一個線上目的地。
YAML
創建 resources/batch_feature.yml:
resources:
features:
transaction_count:
full_name: ${var.catalog}.${var.schema}.transaction_count_7d
description: Transactions per user over a trailing seven days
source:
delta_table_source:
full_name: ${var.source_table}
function:
aggregation_function:
count_function:
input: transaction_id
time_window:
sliding:
window_duration: 604800s
slide_duration: 86400s
entities:
- name: user_id
timeseries_column:
name: transaction_time
transaction_count_30d:
full_name: ${var.catalog}.${var.schema}.transaction_count_30d
description: Transactions per user over a trailing 30 days
source:
delta_table_source:
full_name: ${var.source_table}
function:
aggregation_function:
count_function:
input: transaction_id
time_window:
sliding:
window_duration: 2592000s
slide_duration: 86400s
entities:
- name: user_id
timeseries_column:
name: transaction_time
materialization_sets:
transactions:
materialized_features:
# Batch offline materialized feature of transaction_count.
- feature_name: ${resources.features.transaction_count.full_name}
offline_store_config:
catalog_name: ${var.catalog}
schema_name: ${var.schema}
table_name_prefix: transactions_offline
cron_schedule_trigger:
mode: DERIVED
# Batch online materialized feature of transaction_count.
- feature_name: ${resources.features.transaction_count.full_name}
online_store_config:
online_store_name: ${var.online_store}
catalog_name: ${var.catalog}
schema_name: ${var.schema}
table_name_prefix: transactions_online
cron_schedule_trigger:
mode: DERIVED
# Batch offline materialized feature of transaction_count_30d.
- feature_name: ${resources.features.transaction_count_30d.full_name}
offline_store_config:
catalog_name: ${var.catalog}
schema_name: ${var.schema}
table_name_prefix: transactions_offline
cron_schedule_trigger:
mode: DERIVED
# Batch online materialized feature of transaction_count_30d.
- feature_name: ${resources.features.transaction_count_30d.full_name}
online_store_config:
online_store_name: ${var.online_store}
catalog_name: ${var.catalog}
schema_name: ${var.schema}
table_name_prefix: transactions_online
cron_schedule_trigger:
mode: DERIVED
Python
將匯入新增至 resources/__init__.py:
from databricks.bundles.features import (
AggregationFunction,
CountFunction,
DataSource,
DeltaTableSource,
EntityColumn,
Feature,
Function,
SlidingWindow,
TimeWindow,
TimeseriesColumn,
)
from databricks.bundles.materialization_sets import (
CronSchedule,
CronScheduleMode,
MaterializationSet,
MaterializedFeature,
OfflineStoreConfig,
OnlineStoreConfig,
)
然後將此加入主 load_resources體:
resources.add_feature(
'transaction_count',
Feature(
full_name='${var.catalog}.${var.schema}.transaction_count_7d',
description='Transactions per user over a trailing seven days',
source=DataSource(delta_table_source=DeltaTableSource(full_name='${var.source_table}')),
function=Function(
aggregation_function=AggregationFunction(
count_function=CountFunction(input='transaction_id'),
time_window=TimeWindow(
sliding=SlidingWindow(window_duration='604800s', slide_duration='86400s')
),
)
),
entities=[EntityColumn(name='user_id')],
timeseries_column=TimeseriesColumn(name='transaction_time'),
),
)
resources.add_feature(
'transaction_count_30d',
Feature(
full_name='${var.catalog}.${var.schema}.transaction_count_30d',
description='Transactions per user over a trailing 30 days',
source=DataSource(delta_table_source=DeltaTableSource(full_name='${var.source_table}')),
function=Function(
aggregation_function=AggregationFunction(
count_function=CountFunction(input='transaction_id'),
time_window=TimeWindow(
sliding=SlidingWindow(window_duration='2592000s', slide_duration='86400s')
),
)
),
entities=[EntityColumn(name='user_id')],
timeseries_column=TimeseriesColumn(name='transaction_time'),
),
)
resources.add_materialization_set(
'transactions',
MaterializationSet(
materialized_features=[
# Batch offline materialized feature of transaction_count.
MaterializedFeature(
feature_name='${resources.features.transaction_count.full_name}',
offline_store_config=OfflineStoreConfig(
catalog_name='${var.catalog}',
schema_name='${var.schema}',
table_name_prefix='transactions_offline',
),
cron_schedule_trigger=CronSchedule(mode=CronScheduleMode.DERIVED),
),
# Batch online materialized feature of transaction_count.
MaterializedFeature(
feature_name='${resources.features.transaction_count.full_name}',
online_store_config=OnlineStoreConfig(
online_store_name='${var.online_store}',
catalog_name='${var.catalog}',
schema_name='${var.schema}',
table_name_prefix='transactions_online',
),
cron_schedule_trigger=CronSchedule(mode=CronScheduleMode.DERIVED),
),
# Batch offline materialized feature of transaction_count_30d.
MaterializedFeature(
feature_name='${resources.features.transaction_count_30d.full_name}',
offline_store_config=OfflineStoreConfig(
catalog_name='${var.catalog}',
schema_name='${var.schema}',
table_name_prefix='transactions_offline',
),
cron_schedule_trigger=CronSchedule(mode=CronScheduleMode.DERIVED),
),
# Batch online materialized feature of transaction_count_30d.
MaterializedFeature(
feature_name='${resources.features.transaction_count_30d.full_name}',
online_store_config=OnlineStoreConfig(
online_store_name='${var.online_store}',
catalog_name='${var.catalog}',
schema_name='${var.schema}',
table_name_prefix='transactions_online',
),
cron_schedule_trigger=CronSchedule(mode=CronScheduleMode.DERIVED),
),
]
),
)
步驟 4:新增串流功能
此 Feature View 會針對每個鍵計算過去 90 分鐘內的事件數,並持續實體化至線上商店。 它寫入一個目的地,因此其集合宣告單一成員。
串流物質化特徵成員攜帶streaming_mode,且沒有cron_schedule_trigger。 關於這些模式及其代表,請參閱 materialization_set.materialized_features.streaming_mode。
這是第二個實體化集合,而非加入 步驟 3 定義的實體化特徵,因為實體化集合無法將串流模式下實體化的功能視圖與批次實體化的功能視圖混合使用。 帶有鋸齒窗的串流特徵視圖是例外。 其批次與串流實體化特徵屬於同一組,如同 步驟 5。
功能檢視會依名稱讀取串流。 串流會獨立建立並設定版本,置於套件之外,方式與批次功能所讀取的 Delta 表相同。 若要建立串流,請參閱 「建立串流」。
串流功能檢視會將輸入欄位視為進入串流記錄的路徑,而非純欄位名稱,因此請將範例所示 value.*位置替換串流結構所使用的名稱。
YAML
創建 resources/streaming_feature.yml:
resources:
features:
event_count_90m:
full_name: ${var.catalog}.${var.schema}.event_count_90m
description: Events per key over a trailing 90 minutes
source:
stream_source:
full_name: ${var.stream}
function:
aggregation_function:
count_function:
input: value.amount
time_window:
rolling:
window_duration: 5400s
entities:
- name: value.key
timeseries_column:
name: value.event_timestamp
materialization_sets:
events_online:
materialized_features:
- feature_name: ${resources.features.event_count_90m.full_name}
online_store_config:
online_store_name: ${var.online_store}
catalog_name: ${var.catalog}
schema_name: ${var.schema}
table_name_prefix: event_count_90m_online
streaming_mode:
mode: STREAMING_MODE_TYPE_RTM
Python
將 StreamSource 和 RollingWindow 加入從 StreamingMode 匯入的項目中,並將 StreamingModeStreamingModeType 和 databricks.bundles.materialization_sets 加入從 load_resources 匯入的項目中,然後將以下內容加入 databricks.bundles.features:
resources.add_feature(
'event_count_90m',
Feature(
full_name='${var.catalog}.${var.schema}.event_count_90m',
description='Events per key over a trailing 90 minutes',
source=DataSource(stream_source=StreamSource(full_name='${var.stream}')),
function=Function(
aggregation_function=AggregationFunction(
count_function=CountFunction(input='value.amount'),
time_window=TimeWindow(rolling=RollingWindow(window_duration='5400s')),
)
),
entities=[EntityColumn(name='value.key')],
timeseries_column=TimeseriesColumn(name='value.event_timestamp'),
),
)
resources.add_materialization_set(
'events_online',
MaterializationSet(
materialized_features=[
MaterializedFeature(
feature_name='${resources.features.event_count_90m.full_name}',
online_store_config=OnlineStoreConfig(
online_store_name='${var.online_store}',
catalog_name='${var.catalog}',
schema_name='${var.schema}',
table_name_prefix='event_count_90m_online',
),
streaming_mode=StreamingMode(
mode=StreamingModeStreamingModeType.STREAMING_MODE_TYPE_RTM
),
)
]
),
)
步驟 5:新增帶有鋸齒視窗的串流特徵
將串流功能檢視實體化,其彙整使用鋸齒槽視窗,會產生三個實體化特徵:離線批次實體化特徵、線上批次實體化特徵,以及線上串流實體化特徵。 伺服器要求三者都需要,且宣告部分具體化特徵會被拒絕。 這兩個批次實體化功能必須使用 DERIVED 作為 cron_schedule_trigger,這樣 Azure Databricks 才能從功能定義中選擇排程。 批次線上實體化功能與串流線上實體化功能必須共享相同的線上商店配置。 不符合這些要求的實體化集會在部署時被拒絕。 完整清單請參閱 「實體化集合必須符合的規則」。
YAML
創建 resources/streaming_feature_sawtooth.yml:
resources:
features:
amount_sum:
full_name: ${var.catalog}.${var.schema}.amount_sum_3d
description: Sum of amount per key over a sawtooth window
source:
stream_source:
full_name: ${var.stream}
function:
aggregation_function:
sum:
input: value.amount
time_window:
sawtooth:
window_duration: 259200s
entities:
- name: value.key
timeseries_column:
name: value.event_timestamp
materialization_sets:
amounts:
materialized_features:
# Batch offline materialized feature.
- feature_name: ${resources.features.amount_sum.full_name}
offline_store_config:
catalog_name: ${var.catalog}
schema_name: ${var.schema}
table_name_prefix: amount_sum_offline
cron_schedule_trigger:
mode: DERIVED
# Batch online materialized feature.
- feature_name: ${resources.features.amount_sum.full_name}
online_store_config:
online_store_name: ${var.online_store}
catalog_name: ${var.catalog}
schema_name: ${var.schema}
table_name_prefix: amount_sum_online
cron_schedule_trigger:
mode: DERIVED
# Streaming online materialized feature.
- feature_name: ${resources.features.amount_sum.full_name}
online_store_config:
online_store_name: ${var.online_store}
catalog_name: ${var.catalog}
schema_name: ${var.schema}
table_name_prefix: amount_sum_online
streaming_mode:
mode: STREAMING_MODE_TYPE_RTM
Python
將 SawtoothWindow 和 SumFunction 加入從 databricks.bundles.features 匯入的項目,然後將此加入 load_resources:
resources.add_feature(
'amount_sum',
Feature(
full_name='${var.catalog}.${var.schema}.amount_sum_3d',
description='Sum of amount per key over a sawtooth window',
source=DataSource(stream_source=StreamSource(full_name='${var.stream}')),
function=Function(
aggregation_function=AggregationFunction(
sum=SumFunction(input='value.amount'),
time_window=TimeWindow(sawtooth=SawtoothWindow(window_duration='259200s')),
)
),
entities=[EntityColumn(name='value.key')],
timeseries_column=TimeseriesColumn(name='value.event_timestamp'),
),
)
resources.add_materialization_set(
'amounts',
MaterializationSet(
materialized_features=[
# Batch offline materialized feature.
MaterializedFeature(
feature_name='${resources.features.amount_sum.full_name}',
offline_store_config=OfflineStoreConfig(
catalog_name='${var.catalog}',
schema_name='${var.schema}',
table_name_prefix='amount_sum_offline',
),
cron_schedule_trigger=CronSchedule(mode=CronScheduleMode.DERIVED),
),
# Batch online materialized feature.
MaterializedFeature(
feature_name='${resources.features.amount_sum.full_name}',
online_store_config=OnlineStoreConfig(
online_store_name='${var.online_store}',
catalog_name='${var.catalog}',
schema_name='${var.schema}',
table_name_prefix='amount_sum_online',
),
cron_schedule_trigger=CronSchedule(mode=CronScheduleMode.DERIVED),
),
# Streaming online materialized feature.
MaterializedFeature(
feature_name='${resources.features.amount_sum.full_name}',
online_store_config=OnlineStoreConfig(
online_store_name='${var.online_store}',
catalog_name='${var.catalog}',
schema_name='${var.schema}',
table_name_prefix='amount_sum_online',
),
streaming_mode=StreamingMode(
mode=StreamingModeStreamingModeType.STREAMING_MODE_TYPE_RTM
),
),
]
),
)
步驟 6:驗證與規劃
檢查設定,其內容會讀取 databricks.yml 及該套件載入的所有資源檔案:
databricks bundle validate
validate 也是判斷某欄位是否被接受的最便宜方式。 它會列出該組合無法識別的任何欄位,並附上其路徑、來源檔案和行號:
Warning: unknown field: bogus_field
at resources.features.transaction_count.function.aggregation_function.time_window.sliding
in resources/batch_feature.yml:16:15
未識別欄位會被報告為警告,其值會被刪除,因此請讀取輸出而非僅讀取退出代碼。 傳入 --strict 以將警告視為錯誤處理。 關於每個視窗、函數與來源所接受的欄位,請參閱 feature。
伺服器強制執行的規則,例如具有鋸齒形視窗的特徵視圖必須宣告所有三個具體化特徵,此處不會檢查,而會顯示在 databricks bundle deploy 上。
接著在執行前檢視部署所做的事情:
databricks bundle plan
首次部署時,所有七個資源(四個功能檢視與三個實體化集)都會回報 create:
create features.amount_sum
create features.event_count_90m
create features.transaction_count
create features.transaction_count_30d
create materialization_sets.amounts
create materialization_sets.events_online
create materialization_sets.transactions
Plan: 7 to add, 0 to change, 0 to delete, 0 unchanged
也要在每一次後續變更時執行 plan ,看看計畫中的行動。
步驟 7:部署
將套件部署到你的工作空間:
databricks bundle deploy
部署程序會在建立每個資源時回報它:
Uploading bundle files to /Workspace/Users/someone@example.com/.bundle/feature-store-quickstart/default/files...
Created features.transaction_count_30d
Created features.transaction_count
Created features.amount_sum
Created features.event_count_90m
Created materialization_sets.transactions
Created materialization_sets.events_online
Created materialization_sets.amounts
Files: 4 uploaded, 0 deleted
Resources: 7 created, 0 changed, 0 deleted, 0 unchanged
步驟 8:檢查已建立的內容
databricks bundle summary -o json 報告您的套件配置,並附有上次部署時記錄的資源識別碼,例如實體化集 ID。 對於 步驟 5 中定義的實體化集:
{
"id": "1234567890abcdef1234567890abcdef",
"materialized_features": [
{
"cron_schedule_trigger": {
"mode": "DERIVED"
},
"feature_name": "${resources.features.amount_sum.full_name}",
"offline_store_config": {
"catalog_name": "main",
"schema_name": "features_demo",
"table_name_prefix": "amount_sum_offline"
}
},
{
"cron_schedule_trigger": {
"mode": "DERIVED"
},
"feature_name": "${resources.features.amount_sum.full_name}",
"online_store_config": {
"catalog_name": "main",
"online_store_name": "my_online_store",
"schema_name": "features_demo",
"table_name_prefix": "amount_sum_online"
}
},
{
"feature_name": "${resources.features.amount_sum.full_name}",
"online_store_config": {
"catalog_name": "main",
"online_store_name": "my_online_store",
"schema_name": "features_demo",
"table_name_prefix": "amount_sum_online"
},
"streaming_mode": {
"mode": "STREAMING_MODE_TYPE_RTM"
}
}
]
}
要查看伺服器上儲存了什麼,請使用功能工程指令,並使用 databricks bundle summary 輸出中的 id 讀回該集合:
databricks feature-engineering get-materialization-set <materialization-set-id>
回應會攜帶伺服器為同一集合儲存的內容,包括它產生的欄位:
{
"materialization_set_id": "1234567890abcdef1234567890abcdef",
"materialized_features": [
{
"cron_schedule_trigger": {
"cron_expression": "0 0 2 * * ?",
"mode": "DERIVED",
"timezone_id": "UTC"
},
"feature_name": "main.features_demo.amount_sum_3d",
"is_online": true,
"job_id": 123456789012345,
"materialization_set_id": "1234567890abcdef1234567890abcdef",
"materialized_feature_id": "22222222222222222222222222222222",
"online_store_config": {
"catalog_name": "",
"online_store_name": "my_online_store",
"schema_name": "",
"table_name_prefix": ""
},
"pipeline_schedule_state": "ACTIVE",
"table_name": "main.features_demo.amount_sum_online_ceq88t"
},
{
"feature_name": "main.features_demo.amount_sum_3d",
"is_online": true,
"job_id": 234567890123456,
"materialization_set_id": "1234567890abcdef1234567890abcdef",
"materialized_feature_id": "33333333333333333333333333333333",
"online_store_config": {
"catalog_name": "",
"online_store_name": "my_online_store",
"schema_name": "",
"table_name_prefix": ""
},
"pipeline_id": "12345678-1234-1234-1234-123456789012",
"pipeline_schedule_state": "ACTIVE",
"streaming_mode": {
"mode": "STREAMING_MODE_TYPE_RTM"
},
"table_name": "main.features_demo.amount_sum_online_qiohun"
},
{
"cron_schedule_trigger": {
"cron_expression": "0 0 2 * * ?",
"mode": "DERIVED",
"timezone_id": "UTC"
},
"feature_name": "main.features_demo.amount_sum_3d",
"is_online": false,
"job_id": 123456789012345,
"materialization_set_id": "1234567890abcdef1234567890abcdef",
"materialized_feature_id": "11111111111111111111111111111111",
"pipeline_schedule_state": "ACTIVE",
"table_name": "main.features_demo.amount_sum_offline_ceq88t"
}
]
}
步驟 9:變更設定
以下範例變更了先前步驟中的功能檢視與實體化集合,並展示每次變更在部署時所產生的影響。 除非範例指定不同的起始點,否則每個範例都從 第 7 步的部署開始。 每一個都是對宣告資源的檔案進行編輯,或者如果你把所有東西都宣告在同一個檔案裡,則是對 resources 映射 databricks.yml 所做的編輯。 在 Python 標籤中,編輯內容會放在 resources/__init__.py 中。 關於哪些變更是原地套用、哪些是取代資源,哪些是被拒絕的,請參閱 〈變更特徵視圖 〉與 〈變更實體化集〉。
移除 Feature View 中所有實體化的功能
從實體化集合中移除某個特徵視圖中的所有實體化特徵,會即時套用,因此不會重新計算特徵資料。 請參閱 變更實體化集。
以下內容採用在transaction_count中為特徵檢視transaction_count_30d和transactions定義的集合,並移除特徵檢視transaction_count_30d的所有具體化特徵。 同一組中 的 transaction_count 具體化特徵保持不變:
YAML
materialization_sets:
transactions:
materialized_features:
# Batch offline materialized feature of transaction_count.
- feature_name: ${resources.features.transaction_count.full_name}
offline_store_config: # As in Step 3.
cron_schedule_trigger:
mode: DERIVED
# Batch online materialized feature of transaction_count.
- feature_name: ${resources.features.transaction_count.full_name}
online_store_config: # As in Step 3.
cron_schedule_trigger:
mode: DERIVED
- # Batch offline materialized feature of transaction_count_30d.
- - feature_name: ${resources.features.transaction_count_30d.full_name}
- offline_store_config: # As in Step 3.
- cron_schedule_trigger:
- mode: DERIVED
- # Batch online materialized feature of transaction_count_30d.
- - feature_name: ${resources.features.transaction_count_30d.full_name}
- online_store_config: # As in Step 3.
- cron_schedule_trigger:
- mode: DERIVED
Python
resources.add_materialization_set(
'transactions',
MaterializationSet(
materialized_features=[
# Batch offline materialized feature of transaction_count.
MaterializedFeature(...), # As in Step 3.
# Batch online materialized feature of transaction_count.
MaterializedFeature(...), # As in Step 3.
- # Batch offline materialized feature of transaction_count_30d.
- MaterializedFeature(...), # As in Step 3.
- # Batch online materialized feature of transaction_count_30d.
- MaterializedFeature(...), # As in Step 3.
]
),
)
databricks bundle plan 將該集合作為更新回報:
update materialization_sets.transactions
Plan: 0 to add, 1 to change, 0 to delete, 6 unchanged
變更特徵檢視中不可變的欄位
當實體化集依賴功能檢視時,變更功能檢視中不可變欄位的部署時會失敗。 替換特徵檢視會先刪除它,且在特徵檢視已具象化特徵時,刪除會被拒絕。 關於哪些欄位是不可變的,以及建議的變更方式,請參閱 「變更 Feature View」。
以下範例將在 transaction_count 中定義的特徵檢視擴大其滑動視窗的持續時間,而此持續時間是不可變的欄位:
YAML
features:
transaction_count:
# The other keys, as in Step 3.
function:
aggregation_function:
count_function:
input: transaction_id
time_window:
sliding:
- window_duration: 604800s
+ window_duration: 1209600s
slide_duration: 86400s
Python
resources.add_feature(
'transaction_count',
Feature(
# The other arguments, as in Step 3.
function=Function(
aggregation_function=AggregationFunction(
count_function=CountFunction(input='transaction_id'),
time_window=TimeWindow(
- sliding=SlidingWindow(window_duration='604800s', slide_duration='86400s')
+ sliding=SlidingWindow(window_duration='1209600s', slide_duration='86400s')
),
)
),
),
)
databricks bundle plan 回報重新建立作業:
recreate features.transaction_count
Plan: 1 to add, 0 to change, 1 to delete, 6 unchanged
databricks bundle deploy 會要求確認,並說明為何變更無法按原樣執行:
This action will result in the deletion or recreation of the following Feature Views.
A Feature View that has materialized features in a materialization set cannot be deleted or
recreated. Remove its materialized features from the set in a separate deploy first, then delete
or recreate the Feature View. Deleting or recreating a Feature View that a FeatureSpec serves is
not blocked, and leaves the FeatureSpec stale:
recreate resources.features.transaction_count
如果您核准,部署將失敗,因為該組 transactions 仍保留功能檢視的兩個具體化功能:
Error: cannot recreate resources.features.transaction_count: deleting old id=main.features_demo.transaction_count_7d: Cannot delete feature 'main.features_demo.transaction_count_7d' because it has 2 materialized feature(s). Delete all materialized features first before deleting the feature. (400 BAD_REQUEST)
要進行變更,請依照 「變更功能檢視」中說明的步驟部署兩次:
從集合中移除功能檢視的實體化特徵,然後部署。 該集合會就地更新,例如 移除特徵檢視中所有實體化特徵。
變更功能檢視,將其實體化的功能重新加入集合,然後部署。 計畫會為變更後的視窗重建特徵檢視,並重新建立該集合,因為將實體化特徵加入集合會取代該特徵:
Warn: planning resources.materialization_sets.transactions: the member for feature "main.features_demo.transaction_count_7d" is not in the deployed set, and members cannot be added to an existing set, so the whole set will be recreated: every member is dropped and recomputed Warn: planning resources.materialization_sets.transactions: the member for feature "main.features_demo.transaction_count_7d" is not in the deployed set, and members cannot be added to an existing set, so the whole set will be recreated: every member is dropped and recomputed recreate features.transaction_count recreate materialization_sets.transactions Plan: 2 to add, 0 to change, 2 to delete, 5 unchanged核准後,部署會重新建立功能檢視,然後重新建立集合:
Recreated features.transaction_count Recreated materialization_sets.transactions Files: 3 uploaded, 0 deleted Resources: 2 created, 0 changed, 2 deleted, 5 unchanged
變更物化集合的不可變欄位
變更集合中任何實體化特徵的欄位,會取代整個實體化集合,包括編輯未觸及的實體化特徵。 請參閱 變更實體化集。
以下內容會採用在 transactions 中定義的集合 ,並變更其線上目的地的 table_name_prefix,而此屬性是不可變的。 這兩個線上物化特徵都帶有此前綴,因為該集合只有一個線上目的地:
YAML
materialization_sets:
transactions:
materialized_features:
# Batch offline materialized feature of transaction_count.
- feature_name: ${resources.features.transaction_count.full_name}
offline_store_config: # As in Step 3.
cron_schedule_trigger:
mode: DERIVED
# Batch online materialized feature of transaction_count.
- feature_name: ${resources.features.transaction_count.full_name}
online_store_config:
online_store_name: ${var.online_store}
catalog_name: ${var.catalog}
schema_name: ${var.schema}
- table_name_prefix: transactions_online
+ table_name_prefix: transactions_online_v2
cron_schedule_trigger:
mode: DERIVED
# Batch offline materialized feature of transaction_count_30d.
- feature_name: ${resources.features.transaction_count_30d.full_name}
offline_store_config: # As in Step 3.
cron_schedule_trigger:
mode: DERIVED
# Batch online materialized feature of transaction_count_30d.
- feature_name: ${resources.features.transaction_count_30d.full_name}
online_store_config:
online_store_name: ${var.online_store}
catalog_name: ${var.catalog}
schema_name: ${var.schema}
- table_name_prefix: transactions_online
+ table_name_prefix: transactions_online_v2
cron_schedule_trigger:
mode: DERIVED
Python
resources.add_materialization_set(
'transactions',
MaterializationSet(
materialized_features=[
# Batch offline materialized feature of transaction_count.
MaterializedFeature(...), # As in Step 3.
# Batch online materialized feature of transaction_count.
MaterializedFeature(
feature_name='${resources.features.transaction_count.full_name}',
online_store_config=OnlineStoreConfig(
online_store_name='${var.online_store}',
catalog_name='${var.catalog}',
schema_name='${var.schema}',
- table_name_prefix='transactions_online',
+ table_name_prefix='transactions_online_v2',
),
cron_schedule_trigger=CronSchedule(mode=CronScheduleMode.DERIVED),
),
# Batch offline materialized feature of transaction_count_30d.
MaterializedFeature(...), # As in Step 3.
# Batch online materialized feature of transaction_count_30d.
MaterializedFeature(
feature_name='${resources.features.transaction_count_30d.full_name}',
online_store_config=OnlineStoreConfig(
online_store_name='${var.online_store}',
catalog_name='${var.catalog}',
schema_name='${var.schema}',
- table_name_prefix='transactions_online',
+ table_name_prefix='transactions_online_v2',
),
cron_schedule_trigger=CronSchedule(mode=CronScheduleMode.DERIVED),
),
]
),
)
databricks bundle plan 會回報該集合的重建,因此其兩個批次離線實體化特徵也會被丟棄並重新計算:
recreate materialization_sets.transactions
Plan: 1 to add, 0 to change, 1 to delete, 6 unchanged
將實體化特徵新增至現有集合
將實體化特徵加入已部署的實體化集合會取代該集合,因此該集合中的每個實體化特徵都會被重新計算。 請參閱 變更實體化集。
以下內容以前述在transactions後的集合transaction_count_30d為基礎,並將特徵檢視 transaction_count_30d 的物化特徵重新加入:
YAML
materialization_sets:
transactions:
materialized_features:
# Batch offline materialized feature of transaction_count.
- feature_name: ${resources.features.transaction_count.full_name}
offline_store_config: # As in Step 3.
cron_schedule_trigger:
mode: DERIVED
# Batch online materialized feature of transaction_count.
- feature_name: ${resources.features.transaction_count.full_name}
online_store_config: # As in Step 3.
cron_schedule_trigger:
mode: DERIVED
+ # Batch offline materialized feature of transaction_count_30d.
+ - feature_name: ${resources.features.transaction_count_30d.full_name}
+ offline_store_config: # As in Step 3.
+ cron_schedule_trigger:
+ mode: DERIVED
+ # Batch online materialized feature of transaction_count_30d.
+ - feature_name: ${resources.features.transaction_count_30d.full_name}
+ online_store_config: # As in Step 3.
+ cron_schedule_trigger:
+ mode: DERIVED
Python
resources.add_materialization_set(
'transactions',
MaterializationSet(
materialized_features=[
# Batch offline materialized feature of transaction_count.
MaterializedFeature(...), # As in Step 3.
# Batch online materialized feature of transaction_count.
MaterializedFeature(...), # As in Step 3.
+ # Batch offline materialized feature of transaction_count_30d.
+ MaterializedFeature(...), # As in Step 3.
+ # Batch online materialized feature of transaction_count_30d.
+ MaterializedFeature(...), # As in Step 3.
]
),
)
databricks bundle plan 對每一項新增的實體化特徵發出警告,並回報重建:
Warn: planning resources.materialization_sets.transactions: the member for feature "main.features_demo.transaction_count_30d" is not in the deployed set, and members cannot be added to an existing set, so the whole set will be recreated: every member is dropped and recomputed
Warn: planning resources.materialization_sets.transactions: the member for feature "main.features_demo.transaction_count_30d" is not in the deployed set, and members cannot be added to an existing set, so the whole set will be recreated: every member is dropped and recomputed
recreate materialization_sets.transactions
Plan: 1 to add, 0 to change, 1 to delete, 6 unchanged
databricks bundle deploy 在重新建立該集合前,會要求確認:
This action will result in the deletion or recreation of the following materialization sets.
Every materialized feature in a recreated set is dropped and recomputed from scratch, including
the members whose configuration did not change:
recreate resources.materialization_sets.transactions
移除部分特徵檢視的實體化特徵
從實體化集移除部分但非全部特徵檢視的實體化特徵會被拒絕。 請參閱 變更實體化集。
以下內容使用在 amount_sum 中為特徵檢視 amounts 定義的集合 ,該檢視具有鋸齒視窗,並且只移除其批次離線實體化特徵:
YAML
materialization_sets:
amounts:
materialized_features:
- # Batch offline materialized feature.
- - feature_name: ${resources.features.amount_sum.full_name}
- offline_store_config: # As in Step 5.
- cron_schedule_trigger:
- mode: DERIVED
# Batch online materialized feature.
- feature_name: ${resources.features.amount_sum.full_name}
online_store_config: # As in Step 5.
cron_schedule_trigger:
mode: DERIVED
# Streaming online materialized feature.
- feature_name: ${resources.features.amount_sum.full_name}
online_store_config: # As in Step 5.
streaming_mode:
mode: STREAMING_MODE_TYPE_RTM
Python
resources.add_materialization_set(
'amounts',
MaterializationSet(
materialized_features=[
- # Batch offline materialized feature.
- MaterializedFeature(...), # As in Step 5.
# Batch online materialized feature.
MaterializedFeature(...), # As in Step 5.
# Streaming online materialized feature.
MaterializedFeature(...), # As in Step 5.
]
),
)
此編輯會移除 amount_sum 的批次離線實體化特徵,且該刪除同時也會刪除同一特徵檢視中其他兩個實體化特徵,而這並非此次編輯所要求的。 部署被拒絕:
Error: cannot update resources.materialization_sets.amounts: updating id=1234567890abcdef1234567890abcdef: Deleting materialized features '11111111111111111111111111111111' of feature 'main.features_demo.amount_sum_3d' also deletes materialized features '22222222222222222222222222222222', '33333333333333333333333333333333', which the request does not delete. Delete the unrequested materialized features as well. (400 INVALID_PARAMETER_VALUE)
步驟 10:清理
實體化管線會依其設定的排程持續執行,直到該集合被刪除。 完成後移除此 bundle:
databricks bundle destroy
這會刪除套件所包含的每個檔案中宣告的實體化集與特徵檢視。 在與這些資料表、管線和工作共用的最後一個實體化特徵消失後,背景程序會將其移除,因此相關成本停止之前會有短暫延遲。 只有建立實體化功能的使用者才能刪除該功能,因此請使用部署該套件的相同身份執行 databricks bundle destroy 。 參見 delete_materialized_feature()。
關於實體化成本及歸因方式,請參閱 Feature Store 成本管理。
Limitations
- 串流不是一個 bundle 資源。 在套件之外建立它,並以名稱來引用它。
- 將實體化特徵加入現有實體化集合會重新建立該集合,因此該集合中的每個實體化特徵都會被重新計算。
- 在
feature_view_source或custom_udf上,或使用request_source函數的特徵視圖,無法實體化。 請參見 Materialize 特徵視圖。
下一步
- 若要從 CI/CD 工作流程部署這些資源,請參閱 CI/CD 以 GitHub Actions 進行功能檢視。
- 若要從套件中提供這些特徵,請參閱 使用宣告式自動化套件服務特徵檢視。 若要使用 Python API 提供這些功能檢視,請參閱「提供功能檢視」與「範例:部署並查詢功能服務端點」。
- 若要在這些特徵上訓練模型,請參閱 「使用特徵視圖訓練模型」。
- 完整設定參考請參閱 feature 與 materialization_set。
- 若要了解實體化如何運作,請參閱 〈實體化特徵檢視〉。