System errors

A system error is the record Lakewatch creates when a detection rule or datasource fails, or when it finds a configuration problem in one. Each system error groups its repeated occurrences so you can investigate related system health and configuration issues together. System errors are separate from alerts, which group signals that indicate potential security threats.

How are system errors created?

System errors are created automatically through grouping. When a detection rule or datasource raises a new occurrence, Lakewatch attaches it to an existing system error rather than opening a new system error for every occurrence. An occurrence attaches to an existing system error (one that's not resolved or marked as a duplicate) when both of the following conditions are met:

• The occurrence has the same summary and was raised by the same detection rule or datasource as the occurrence that opened the existing system error.
• The existing system error was opened within the last 24 hours.

Lakewatch creates a new system error when no matching system error exists, the most recent matching system error was opened more than 24 hours ago, or the matching system error is resolved or marked as a duplicate. The 24-hour grouping window starts when the system error is opened. New occurrences don't extend it.

System errors can't be created manually from the UI.

What triggers system errors?

Lakewatch opens system errors when a detection rule or datasource job runs into an error or health issue. Common scenarios include:

• Detection rule execution errors: A rule fails while reading its source data, evaluating its detection logic, or processing or writing its output. For example, a rule whose source table was deleted fails to read its source data.
• Datasource pipeline errors: A datasource pipeline run fails.

Lakewatch also opens system errors for configuration issues that it finds while it sets up or updates a rule or datasource. For example, a rule fails validation, a datasource's existing-table source is unavailable, or a parser can't be found. These system errors don't list individual occurrences on the Errors tab.

Working with system errors

Each system error has the following editable attributes:

• Assignee: The user responsible for addressing the system error.
• State: One of Open, In progress, Mitigated, Resolved, or Duplicate. When a rule that failed validation passes validation or is deleted, Lakewatch sets its system error to Resolved. Other system errors keep their state until you change it.
• Severity: Lakewatch sets new system errors to Medium. You can manually update severity at any time. Valid values: Informational, Low, Medium, High, or Critical.

Each system error also has the following read-only attributes:

• Permalink: The system error's ID, such as SYS-42, with a link you can copy.
• Title: The summary of the occurrence that opened the system error.
• Description: The details of the most recent occurrence that included details, such as the error message or a suggested fix. Lakewatch updates the description when a new occurrence with details attaches.
• Rule or Datasource: The detection rule or datasource that raised the occurrences.
• Errors: The occurrences attached to the system error and their count. System errors for configuration issues don't list individual occurrences.
• First event: Timestamp of the earliest occurrence attached to the system error.
• Last event: Timestamp of the most recent occurrence attached to the system error.
• Created by: The identity that created the system error. For a failure in a detection rule or datasource job, this is the run-as identity of that rule or datasource. For a configuration issue, it's a Lakewatch service identity.
• Created at: Timestamp of when the system error was created.
• Updated at: Timestamp of when the system error was last modified.

Work with system errors in the UI

System errors appear as a top-level item in the Lakewatch navigation. On this page, you can:

• View a list of system errors. By default, the list shows Open, In progress, and Mitigated system errors with a severity of Low or higher. To see resolved, duplicate, or informational system errors, change the State and Severity filters.
• Search system errors by title, description, or comment text, or find a specific system error by its ID, such as SYS-42.
• Filter system errors by state, severity, assignee, detection rule, datasource, and creation time. Search and filter selections persist across navigation within Lakewatch, and the active filter state is reflected in the URL. Copy the URL to share or bookmark a filtered view.
• Sort by the Events, Created, or Updated column, and resize columns. The list also includes an Exported to column.
• Open a system error to view its details and edit its attributes.

System error details

Click the title of a system error to open the details page. There, you can view and edit its attributes and see its occurrences on the Errors tab.

Click the summary of an occurrence to open its details in a side panel. The panel shows the resource involved and the error message or a suggested fix.

You can also add comments to a system error from the details page. Comments support Markdown syntax. Use the comment order control to switch between Newest first and Oldest first.

Exporting system errors

Export system errors to an external system, such as Jira, ServiceNow, Tines, or a webhook. Export is configured for each environment in the System errors section of Settings. For setup and options, see Configure settings.

Troubleshoot datasource run failures

When a datasource run fails, Lakewatch opens a system error for the affected datasource. The system error's Datasource attribute identifies the datasource, and its occurrences include the run's error message, which describes the failure, such as a pipeline error or a source-connectivity problem.

To investigate and recover:

• Open the datasource's system error to read its summary and the error message on the Errors tab, or query the occurrences in the system_events view filtered on resource_type = 'Datasource'.
• On the datasource's Run history tab, click View job runs to open the underlying Databricks job runs for run history and logs.
• For a source-access failure, restore the run-as identity's access to the source. If the identity is unchanged, the next scheduled run recovers automatically. To reassign the run-as identity, open the datasource's Configuration tab and select a new principal in the Run as card. Changes on the Configuration tab save automatically. See Troubleshoot datasource errors.
• After runs succeed again, the system error stays open. Change its state to Resolved when you've confirmed the fix.

Query occurrences in SQL

The <UC-CATALOG>.__lakewatch.system_events view contains one row per occurrence raised by detection rule and datasource jobs in that catalog, where <UC-CATALOG> is your Lakewatch environment's catalog. Query it to analyze occurrences, correlate them with system errors, and build custom reports or dashboards.

Schema

Additional resources

• Monitor pipeline and rule health