Implement configuration ingestion

[adamzip]

#5494

[Copilot]

Pull request overview

This PR implements configuration ingestion functionality to allow the Product Construction Service to validate and persist configuration data (subscriptions, channels, default channels, and branch merge policies) from external sources. The implementation introduces a new interface for externally synced entities, validation logic for entity uniqueness and field constraints, and CRUD operations for managing configuration data.

Key changes:

• Introduces IExternallySyncedEntity<TId> interface to standardize entity identification across YAML models and database models
• Implements comprehensive validation framework for configuration entities with uniqueness checking
• Adds CRUD operations in SqlBarClient and ConfigurationIngestor to create, update, and delete configuration entities with proper namespace isolation

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 14 comments.

Show a summary per file

File	Description
src/Microsoft.DotNet.Darc/DarcLib/Models/Yaml/IExternallySyncedEntity.cs	New interface defining contract for externally synced entities with unique identifiers
src/Microsoft.DotNet.Darc/DarcLib/Models/Yaml/SubscriptionYaml.cs	Implements IExternallySyncedEntity using subscription Id as unique identifier
src/Microsoft.DotNet.Darc/DarcLib/Models/Yaml/ChannelYaml.cs	Implements IExternallySyncedEntity using channel name as unique identifier
src/Microsoft.DotNet.Darc/DarcLib/Models/Yaml/DefaultChannelYaml.cs	Implements IExternallySyncedEntity using repository/branch/channel tuple; adds Id field for ingestion
src/Microsoft.DotNet.Darc/DarcLib/Models/Yaml/BranchMergePoliciesYaml.cs	Implements IExternallySyncedEntity using repository/branch tuple as unique identifier
src/Maestro/Maestro.Data/Models/Repository.cs	Adds UniqueId property to RepositoryBranch for consistency with IExternallySyncedEntity pattern
src/Maestro/Maestro.Data/Models/DefaultChannel.cs	Implements IExternallySyncedEntity on database model for alignment with YAML model
src/Maestro/Maestro.DataProviders/ISqlBarClient.cs	Adds interface methods for subscription CRUD operations used in configuration ingestion
src/Maestro/Maestro.DataProviders/SqlBarClient.cs	Implements CRUD methods with conflict detection, validation, and conversion between client/DAO models
src/Maestro/Maestro.DataProviders/Exceptions/EntityConflictException.cs	New exception type for handling duplicate entity conflicts during ingestion
src/Maestro/Maestro.DataProviders/ConfigurationIngestor/IConfigurationIngestor.cs	Updated interface signature to return entity changes and require namespace parameter
src/Maestro/Maestro.DataProviders/ConfigurationIngestor/ConfigurationIngestor.cs	Core ingestion orchestration including validation, entity comparison, and database persistence
src/Maestro/Maestro.DataProviders/ConfigurationIngestor/ConfigurationData.cs	Changed to use YAML models instead of DAO models for external configuration representation
src/Maestro/Maestro.DataProviders/ConfigurationIngestor/ConfigurationDataUpdate.cs	New record types capturing created, updated, and removed entities for each configuration type
src/Maestro/Maestro.DataProviders/ConfigurationIngestor/ConfigurationDataHelper.cs	Helper methods for computing entity differences and converting between YAML and DAO models
src/Maestro/Maestro.DataProviders/ConfigurationIngestor/Validations/EntityValidator.cs	Generic validator for checking entity uniqueness across IExternallySyncedEntity collections
src/Maestro/Maestro.DataProviders/ConfigurationIngestor/Validations/SubscriptionValidator.cs	Updated to validate collections and work with SubscriptionYaml instead of DAO models
src/Maestro/Maestro.DataProviders/ConfigurationIngestor/Validations/ChannelValidator.cs	Updated to validate collections and work with ChannelYaml instead of DAO models
src/Maestro/Maestro.DataProviders/ConfigurationIngestor/Validations/DefaultChannelValidator.cs	Updated to validate collections and work with DefaultChannelYaml instead of DAO models
src/Maestro/Maestro.DataProviders/ConfigurationIngestor/Validations/BranchMergePolicyValidator.cs	Updated to validate collections and work with BranchMergePoliciesYaml instead of DAO models

Comments suppressed due to low confidence (6)

src/Maestro/Maestro.DataProviders/SqlBarClient.cs:641

• This assignment assigns Enabled to itself.

        existingSubscription.SourceRepository = existingSubscription.SourceRepository;

src/Maestro/Maestro.DataProviders/SqlBarClient.cs:642

• This assignment assigns SourceEnabled to itself.

        existingSubscription.TargetRepository = existingSubscription.TargetRepository;

src/Maestro/Maestro.DataProviders/SqlBarClient.cs:643

• This assignment assigns SourceDirectory to itself.

        existingSubscription.Enabled = existingSubscription.Enabled;

src/Maestro/Maestro.DataProviders/SqlBarClient.cs:644

• This assignment assigns TargetDirectory to itself.

        existingSubscription.SourceEnabled = existingSubscription.SourceEnabled;

src/Maestro/Maestro.DataProviders/SqlBarClient.cs:646

• This assignment assigns PullRequestFailureNotificationTags to itself.

        existingSubscription.TargetDirectory = existingSubscription.TargetDirectory;

src/Maestro/Maestro.DataProviders/SqlBarClient.cs:546

• The contents of this container are never initialized.

        var newSubscriptionHashes = new HashSet<string>();

[premun]

I wonder if it'd be better if we had this on the DAOs?
At least it would be immediately visible if we ever add a field to the DAO.

[adamzip]

Agreed, that would've been a good place. What bothered me is that this isn't a pure converter (i.e. One type of entity in -> Another type of entity out), but it requires other contextual data like existingChannelsByName.
It makes sense in the context of the ingestor because we do query all the existing channels, but if you're developing a feature somewhere else in the codebase and you see that Subscription has a converter that requires existingChannelsByName, we don't have a way to force the caller to truly provide a list of all existing channels. So I find its reusability impractical and/or dangerous

[premun]

I think source repository can change? Maybe not for codeflow subscriptions only

[adamzip]

Suggested change

	.Where(e => dbEntitiesById.ContainsKey(e.UniqueId))];
	.Where(e => dbEntitiesById.ContainsKey(e.UniqueId))];