Understanding Golden Paths: A Practical Guide.

After discussing how to release a Platform as a Product, in this post we provide you with a practical guide to understanding golden paths. To clarify the concept of golden paths, let's explore an example of building a golden path to introduce relational databases as a new capability in the platform.

Understanding the Scenario

A Microservices Adoption Nightmare

Your organization embarked on a modernization initiative aimed at boosting two critical dimensions: delivery velocity and scalability. The CTO identified microservices architecture as the core driver for achieving these business objectives.

As part of this initiative, development teams were given full responsibility for their technical stack, empowered by a DevOps approach and one of the core principles of microservices: the ability to select the best technology for their specific service use case.

Initially, this approach seemed ideal. Development teams began releasing features using various programming languages and selecting different datastores for their microservices.
While many teams relied on Postgres for relational use cases and Redis as a key-value store, other teams started adopting technologies such as DuckDB, MySQL, MongoDB, and ElasticSearch.

The Seemingly Right Path for Microservices Architecture

• Technology Autonomy: Teams were empowered to select the right technology for each microservice, allowing them to deliver isolated features that weren’t coupled, could scale independently, and were optimized for performance.
• Cultural Shift to DevOps: Teams were responsible for running what they built, driving automation and shifting deployment leftward.

The Unintended Consequences

However, what initially seemed like the right path quickly led to a technology sprawl that became the root cause of several critical issues:

• Increased Cognitive Load on Development Teams:
  • Teams had to manage the deployment of various datastores, including handling observability, backup, disaster recovery, scalability, tuning, and more.
  • This complexity added significant cognitive load, detracting from their ability to focus on delivering business value.
• Challenges for Security and Compliance Teams:
  • The diverse array of technologies required different security and compliance approaches, complicating efforts to ensure consistency and control across the organization.
• Operational Concerns for SREs:
  • The SRE teams were faced with managing a wide variety of technologies, each with tight SLAs.
  • This added operational risk and complexity, which impacted the stability and reliability of services in production.

Negative Outcomes

Instead of achieving the intended goals of increased delivery velocity and scalability, the organization faced several setbacks:

• Slower Delivery Velocity and Scalability Issues:
  • Difficulty scaling teams or rotating individual contributors due to the specialized nature of each tech stack.
  • Challenges in contributing to certain services because of the diverse technical stacks.
  • Problems finding new team members with the requisite skills to be productive in this complex environment.
  • Teams and contributors becoming stuck in particular services or tech stacks, reducing overall agility.
• Operational Risks:
  • Difficulty meeting SLAs due to the complexity of managing services in production.
  • Increased technical debt, making it harder to upgrade and maintain the tech stack over time.
• Compliance and Security Blockers:
  • Certain teams slowing down or blocking new features due to operational or security concerns, hindering overall progress.

Platform Engineering to the Rescue

Faced with these challenges, the organization recognized the need to standardize and reduce technology sprawl. Regarding datastores, the decision was made that any operational service requiring a relational datastore must use Postgres.

Initially, the idea was to create a cross-functional DevOps team to centralize the deployment of these common services.

However, the CTO soon realized that the issue was not just about managing individual technical components. Instead, it was about understanding the role of a relational database within the broader organizational architecture.

Key Insights from the CTO:

• It’s Not About Individual Components: The relational database’s role within the organization must be conceived holistically, rather than just focusing on spinning up databases.
• Development Teams Need a Higher Abstraction Level: It’s not just about providing a single flavor of databases for relational use cases; it’s about how the architecture leverages this technology to deliver business value.
• Cognitive Load is a Real Issue: Development teams need to concentrate on their value streams to maximize delivery speed and efficiency.

Platform Engineering should encompass the entire end-to-end value chain to effectively deliver products and applications to end users. A relational database capability in your platform is not just about offering a Database as a Service (DBaaS); it’s about understanding the role of the relational database within your architecture and the business value it provides.

Recognizing these insights, the CTO launched a Platform Engineering initiative to address the organization’s challenges.

As part of this initiative, the Platform Team assumed responsibility for making the "relational datastore" capability available as part of the Internal Developer Platform (IDP).

The goal is to provide development teams with a golden path that allows them to efficiently leverage this capability, reducing complexity and driving standardization across the organization.

Thinking of Relational Databases as a Golden Path

Databases in Operational Services

A key principle of microservices architecture is that each service manages its own data. Services should not share a datastore to avoid unintentional coupling, which could result in dependencies that hinder independent deployments and scalability.

This best practice necessitates the ability for development teams to spin up databases quickly and easily, while ensuring compliance with security and deployment requirements.

But deploying a “production-ready” Postgres database is a complex endeavor, involving decisions related to security, deployment, availability, backups, and secrets, among others.

The golden path for relational databases does not have to expose all these elements for developer teams, otherwise their cognitive load will increase, but must take care of these needs to ensure that this is the paved road to production.

The relational database golden path must encapsulate all the operational decisions made at your company to transform an “out of the box” postgres into the specific set of configuration and decisions to run a relational database in your organization.

What Developer Teams Actually Need from the Platform

Developers don’t need to manage the full complexity of setting up and maintaining relational databases. Instead, they should only need to configure elements that directly influence the development process, such as:

• Computing Size, Storage, Number of Instances
• Encoding
• Optional Features like PostGIS

The platform, in turn, provides a connection string to the database, allowing the service to operate smoothly within the organization’s compliance framework. This setup abstracts the complexity away from the developers, enabling them to focus on building features rather than managing infrastructure.

Simplifying Configuration with High-Level Attributes

Given the elements that developers need to configure, there’s an opportunity to simplify the process even further. The platform team can design high-level attributes for the capability that are then translated into the necessary low-level configurations.

The platform team could take three attributes:

• Computing size
• Storage
• Number of instances

And encapsulate them into a single high-level attribute called SIZE.

This attribute could use values like SMALL, MEDIUM, LARGE, EXTRA-LARGE, which the golden path would then translate into the appropriate configurations for computing power, storage capacity, and number of instances.

Benefits of This Approach

• Reduced Cognitive Load:
  • By reducing the number of decisions developers need to make, this approach minimizes cognitive load. Developers only need to choose a high-level configuration, leaving the platform to handle the detailed adjustments.
• Environment-Specific Adjustments:
  • This method also allows the platform team to adjust the underlying configurations depending on the environment without requiring changes from the development teams.
  • For example, an extra large relational database could be significantly smaller in development environments than in production. But regardless of the stage, the developer teams will always refer to an extra large relational database.

Golden paths provide a means to reduce cognitive load in various ways, preserving the ability to make low-level decisions when necessary while streamlining the overall configuration process.

By offering these high-level abstractions, the platform enables developers to focus on what they do best—delivering business value—while maintaining the flexibility to adapt to different environments and requirements.

What is the Actual Role of the Relational Database in your Architecture

The platform team needs to grasp the intended use and expected outcomes of that database within the broader architectural context. For example:

• Outbox Pattern:
  • The architecture may define an event-driven solution based on Change Data Capture (CDC) on top of Postgres, following an Outbox design pattern.
  • This ensures reliable event publishing by storing events in a dedicated table that can be consumed asynchronously.
• Materialized View Pattern:
  • This involves pre-populating views with data from multiple services.
  • Such views can be useful when a request requires data from multiple sources, making real-time data aggregation inefficient.
• Pushing Data to Analytical Warehouses:
  • The architecture might define specific views that are pushed in real-time to an analytical warehouse for further analysis and reporting.

By understanding the specific purpose and mission of these technical requirements, the platform team can create more useful golden paths. This approach allows the team to better encapsulate the cognitive load for development teams, increasing the platform’s value and enhancing its mission.

Aligning the relational database capabilities with the broader business objectives ensures that the platform not only meets technical requirements but also delivers measurable business outcomes. This strategic alignment is crucial for demonstrating the value of Platform Engineering to the wider organization.

A New Collaboration Model Converging in the Platform Team

Golden Paths can benefit multiple personas within your organization and must be designed considering not only the feedback from different teams but also their actual needs.

Introducing new capabilities, such as relational databases, into the platform requires therefore a new collaboration model:

• Platform Team:
  • Captures needs and feedback from all stakeholders, including development, operations, security, and architecture teams.
• Developer Teams:
  • Provide feedback on what they need from a development perspective, ensuring that the platform meets their requirements.
  • Expect a path to build faster by having a clear pathway to supported tooling
• Operation and Security Teams:
  • Define needs to ensure that the relational database is compliant and production-ready.
  • Expect the implementation of policies related to security, observability and even cost-management
• Architecture Teams:
  • Align the platform’s capabilities with the organization’s broader architectural goals.

Golden paths are the center of this relationship, as depicted in the following diagram with a sample approach for a relational database golden path:

Continuous feedback loops from development, operations, and architecture teams are vital in ensuring that the relational database capabilities evolve in response to changing business needs and technological advancements. This ongoing dialogue helps maintain the platform’s relevance and effectiveness.

Golden Paths and Abstraction Levels

Platforms should be composable, offering different levels of abstraction to empower flexibility and reusability:

• Relational Database Golden Path:
  • This might be the starting point, but platforms can go further by creating higher-level abstractions.
  • For example, your organization might determine that a standard service should include several cloud services alongside the database, such as a source code repository or a messaging queue.
• Service Archetypes:
  • Just as software architecture uses archetypes, platform engineering can encapsulate the necessary infrastructure for a service, composed of various low-level golden paths.
  • This overarching golden path deploys a service as a meaningful architectural quantum within the organization.

This composability maximizes platform adoption by allowing teams to choose the appropriate level of abstraction without deviating from the platform’s established standards and paved road.

As technology evolves, the platform should be flexible enough to accommodate future architectural changes or the adoption of new database technologies. This flexibility ensures that the platform remains a valuable asset to the organization over time.

If you’re interested in Platform Engineering and want to keep diving deeper into this topic, here’s some content you might find interesting:

• What Platform Engineering is (and what it isn’t)
• Core Elements of Platform Engineering
• Key Benefits of Platform Engineering
• Barriers to Adoption of Platform Engineering
• How to Release a Platform as a Product?