-
Notifications
You must be signed in to change notification settings - Fork 5.3k
Clarify namespace sameness control via optional policy #6748
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
thockin
wants to merge
1
commit into
kubernetes:master
Choose a base branch
from
thockin:sameness-impl-policy
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+54
−22
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,38 +1,42 @@ | ||
# Namespace Sameness - SIG Multicluster Position Statement | ||
|
||
Author: Jeremy Olmsted-Thompson (**[@jeremyot](https://github.com/jeremyot)**), Google | ||
Last Edit: 2020/04/20 | ||
Last Edit: 2023/04/22 | ||
Status: RELEASED | ||
|
||
## Goal | ||
|
||
To establish a normative statement for multi-cluster namespace semantics and | ||
governance as a building block for further development which will require | ||
specifying behaviors across clusters. | ||
|
||
## Context | ||
|
||
Users are reaching for multi-cluster deployments for a | ||
[variety of reasons](http://bit.ly/k8s-multicluster-conversation-starter-doc). | ||
However, Kubernetes treats the cluster boundary as the edge of the universe. | ||
There are currently no standard practices for how to extend the Kubernetes | ||
resource model across multiple clusters. Without common patterns we can’t build | ||
portable tooling to facilitate multi-cluster capabilities and know that behavior | ||
will be consistent for each user. | ||
However, Kubernetes has historically treated the cluster boundary as the edge | ||
of the universe. Without common patterns we can’t build portable tooling to | ||
facilitate multi-cluster capabilities and know that behavior will be consistent | ||
for each user. | ||
|
||
## Scope | ||
|
||
A single organization may need multiple, disjoint sets of clusters. They may, | ||
for example, represent different phases of the development lifecycle (dev, | ||
staging, prod) or support unrelated projects. Each organization governs their | ||
own clusters in isolation, so the scope of a namespace can only reasonably be | ||
declared within the organizational boundary. The scope of namespace identity is | ||
defined as the union of clusters, governed by a single authority, that are | ||
defined as the set of all clusters, governed by a single authority, that are | ||
expected to work together. An authority is a company, organization, team, | ||
individual, or other entity which is entrusted to manage these clusters and, in | ||
particular, to create namespaces in them. | ||
|
||
## Position | ||
**For a set of related clusters governed by a single authority, all namespaces of | ||
a given name are considered to be the same namespace. A single namespace should | ||
have a consistent owner across the set of clusters.** | ||
|
||
**Within a set of related clusters which are governed by a single authority (a | ||
"clusterset"), samely named namespaces in all clusters are considered to be | ||
related, to have the same owner(s), and to represent the same intention, | ||
unless otherwise dictated by some overriding policy, including, but not limited | ||
to, implementation-specific configuration.** | ||
|
||
## Appendix | ||
|
||
|
@@ -46,12 +50,12 @@ Consider an organization that runs two clusters, "A" and "B". They have a | |
cluster-ops team which is responsible for keeping those clusters alive and | ||
healthy. They also have app-teams, "foo" and "bar" which use those clusters. | ||
|
||
Cluster-ops owns the creation of namespaces in both clusters. They can choose | ||
Cluster-ops owns the creation of namespaces in all clusters. They can choose | ||
to delegate the ability to create namespaces to foo-team and bar-team, but any | ||
namespace name that is allocated in "reserved" in all clusters. How the | ||
namespace name that is allocated is "reserved" in all clusters. How the | ||
delegation works is an area for innovation, but some possible examples | ||
include: | ||
* A self-service portal which allocate a name in a global database and | ||
* A self-service portal which allocates a name in a global database and | ||
creates the namespace on the user's behalf | ||
* Tooling that allocates a name in a global database and issues a credential | ||
to the user which is checked in an admission controller | ||
|
@@ -79,7 +83,7 @@ If there are special RBAC rules that need to be applied in some clusters (e.g. | |
clusters in EU have more limited access), the LDAP-to-RBAC sync implementation | ||
can apply specializations based on whatever criteria it understands. | ||
|
||
### Example 3: Multi-cluster services | ||
### Example 3: Multi-cluster Services | ||
|
||
Consider the same organization from previous examples. They have enabled an | ||
implementation of multi-cluster Services, which lets them access backends which | ||
|
@@ -88,14 +92,42 @@ are running on other clusters in the group. | |
As in example 2, cluster-ops run a per-cluster metrics service, It does not | ||
make sense for clients in cluster A to access the metrics server of cluster B. | ||
Even though this runs in the same "metrics" namespace in both clusters (and | ||
thus namespce sameness applies), the implementation of multi-cluster services | ||
thus namespace sameness applies), the implementation of multi-cluster services | ||
probably should not be on-by-default. The details of how multi-cluster | ||
services will work are an area for innovation, but ideas include: | ||
* Opt-in: services must be "exported" to be merged across clusters | ||
* Opt-out: services or namespace can be opted out of service merging | ||
services work are out of scope for this document, but valid approaches could | ||
include: | ||
* Opt-in: services must be explicitly "exported" to be merged across clusters | ||
* Opt-out: services or namespaces can be opted out of service merging | ||
* Different discovery: merged services and "raw" services use different names | ||
or other discovery mechanisms | ||
|
||
However the implementation works, the metrics service is not automatically | ||
merged across clusters, though the LDAP-to-RBAC sync from example 2 can still | ||
apply consistent policies. | ||
However the implementation works, the metrics service is not _necessarily_ | ||
merged across clusters, even if the LDAP-to-RBAC sync from example 2 still | ||
applies consistent RBAC policies. Those are independent capabilities. | ||
|
||
### Example 4: Authority sameness policy | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. A comment thread got lost on the push: |
||
|
||
Consider the same organization from previous examples, but with more clusters: | ||
"A", "B", "C", and "D". Business rules dictate that the foo-team and the | ||
bar-team have a common security posture and may share clusters, but the | ||
billing-team has stricter requirements and must not share clusters with other | ||
teams. Cluster-ops wants to assign foo-team's and bar-team's namespaces to | ||
clusters A and B, and billing-team's namespaces to clusters C and D. They also | ||
want to ensure that the opposite is never true - foo-team's and bar-team's | ||
namespaces cannot be used from clusters C or D, and billing-team's namespaces | ||
cannot be used from clusters A or B. | ||
|
||
What exactly it means to "be used" depends on each specific multi-cluster | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. A comment thread got lost on the push: |
||
capability. For multi-cluster Services, the billing-team's endpoints from | ||
clusters C and D would be eligible for merging, but billing endpoints from | ||
clusters A and B would not. This can be used by implementations to add another | ||
layer of safety - even if cluster A were compromised and the billing-team's | ||
namespaces was created, the MCS implementation can use the policy described | ||
above to know that it should NOT merge billing endpoints from cluster A. | ||
|
||
As an implementation choice, the authority may offer controls which govern | ||
which cluster-namespaces are considered for sameness and which are not. | ||
The details of how these policies might be expressed could include: | ||
* In-cluster namespace labels or annotations | ||
* In-cluster custom-resources | ||
* An external control-plane API |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this section could benefit from a larger update to reflect concrete implementations such as
ServiceExport
,ServiceImport
and.clusterset.local
DNS.Do any implementations actually support a service opt-out pattern, where a default or configured behavior is for all services in a given namespace to global/merged/same, but some could be reserved/un-exported, or perhaps more confusingly, exported but "not-same"?
"Export all services in namespace" feels like a reasonable option, and if more granularity is needed, following a service opt-in pattern with manual ServiceExports definitions instead or moving the cluster-local services to a local namespace feels preferable and less confusing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I understand namespace sameness to allow for only the first, not the second (I would prefer the line here should say what is being opted out of is, as originally, "service merging", or even better "exporting", but definitely not as proposed here "sameness"). Most (all?) MCS implementations today are opt-in so implementation wise opt-out is only its corollary: by not having ServiceExports made in 1-N clusters with the same named namespace/service in the first place or removing them. Istio et al probably have more to say on opt out patterns of behavior since those implementations are more on-by-default.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll update. At the time this was written, MCS semantics had not really been pinned down. That said, I like it as an example, but I don't want to inline that spec here.
Will try to balance.