Skip to content
This repository was archived by the owner on Jun 23, 2020. It is now read-only.

Commit 6e92f93

Browse files
authored
Merge pull request #39 from oracle/ol-tidy-docs
Consolidate docs into a single README.md file for better discoverability
2 parents 17b813b + f072ce4 commit 6e92f93

File tree

3 files changed

+212
-220
lines changed

3 files changed

+212
-220
lines changed

README.md

Lines changed: 212 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,18 +4,222 @@
44

55
This project implements a flexvolume driver for Kubernetes clusters
66
running on Oracle Cloud Infrastructure (OCI). It enables mounting of [OCI block
7-
storage volumes][1] to Kubernetes pods via the [Flexvolume][2] plugin interface.
7+
storage volumes][1] to Kubernetes Pods via the [Flexvolume][2] plugin interface.
88

99
We recommend you use this driver in conjunction with the OCI Volume Provisioner.
1010
See the [oci-volume-provisioner][3] for more information.
1111

1212
## Install / Setup
1313

14-
See [here](docs/install.md) for information about installing and configuring the volume driver.
14+
We publish the OCI flexvolume driver as a single binary that needs to be
15+
installed on every node in your Kubernetes cluster.
1516

16-
## Getting started
17+
### Ansible
1718

18-
See [here](docs/tutorial.md) for a tutorial and guide to using the flexvolume volume driver.
19+
The recommended way to install the driver is with Ansible.
20+
21+
Compile the oci binary or download the latest release from Github.
22+
23+
```
24+
make build
25+
```
26+
27+
Now you can use Ansible to deploy the driver to your cluster
28+
29+
```
30+
cd ansible
31+
```
32+
33+
Create an inventory file
34+
35+
```
36+
cp hosts.example hosts
37+
```
38+
39+
Add the details for all the masters and nodes in your cluster
40+
(the oci-flexvolume-driver needs to be installed on all masters and workers)
41+
42+
```
43+
[all:vars]
44+
ansible_ssh_user=opc
45+
46+
[masters]
47+
...
48+
49+
[workers]
50+
...
51+
```
52+
53+
#### Run the playbook.
54+
55+
```
56+
ansible-playbook -i hosts \
57+
--private-key=generated/instances_id_rsa site.yaml
58+
```
59+
60+
### Manually
61+
62+
The driver should be installed in the volume plugin path on **every**
63+
node in your Kubernetes cluster at the following location:
64+
`/usr/libexec/kubernetes/kubelet-plugins/volume/exec/oracle~oci/oci`.
65+
66+
NOTE: If running kube-controller-managers in a container you _must_ ensure that
67+
the plugin directory is mounted into the container. See:
68+
https://gitlab-odx.oracle.com/odx/oke-prime/merge_requests/15 for specifics.
69+
70+
#### Configuration
71+
72+
The driver requires API credentials for a OCI account with the ability
73+
to attach and detach [OCI block storage volumes][1] from to/from the appropriate
74+
nodes in the cluster.
75+
76+
These credentials should be provided via a YAML file present on **master** nodes
77+
in the cluster at `/usr/libexec/kubernetes/kubelet-plugins/volume/exec/oracle~oci/config.yaml`
78+
in the following format:
79+
80+
```yaml
81+
---
82+
auth:
83+
tenancy: ocid1.tenancy.oc1..aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
84+
compartment: ocid1.compartment.oc1..aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
85+
user: ocid1.user.oc1..aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
86+
region: us-phoenix-1
87+
key: |
88+
-----BEGIN RSA PRIVATE KEY-----
89+
<snip>
90+
-----END RSA PRIVATE KEY-----
91+
fingerprint: d4:1d:8c:d9:8f:00:b2:04:e9:80:09:98:ec:f8:42:7e
92+
```
93+
94+
If `"region"` and/or `"compartment"` are not specified in the config file
95+
they will be retrieved from the hosts [OCI metadata service][4].
96+
97+
#### Extra configuration values
98+
99+
You can set these in the environment to override the default values.
100+
101+
* `OCI_FLEXD_DRIVER_LOG_DIR` - Directory where the log file is written (Default: `/usr/libexec/kubernetes/kubelet-plugins/volume/exec/oracle~oci`)
102+
* `OCI_FLEXD_DRIVER_DIRECTORY` - Directory where the driver binary lives (Default:
103+
`/usr/libexec/kubernetes/kubelet-plugins/volume/exec/oracle~oci`)
104+
105+
# Tutorial
106+
107+
This guide will walk you through creating a Pod with persistent storage. It assumes
108+
that you have already installed the flexvolume driver in your cluster.
109+
110+
See [examples/nginx.yaml](examples/nginx.yaml) for a finished Kubernetes manifest that ties all these concepts
111+
together.
112+
113+
1. Create a block storage volume. This can be done using the `oci` [CLI][5] as follows:
114+
115+
```bash
116+
$ oci bv volume create \
117+
--availability-domain="aaaa:PHX-AD-1" \
118+
--compartment-id "ocid1.compartment.oc1..aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
119+
```
120+
121+
2. Add a volume to your `pod.yml` in the format below and named with the last
122+
section of your volume's OCID (see limitations). E.g. a volume with the OCID
123+
124+
```
125+
ocid1.volume.oc1.phx.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
126+
```
127+
128+
Would be named `aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa`
129+
in the pod.yml as shown below.
130+
131+
```yaml
132+
volumes:
133+
- name: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
134+
flexVolume:
135+
driver: "oracle/oci"
136+
fsType: "ext4"
137+
```
138+
139+
3. Add volume mount(s) in the appropriate container(s) in your as follows:
140+
141+
```yaml
142+
volumeMounts:
143+
- name: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
144+
mountPath: /usr/share/nginx/html
145+
```
146+
(Where `"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"` is the
147+
last '.' sparated section of the volume OCID.)
148+
149+
### Fixing a Pod to a Node
150+
151+
It's important to note that a block volume can only be attached to a Node that runs
152+
in the same AD. To get around this problem, you can use a nodeSelector to ensure
153+
that a Pod is scheduled on a particular Node.
154+
155+
This following example shows you how to do this.
156+
157+
```yaml
158+
apiVersion: v1
159+
kind: Pod
160+
metadata:
161+
name: nginx
162+
labels:
163+
app: nginx
164+
spec:
165+
containers:
166+
- name: nginx
167+
image: nginx
168+
ports:
169+
- containerPort: 80
170+
volumeMounts:
171+
- name: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
172+
mountPath: /usr/share/nginx/html
173+
nodeSelector:
174+
node.info/availability.domain: 'UpwH-US-ASHBURN-AD-1'
175+
volumes:
176+
- name: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
177+
flexVolume:
178+
driver: "oracle/oci"
179+
fsType: "ext4"
180+
```
181+
182+
## Debugging
183+
184+
The flexvolume driver writes logs to `/usr/libexec/kubernetes/kubelet-plugins/volume/exec/oracle~oci/oci_flexvolume_driver.log` by default.
185+
186+
## Assumptions
187+
188+
- If a Flexvolume is specified for a Pod, it will only work with a single
189+
replica. (or if there is more than one replica for a Pod, they will all have
190+
to run on the same Kubernetes Node). This is because a volume can only be
191+
attached to one instance at any one time. Note: This is in common with both
192+
the Amazon and Google persistent volume implementations, which also have the
193+
same constraint.
194+
195+
- If nodes in the cluster span availability domain you must make sure your Pods are scheduled
196+
in the correct availability domain. This can be achieved using the label selectors with the zone/region.
197+
198+
Using the [oci-volume-provisioner][6]
199+
makes this much easier.
200+
201+
- For all nodes in the cluster, the instance display name in the OCI API must
202+
match with the instance hostname, start with the vnic hostnamelabel or match the public IP.
203+
This relies on the requirement that the nodename must be resolvable.
204+
205+
## Limitations
206+
207+
Due to [kubernetes/kubernetes#44737][7] ("Flex volumes which implement
208+
`getvolumename` API are getting unmounted during run time") we cannot implement
209+
`getvolumename`. From the issue:
210+
211+
> Detach call uses volume name, so the plugin detach has to work with PV Name
212+
213+
This means that the Persistent Volume (PV) name in the `pod.yml` _must_ be
214+
the last part of the block volume OCID ('.' separated). Otherwise, we would
215+
have no way of determining which volume to detach from which worker node. Even
216+
if we were to store state at the time of volume attachment PV names would have
217+
to be unique across the cluster which is an unreasonable constraint.
218+
219+
The full OCID cannot be used because the PV name must be shorter than 63
220+
characters and cannot contain '.'s. To reconstruct the OCID we use the region
221+
of the master on which `Detach()` is exected so this blocks support for cross
222+
region clusters.
19223

20224
## Support
21225

@@ -25,3 +229,7 @@ Please checkout our documentation. If you find a bug, please raise an
25229
[1]: https://docs.us-phoenix-1.oraclecloud.com/Content/Block/Concepts/overview.htm
26230
[2]: https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md
27231
[3]: https://github.com/oracle/oci-volume-provisoner
232+
[4]: https://docs.us-phoenix-1.oraclecloud.com/Content/Compute/Tasks/gettingmetadata.htm
233+
[5]: https://docs.us-phoenix-1.oraclecloud.com/Content/API/SDKDocs/cli.htm
234+
[6]: https://github.com/oracle/oci-volume-provisioner
235+
[7]: https://github.com/kubernetes/kubernetes/issues/44737

docs/install.md

Lines changed: 0 additions & 96 deletions
This file was deleted.

0 commit comments

Comments
 (0)