4
4
5
5
This project implements a flexvolume driver for Kubernetes clusters
6
6
running on Oracle Cloud Infrastructure (OCI). It enables mounting of [ OCI block
7
- storage volumes] [ 1 ] to Kubernetes pods via the [ Flexvolume] [ 2 ] plugin interface.
7
+ storage volumes] [ 1 ] to Kubernetes Pods via the [ Flexvolume] [ 2 ] plugin interface.
8
8
9
9
We recommend you use this driver in conjunction with the OCI Volume Provisioner.
10
10
See the [ oci-volume-provisioner] [ 3 ] for more information.
11
11
12
12
## Install / Setup
13
13
14
- See [ here] ( docs/install.md ) for information about installing and configuring the volume driver.
14
+ We publish the OCI flexvolume driver as a single binary that needs to be
15
+ installed on every node in your Kubernetes cluster.
15
16
16
- ## Getting started
17
+ ### Ansible
17
18
18
- See [ here] ( docs/tutorial.md ) for a tutorial and guide to using the flexvolume volume driver.
19
+ The recommended way to install the driver is with Ansible.
20
+
21
+ Compile the oci binary or download the latest release from Github.
22
+
23
+ ```
24
+ make build
25
+ ```
26
+
27
+ Now you can use Ansible to deploy the driver to your cluster
28
+
29
+ ```
30
+ cd ansible
31
+ ```
32
+
33
+ Create an inventory file
34
+
35
+ ```
36
+ cp hosts.example hosts
37
+ ```
38
+
39
+ Add the details for all the masters and nodes in your cluster
40
+ (the oci-flexvolume-driver needs to be installed on all masters and workers)
41
+
42
+ ```
43
+ [all:vars]
44
+ ansible_ssh_user=opc
45
+
46
+ [masters]
47
+ ...
48
+
49
+ [workers]
50
+ ...
51
+ ```
52
+
53
+ #### Run the playbook.
54
+
55
+ ```
56
+ ansible-playbook -i hosts \
57
+ --private-key=generated/instances_id_rsa site.yaml
58
+ ```
59
+
60
+ ### Manually
61
+
62
+ The driver should be installed in the volume plugin path on ** every**
63
+ node in your Kubernetes cluster at the following location:
64
+ ` /usr/libexec/kubernetes/kubelet-plugins/volume/exec/oracle~oci/oci ` .
65
+
66
+ NOTE: If running kube-controller-managers in a container you _ must_ ensure that
67
+ the plugin directory is mounted into the container. See:
68
+ https://gitlab-odx.oracle.com/odx/oke-prime/merge_requests/15 for specifics.
69
+
70
+ #### Configuration
71
+
72
+ The driver requires API credentials for a OCI account with the ability
73
+ to attach and detach [ OCI block storage volumes] [ 1 ] from to/from the appropriate
74
+ nodes in the cluster.
75
+
76
+ These credentials should be provided via a YAML file present on ** master** nodes
77
+ in the cluster at ` /usr/libexec/kubernetes/kubelet-plugins/volume/exec/oracle~oci/config.yaml `
78
+ in the following format:
79
+
80
+ ``` yaml
81
+ ---
82
+ auth :
83
+ tenancy : ocid1.tenancy.oc1..aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
84
+ compartment : ocid1.compartment.oc1..aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
85
+ user : ocid1.user.oc1..aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
86
+ region : us-phoenix-1
87
+ key : |
88
+ -----BEGIN RSA PRIVATE KEY-----
89
+ <snip>
90
+ -----END RSA PRIVATE KEY-----
91
+ fingerprint : d4:1d:8c:d9:8f:00:b2:04:e9:80:09:98:ec:f8:42:7e
92
+ ` ` `
93
+
94
+ If ` " region" ` and/or ` " compartment" ` are not specified in the config file
95
+ they will be retrieved from the hosts [OCI metadata service][4].
96
+
97
+ #### Extra configuration values
98
+
99
+ You can set these in the environment to override the default values.
100
+
101
+ * ` OCI_FLEXD_DRIVER_LOG_DIR` - Directory where the log file is written (Default: `/usr/libexec/kubernetes/kubelet-plugins/volume/exec/oracle~oci`)
102
+ * `OCI_FLEXD_DRIVER_DIRECTORY` - Directory where the driver binary lives (Default:
103
+ ` /usr/libexec/kubernetes/kubelet-plugins/volume/exec/oracle~oci` )
104
+
105
+ # Tutorial
106
+
107
+ This guide will walk you through creating a Pod with persistent storage. It assumes
108
+ that you have already installed the flexvolume driver in your cluster.
109
+
110
+ See [examples/nginx.yaml](examples/nginx.yaml) for a finished Kubernetes manifest that ties all these concepts
111
+ together.
112
+
113
+ 1. Create a block storage volume. This can be done using the `oci` [CLI][5] as follows :
114
+
115
+ ` ` ` bash
116
+ $ oci bv volume create \
117
+ --availability-domain="aaaa:PHX-AD-1" \
118
+ --compartment-id "ocid1.compartment.oc1..aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
119
+ ` ` `
120
+
121
+ 2. Add a volume to your `pod.yml` in the format below and named with the last
122
+ section of your volume's OCID (see limitations). E.g. a volume with the OCID
123
+
124
+ ```
125
+ ocid1.volume.oc1.phx.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
126
+ ```
127
+
128
+ Would be named `aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa`
129
+ in the pod.yml as shown below.
130
+
131
+ ```yaml
132
+ volumes:
133
+ - name: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
134
+ flexVolume:
135
+ driver: "oracle/oci"
136
+ fsType: "ext4"
137
+ ```
138
+
139
+ 3 . Add volume mount(s) in the appropriate container(s) in your as follows:
140
+
141
+ ``` yaml
142
+ volumeMounts :
143
+ - name : " aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
144
+ mountPath : /usr/share/nginx/html
145
+ ` ` `
146
+ (Where ` " aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" ` is the
147
+ last '.' sparated section of the volume OCID.)
148
+
149
+ ### Fixing a Pod to a Node
150
+
151
+ It's important to note that a block volume can only be attached to a Node that runs
152
+ in the same AD. To get around this problem, you can use a nodeSelector to ensure
153
+ that a Pod is scheduled on a particular Node.
154
+
155
+ This following example shows you how to do this.
156
+
157
+ ` ` ` yaml
158
+ apiVersion : v1
159
+ kind : Pod
160
+ metadata :
161
+ name : nginx
162
+ labels :
163
+ app : nginx
164
+ spec :
165
+ containers :
166
+ - name : nginx
167
+ image : nginx
168
+ ports :
169
+ - containerPort : 80
170
+ volumeMounts :
171
+ - name : aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
172
+ mountPath : /usr/share/nginx/html
173
+ nodeSelector :
174
+ node.info/availability.domain : ' UpwH-US-ASHBURN-AD-1'
175
+ volumes :
176
+ - name : aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
177
+ flexVolume :
178
+ driver : " oracle/oci"
179
+ fsType : " ext4"
180
+ ` ` `
181
+
182
+ ## Debugging
183
+
184
+ The flexvolume driver writes logs to ` /usr/libexec/kubernetes/kubelet-plugins/volume/exec/oracle~oci/oci_flexvolume_driver.log` by default.
185
+
186
+ # # Assumptions
187
+
188
+ - If a Flexvolume is specified for a Pod, it will only work with a single
189
+ replica. (or if there is more than one replica for a Pod, they will all have
190
+ to run on the same Kubernetes Node). This is because a volume can only be
191
+ attached to one instance at any one time. Note : This is in common with both
192
+ the Amazon and Google persistent volume implementations, which also have the
193
+ same constraint.
194
+
195
+ - If nodes in the cluster span availability domain you must make sure your Pods are scheduled
196
+ in the correct availability domain. This can be achieved using the label selectors with the zone/region.
197
+
198
+ Using the [oci-volume-provisioner][6]
199
+ makes this much easier.
200
+
201
+ - For all nodes in the cluster, the instance display name in the OCI API must
202
+ match with the instance hostname, start with the vnic hostnamelabel or match the public IP.
203
+ This relies on the requirement that the nodename must be resolvable.
204
+
205
+ # # Limitations
206
+
207
+ Due to [kubernetes/kubernetes#44737][7] ("Flex volumes which implement
208
+ ` getvolumename` API are getting unmounted during run time") we cannot implement
209
+ `getvolumename`. From the issue :
210
+
211
+ > Detach call uses volume name, so the plugin detach has to work with PV Name
212
+
213
+ This means that the Persistent Volume (PV) name in the `pod.yml` _must_ be
214
+ the last part of the block volume OCID ('.' separated). Otherwise, we would
215
+ have no way of determining which volume to detach from which worker node. Even
216
+ if we were to store state at the time of volume attachment PV names would have
217
+ to be unique across the cluster which is an unreasonable constraint.
218
+
219
+ The full OCID cannot be used because the PV name must be shorter than 63
220
+ characters and cannot contain '.'s. To reconstruct the OCID we use the region
221
+ of the master on which `Detach()` is exected so this blocks support for cross
222
+ region clusters.
19
223
20
224
# # Support
21
225
@@ -25,3 +229,7 @@ Please checkout our documentation. If you find a bug, please raise an
25
229
[1] : https://docs.us-phoenix-1.oraclecloud.com/Content/Block/Concepts/overview.htm
26
230
[2] : https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md
27
231
[3] : https://github.com/oracle/oci-volume-provisoner
232
+ [4] : https://docs.us-phoenix-1.oraclecloud.com/Content/Compute/Tasks/gettingmetadata.htm
233
+ [5] : https://docs.us-phoenix-1.oraclecloud.com/Content/API/SDKDocs/cli.htm
234
+ [6] : https://github.com/oracle/oci-volume-provisioner
235
+ [7] : https://github.com/kubernetes/kubernetes/issues/44737
0 commit comments