Skip to content

Commit d1e7474

Browse files
ephraimbuddyephraimbuddy
authored
Update the release process doc (#59)
The significant change here is copying the packages from svn into the dist folder instead of rebuilding packages for PyPI upload after voting. 2 other essential changes: 1. Unified the release docs into one piece 2. Added email template instead of relying on the user modifying Airflow email template
1 parent 183059c commit d1e7474

File tree

3 files changed

+170
-77
lines changed

3 files changed

+170
-77
lines changed

‎README.md

Lines changed: 1 addition & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -89,41 +89,4 @@ with airflow_client.client.ApiClient(configuration) as api_client:
8989
print("Exception when calling ConfigApi->get_config: %s\n" % e)
9090
```
9191

92-
See [README](./airflow_client/README.md#documentation-for-api-endpoints) for full client API documentation.
93-
94-
## Release Process
95-
96-
Typically releases are done coinciding with major and minor releases to Airflow. Therefore a release of (for example) 2.3.0 of this client
97-
would correspond with 2.3.X of Airflow.
98-
99-
The Python client is generated using Airflow's [openapi spec](https://github.com/apache/airflow/blob/master/clients/gen/python.sh).
100-
To update the client for new APIs do the following steps:
101-
102-
```bash
103-
# clone this repo
104-
git clone git@github.com:apache/airflow-client-python.git
105-
106-
# clone Airflow repo (if not already)
107-
git clone git@github.com:apache/airflow.git
108-
```
109-
Edit the file `airflow/airflow/api_connexion/openapi/v1.yaml`
110-
Make sure it has the following `securitySchema`s listed under security `section`
111-
```yaml
112-
security:
113-
- Basic: []
114-
- GoogleOpenId: []
115-
- Kerberos: []
116-
```
117-
If your deployment of Airflow uses any different authentication mechanism than the three listed above, you might need to make further changes to the `v1.yaml` and generate your own client, see [OpenAPI Schema specification](https://swagger.io/docs/specification/authentication/) for details.
118-
(*These changes should not be commited to the upstream `v1.yaml` [as it will generate misleading openapi documentaion](https://github.com/apache/airflow/pull/17174)*)
119-
120-
Update the VERSION string in setup.py with the corresponding new version string.
121-
122-
```bash
123-
cd airflow
124-
125-
# bump up the version in python.sh & run the following command
126-
./clients/gen/python.sh airflow/api_connexion/openapi/v1.yaml ../airflow-client-python/airflow_client
127-
128-
# raise a PR in github for both the repos (airflow & airflow-client-python)
129-
```
92+
See [README](./airflow_client/README.md#documentation-for-api-endpoints) for full client API documentation.

‎dev/README_RELEASE_CLIENT.md

Lines changed: 168 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,58 @@
1717
under the License.
1818
-->
1919

20-
You can find the prerequisites to release Apache Airflow Python Client in [README.md](README.md).
20+
# Release Process
21+
22+
Typically, releases are done coinciding with major and minor releases to Airflow. Therefore, a release of (for example)
23+
2.3.0 of this client would correspond with 2.3.X of Airflow.
24+
25+
The Python client is generated using Airflow's [openapi spec](https://github.com/apache/airflow/blob/master/clients/gen/python.sh).
26+
To update the client for new APIs do the following steps:
27+
28+
```bash
29+
# clone this repo
30+
git clone git@github.com:apache/airflow-client-python.git
31+
cd airflow-client-python
32+
export CLIENT_REPO_ROOT=$(pwd)
33+
cd ..
34+
35+
# clone Airflow repo (if not already)
36+
git clone git@github.com:apache/airflow.git
37+
cd airflow
38+
export AIRFLOW_REPO_ROOT=$(pwd)
39+
```
40+
Edit the file `airflow/airflow/api_connexion/openapi/v1.yaml`
41+
Make sure it has the following `securitySchema`s listed under security `section`
42+
```yaml
43+
security:
44+
- Basic: []
45+
- GoogleOpenId: []
46+
- Kerberos: []
47+
```
48+
If your deployment of Airflow uses any different authentication mechanism than the three listed above, you might need to make further changes to the `v1.yaml` and generate your own client, see [OpenAPI Schema specification](https://swagger.io/docs/specification/authentication/) for details.
49+
(*These changes should not be commited to the upstream `v1.yaml` [as it will generate misleading openapi documentaion](https://github.com/apache/airflow/pull/17174)*)
50+
51+
52+
```bash
53+
54+
# bump up the version in python.sh & run the following command
55+
${AIRFLOW_REPO_ROOT}/clients/gen/python.sh airflow/api_connexion/openapi/v1.yaml ${CLIENT_REPO_ROOT}/airflow-client-python/airflow_client
56+
57+
cd ${CLIENT_REPO_ROOT}
58+
```
59+
60+
- Set your version in `setup.py` (without the RC tag)
61+
62+
- Get a diff between the last airflow version and the current airflow version
63+
which this release is based on:
64+
65+
```shell script
66+
git log 2.3.0..2.5.0 --pretty=oneline
67+
```
68+
69+
- Update CHANGELOG.md with the details.
70+
- Raise a PR in airflow-client-python
71+
- Merge the above PR when approved before proceeding
2172

2273
# Prepare the Apache Airflow Python Client Package RC
2374

@@ -28,22 +79,21 @@ renaming – i.e. the contents of the files must be the same between voted relea
2879
of this the version in the built artifacts that will become the official Apache releases must not include the rcN suffix.
2980

3081
- Set environment variables
31-
82+
3283
```shell script
3384
# Set Version
3485
export VERSION=2.0.0rc1
3586
export VERSION_WITHOUT_RC=${VERSION%rc?}
87+
# Set the airflow version that this release is based
88+
export AIRFLOW_VERSION=2.1.4
3689
3790
3891
# Example after cloning
39-
git clone https://github.com/apache/airflow-client-python.git airflow-client
40-
cd airflow-client
92+
git clone https://github.com/apache/airflow-client-python.git
93+
cd airflow-client-python
4194
export CLIENT_REPO_ROOT=$(pwd)
4295
```
4396

44-
- Set your version to 2.0.0 in `setup.py` (without the RC tag)
45-
- Commit the version change.
46-
4797
- Tag your release
4898

4999
```shell script
@@ -74,22 +124,22 @@ of this the version in the built artifacts that will become the official Apache
74124
- Rename the sdist
75125

76126
```shell script
77-
mv dist/apache-airflow-client-${VERSION_WITHOUT_RC}.tar.gz apache-airflow-client-${VERSION_WITHOUT_RC}-bin.tar.gz
78-
mv dist/apache_airflow_client-${VERSION_WITHOUT_RC}-py3-none-any.whl apache_airflow_client-${VERSION_WITHOUT_RC}-py3-none-any.whl
127+
mv dist/apache-airflow-client-${VERSION_WITHOUT_RC}.tar.gz dist/apache-airflow-client-${VERSION_WITHOUT_RC}-bin.tar.gz
128+
mv dist/apache_airflow_client-${VERSION_WITHOUT_RC}-py3-none-any.whl dist/apache_airflow_client-${VERSION_WITHOUT_RC}-py3-none-any.whl
79129
```
80130

81131
- Generate SHA512/ASC (If you have not generated a key yet, generate it by following instructions on
82132
http://www.apache.org/dev/openpgp.html#key-gen-generate-key)
83133

84134
```shell script
85-
${CLIENT_REPO_ROOT}/dev/sign.sh apache-airflow-client-${VERSION_WITHOUT_RC}-source.tar.gz
86-
${CLIENT_REPO_ROOT}/dev/sign.sh apache-airflow-client-${VERSION_WITHOUT_RC}-bin.tar.gz
87-
${CLIENT_REPO_ROOT}/dev/sign.sh apache_airflow_client-${VERSION_WITHOUT_RC}-py3-none-any.whl
135+
pushd dist
136+
${CLIENT_REPO_ROOT}/dev/sign.sh *
137+
popd
88138
```
89139

90140
- Push the artifacts to ASF dev dist repo
91141

92-
```
142+
```shell script
93143
# First clone the repo
94144
svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
95145
@@ -102,6 +152,8 @@ mv ${CLIENT_REPO_ROOT}/apache{-,_}*client-${VERSION_WITHOUT_RC}* ${VERSION}/
102152
cd ${VERSION}
103153
svn add *
104154
svn commit -m "Add artifacts for Apache Airflow Python Client ${VERSION}"
155+
cd ${CLIENT_REPO_ROOT}
156+
rm -rf airflow-dev
105157
```
106158

107159
## Prepare PyPI convenience "snapshot" packages
@@ -138,7 +190,10 @@ To do this we need to
138190
pip install -i https://test.pypi.org/simple/ apache-airflow-client==${VERSION}
139191

140192
- Upload the package to PyPi's production environment:
141-
`twine upload -r pypi dist/*`
193+
194+
```shell script
195+
twine upload -r pypi dist/*
196+
```
142197

143198
- Again, confirm that the package is available here:
144199
https://pypi.python.org/pypi/apache-airflow-client
@@ -148,12 +203,63 @@ is not supposed to be used by and advertised to the end-users who do not read th
148203

149204
- Push Tag for the release candidate
150205

151-
```shell script
152-
git push origin ${VERSION}
153-
```
206+
```shell script
207+
git push origin ${VERSION}
208+
```
154209

155210
## Prepare Vote email on the Airflow Client release candidate
156-
See Airflow process documented [here](https://github.com/apache/airflow/blob/master/dev/README_RELEASE_AIRFLOW.md#prepare-vote-email-on-the-apache-airflow-release-candidate) (just replace Airflow with Airflow Client).
211+
212+
Subject:
213+
214+
```shell script
215+
cat <<EOF
216+
[VOTE] Release Airflow Python Client ${VERSION_WITHOUT_RC} from ${VERSION}
217+
EOF
218+
```
219+
220+
Body:
221+
222+
```shell script
223+
cat <<EOF
224+
Hey fellow Airflowers,
225+
226+
I have cut the first release candidate for the Airflow Python Client ${VERSION}.
227+
The client consists of APIs corresponding to REST APIs available in
228+
*Apache Airflow ${AIRFLOW_VERSION}*. This email is calling for a vote on
229+
the release, which will last for 72 hours. Consider this my (binding) +1.
230+
231+
Airflow Client ${VERSION} is available at:
232+
https://dist.apache.org/repos/dist/dev/airflow/clients/python/${VERSION}/
233+
234+
Or also available at PyPI:
235+
https://pypi.org/project/apache-airflow-client/${VERSION}/
236+
237+
*apache-airflow-client-${VERSION}-source.tar.gz* is a source release that comes with
238+
INSTALL instructions.
239+
*apache-airflow-client-${VERSION}-bin.tar.gz* is the binary Python "sdist" release.
240+
241+
Public keys are available at:
242+
https://dist.apache.org/repos/dist/release/airflow/KEYS
243+
244+
Only votes from PMC members are binding, but the release manager should
245+
encourage members of the community to test the release and vote with
246+
"(non-binding)".
247+
248+
*Changelog:*
249+
250+
*Major changes:*
251+
...
252+
253+
*Major fixes:*
254+
...
255+
256+
*New API supported:*
257+
...
258+
259+
Cheers,
260+
<your name>
261+
EOF
262+
```
157263

158264
# Verify the release candidate by PMCs
159265
See Airflow process documented [here](https://github.com/apache/airflow/blob/master/dev/README_RELEASE_AIRFLOW.md#verify-the-release-candidate-by-pmcs).
@@ -167,8 +273,6 @@ See Airflow process documented [here](https://github.com/apache/airflow/blob/mas
167273
## Signature check
168274
See Airflow process documented [here](https://github.com/apache/airflow/blob/master/dev/README_RELEASE_AIRFLOW.md#signature-check).
169275

170-
171-
172276
# Verify release candidates by Contributors
173277
This can be done (and we encourage to) by any of the Contributors. In fact, it's best if the
174278
actual users of Airflow Client test it in their own staging/test installations. Each release candidate
@@ -187,23 +291,55 @@ that the client works as you expected.
187291

188292
## Summarize the voting for the Apache Airflow client release
189293

190-
See Airflow process documented [here](https://github.com/apache/airflow/blob/master/dev/README_RELEASE_AIRFLOW.md#publish-the-final-apache-airflow-release) (just replace Airflow with Airflow Client).
294+
```shell script
295+
Hello,
296+
297+
Apache Airflow Python Client 2.5.0 (based on RC1) has been accepted.
298+
299+
3 "+1" binding votes received:
300+
- Ephraim Anierobi
301+
- Jarek Potiuk
302+
- Jed Cunningham
303+
304+
305+
1 "+1" non-binding votes received:
306+
307+
- Pierre Jeambrun
308+
309+
Vote thread:
310+
https://lists.apache.org/thread/1qcj0r67dff3zg0w2vyfhr30fx9xtp3y
311+
312+
I'll continue with the release process, and the release announcement will follow shortly.
313+
314+
Cheers,
315+
<your name>
316+
```
191317

192318
## Publish release to SVN
193319

194320
```shell script
195-
# First clone the repo
321+
# Go to Airflow python client sources first
322+
cd <YOUR_AIRFLOW_CLIENT_REPO_ROOT>
323+
export CLIENT_REPO_ROOT="$(pwd)"
324+
cd ..
325+
# Clone the AS
326+
[ -d asf-dist ] || svn checkout --depth=immediates https://dist.apache.org/repos/dist asf-dist
327+
svn update --set-depth=infinity asf-dist/{release,dev}/airflow
328+
CLIENT_DEV_SVN="${PWD}/asf-dist/dev/airflow/clients/python"
329+
CLIENT_RELEASE_SVN="${PWD}/asf-dist/release/airflow/clients/python"
330+
cd "${CLIENT_RELEASE_SVN}"
331+
196332
export RC=2.0.0rc1
197333
export VERSION=${RC/rc?/}
198-
svn checkout https://dist.apache.org/repos/dist/release/airflow airflow-release
199334
200335
# Create new folder for the release
201-
cd airflow-release/clients/python
202336
svn mkdir ${VERSION}
203337
cd ${VERSION}
204338
205339
# Move the artifacts to svn folder & commit
206-
for f in ../../../../airflow-dev/clients/python/$RC/*; do svn cp $f . ; done
340+
for f in ${CLIENT_DEV_SVN}/$RC/*; do
341+
svn cp $f . ;
342+
done
207343
svn commit -m "Release Apache Airflow Python Client ${VERSION} from ${RC}"
208344
209345
# Remove old release
@@ -220,11 +356,15 @@ Verify that the packages appear in [airflow](https://dist.apache.org/repos/dist/
220356

221357
At this point we release an official package:
222358

223-
- Clean & Build the package:
359+
- Copy the packages from the SVN into the dist folder in CLIENT_REPO_ROOT:
224360

225361
```shell script
226-
git clean -fxd
227-
python setup.py sdist bdist_wheel
362+
rm -rf ${CLIENT_REPO_ROOT}/dist
363+
mkdir ${CLIENT_REPO_ROOT}/dist
364+
cp ${CLIENT_RELEASE_SVN}/${VERSION}/apache-airflow-client-${VERSION}-bin.tar.gz ${CLIENT_REPO_ROOT}/dist/
365+
cp ${CLIENT_RELEASE_SVN}/${VERSION}/apache_airflow_client-${VERSION}-py3-none-any.whl ${CLIENT_REPO_ROOT}/dist/
366+
# Remove the -bin
367+
mv ${CLIENT_REPO_ROOT}/dist/apache-airflow-client-${VERSION}-bin.tar.gz ${CLIENT_REPO_ROOT}/dist/apache-airflow-client-${VERSION}.tar.gz
228368
```
229369

230370
- Verify the artifacts that would be uploaded:
@@ -250,16 +390,6 @@ At this point we release an official package:
250390

251391
- Again, confirm that the package is available here: https://pypi.python.org/pypi/apache-airflow-client
252392

253-
## Update CHANGELOG.md
254-
255-
- Get a diff between the last version and the current version:
256-
257-
```shell script
258-
git log 1.0.0..2.0.0 --pretty=oneline
259-
```
260-
261-
- Update CHANGELOG.md with the details, and commit it.
262-
263393
- Push Tag for the final version
264394

265395
```shell script

‎dev/sign.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,6 @@ SIGN_WITH="${SIGN_WITH:-apache.org}"
2929

3030
for name in "${@}"
3131
do
32-
gpg --armor --local-user "$SIGN_WITH" --output "${name}.asc" --detach-sig "${name}"
32+
gpg --yes -armor --local-user "$SIGN_WITH" --output "${name}.asc" --detach-sig "${name}"
3333
shasum -a 512 "${name}" > "${name}.sha512"
3434
done

0 commit comments

Comments
 (0)