Skip to content

Fix: Resolve protobuf generation bloat with sparse mode filtering #1525

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Naq302 wants to merge 9 commits into
base: master
Choose a base branch
from
Open

Fix: Resolve protobuf generation bloat with sparse mode filtering #1525

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
13c02d5
fix: improve THORChain genMsgs.sh script robustness and error handling
Naq302 Aug 26, 2025
2b64e59
Merge branch 'master' into fix/improve-genmsg-scripts
Naq302 Sep 5, 2025
e0cc58c
Updates to scripts and reverted protos
Naq302 Sep 5, 2025
8e0eead
Merge remote-tracking branch 'origin/master' into fix/improve-genmsg-…
Naq302 Sep 5, 2025
a079d62
resolve protobuf generation bloat with sparse mode filtering
Naq302 Sep 5, 2025
cb614fe
Code Rabbit Feedback
Naq302 Sep 5, 2025
1ca3399
Reduced clone time and adding back in coin for thorchain. wip
Naq302 Sep 5, 2025
4f9cad6
Fix THORChain protobuf bloat using minimal common.proto
Naq302 Sep 5, 2025
4d2ddf2
CR feedback
Naq302 Sep 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 33 additions & 7 deletions packages/xchain-mayachain/genMsgs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,29 +1,55 @@
#!/bin/bash

# this script checks out mayanode master and generates the proto3 typescript buindings for MsgDeposit and MsgSend
# This script updates MAYAChain Protobuf bindings for MsgDeposit and MsgSend

MSG_COMPILED_OUTPUTFILE=src/types/proto/MsgCompiled.js
MSG_COMPILED_TYPES_OUTPUTFILE=src/types/proto/MsgCompiled.d.ts

TMP_DIR=$(mktemp -d)

tput setaf 2
echo "Checking out https://gitlab.com/mayachain/mayanode to $TMP_DIR"
echo "Checking out https://gitlab.com/mayachain/mayanode to $TMP_DIR"
tput sgr0
(cd $TMP_DIR && git clone https://gitlab.com/mayachain/mayanode)
(cd "$TMP_DIR" && git clone https://gitlab.com/mayachain/mayanode)

# Generate msgs
# Verify proto files exist
tput setaf 2
echo "Checking proto files"
tput sgr0
ls "$TMP_DIR/mayanode/proto/mayachain/v1/common/common.proto" || echo "common.proto missing"
ls "$TMP_DIR/mayanode/proto/mayachain/v1/x/mayachain/types/msg_deposit.proto" || echo "msg_deposit.proto missing"
ls "$TMP_DIR/mayanode/proto/mayachain/v1/x/mayachain/types/msg_send.proto" || echo "msg_send.proto missing"

# Download cosmos/base/v1beta1/coin.proto from cosmossdk if needed
if [ ! -f "$TMP_DIR/mayanode/third_party/proto/cosmos/base/v1beta1/coin.proto" ]; then
tput setaf 2
echo "Downloading cosmos/base/v1beta1/coin.proto from cosmossdk"
tput sgr0
mkdir -p "$TMP_DIR/mayanode/third_party/proto/cosmos/base/v1beta1"
curl -o "$TMP_DIR/mayanode/third_party/proto/cosmos/base/v1beta1/coin.proto" \
"https://raw.githubusercontent.com/cosmos/cosmos-sdk/main/proto/cosmos/base/v1beta1/coin.proto"
fi
Copy link
Contributor

@coderabbitai coderabbitai bot Aug 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Harden coin.proto download (fail on HTTP errors, retries, clear messaging)

Current curl lacks -f/-L and doesn’t abort on failures; a partial/empty file will break generation.

Apply this diff:

-# Download cosmos/base/v1beta1/coin.proto from cosmossdk if needed
-if [ ! -f "$TMP_DIR/mayanode/third_party/proto/cosmos/base/v1beta1/coin.proto" ]; then
-  tput setaf 2
-  echo "Downloading cosmos/base/v1beta1/coin.proto from cosmossdk"
-  tput sgr0
-  mkdir -p "$TMP_DIR/mayanode/third_party/proto/cosmos/base/v1beta1"
-  curl -o "$TMP_DIR/mayanode/third_party/proto/cosmos/base/v1beta1/coin.proto" \
-    "https://raw.githubusercontent.com/cosmos/cosmos-sdk/main/proto/cosmos/base/v1beta1/coin.proto"
-fi
+# Download cosmos/base/v1beta1/coin.proto from cosmossdk if needed
+COSMOS_COIN_PROTO="$TMP_DIR/mayanode/third_party/proto/cosmos/base/v1beta1/coin.proto"
+if [ ! -f "$COSMOS_COIN_PROTO" ]; then
+  tput setaf 2
+  echo "Downloading cosmos/base/v1beta1/coin.proto from cosmossdk"
+  tput sgr0
+  mkdir -p "$(dirname "$COSMOS_COIN_PROTO")"
+  if ! curl -fSL --retry 3 --retry-delay 2 -o "$COSMOS_COIN_PROTO" \
+    "https://raw.githubusercontent.com/cosmos/cosmos-sdk/main/proto/cosmos/base/v1beta1/coin.proto"; then
+    echo "Error: Failed to download cosmos coin.proto"
+    exit 1
+  fi
+  echo "✓ Downloaded cosmos coin.proto"
+else
+  echo "✓ cosmos coin.proto already exists"
+fi
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# Download cosmos/base/v1beta1/coin.proto from cosmossdk if needed
if [ ! -f "$TMP_DIR/mayanode/third_party/proto/cosmos/base/v1beta1/coin.proto" ]; then
tput setaf 2
echo "Downloading cosmos/base/v1beta1/coin.proto from cosmossdk"
tput sgr0
mkdir -p "$TMP_DIR/mayanode/third_party/proto/cosmos/base/v1beta1"
curl -o "$TMP_DIR/mayanode/third_party/proto/cosmos/base/v1beta1/coin.proto" \
"https://raw.githubusercontent.com/cosmos/cosmos-sdk/main/proto/cosmos/base/v1beta1/coin.proto"
fi
# Download cosmos/base/v1beta1/coin.proto from cosmossdk if needed
COSMOS_COIN_PROTO="$TMP_DIR/mayanode/third_party/proto/cosmos/base/v1beta1/coin.proto"
if [ ! -f "$COSMOS_COIN_PROTO" ]; then
tput setaf 2
echo "Downloading cosmos/base/v1beta1/coin.proto from cosmossdk"
tput sgr0
mkdir -p "$(dirname "$COSMOS_COIN_PROTO")"
if ! curl -fSL --retry 3 --retry-delay 2 -o "$COSMOS_COIN_PROTO" \
"https://raw.githubusercontent.com/cosmos/cosmos-sdk/main/proto/cosmos/base/v1beta1/coin.proto"; then
echo "Error: Failed to download cosmos coin.proto"
exit 1
fi
echo "✓ Downloaded cosmos coin.proto"
else
echo "✓ cosmos coin.proto already exists"
fi
🤖 Prompt for AI Agents 
In packages/xchain-mayachain/genMsgs.sh around lines 23 to 31, harden the
coin.proto download: use curl with -f -L -S and retry flags (e.g., --retry and
--retry-delay) so HTTP errors and redirects fail fast and retry, write to a
temporary file and only move it into place on successful curl to avoid partial
files, remove any temp file on failure, check curl's exit code and exit the
script non-zero on failure, and update the echo/tput messages to clearly report
success or the HTTP/error returned.


# Generate Protobuf JS bindings with include path
tput setaf 2
echo "Generating $MSG_COMPILED_OUTPUTFILE"
tput sgr0
yarn run pbjs -w commonjs -t static-module $TMP_DIR/mayanode/proto/mayachain/v1/common/common.proto $TMP_DIR/mayanode/proto/mayachain/v1/x/mayachain/types/msg_deposit.proto $TMP_DIR/mayanode/proto/mayachain/v1/x/mayachain/types/msg_send.proto $TMP_DIR/mayanode/third_party/proto/cosmos/base/v1beta1/coin.proto -o $MSG_COMPILED_OUTPUTFILE
yarn pbjs -w commonjs -t static-module \
-p "$TMP_DIR/mayanode/proto" \
-p "$TMP_DIR/mayanode/third_party/proto" \
"$TMP_DIR/mayanode/proto/mayachain/v1/common/common.proto" \
"$TMP_DIR/mayanode/proto/mayachain/v1/x/mayachain/types/msg_deposit.proto" \
"$TMP_DIR/mayanode/proto/mayachain/v1/x/mayachain/types/msg_send.proto" \
"$TMP_DIR/mayanode/third_party/proto/cosmos/base/v1beta1/coin.proto" \
-o "$MSG_COMPILED_OUTPUTFILE" 2>pbjs_errors.txt

# Generate TypeScript definitions
tput setaf 2
echo "Generating $MSG_COMPILED_TYPES_OUTPUTFILE"
tput sgr0
yarn run pbts $MSG_COMPILED_OUTPUTFILE -o $MSG_COMPILED_TYPES_OUTPUTFILE
yarn pbts "$MSG_COMPILED_OUTPUTFILE" -o "$MSG_COMPILED_TYPES_OUTPUTFILE" 2>pbts_errors.txt

tput setaf 2
echo "Removing $TMP_DIR/mayanode"
tput sgr0
rm -rf $TMP_DIR
rm -rf "$TMP_DIR"
Loading