feat: migrate tools support pika v3.5.0 #2984

pro-spild · 2024-12-24T04:55:12Z

feature

pika-migrate-tools支持3.5.0。

Test

写入SET/HSET/LPUSH/SADD/ZADD 各10000000条，然后运行migrate-tool。在迁移期间写入/DEL/HDEL/LPOP/SREM/ZREM各5000000条。在sourceDB和targetDB对操作的key进行一致性测试，结果均一致。

coderabbitai · 2024-12-24T04:55:21Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

wangshao1 · 2025-01-09T06:20:52Z

src/pika_repl_bgworker.cc

+      int timestamp = std::atoi(argv[2].data());
+      std::string value = argv[3];
+
+      int seconds = timestamp - time(NULL);


这个地方会不会溢出?

不会， ParseAndSendPikaCommand最后调用的是RedisSender::SendRedisCommand()，当其超过100000条待发送命令的时候就会被阻塞在这。

wangshao1 · 2025-01-09T06:24:57Z

src/pika_sender.cc

+          s = cli_->Recv(&resp);
+          if (resp[0] == "OK") {
+          } else {
+            LOG(FATAL) << "Connect to redis(" << ip_ << ":" << port_ << ") Invalid password";


FATAL级别的日志会直接abort进程，所以我理解这个地方打印个ERROR的日志加重试吧。

以修改为ERROR

wangshao1 · 2025-01-09T06:26:30Z

src/pika_sender.cc

+          s = cli_->Recv(&resp);
+          if (s.ok()) {
+            if (resp[0] == "NOAUTH Authentication required.") {
+              LOG(FATAL) << "Ping redis(" << ip_ << ":" << port_ << ") NOAUTH Authentication required";


同上，FATAL级别日志会导致进程直接退出

wangshao1 · 2025-01-09T06:28:34Z

src/pika_sender.cc

+    elements_--;
+    LoadKey(key);
+    cli_->Close();
+    LOG(INFO) <<  s.ToString().data();


LOG（INFO）<< s.ToString()就可以了.
另外，这里应该得判断下返回值，如果返回结果不符合预期，我理解需要打印出具体失败的key和类型，而且是需要一直重试直到成功的，要不就丢数据了。

这里失败会调用LoadKey将key重新存到缓冲区，等待下次重新发送

wangshao1 · 2025-01-09T07:05:15Z

src/pika_sender.cc

+    if (!s.ok()) {
+      delete cli_;
+      cli_ = NULL;
+      LOG(WARNING) << "Can not connect to " << ip_ << ":" << port_ << ", status: " << s.ToString();


如果是connect失败需要一直重试的话，我理解重试connect即可，感觉不需要重新new对象。

done
将connectRedis的逻辑修改为重新connect，cli_初始化放在pikaSender构造函数中

wangshao1 · 2025-01-09T07:07:32Z

src/pika_sender.cc

+        pstd::Status s = cli_->Send(&cmd);
+
+        if (s.ok()) {
+          s = cli_->Recv(&resp);


auth返回值是不是需要判断下，如果是穿的密码是错的，就不用重试了。

wangshao1 · 2025-01-09T07:20:23Z

src/pika_sender.cc

+  LOG(INFO) << "Start sender thread...";
+
+  if (cli_ == NULL) {
+    ConnectRedis();


如果ConnectRedis返回时，cli不能保证一定初始化成功，那我理解你在外面就需要考虑cli是空指针的问题。或者你就直接让进程退出，比如auth 命令返回密码错误，或者服务端需要auth但migrate启动的时候没有设置密码，这种直接就进程退出。

wangshao1 · 2025-01-09T07:25:12Z

src/redis_sender.cc

+      return 0;
+    }
+
+    cli_->Close();


delete cli_

Updated the logic of connecting to Redis in redisSender and pikaSender

chenbt-hz · 2025-02-28T02:06:10Z

src/pika_conf.cc

+
+    // pika-migrate-tool only support 1 db
+    if (databases_ != 1) {
+      LOG(FATAL) << "pika-migrate-tool only support 1 db";


这块会导致CI过不了
还有个问题：现在pika默认配置就是3db、一般线上都是6db，这种情况怎么处理合适？

wangshao1 · 2025-02-28T09:02:02Z

刚发现，你的改动好像是在pika/src或者是pika/include下，是不应该修改 pika/tools/migrate_tool下呢？

Issues-translate-bot · 2025-02-28T09:02:17Z

Bot detected the issue body's language is not English, translate it automatically.

I just found out that your changes seem to be under pika/src or pika/include. Shouldn't you modify pika/tools/migrate_tool?

pro-spild · 2025-02-28T09:30:05Z

刚发现，你的改动好像是在pika/src或者是pika/include下，是不应该修改 pika/tools/migrate_tool下呢？
3.5当时直接在src下面改了，我不知道重新commit会不会导致你以前的review记录没了，所以想着review结束了再和4.0一样放到tools下面。当时unstable分支下面有这个工具，3.5没有，我就直接在src下面改了。

Issues-translate-bot · 2025-02-28T09:30:18Z

Bot detected the issue body's language is not English, translate it automatically.

I just found out that your changes seem to be under pika/src or pika/include. Shouldn't you modify pika/tools/migrate_tool?
3.5 was changed directly under src at that time. I don’t know if recommit will cause your previous review record to be lost, so I thought that after the review ended, I put it under tools like 4.0. At that time, there was this tool under the unstable branch, but it didn't have it in 3.5, so I changed it directly under src.

guozhihao-224 · 2025-03-03T04:02:59Z

这里我尝试使用该工具进行迁移，发现其仅将sst文件下载到migrate-tool本地，但是没有将命令转发给下游

Issues-translate-bot · 2025-03-03T04:03:10Z

Bot detected the issue body's language is not English, translate it automatically.

Here I tried to use the tool for migration and found that it only downloads the sst file to migrate-tool locally, but does not forward the command to the downstream

pro-spild · 2025-03-04T06:50:17Z

这里我尝试使用该工具进行迁移，发现其仅将sst文件下载到migrate-tool本地，但是没有将命令转发给下游

@guozhihao-224 可以贴一下pika_migrate用的conf吗？还有使用场景是啥样的呢？

Issues-translate-bot · 2025-03-04T06:50:30Z

Bot detected the issue body's language is not English, translate it automatically.

Here I tried using the tool for migration and found that it only downloads the sst file to migrate-tool locally, but does not forward the command to the downstream

@guozhihao-224 Can you post the conf used for pika_migrate? What are the usage scenarios?

guozhihao-224 · 2025-03-04T09:46:34Z

这里我尝试使用该工具进行迁移，发现其仅将 sst 文件下载到 migrate-tool 本地，但是没有将命令转发给下游

@guozhihao-224可以贴一下pika_migrate用的conf吗？还有使用场景是啥样的呢？
是pika3.5.5 迁移到 3.5.5版本，100GB量级的数据

这里测试后，发现有将命令往下转发给下游，但是我发现，设置redis-sender-num大于8的话，就会在转发一部分数据后卡住不动了，设置成其他值会的比如5 — 8 之间，也会突然出现卡住不执行的状态。有试过设置为2能够跑完迁移任务，但是设置为2后，迁移时间太长了，100GB的数据的迁移1个多小时

是否有死锁的情况出现
对应的conf：
pika-migrate.txt

Issues-translate-bot · 2025-03-04T09:46:48Z

Bot detected the issue body's language is not English, translate it automatically.

Here I tried using the tool for migration and found that it only downloads the sst file to migrate-tool locally, but does not forward the command to the downstream

@guozhihao-224 Can you post the conf used by pika_migrate? What are the usage scenarios?
After testing here, I found that the command is forwarded downstream to the downstream, but I found that if the redis-sender-num is greater than 8, it will be stuck after forwarding part of the data, but there is no problem with setting 8. This may be due to the influence of that direction setting.

Corresponding conf:
pika-migrate.txt

pro-spild · 2025-04-03T05:30:23Z

这里我尝试使用该工具进行迁移，发现其仅将 sst 文件下载到 migrate-tool 本地，但是没有将命令转发给下游

@guozhihao-224可以贴一下pika_migrate用的conf吗？还有使用场景是啥样的呢？
是pika3.5.5 迁移到 3.5.5版本，100GB量级的数据

这里测试后，发现有将命令往下转发给下游，但是我发现，设置redis-sender-num大于8的话，就会在转发一部分数据后卡住不动了，设置成其他值会的比如5 — 8 之间，也会突然出现卡住不执行的状态。有试过设置为2能够跑完迁移任务，但是设置为2后，迁移时间太长了，100GB的数据的迁移1个多小时

是否有死锁的情况出现对应的conf： pika-migrate.txt

这边可能还需要提供一下pika_migrate的日志，我这边测试5-8之间同量级的数据均没有出现卡住的情况。

Issues-translate-bot · 2025-04-03T05:30:34Z

Bot detected the issue body's language is not English, translate it automatically.

Here I tried using the tool for migration and found that it only downloads the sst file to migrate-tool locally, but does not forward the command to the downstream

@guozhihao-224 Can you post the conf used by pika_migrate? What are the usage scenarios?
is pika3.5.5 migrated to 3.5.5 version, 100GB data

After testing here, I found that the command is forwarded downstream to the downstream, but I found that if the redis-sender-num is greater than 8, it will be stuck after forwarding part of the data. If the setting to other values, such as between 5 and 8, it will suddenly become stuck and not executed. I have tried setting it to 2 to complete the migration task, but after setting it to 2, the migration time is too long, and the migration of 100GB of data will take more than an hour.

Whether there is a deadlock? Corresponding conf: pika-migrate.txt

Here, you may also need to provide the log of pika_migrate. I tested that the data of the same magnitude between 5-8 did not get stuck.

github-actions bot added Invalid PR Title ✏️ Feature New feature or request 📒 Documentation Improvements or additions to documentation labels Dec 24, 2024

pro-spild changed the title ~~feature: migrate tools support pika v3.5.0~~ feat: migrate tools support pika v3.5.0 Dec 24, 2024

github-actions bot removed the Invalid PR Title label Dec 24, 2024

wangshao1 requested review from chenbt-hz and wangshao1 January 3, 2025 12:09

feature: migrate tools support pika v3.5.0

e744786

pro-spild force-pushed the pika-migrate-v3.5.0 branch from 188566a to e744786 Compare January 6, 2025 09:03

wangshao1 reviewed Jan 9, 2025

View reviewed changes

Modified some log levels of redisSender and pikaSender,

014bc67

Updated the logic of connecting to Redis in redisSender and pikaSender

chenbt-hz reviewed Feb 28, 2025

View reviewed changes

wangshao1 requested a review from baixin01 March 7, 2025 10:10

Simplify the number of files in tools/pika-migrate

5c1cb48

feat: migrate tools support pika v3.5.0 #2984

Are you sure you want to change the base?

feat: migrate tools support pika v3.5.0 #2984

Uh oh!

Conversation

pro-spild commented Dec 24, 2024

feature

Test

Uh oh!

coderabbitai bot commented Dec 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wangshao1 commented Feb 28, 2025

Uh oh!

Issues-translate-bot commented Feb 28, 2025

Uh oh!

pro-spild commented Feb 28, 2025

Uh oh!

Issues-translate-bot commented Feb 28, 2025

Uh oh!

guozhihao-224 commented Mar 3, 2025

Uh oh!

Issues-translate-bot commented Mar 3, 2025

Uh oh!

pro-spild commented Mar 4, 2025

Uh oh!

Issues-translate-bot commented Mar 4, 2025

Uh oh!

guozhihao-224 commented Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Issues-translate-bot commented Mar 4, 2025

Uh oh!

pro-spild commented Apr 3, 2025

Uh oh!

Issues-translate-bot commented Apr 3, 2025

Uh oh!

Uh oh!

coderabbitai bot commented Dec 24, 2024 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)

guozhihao-224 commented Mar 4, 2025 •

edited

Loading