Error when sending large number of requests to curl_cffi when using locust for pressure testing

[GeraldWhitehead]

When I use locust for pressure testing, I use curl_cffi to send the request because I want to simulate fingerprinting. But when I use curl_cffi to send request, the qps can't go up, finally I added gevent.get_hub().threadpool.maxsize = 500 in the code, the qps can go up, but after a while, it will give me an error failed to send heartbeat, setting state to missing. (This error is reported by locust, but for the same request, there is no problem using the requests library). During the execution, the machine has thousands of threads (only dozens using requests).
Can you please tell me how I can make the change, I don't know why this is happening?
I also asked locust and waiting for his reply. But I suspect that it's more the use of curl_cffi to send the request that's causing the problem (since there's no problem with requests)

[lexiforest]

Did you set thread="gevent"?

[lexiforest]

OK, I will try on my machine later. If you already have the scripts to reproduce, please kindly add them here.

[GeraldWhitehead]

OK, I will try on my machine later. If you already have the scripts to reproduce, please kindly add them here.

• master：locust -f test.py --master
• worker: locust -f test.py --processes -1 --worker

from gevent import monkey

monkey.patch_all()
import gevent, sys, traceback
from locust import User, task, TaskSet, events

from curl_cffi import requests
from loguru import logger

gevent.get_hub().threadpool.maxsize = 200


# 设置全局异常处理器
def global_exception_handler(exctype, value, tb):
    tb_str = "".join(traceback.format_exception(exctype, value, tb))
    logger.error(f"【全局异常】{tb_str}")


sys.excepthook = global_exception_handler


class UserStressTest(TaskSet):
    def on_start(self):
        self.h_client = requests.Session(thread="gevent", timeout=10, impersonate="chrome", verify=False)

    @task
    def transaction(self):
        request = {
            "method": "GET",
            "path": "https://httpbin.org/delay/3",
            "headers": {
                "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
                "Accept-Encoding": "gzip, deflate, br, zstd",
                "accept": "application/json, text/plain, */*",
                "accept-language": "zh-CN,zh;q=0.9",
                "origin": "https://p5fun.com",
                "priority": "u=1, i",
                "referer": "https://p5fun.com/",
                "sec-ch-ua": '"Not(A:Brand";v="99", "Google Chrome";v="133", "Chromium";v="133"',
                "sec-ch-ua-mobile": "?0",
                "sec-ch-ua-platform": '"Windows"',
                "sec-fetch-dest": "empty",
                "sec-fetch-mode": "cors",
                "sec-fetch-site": "cross-site",
                "pragma": "no-cache",
                "cookie": "",
            }
        }
        self.public_task(request)

    def public_task(self, request):
        gevent.sleep(3)
        try:
            resp = self.h_client.request(
                method=request["method"],
                url=request["path"]
            )
            assert resp.ok == True, f"status:{resp.status_code}, 未知原因: 日志查看"

        except Exception as ex:
            events.request.fire(
                request_type=request["method"],
                name=request["path"],
                response_time=0,
                response_length=0,
                exception=str(ex))
        else:
            events.request.fire(
                request_type=request["method"],
                name=request["path"],
                response_time=int(resp.elapsed * 1000),
                response_length=len(resp.content),
            )


class TestLocustFile(User):
    host = "https://httpbin.org"
    tasks = [UserStressTest]

[GeraldWhitehead]

OK, I will try on my machine later. If you already have the scripts to reproduce, please kindly add them here.

Hello, do you have a reproduction on your side?

[lexiforest]

I wouldn't use httpbin.org as the target. It's both unethical and unreliable. Is it reproducible with localhost?

[GeraldWhitehead]

httpbin.org

I'm just giving an example, the pressure test uses a local interface, you can replace httpbin.org with an arbitrary address