The retrieval speed of v2.6.2 is much slower than v2.6.0

[chaoyifei]

environment:

NAME READY STATUS RESTARTS AGE
milvus-release-etcd-0 1/1 Running 1 (4h47m ago) 25h
milvus-release-etcd-1 1/1 Running 1 (4h47m ago) 25h
milvus-release-etcd-2 1/1 Running 1 (4h47m ago) 25h
milvus-release-milvus-datanode-86d9d96b99-fks4j 1/1 Running 0 72m
milvus-release-milvus-mixcoord-57c7c7d945-pblr7 1/1 Running 0 72m
milvus-release-milvus-proxy-7f59c984c6-lr9vb 1/1 Running 0 72m
milvus-release-milvus-querynode-0-85695ff9f7-clb4n 1/1 Running 0 112s
milvus-release-milvus-querynode-0-85695ff9f7-nd5s7 1/1 Running 0 64m
milvus-release-milvus-streamingnode-6bf9f77fc7-t9g9w 1/1 Running 0 72m

INDEX: IVF-FLAT

JSON INDEX: file_indo_width_idx, file_info_height_idx

After upgrading from version 2.6.0 to version 2.6.2 with the same configuration, the retrieval speed is 6 times slower！

[xiaofan-luan]

this might be expected, if you json shredding happened and split data into multiple field instead of store it in field in 2.6.

can you share your schema and your query?

[xiaofan-luan]

@chaoyifei
I don't think test performnace on attu makes too much sense.
usually milvus search/query takes tens milliseconds, and the front end is way slower than milvus

[chaoyifei]

I found that backend testing is slow now, later went to Attu testing and found that 2.6.0 uses 200ms, while 2.6.2 requires over 3000 milliseconds .The total amount of data is 30 million,use three partitions: default partition, large image partition, and small image partition
The data as follows: 0 ，15 million,15 million.

[chaoyifei]

When I manually increased the number of dataNodes, the speed was restored

[xiaofan-luan]

can you share your code to do search/query?

I think the problem is index build speed can not catch up

[chaoyifei]

What is the role of ConsistencyLevel? The default Bound will be slower，Index has created completed

[zhagnlu]

common.enabledJSONKeyStats Did you change this param from false to true ? The json index building is async process, so the query is keep slow or become normal after?

[chaoyifei]

I did not set this param，it should be the default

[xiaofan-luan]

I did not set this param，it should be the default

then I guess it's nothing to do with 2.6.2
more like index build is not done.

[chaoyifei]

My operation steps here:

1. Upgrade v2.6.2
2. Delete all indexes
3. Create index
4. Wait for the index to be fully created before loading
5. Search test
   I noticed that the speed has slowed down, downgraded to 2.6.0, but it works well

[xiaofan-luan]

waht is your query looks like? can you share the code?
any filters?
waht is output field?
search or query?

I don't think any query should cost that long, no matter 2.6.0 or 2.6.2

[chaoyifei]

POST: /api/v1/collections/file/search
body:

{
    "output_fields": [
        "file_id",
        "file_info"
    ],
    "limit": 15,
    "data": [
        {
            "anns_field": "embedding",
            "data": [] // 768dim random,
            "params": {}
        }
    ],
    "filter": "",
    "consistency_level": "Bounded"
}

[xiaofan-luan]

I don't think there is much different on this case.
but we will try to reproduce in house

[xiaofan-luan]

@yanliang567

please help to verify the retrieve performance of 2.6.0 versus 2.6.2