Sliding window KV cache #12975
Sliding window KV cache #12975
Conversation
Summary: Make the KV cache a sliding window by default. If this is not desired user can check number of tokens generated in the `should_stop` callback. The interface still only accepts a single cache length, but internally the class stores cache length and cache position independently for each layer, in preparation for local-global attention. Reviewed By: billmguo Differential Revision: D79128246
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12975
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 1d8bfd8 with merge base 8b2ddb2 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D79128246 |
This PR needs a
|
Summary: Make the KV cache a sliding window by default. If this is not desired user can check number of tokens generated in the
should_stopcallback. The interface still only accepts a single cache length, but internally the class stores cache length and cache position independently for each layer, in preparation for local-global attention.Reviewed By: billmguo
Differential Revision: D79128246