BATON: Enhancing Batch-wise Inference Efficiency for Large Language Models via Dynamic Re-batching

Devs

BATON: Enhancing Batch-wise Inference Efficiency for Large Language Models via Dynamic Re-batching | Read Paper on Bytez