Using MarlinLinearKernel for CompressedTensorsWNA16
Model loading took 8.02 GiB memory and 98.721582 seconds
GPU KV cache size: 519,568 tokens, Maximum concurrency for 16,384 tokens per request: 31.71x