TensorFold: Speculative Decoding That Never Changes a Token
What TensorFold is TensorFold is an open-source inference server for Apple Silicon and NVIDIA GPUs that speeds up decoding with draft tokens, and promises that the drafts never change a single token of the reply. It comes from @ashxhart, the repository is MIT licensed, and it serves an OpenAI-compatible API on your own machine. The … Read more