I have now open sourced AMQL - a set of .NET AI model engineering capabilities: https://github.com/simonsoanes/AMQL
This tool has the ability to create models, fine tune/train them, merge them, create MoE clusters, prune them, import and export different formats (safetensors and gguf), quantise a model and introspect inference operations to find exactly what pathway triggered a particular token response. It can also export any changes to models as low-rank adapters (LoRA) to be loaded.
It also has its own format compatible with VIndex3 which is model agnostic, however I had to add support for variable quantisation.
This is a tool I developed for my own model engineering needs when building Ariadne (both the platform and the foundation models that underpin it). It allows investigation of hallucinations (over-fit for compliance due to post-training) and even fixing specific ones by adjusting the tensors and logits directly. It also has the ability to convert a model to a classifier or embedding model.
Most of these things are possible in the Python ecosystem already, but this combines them into a single utility that can do so at fairly high performance and both with and without using GPU compute. Apologies that it only supports CPU and CUDA, I do not have AMD GPU systems available to test any ROCm code on.
This is an R&D level tool; it isn't intended for normal users, however I hope it's of use to someone. It is part of an automated workflow normally.
Its natural partner is TensorSharp, which provides a higher-performance inference server. Alongside Zhongkai Fu’s excellent ongoing work, I’ve contributed tensor parallelism, dynamic LoRA adapters and several other features to it.
Permalink