nullify

I have now open sourced AMQL - a set of .NET AI model engineering capabilities: https://github.com/simonsoanes/AMQL 

This tool has the ability to create models, fine tune/train them, merge them, create MoE clusters, prune them, import and export different formats (safetensors and gguf), quantise a model and introspect inference operations to find exactly what pathway triggered a particular token response.  It can also export any changes to models as low-rank adapters (LoRA) to be loaded.

It also has its own format compatible with VIndex3 which is model agnostic, however I had to add support for variable quantisation.

This is a tool I developed for my own model engineering needs when building Ariadne (both the platform and the foundation models that underpin it).  It allows investigation of hallucinations (over-fit for compliance due to post-training) and even fixing specific ones by adjusting the tensors and logits directly.  It also has the ability to convert a model to a classifier or embedding model.

Most of these things are possible in the Python ecosystem already, but this combines them into a single utility that can do so at fairly high performance and both with and without using GPU compute.  Apologies that it only supports CPU and CUDA, I do not have AMD GPU systems available to test any ROCm code on.

This is an R&D level tool; it isn't intended for normal users, however I hope it's of use to someone. It is part of an automated workflow normally.

Its natural partner is TensorSharp, which provides a higher-performance inference server. Alongside Zhongkai Fu’s excellent ongoing work, I’ve contributed tensor parallelism, dynamic LoRA adapters and several other features to it.

I've recently decided to open source some groupware software I built to replace applications that were running on PHP on my home network.  These are brand new apps built to my own standards  and running on .NET 10 with the fewest possible dependencies (for example, the e-mail app doesn't store your password at all and supports client side end to end encryption using OpenPGP) - even the icons in the webmail are custom made.

The CalDav server is designed to support the webmail client and android/apple phones, and supports push notifications for calendar and contact changes, as well as SyncML.

https://github.com/simonsoanes/CalDav - Nullify CalDav, my CalDav and CardDav server supporting calendars, contacts, tasks and notes.

https://github.com/simonsoanes/Webmail - Nullify WebMail

These applications are going to be monitored and maintained by my AI platform, so will get ongoing updates and be kept on a current version of all libraries, however most functionality was custom written from scratch with no dependencies - for example, the PGP implementation is from scratch (which does mean I've not necessarily provided all historic encryption and signing algorithms and it hasn't been audited, but it does pass standard interop tests).

I kept having out of memory situations where I was unable to finish fine-tuning or quantization jobs when I had hundreds of GB's of video memory seemingly free - I kept getting:

Failed to load checkpoint: Some modules are dispatched on the CPU or the disk

It turns out that in a unified memory Grace Blackwell system you need to drop your OS cache or it can consume too much of the unified memory, resulting in paging to disk instead of GPU use.

sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'