NullClaw

After Clawdbot/Moltbot/OpenClaw gained popularity because it introduced channels like e-mail and WhatsApp to contact your AI assistant was very attractive, I wanted to see what it would take to run your own offline personal AI agent. The downside of OpenClaw is you still need to connect it to models that do the AI features and where you were running the bot it had full access and further more, anyone could hijack your AI assistant by invoking along the same channels.

So if you really want to run your models offline and on own hardware, you need a beefy machine that will do so. However that brings other problems I will detail later on in the post. For now I wanted to run a secure and low resources process that would give me free access to start playing around with this and get a better feel for things.

Enter NullClaw. It is a wonderful small binary that just connects to OpenRouter. So on OpenRouter there are plenty of free models to choose from, so just connect one that fits your use case best (mine was mainly coding, therefore I went with Qwen: Qwen3 Coder 480B A35B (free)). Then you just connect for example Zed editor to your NullClaw instance and then you can code with your “own” AI assistant.

Local offline LLMs

Of course you can also run the entire model for free for yourself. That requires a serious investment of hardware though. Not only that, you might need to train it more (inference it more is a better term) to better fit you.

Therein also lies the problem. Something that exists in Machine Learning is the fact that you can overfit a model/function. That means it exactly fits the model but that leaves no room for actual learning, doing actual useful things. For example, let us say you are training a model to respond to textual input and provide meaningful output. So you have a bunch of text that represents dialogue with questions, answers or maybe even chat transcripts between two people to make it more organic. The problem with overfitting is that if you use this model it will only respond ever to the exact sentences that were given as part of the inference stage and it will always respond exactly with whatever comes next in that set of data.

Not exactly useful. So now think back to the models that might do inference as you use them in order for them to “learn” to do their job better. It might make your model overfit to you. So it will not give you best practices for example, since you do it your own way and it will adjust to that. It might not give you information that might be new and factually correct to you because you never looked for that particular piece of information, one cannot know their own incompetence.

Echo chambers

This will lead to super echo chambers where a virtual text prediction generator is feeding you the idea that you are the centre of the universe and are the best human that ever was made. Especially now with all this “great idea”, “that sounds wonderful”, and other positive lead ins of answers given back by the likes of ChatGPT.

There is a reason something called AI induced psychosis now exists.

Dark forests

So now even more dark forests might be coming up, as lots of people will run their own model of something that will be tailored to themselves and people will be disconnecting from social structures more and more.

• Log statistics of devices across a bunch of metrics like CPU, memory, Network and Disk I/O, etc.
• Aggregate these in a central place
• Make forecast / anomaly detection on this data

The reason for this was the relative recent SSH backdoor that almost was not caught. The reason it was caught was because of a developer was excessively testing a PostgreSQL lab and noticed significant sudden increase in SSH login.

This is written more like a reference for myself to look back on, rather than a suggestion to implement it this way or stating it is the only || best way to do this.

My work loved this as a feature

In order to implement this feature a process was needed that would harvest the data, send it somewhere to aggregate and then visualize and potentially let loose some Machine Learning on it. Because of my interests I knew of tools that were used in this space; Fluentbit, Prometheus, OpenTelemetry, Grafana, Kibana, and some more. I decided to start with making a mock up of sorts. A flow of how things would look hooked up.

Prometheus

Prometheus did not operate the way I expected. I thought it would expose a port / URI / API that you could connect to or send your, structured, data to. This is not how it works. It does so by getting the data from places and ingesting it that way. So rather than waiting for an agent to send data in ( á la DataDog, New Relic and their ilk), it goes out to what is configured and gets the data from endpoints.

This did not necessarily throw a wrench in the works, but it did cause a pause. So I wanted to see if a central place could be used. Prometheus also supports something called remote_write and remote_read, so using that I found a place to store data: Google Cloud Spanner. This is a relatively cheap service that makes it so all data can be sent in a central store and multiple Prometheus instances can read from the data store, and write to it.

Fluentbit

Fluentbit can be configured to immediately do a remote_write, so that made it nicer to work with. In this case I went with truestreet by Google. It was running in a container and would send the data to the central place from each device. Then in essence all that is needed to run the Prometheus and rest anywhere is another one of these exactly the same truestreet containers and use it to do remote_read against.

Also it has a nice Node exporter to automatically export almost all statistics that were relevant or interesting.

Grafana

I went with Grafana to visualize the data. It can also run anomaly detection and custom scripts, but that is for later. What I did encounter was installing, configuring and running Grafana in a secure manner is an ungodly amount of work. So I went with just running it locally on my dev machine for now. That seemed to be the easiest.

Recap

To recap what I ended up doing was running fluentbit as a process on the device itself, talking to a container running truestreet that sent the data to a Google Cloud Spanner instance. Then anywhere else I wanted to have the data available to look into, all I needed was a docker compose file that spun up: – truestreet container – Prometheus configured to do remote_read of the truestreet container – Grafana with Prometheus as the data source

Github.com

So it is 2026 as I write this post and still GitHub.com does not have a AAAA record resolving to a public IPv6 address. This means if you are on a server, like the one I got, that only has IPv6 egress networking capabilities, then you are screwed. Since you cannot connect over IPv4 to any service. Luckily for us there exists a very nice service that resolves to public NAT addresses for IPv4 only services like GitHub. So that was solved.

Every other process

So two problems why networks fail, are either it is DNS or it is BGP. Well in my case it is DNS. The problem is that by default systemd-resolved runs on local loopback 127.0.0.1 which is IPv4. That got settled quickly with just running the following config in /etc/system/resolved.conf.d/ipv6.conf:

[Resolve]
DNS=2a00:1098:2c::1 2a01:4f8:c2c:123f::1 2a00:1098:2b::1
FallbackDNS=2a01:4ff:ff00::add:2 2a01:4ff:ff00::add:1
#Domains=
#DNSSEC=no
#DNSOverTLS=no
#MulticastDNS=no
#LLMNR=no
#Cache=no-negative
#CacheFromLocalhost=no
DNSStubListener=no
DNSStubListenerExtra=[::1]:53
#ReadEtcHosts=yes
#ResolveUnicastSingleLabel=no
#StaleRetentionSec=0

This will run the process listening only on the local loopback IPv6 address. However it still resolves A records if you ask it to of course.

The actual problem I ran into was that a Node instance wanting to send mails with nodemailer resolved the service I was using to an IPv4 address.

I tried to give the --dns-result-order=ipv6first to the node process but that did nothing, even though my test code within that loaded instance resolved the IPv6 address. Well the culprit is actually nodemailer itself. It just always resolves the IPv4 address first and if it got an address then it went with that. This was version 6 however and they are at version 8 nowadays. So I looked at the latest code and it does combine the addresses in total, but does not respect the option you gave to Node for what to do first.

I just hacked the code to resolve only IPv6 addresses and it was “solved”.

Problem not solved

The problem was not really solved because there is still no way to give to every process only IPv6 addresses by default. You can edit the /etc/gai.conf file. Then you need to have IPv6 addresses for your DNS servers only and then you still need to potentially configure NetworkManager to prefer IPv6 first. Even so, after all that work, every language has libraries and implementations that let's you bypass that.

Like we saw with nodemailer and NodeJS. It is nigh impossible to have an IPv6 only setup in the real world, rather than in just a homelab situation where you control everything.

What I think I will do is in the coming days is either make a full fledged PR for Rosé Pine or just have it on my own GitHub with a fork, so I can just maintain it there. I am not sure yet.

My DE looks very nice though currently. The last thing I need to figure out is the default background for things like the file manager and such.

• SpaceVim
• NvChad
• LunarVim
• AstroVim –> AstroNvim

I did that meme of the graph where noob uses little to no stuff, in the middle you do all this fancy stuff and then at the right you are at the other extreme and you use little to no stuff again. I am at the right side again.

When I started I found the fact you had to learn the config and use Plug or Vundle. Then in recent years the Lazy.nvim came to the market, and everyone is using that now. Well with NeoVim 0.12 release we get native package manager.

So I decided to redo my config from scratch and instead of having a tonne of extra features that I will never use, just pull in the features I like and actually use and whenever I run into a problem of how do I this or how should do I that, just implement it at that time.

My current config of init.lua for my NeoVim is about 145 lines long and most of that is enabling different LSPs and TreeSitters. These are the plugins I use currently:

vim.pack.add({
  { src = "https://github.com/catppuccin/nvim" },
  { src = "https://github.com/stevearc/oil.nvim" },
  { src = "https://github.com/echasnovski/mini.pick" },
  { src = "https://github.com/echasnovski/mini.comment" },
  { src = "https://github.com/echasnovski/mini.pairs" },
  { src = "https://github.com/echasnovski/mini.icons" },
  { src = "https://github.com/nvim-treesitter/nvim-treesitter" },
  { src = "https://github.com/folke/twilight.nvim" },
  { src = "https://github.com/folke/todo-comments.nvim" },
  { src = "https://github.com/MeanderingProgrammer/render-markdown.nvim" },
  { src = "https://github.com/michaelrommel/nvim-silicon" },
})

These are the bare minimum I feel I use to do my programming in a nice flow. As you can see, there is no Mason, one can just follow the instructions contained in the LSP config lua files I feel. I rarely setup a complete dev flow to test out a new language. Usually in that case I just run a simple Docker container that has the config and tools I need.

Meanwhile, one still cannot send a application/json mimetype request to PHP server out of the box and get the variables in:

function auth() {
    $username = $_POST["username"];
   // The rest
}

Whereas, as far as I can tell the only change needed is right here. Just add the handler for the JSON one based on the example code that is already there. They did not do this for the last 15 years, where application/json has definitely become of the main stays of web development. No let us focus on bringing functional programming to PHP of all things. At least these days you cannot use the argument anymore that PHP is not a compiled language.

Pop!_OS shenanigans

So I ran Pop!_OS from System76 for years now. I had a brief stint with Arch in the form of EndeavourOS again, but like I said in the past; You need a stable driver for your work life. So I ran this for quite some time, being quite content with it. They opted to go and make their own Desktop Environment (DE) named Cosmic. Which has been in alpha state for years now, it is slowly coming out to a beta point though. I am also using ghostty these days and I wanted to update it. I reran my zig command to build it and it failed. It needed a newer version of zig. No problem. I installed the newer version of zig. It failed again with the build but now it was a compile error. Something to do with libadwaita in the end, it needed to be a higher version. It is what is being used to make the actual widgets on screen in GTK based systems. Well I tried to compile newer libadwaita version from the repo, but it needed a newer glib version. I could not begin to compile that.

New plan. I happened to look at the Pop!_OS repositories and saw they now had noble alongside jammy. I switched everything to noble in the /etc/apt/sources.list.d/ and started my upgrade journey. Well 1000s of packages, and about 5 hrs of running apt --fix-broken install followed by dpkg --force-breaks install, dpkg --force-depends remove and dpkg --configure -a later, I had a working system again. In the meantime I rebooted, and did not even get a screen. It was just black, luckily Pop!_OS has a old kernel entry in Grub and even a recovery partition.

Then I was greeted with the Cosmic desktop. I did not ask for this, but as part of the upgrade to noble you get Cosmic DE for free. Well needless to say it did not work smoothly all of it. The one thing that was possible now though, was to upgrade ghostty, only to find out it did not work anymore in this DE. When I switch tabs now, it crashes. Fantastic. Okay back to kitty, that also does not work anymore. Even worse. I find out Alt+` does not work anymore. I tried to dconf editor new short cuts for things, but the fact is Cosmic just did not code it in. So it won't work. Now I thought I could live without that, but no it turns out I use that shortcut all the time. It is to switch to other windows of the same group essentially. So between all your Firefox windows, or for me, all the terminal windows I have open.

Cosmic

It did look nice though. I cannot remember this now anymore, but I believe it did support different scaling on two monitors. Something I found out X11 does not do. It felt snappy, but also with a hint of uneasiness about it. I think they severely underestimated what it takes to write/implement a DE.

Before the big move

Well I wanted to keep all my important data. So I copied everything over to a spare server with some room and SSHFS mounted a directory to copy stuff over. Well actually I mv-ed things over first. That failed and then that made me lose some data already. Next I forgot the two most important things to copy: – SSH keys – GPG keys

Luckily I had my GPG key from my EndeavourOS disaster, and my SSH keys are not being used anywhere else other than my own servers which I can log in to and GitHub which is easily fixed.

So when I was installing my new system and setting it all up, that feeling of forgetting something crept up and manifested itself nicely and then you have that sinking pit in your stomach feeling of will this be horrible or not. It all turned out okay in the end.

Budgie

I tried out Solus Budgie a while back and I loved it. It felt so snappy, clean, and good. I thought if I ever have to move to another system, it will be that one. So now was the time. Sadly Budgie is not on Wayland yet. So that meant X11. I did not see the harm in this. Turns out X11 cannot handle different scaling factors on different monitors. So either my 4k laptop screen looks fine and my monitor is absolutely huge, or my monitor looks great but my 4k laptop screen is absolutely tiny and illegible. So I turn to Solus GNOME. It is wayland after install. I have customised it again to be all Catppuccin and I am cozy and calm again behind my laptop. The only thing I did not get to work on either ghostty or kitty again here are the italics. So then I stuck with the GNOME terminal, and it seems to be working now. I still like VictorMono as a font and I configured AstroNvim to have comments be italicized so they look all nice and cursive. You can extend this to function definitions and what have you.

Conclusion

I will probably always forget something, but next time I really need to save my keys. I am currently looking at my nice icons and cozyness that makes working at least a little nicer. An added benefit is that my Firefox is working again normally too. It can finally not stop breaking when I wanted to skip to a section in the video.

Docker, Rust, WASM, and many other things are on that list for me. Some I have figured out, some maybe not yet really.

Well the big selling point that Ghostty has is it is fast, feature rich and native. That last part is really important. Kitty is sort of native, but not when it concerns it tabs for instance. They are a non-native widget.

This means that if you install Ghostty on Windows, Linux and MacOS you can re-use your configuration and get the same experience.

So I am switching to Ghostty with everything. On a side-note I am still debating if using tmux improves my flow or not.

What is it?

Well the project is making palette for all kinds of niche/esoteric platforms. Like make one for Windows 3.11, YSMenu for the Nintendo DS or MS-DOS. Those types of situations where you have to use the platform, I like it to be personalised. It is really difficult to make it all cohesive. This is because for example for Windows 3.11, you only have 8-bit 256 colors. However that relies on dithering and on a modern monitor dithering does not do the same as on a CRT monitor for example.

CRT and AI

I tried asking some AI LLMs out there (Gemini, Claude, ChatGPT and the like) if you can have a VM running Windows 3.11 emulate a CRT monitor. Options that came back were pipe the video output into ffmpeg, no it is not possible and use VBoxManage to control the framebuffer output, maybe script it with Python.

Of those mentioned, one kind of worked. I did record my session from VirtualBox, then pipe it into ffmpeg with some filters and streamed that into VLC. It showed me some nice video and it worked albeit with a slight delay, but I thought there has to be a better way. Luckily for me there are more emulation setups our there, then just VirtualBox. I looked at DosBox and DosBox-X etc, but it did not allow me to do simple things to the video output.

Then I remembered PCem as a project. It was very difficult to get going. I could not compile it and I gave up on this. I did not want to spend my time fixing compiler errors in my spare time. Props to them for making an awesome emulator, but I cannot get it to work in a jiffy on Linux.

Then somebody mentioned 86Box, and I remembered that one as well. I had all those things already bookmarked, but never was able to actually work on it.

86Box to the rescue

Well it was not easy to get this one going, but at least they supplied a working binary out the gate. I downloaded a release, then the mandatory romset and nothing worked yet. Turned out you also need a nice manager to run everything. I went with 86BoxManager that could run cross platform.

Now I had my manager that could create VMs and then I could configure that to be hyper specific. That is so overwhelming. I just tried to get going, and I got a working system after clicking around just a bit. I installed MS-DOS 6.22 first and then went on installing Windows 3.11.

This was grand. I felt like I was right back in my childhood again, just clicking away in Windows 3.11. Barely understanding what I was doing but I could open Paint and I could change the way everything looked and I could run a calculator on my machine.

Back then I already fell in love of personalizing the computer. That was one of my first creative outlets I think. I always love tinkering with the settings, seeing what looked nice and trying out stuff.

I also recently learned there is such a thing as #DOSCember so maybe next year I will participate.

CRT

Okay but now the story isn't ready yet, because I needed CRT. Well there exist another great wonderful project from libretro. It contains all the shaders you ever wanted to make something emulate something else. Lots of people contributed their best versions of what they think is the best one.

The moment I applied on the shaders to my machine that sort of worked, was this one, it immediately looked better. It felt like home. The colors blended already nicely. The contrast with the modern looking VirtualBox I had running alongside it was stark indeed.

Conclusion

I am just happy I live in an age where I can relatively easily go back in time to my childhood and try out things again. I can also close off certain chapters again, by completing some task I might have had in my head for years on end.

Also AI is nice and all, but it definitely has its limitations. Sometimes you just got to work stuff out yourself.