I was just working on a small PoC (Proof of Concept) to do some nginx testing. I wanted to have a working nginx, two simple Python APIs and then a client that would send requests to the nginx instance that would be load balancing to either of those Python API instances. For some reason it would not collect the statistics correctly. Hereunder a simplified example but it will contain the core concept of what I was trying to do.

Concurrency

So in Python it is very simple to do concurrency. You get an executor and then it will just be available. In code:

from concurrent.futures import ProcessPoolExecutor

var = 0


def func(var):
    var = var + 1
    print(var)


with ProcessPoolExecutor() as pool:
    for _ in range(10):
        pool.submit(func, var)

print(var)

If you run it you will see something weird. The var will be constantly 1 and the last one will be 0. What is going on here?

Processes

The key indicator is the ProcessPoolExecutor it will create a separate instance/interpreter/process and run the code inside there. So that var will be complete different instance as well. It took some time for me to realize this. So how to fix this? Either switch to ThreadPoolExecutor, or do the following:

from concurrent.futures import ProcessPoolExecutor, wait
from multiprocessing.managers import SyncManager

var = 0


def update_var():
    global var
    var = var + 1


manager = SyncManager(address=('', 5566), authkey=b"secret")
manager.register('update_var', callable=update_var)
manager.register('get_var', callable=lambda: var)
manager.start()


def func():
    m = SyncManager(address=('', 5566), authkey=b"secret")
    m.register("update_var")
    m.connect()
    m.update_var()
    m.shutdown()


futs = []
with ProcessPoolExecutor() as pool:
    for _ in range(10):
        futs.append(pool.submit(func))
wait(futs)
print(manager.get_var())
manager.shutdown()

That is quite the transformation. So in essence what we need extra is a Manager to synchronize between processes. One for updating the var and one to get the var.

Then every function needs to connect to the Manager and then register everything double.

If one wants to get rid of the global var then you could make a simple class that holds state and instantiate one to the manager as well.

Conclusion

Sometimes something simple turns out to be quite complicated. A bonus solution would be to use shared_memory. That could be something for another post.

#devlife #python

Parts in question

I have a Compaq Pro Linea 4/33 on the bottom. on top of which is the JVC SEA-33. That in turn has a Denon DRM-555 on it, followed by a a Sharp VHS player/recorder.

So the Compaq is a desktop computer sporting a 486 DX2 processor. Then the JVC is a graphic equalizer. The Denon is a cassette deck player/recorder and then a VHS recorder.

The VHS recorder has a SCART connection into which is inserted a device to be able to capture the video and make it possible to output that to my current modern computer and actually record/digitalize the video.

Connection of the audio

I wanted to be able to have the cassette deck be able to easily record tapes so I can make some more mixtapes for in the car for road trips. Luckily for me the manual is easy enough to find online and I started out hooking things up. Nothing worked.

I have the headphone jack of my laptop go out into the SEA-33 and then the SEA-33 goes into a jack to my actual headphones. I hooked the SEA-33 up with the tape deck Play and Rec lines like on the diagram. Nothing. Then I hooked the tape deck up with the line directly from my headphone jack. Nothing again.

I got frustrated and then I remembered something important. These are analog systems. So if you sent a digital signal that is already low powered they are not receiving anything. In essence if I sent volume at 5% then the signal is too low to do anything with it. So I upped the volume to something like 50% and then I saw the lines moving on the tape deck. Success!

So I hooked it up with the Play and Rec lines, and it still worked. Perfect. Now I am pretty sure I can listen to the tape deck whilst recording. However what I can do now is bypass my headphones. So I can up the volume and my headphones do not break.

So yeah, I now have a multimedia tower of epic proportions and definitely would have been so cool back in the day. What a weird mix of technologies.

As a side note I run from the tape deck head phone jack a cable to my mic input of another laptop to be able to digitalize old tapes my parents in law had lying around. So it goes:

headphone jack Dell XPS-15 --> SEA-33 Line In --> SEA-33 Rec --> Denon DRM-555 Line In --> Denon DRM-555 headphone jack --> Mic jack Sony Vaio

#devlife #oldtech

The premise

So the premise is that I worked on a project that was written with Graphene Django, for Django obviously, that essentially is the Python framework implementation behind a GraphQL schema. So you have a GraphQL endpoint where you send your queries and mutations and it will validate the schema and execute the queries and mutations. The two problems with the framework are it is not fully async and it is a lot of meta class programming that implies a lot of configuring but not a lot of control over the implementation details.

This meant when we were hitting our performance issues, we tried everything from low-hanging fruit like using Dataloaders, to optimizing certain queries by not relying on the ORM (Graphene Mongo in this case) and just using PyMongo directly.

Still we got stuck again, and the fact we relied heavily on the Promise implementation in Python to match the Promises/A+ from Javascript land did not help actually. Also there tried to improve some things by making things more streamlined in their concurrency, but alas. So the move is to another framework, preferably in Python, to keep the current dev team.

Enter Strawberry

So Strawberry is a nice framework that does both sync and async and it is a lot of configuring as well, but also lets you control the implementation details. For instance through things called FieldExtensions. These can either be run when the schema is first generated (through the apply method) or when the nodes are being resolved (through either sync resolve or async resolve_async functions). This is a wonderful way to, through a middleware type approach, have a way to tweak the implementation details.

One of the things that kind of was lacking in the Graphene Django implementation setup was a nice automatic control of what fields were allowed to be given as filters. Ideally it should just be always the fields exposed on the Node itself. That however was not the case, you had to manually make it so. Trying to make the new stack better and fixing that particular nuisance, I made a simple BaseExtension class:

class BaseExtension(FieldExtension):

    def apply(self, field: StrawberryField) -> None:
        self.filter_fields = []
        resolved_type: Type[WithStrawberryObjectDefinition] = cast(
            Type[WithStrawberryObjectDefinition], field.resolve_type()
        )
        if resolved_type.__strawberry_definition__.specialized_type_var_map:
            node = cast(
                Type[WithStrawberryObjectDefinition],
                resolved_type.__strawberry_definition__.specialized_type_var_map[
                    "NodeType"
                ],
            )

        for f in node.__strawberry_definition__.fields:
            field.arguments.append(
                StrawberryArgument(
                    python_name=f.name,
                    graphql_name=f.name.replace("_", "")
                    if f.name.startswith("_")
                    else None,
                    type_annotation=StrawberryAnnotation(
                        Optional[f.type]
                        if not isinstance(f.type, StrawberryOptional)
                        else Optional[f.type.of_type]
                    ),
                    description="",
                    default=strawberry.UNSET,
                )
            )
            self.filter_fields.append(f.name)

All this really does is go over the fields defined and add them all as Arguments so that you can filter on them and they will be passed along as kwargs in the resolve functions.

Perfect.

Snag time

So I was porting the NodeTypes and this project also uses Relay. It is a certain implementation of GraphQL itself. Not very important, except for me to say now I had not made any connections yet. As in one Node –> another Node. Which is quite common in GraphQL and in Relay.

When I made the first connection, the schema would not even generate. I was so frustrated, and nothing worked. I could go from resolver function -> relay.ListConnection[NodeType] but not from Node -> relay.ListConnection[NodeType]. It kept complaining about it not being a GraphQLInput Type. I did not want it as an input. I struggled and looked deep into the source code of everything, trying to hack it there. Making it dynamically an input or an output depending on properties, and I suddenly stopped. Since there was no mention of this online whatsoever it had to be a problem I caused and created.

I went to bed, late. Woke up. Paced around a bit. In my head I thought, why is it automatically turning into an argu.....oh I am an idiot.

So I revisited my BaseExtension field that powered my dynamic argument adding stuff. I tweaked it here and there and the following is the fixed version:

class BaseExtension(FieldExtension):
    filter_fields = ["project_id", "change_order_id"]

    def apply(self, field: StrawberryField) -> None:
        self.filter_fields = ["project_id", "change_order_id"]
        resolved_type: Type[WithStrawberryObjectDefinition] = cast(
            Type[WithStrawberryObjectDefinition], field.resolve_type()
        )
        if resolved_type.__strawberry_definition__.specialized_type_var_map:
            node = cast(
                Type[WithStrawberryObjectDefinition],
                resolved_type.__strawberry_definition__.specialized_type_var_map[
                    "NodeType"
                ],
            )
        else:
            node = resolved_type

        for f in node.__strawberry_definition__.fields:
            if inspect.isclass(f.type) and issubclass(
                f.type, strawberry.relay.types.ListConnection
            ):
                continue
            if isinstance(f.type, StrawberryOptional):
                if inspect.isclass(f.type.of_type) and issubclass(
                    f.type.of_type, strawberry.relay.types.ListConnection
                ):
                    continue
            field.arguments.append(
                StrawberryArgument(
                    python_name=f.name,
                    graphql_name=f.name.replace("_", "")
                    if f.name.startswith("_")
                    else None,
                    type_annotation=StrawberryAnnotation(
                        Optional[f.type]
                        if not isinstance(f.type, StrawberryOptional)
                        else Optional[f.type.of_type]
                    ),
                    description="",
                    default=strawberry.UNSET,
                )
            )
            self.filter_fields.append(f.name)

Essentially what I needed to do was check if the type or of_type is a class. If it is check if it is a relay.ListConnection type class and then exclude it from the argument generation. All worked right after this.

Conclusion

I really like this framework. It gives me insight into how they operate and why sometimes a particular query is slow, and they give you the space to fix it. For example I already fixed the fact that we can load all the necessary subparts in one go from a node with the Dataloaders. That was not possible before. However it was still as slow as the old stack, because each node on it's own tried to create this new relay.ListConnection for one Edge essentially.

We already have all the instances needed to make all the edges when doing the Dataloader logic, so implement in that particular spot also the creation of all the edges in one go. Then have a simple mapping of node.id -> Edge and you are done. This sped up things by quite a significant margin.

Something the old stack could not really do. It had no real way of giving you the same tools to do the same thing.

#devlife #python #graphql

Why does it even exist

I am having a hard time figuring out why this even exists as being a thing. It sounds like someone is thinking highly of themselves and go this thing I thought of that nobody has done before needs to be shared with the world and it will make me famous. Then it is something that not only already existed but is used globally and it is the exact same idea that already exists but now somehow transformed into their own idea.

It claims to transform a regular html site into a realtime application by loading in some JavaScript.

That is exactly what JavaScript already does. Not only that, it is apparently realtime, which I think is definitely misused here.

I just do not like these frameworks that try to claim they are superior and did something amazing, where in reality they did nothing really.

Something impressive

If you want to make something truly impressive, use the basic tools that are there and build upon it. Like using construction materials and tools to create a doll house for example. Not try and sell to people see this wood and tools, it can turn any room into one filled with a dollhouse, you just have to create it.

#100DaysToOffload #devlife

Remember REST?

Remember when REST first was a big thing. The idea that you would have stateless calls to the backend that just ships data? That was revolutionary. Then all sites became REST API sites. The SPA got borne out of it. Then everything became an SPA. Whether that fit or not. Handling any type of state/session data became extremely difficult and also the calls to get data from the database through some reverse proxies to ultimately end up in the frontend after several transpiling steps made everything worse in most every way.

There are many other ways to program the application. You had the MVC pattern, MVVM or any of the MV* patterns. These still hold value to this day. I think it is being skipped because it is so old and not new. Not exciting, yeah yeah tech is boring is good post coming. Well not really. Try to make tech exciting and challenging, but not just for the sake of making it difficult for you or following a trend that seems to be forming.

Then there is the fact you do the entire thing in either the backend or frontend that seems to be prevailing now. Which brings with it this idea of following the caravan out into the desert where everyone thinks someone else will know the way to the oasis and they all end up dying in the desert because of it.

I have been in so many projects and every time they reached for a web application with a login screen and everything being behind REST API. The next thing that always has to happen is how to secure the REST API? Then we are always building everything twice, the data models get created in the backend and then have to be exposed to the frontend and because that is in TypeScript we have the data model definitions (schemas) now twice. Once in the backend for let's say Java and Hibernate and in the frontend for Angular. After pointing this out, we still keep on the same train tracks. Why? Well we can charge the customer twice for the same work.

Pick the right tool

Just keep your options open and choose the right tool for the job. Not just take GraphQL which is nothing more than a database explorer disguised as a RPC framework. When I am working on my personal sites, I try to make it as simple as possible and not do double work. So a MVC app works perfect for me.

#100DaysToOffload #devlife

I just have one thought/question; why?

Why indeed

Well it seems like more and more was pulled into the frontend as time passed on, mainly to free up the backend to do the real processing. Like number crunching, data processing and so on. Which makes sense, but now it is weird to say the state of an application should be as close to the user as possible. Also servers are now more than capable of doing both work, and therefore the natural conclusion is that since most of the things were in the frontend anyway now; why not move the whole thing to the frontend?

The thing is that it makes the codebase more difficult to manage, you introduce unwanted complexity let alone making it secure. The whole codebase has so many extra dependencies and vulnerabilities since a lot of Server Side Template Injection (SSTI) plague the landscape now and have been fixed/found already in the old backend languages.

Of course it seems convenient, but let's take the data models that you want to use. You define them either from the frontend perspective or backend perspective first. In the frontend you might only want first name and an e-mail address, for example. Then in the backend you want to have maybe something more than just first name and an email address. How about a password to login? Or a session id, or maybe just an id in general for the databases so you can store multiple combinations of people? Then you have to transform schemas or build them differently. So you get views or something equivalent and suddenly everything is back to this again.

I would steer away from having the whole application in one language with one library/framework since nothing so far checks all the boxes and covers all the bases.

#100DaysToOffload #frontend #JavaScript #devlife

Then there are other options like running a FTP server yourself to share them, but that is not very intuitive to non-tech people. So you have to use a service of some kind, we settled on Dropbox in this case. Now getting away from just helping out a friend.

How about helping someone in my own household. I have some videos/pictures I want to send to another machine very easily and directly. I could use Bluetooth but it is horrible at file transfer specifically. The rest it does quite nicely. So if I transfer from my PC or device to another I usually just start a python3 -m http.server in the directory holding the files and then downloading the files I need.

Every time I run into this problem I find myself asking why are these seemingly so simple and core functionalities so difficult to implement. Even if everyone would run Windows, it is not guaranteed to work because I have a Windows machine in my network and I cannot get it to share files with any other machine.

Casual on the go sharing

Let us say you want to quickly share a file with a friend you happened to run into. You cannot share it via Bluetooth you will be standing there for 10 minutes. You either have to e-mail it, but then what mail address? You have to send it through some sort of messaging service, but then do you have them in that particular service?

Why can I not just drag and drop a file to this person's contact information and they will get the file? This should seem very easy to attain for a mobile phone OS.

If there is another way like NFC to share the file instantly that would be wonderful, but I have not found it yet. This is not limited to files though, what about Spotify songs, YouTube videos, webcomics, or anything else digital you can think of.

#100DaysToOffload #devlife

JVC SEA-33

It is a JVC SEA-33 Graphic Equalizer. This thing was made in the time that everything was analogue, and when mixing was not always done correctly. So what you could do is hook this thing up in your sound flow and change the amplifying or suppressing of certain frequencies to help bring out a better sound experience. Now my laptop has a 3.5mm Jack to two RCA adapter cable, and my headphones has a 6.3mm to two RCA adapter cable to hook everything up. Of course my laptop has a digital soundcard so it is not as effective as it could be, yet still I can tweak things and I just like the fact I have it working. Furthermore I can hook it up again to my record player into my amplifier in the future and then it will all work as it should be.

Kintsugi

This brings me way totally into another area, but we ordered a cake platter and it got delivered shattered. Yeah I know, there is no segue into this. Instead of returning it, we heard of kintsugi which is basically using a golden glue substance to mend broken vases, porcelain- and glassware. I liked this idea and now have a cool set piece which looks much better than the new version we got after reporting this broken delivery to customer service.

Limiting waste

I like the idea of minimizing waste and not just throw out stuff that could be repaired for a smaller fee than buying it new. What I do miss sometimes though is the fact that it is not appreciated by the companies who make their products that you start repairing them instead of buying new stuff. They will make it so difficult and hard to get into stuff or have the parts you can use to repair in a catalog somewhere.

Like the old days when the first PCs came to the market for consumers, everyone got a giant list of part numbers and you could just order more specific parts if you needed them again. Now we actually have to fight for our right to repair stuff we already bought. It should be ours to do with what we want, but that control has long slipped from us consumers.

Knowledge

I also have the idea a certain body of knowledge has been forgotten, or is in the process of being forgotten and that is worrying. Not only because it might set us back in development as a species but also because it might mean we are stuck in a loop or will repeat past mistakes.

Advice

My advice is to just buy old stuff and repurpose it if you feel like it. I still want to buy an old radio from the 60s and put in a Raspberry Pi Zero or equivalent and make it a bluetooth enabled speaker system that can also play YouTube and other songs, yet it still being a wonderful piece to look at. It will be a nice hybrid between analogue and digital, the past being brought into the future.

I have an old PC, 486DX2/50MHz, lying here that will have a running server on the network serving a real site. Obviously behind a TLS offloading proxy but it will still be serving the webpage. I hooked it up to my giant 65” Sony TV and I just loved the fact I held 30 year old tech that still worked and could fulfill a job today.

I also operated a saw that belonged to my great grand father, who I've met, and I learned that it might have been my great great grand father's even. That tool easily has been in my family's hands for more than a 100 years, and I intend to further that lifespan. It is wonderful that still exists. It works beautiful and you cannot buy a similar product anymore these days.

There something beautiful in that old things keep finding purposes and trying to get old things to fulfill a new purpose is fun to do. I do not think my dad in his 20s ever thought my son in his 30s will use this. Who knows what kind of tech I will buy that my children will repurpose in the future.

#100DaysToOffload #devlife

Stat has that information

So my first inclination was that the os.Stat call has that information. It sort of does, in a way, but only for Linux. There is a Sys() method on the fs.FileInfo which returns a specific struct on Linux.

package main

import (
	"fmt"
	"os"
	"syscall"
)

func main() {
	info, err := os.Stat("/tmp/some.file")
	if err != nil {
		fmt.Printf("%s", err)
	}
	if data, ok := info.Sys().(*syscall.Stat_t); ok {
		fmt.Printf("UID: %d\n", data.Uid)
	}
}

Is an example of how you get to that part and if you call syscall.Getuid() you can check it if they are the same and therefore if you at least own the resource. However that does not mean it is writable yet.

Permission bit logic

I tried to finagle some bitwise logic with the permission bits, but again they only work on Linux and truth be told I never trusted myself that I got it to work.

Sidetrack to Java

So in Java there has been this thing since forever. You give it a path, and it tells you if it is writable. It is a static method and easy to use. Why does this not exist in Golang?!?!?

Solution

So I did finally create a solution that was tailored for Unix and Windows separately.

Windows

The Windows solution made me go down a rabbit hole, read up on Win32 API structs and methods on the Microsoft docs and dig deep down in the source of Go itself to figure out what I have access to. I will just show you the code:

package main

import (
	"fmt"
	"syscall"
)

// https://learn.microsoft.com/en-us/windows/win32/fileio/file-access-rights-constants
const FILE_APPEND_FILE = 0x00000002

// https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilew#parameters at dwShareMode
const FILE_LOCK = 0x00000000

func main() {
	// Checks if directory is writable
	if hwnd, err := syscall.CreateFile(syscall.StringToUTF16Ptr("C:\\Windows\\system32"), FILE_APPEND_FILE, FILE_LOCK, nil, syscall.OPEN_EXISTING, syscall.FILE_FLAG_BACKUP_SEMANTICS|syscall.FILE_FLAG_OPEN_REPARSE_POINT, 0); err == nil {
		if err = syscall.CloseHandle(hwnd); err != nil {
			fmt.Printf("%s\n", err)
		}
		fmt.Printf("This directory is writable")
	}
}

So yeah, a syscall to CreateFile to open a handle to a directory. I cannot figure out why you need to create a file to get a handle to be able to tell if you can write to a directory, even more so because CreateDirectoryW is also an actual call in the Win32 API, which is the only one that can actually create a directory. This is so confusing.

Linux / MacOSX

The other solution was sort of similar but much easier. There is a nice syscall to Access.

package main

import (
	"fmt"
	"syscall"
)

func main() {
	// Checks if directory is writable
	if err := syscall.Access("/opt/", syscall.O_RDWR); err == nil {
		fmt.Printf("This directory is writable")
	}
}

Conclusion

I feel like all of this code can be hidden away in the Golang standard library and give us a nice os.IsWritable(path string) bool function signature for it.

#100DaysToOffload #devlife #golang

I ran into this problem because someone was getting a new Apple MacBook with the M1 chip which is ARMv8 or aarch64 and therefore all my previously made Docker images did not work anymore for this person as they did not work for his architecture.

Luckily Alpine and Debian also provide images for the ARMv8 architecture. Yet that meant I had to build two different images and have a multi arch manifest file inside AWS ECR. Which they support and is easy to do, but I did not want to have that plus it made it so difficult to try and reproduce things as I could build cross architecture but not run it yet.

Get yourself QEMU

So the following one line is all you need to make it possible to run as different architectures.

sudo docker run --rm --privileged multiarch/qemu-user-static:register

That will register all the necessary binaries in order to simulate the other architectures. Then for example run the following:

docker run -it --rm multiarch/alpine:aarch64-edge /bin/ash

Then you get an aarch64 image!.

Verify with uname -a and arch

Now you can see what it is like to run on those other architectures. The binaries are all of the form qemu-<ARCH>-static and so there is one for MIPS, SPARC and even PowerPC.

#devlife #devops