The goal was to prove that our Python code was horrible and needed to be switched to Golang implementation ASAP.

Rough start

I could not even start our Docker container because some Werkzeug, Flask and Locust combo made it all not work anymore. So I first had to untangle that mess. It turned out that some older code of Flask used a specific call to a function that does not exist anymore at the provided location.

For all who are interested, the actual error is: cannot import name 'BaseResponse' from 'werkzeug.wrappers'.

After that initial rough start I started out by mapping out how Docker actually works. What happens when you do docker pull or docker login for example. Turns out they are all just HTTP calls to a REST API backend. That returns some data and with that data we continue onward to more calls until all the data has been gotten for docker to actually create the containers and/or images.

Docker API

I wrote the simple Python PoC code for the DockerAPI client. In principle I can use that code now to get any image I want, but I do not use that. So I included that whole code into our Locust framework to make sure the test was always set up correctly, and that subsequent images were deleted.

I ran into the second problem. Images cannot be removed from a Docker registry by default. You have to enable that feature. So when I started talking to our devs, they said just forget about it. Do the setup code once, so that the image exists that is needed in a shared test repository and continue onward.

So I scrapped the entire code out of Locust and began again anew.

Concurrent issues

Next up came the problem that I wanted to only get credentials once, and share those credentials amongst the distributed workers. There were several hosts that each run multiple workers as separate processes. I wanted on each of those hosts, that one call got made by the worker process to get a nice token and share that token in memory with the rest. In comes SharedMemory by Python. I got it to finally work after fixing all my concurrent race condition failures, where there was no synchronised flag to make sure everybody waited on each other.

After all that code, the rest of the devs were that is cool but we do not need it. Just call the login at each start of the flow, it will create credentials and if there are already credentials it will return them. So again rip out the code written so far and start anew.

Finally on my way

Started again with the new flow and now I got a nice test up and running. The data returned was a bit baffling and showed our Python code was not the bottleneck as previously thought, hoped for. It was our Nginx reverse proxy setup. Split out the nginx pods unto their own and updated the config to handle things a bit better and give more threads and workers basically.

Okay after fixing the nginx pods, then ran the tests again and it turned out the Docker registry itself was a bottleneck. It just could not cope in terms of memory usage and freeing up stuff. We use Redis as our cache layer and Google Cloud Storage (GCS) as our bucket to actually store the data retrieved by Docker registry.

Breathing room

We had so much services jammed together in one pod it was crazy. Basically one pod ran the following services: – Nginx – Redis – Docker registry – Flask app

Then there was no control of what pod ran what services, so it could be that one pod ran 2 nginx + redis + docker registry + flask, whilst another ran only docker registry + flask. So back to basics, get one service per pod and split off the docker registry unto it's own node. Now we have the following setup:

• Nodepool A:
  • Three nodes
    • running one pod each of Nginx
    • running one pod each of Flask
    • running one pod total of Redis
• Nodepool B:
  • Three nodes
    • running one pod each of Docker Registry

Now that that was cleared up, the next bottleneck seemed to be Redis? So I turned to Redis and it's config and found out we actually were not using the staging Redis but the production Redis ?!?!?!

I quickly changed that config and made it so there was one node running a dedicated Redis. So the full situation becomes:

• Nodepool A:
  • Three nodes
    • running one pod each of Nginx
    • running one pod each of Flask
• Nodepool B:
  • Three nodes
    • running one pod each of Docker Registry
• Nodepool C:
  • One node
    • running one pod total of Redis

Okay, now can we finally move onward to find out that the Python code itself is so slow?

gunicorn

Well not so fast. Turns out that gunicorn was behaving badly and might do with some optimisation. gunicorn uses different worker classes and if we do not feed it the right ones with the right parameters it might actually be blocking. The reason I started looking down this rabbit hole was because of the gunicorn logs stating they ran out of workers.

After much experimenting on what parameters work best, turns out the best one that worked for us was the following:

CONCURRENCY_SETTING=$(python3 -c 'import multiprocessing as mp; print(mp.cpu_count() * 2)')
exec /usr/local/bin/gunicorn -n internal_auth_secret -w${CONCURRENCY_SETTING} -k gevent --worker-connections=1000 -b 0.0.0.0:8000 internal_auth_secret:app -t 180

Meaning use the gevent type worker class, with 1000 worker connections. Also use a total amount of workers to twice the amount of cores available to us in whatever host we are running as. This also meant it is dynamic to the point where if we would ever upgrade the hardware of the node underlying the pod it will grow with it automatically without us having to make sure we also update the amount of workers.

Conclusion

After fixing all the infrastructure setup of correctly allocating memory and CPU to each of the services, coupled with separating them out to make sure each of them gets the appropriate amount needed. Making sure our nginx was configured correctly. Followed by actually configuring the services in staging correctly to point at services in staging rather than production, followed by configuring the gunicorn service and fine-tuning it, there was still a slight bottleneck.

Yeey, finally Python code is slow and dumb and move on to Golang. Hold on, let us first see what is being the bottleneck. I made some call graphs using the following module https://github.com/daneads/pycallgraph2. It showed that the bottleneck was partly in our shared library code that handled authentication and also the way we were determining when to call that particular function. Finally the culprit has been located.

To fix the shared library code was easy, just improved the for loops and small optimizations in terms of what to store so we do not do a constant looking up of the same values. Cache more in Redis, then also use a Redis connection pool rather than starting up a new connection every time for each query.

To fix the problem of knowing when to call the function in the shared code was a literal one if else statement added to the previously declaring of the variable logic. It was a code fix of 44 characters that resulted in an improvement of the total time spent. The longest before this fix was 465ms on the shared library code path. After fixing both it was only around 60ms. So instead of the code being able to handle roughly 2 per second we could now handle roughly 15 per second per worker per worker_connection.

After that roller-coaster of a ride, I made sure we could handle millions of requests coming in rather than just couple of hundred. The next optimisations lie in Network I/O and other factors. Even if we would move towards Golang implementation it might gain us 1ms max in terms of code maybe, that is even highly optimistic and probably not even realistic. The rest lies in the fact that we have a nginx going to a docker registry talking to another service running somewhere else again on the network that talks to Redis. Those round trip times are starting to add up.

However that is for another time. Right now we got enough to make sure we can get through the next years of running our service. If we need more, just scale the entire setup to include more nodes, until the bottleneck is network throughput/bandwith. Then we will revisit this.

#100DaysToOffload #DevOps #python

I ran into this problem because someone was getting a new Apple MacBook with the M1 chip which is ARMv8 or aarch64 and therefore all my previously made Docker images did not work anymore for this person as they did not work for his architecture.

Luckily Alpine and Debian also provide images for the ARMv8 architecture. Yet that meant I had to build two different images and have a multi arch manifest file inside AWS ECR. Which they support and is easy to do, but I did not want to have that plus it made it so difficult to try and reproduce things as I could build cross architecture but not run it yet.

Get yourself QEMU

So the following one line is all you need to make it possible to run as different architectures.

sudo docker run --rm --privileged multiarch/qemu-user-static:register

That will register all the necessary binaries in order to simulate the other architectures. Then for example run the following:

docker run -it --rm multiarch/alpine:aarch64-edge /bin/ash

Then you get an aarch64 image!.

Verify with uname -a and arch

Now you can see what it is like to run on those other architectures. The binaries are all of the form qemu-<ARCH>-static and so there is one for MIPS, SPARC and even PowerPC.

#devlife #devops

I decided to take matters into my own hands. So first things first, we have sudo privileges but you need to know the password. We do not know the password, everything is done with SSH keys. Secondly there is docker on the system but we can only interact with it via their homegrown tool. Hmmz, how can we exploit this?

A special kind of binary

There are binaries in Linux that are marked s for special, or setuid, either or. These kinds of binaries run with what is known as a different effective uid or euid. For example sudo is one of these programs.

-rwsr-xr-x 1 root root 187K Feb 18  2021 /usr/bin/sudo

That s in the -rws means the setuid bit is set. Of course that is how sudo works, it sets the real uid to the euid and then executes whatever program needed. We can write our own software that does this without asking for passwords.

Take note however of the fact that the euid comes from who actually owns the binary. So you need root privileges in order to make a binary owned by root and so if you already have that then you don't need this exploit.

Go make a binary

So let me try my hand at writing it in Go. The filename is stars.go.

package main

import "syscall"


func main() {
  syscall.Setuid(syscall.Geteuid())
  syscall.Exec("/bin/sh", []string{}, []string{})
}

That is all it takes. Really, that is it. So we build the binary with go build stars.go and then the following on a machine you control to test it out:

sudo chown root:root stars
sudo chmod u+s stars

This creates a binary that has the setuid bit and owned by root but anyone can execute and so run it and BAM we got a root shell.

Well not quite. It did not work for me. So I ditched this way.

Later on I will share what failed to make it work.

Good ol' C

C never fails. So let us write some C code that does the exact same thing.

#include <stdio.h>
#include <sys/types.h>
#include <unistd.h>
#include <stdlib.h>

static uid_t euid;

int main (void) {
  euid = geteuid();
  setuid(euid);
  char* args[] = {"/bin/sh", NULL};
  execvp(args[0], args);
}

As you can see it is not a lot more that is needed to be written. Compile with gcc -o stars stars.c and repeat the same steps above for making the binary the correct state. Run it and BAM we do get a root shell. This was so exciting for me. The proof of concept (PoC) worked.

Actual operation

Back to the server. Since there is access to docker via a docker-compose we create the following docker-compose.yml file:

version: '3.3'

services:
  generic:
      image: rockylinux/rockylinux
      command:
         - "/usr/bin/tail"
         - "-f"
         - "/dev/null"
      volumes:
         - .:/data

Then run the command via their tool to get a container infinitely running and then exec into it. The reason for the RockyLinux container is because I was on CentOS host machine and I wanted to see how RockyLinux was whilst staying relatively close to the host machine. The command is needed otherwise the container will exit immediately. This command is basically trying to endlessly read /dev/null but that will never produce output.

Now navigate to /data and install vim and gcc. I did this with yum but I realized now that actually that should have been dnf, but maybe underwater they are symlinked.

Then after installation create the stars.c file and compile it and fix the binary. Then drop out of the docker container and run the binary on the host system and BAM, root shell, pwned, h4xx0rd and what have you not.

Quickly edited the /etc/sudoers file to give me passwordless sudo and if you really wanted to be fancy you could edit out your wtmp and utmp entries to cover your tracks but I did not feel this was necessary.

Go make a binary, again

So I wanted to get it to work with Go though as I did not understand why it failed. I looked at the official docs and saw it gave back an error object. I thought let me log that out and see what it contains. It contained something along the lines of “operation not permitted”. So a quick search on that with Setuid call and I got the result that Go version 1.15 had a bug that made it not work but Go 1.16 and up had it fixed. This commit has the exact reason.

So I installed Go 1.17 and recompiled and reran and BAM, I got a shell again.

The advantage of making a Golang exploit is you automatically get a statically linked binary that could just be dropped in as long as they ran the general same version of the C library, for example GNU libc, and have the same architecture (both being x86_64 or aarch64 for example) .

So if you wanted to target Alpine (musl instead of glibc) running on Raspberry Pi 4 (aarch64 instead of x86) then you have to cross compile or use some clever tricks I will write in the next post of running docker as a different architecture using QEMU.

Conclusion

This was a fun and simple way of getting around the no root thing. Just remember that Docker shares the kernel with the host and therefore you can easily bypass many restrictions you try to put on your system. For instance making /etc/sudoers read-only (I have root, I can do what I want...) or never sharing passwords only SSH keys.

How to combat this? Well you could use rootless docker that does not do user mapping with the containers. If you do not map users 1-1 with the host you can never actually be root on the host machine and therefore your root user in the container can not create these special binaries on the machine itself.

#devlife #devops #devsecops #secops

5 years

I would say even 5 years ago it was really bad. It was not secure and frameworks were lacking professionalism. To be honest I would say PHP is still bad today.

Criticisms

The criticisms to bad structure, SQL injections and general security related things are all batted away with one statement: use frameworks. Just use a framework and magically all things are fixed. That is a lie. The fundamental nature of PHP will make it be bad, no matter how many cool awesome things you build on top of it.

Also frameworks themselves are also not foolproof and still will suffer from security flaws. Aside from this there are some other problems. Just search for all the CVE's of your favourite framework. They still occur across the board.

Concurrency / parallelism

The main problem with PHP is that it is not async in any way. It does not support threads, multiprocess or other constructs to make it concurrent or scalable in that sense. It just does not utilize the CPU efficiently. This also is combated in the original article by stating just throw more servers at it. Also the database is the bottleneck in all architectures, just scale that first then you can scale the application too.

This is just nonsense. The database is hardly ever the real bottleneck. I made many applications, enterprise and non-enterprise, and I did not once run into the issue the database could not cope with the amount that was requested. Inefficient queries, sure, lots of request to the webserver, sure, but not that the database was struggling and the application was doing just fine.

The fact you should not have to throw more servers at it is what is important. I worked on a simple Java application using Spring Boot and HATEOAS. This application is used by 100s of concurrent users. I only need to run one instance of this application to handle all of that. It is not even one giant instance with lots of resources, it has 2vCPU and 4Gb memory. That is all that is needed to support complex workflows for 100s of concurrent users. The database is a modest t3.small instance.

Code itself

Of course PHP has typehinting these days and namespacing and all good things, but it still suffers from the same problem outlined in this article subtracted with the addendum of this article. The core is trying to get better but it still is weak. I think the author of the article is referring to this ten year opinion.

The tools you get are still slightly different from normal tools. So the things that get built with them are all slightly wonky.

Deployments

To deploy PHP is horrible. Same goes for Python, because it is not a compiled language. So to make artifacts and deploy consistent code is actually quite difficult. Putting them in a container does not help as you always need a webserver to serve the PHP code. Same for Python where you need something like Gunicorn or similar to wrap around the code. Unless you wrote a server yourself.

It also just is not fun to write PHP.

#code #devops

One is based on Alpine, the other on Debian. Both will install basic tools like the needed SSL certificates to connect to HTTPS sites, wget, vim and sudo. The image always runs with a user instead of root, although to be honest the fact the user has access to adm and therefore can run as root makes this futile. However this image can easily be amended when used for production to remove the sudo part of things and just have a user run as a non-root user.

The reason for the different users is so people can become them when running the docker images and so hot reloading and source code ownership does not change when mounting the folders inside the docker container.

Also we install a standard shell, in this case the BaSH shell is chosen. This can be any arbitrary shell. The reason for this is so when people want to exec into the docker image they don't need to guess is it based on Alpine so I have to choose ash or Debian so I choose bash. Now there is always bash.

Tip: if you ever want a surefire way to get into a docker image without checking what the underlying system is use /bin/sh . That will always execute and you can then check what other shells are installed.

All these templates look to do is to instill best practices and routines.

Alpine

FROM alpine:3.13.5
LABEL maintainer="stealthycoder@stealthycoder.com"

RUN apk update && \
    apk upgrade && \
    apk add ca-certificates \
    sudo \
    vim \
    bash \
    wget && \
    adduser -u 1000 -D stealthy-adm && \
    adduser -u 1001 -D stealthy && \
    adduser -u 1002 -D stealthy-alt && \
    adduser -u 501 -D stealthy-mac && \
    echo "%adm ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers && \
    echo "Set disable_coredump false" >> /etc/sudo.conf && \
    addgroup stealthy-adm adm && \
    addgroup stealthy adm && \
    addgroup stealthy-alt adm && \
    addgroup stealthy-mac adm && \
    mkdir -p /srv/http && \
    echo -e '#!/usr/bin/env bash\n\
    sudo chown -R $(id -u):$(id -g) /srv \n\
    pushd &>/dev/null .boot\n\
    shopt -s globstar \n\
    compgen -G **/*.sh > /dev/null && \
    for f in $(ls **/*.sh)\n\
    do\n\
        tr -d ''"\\015"'' < "$f" > "/var/lib/augmented/$f"\n\
        chmod +x "/var/lib/augmented/$f"\n\
        bash "/var/lib/augmented/$f"\n\
    done\n\
    popd &>/dev/null\n\
    /usr/bin/env bash -l -c "$*" \n\
' >> /srv/entrypoint.sh  && \
    chmod +x /srv/entrypoint.sh && \
    chown stealthy:stealthy -R /srv && \
    rm -r /var/cache/apk/*

ENTRYPOINT [ "/srv/entrypoint.sh" ]
WORKDIR /srv/http
USER stealthy:stealthy

CMD ["/bin/bash"]

This template can be extended by installing more packages, and the entryfile can be extended with more logic to run on startup. There is a way to get more scripts in there with the .boot being mounted. This line:

tr -d ''"\\015"'' < "$f" > "/var/lib/augmented/$f"\n\

just removes the CR from CRLF endings when it is mounted in Windows, and therefore it can still run when executed in Linux.

Debian

FROM debian:testing-20210329-slim
LABEL maintainer="stealthycoder@stealthycoder.com"

RUN apt update && \
    apt upgrade -y && \
    apt install -y --no-install-recommends ca-certificates \
    sudo \
    vim \
    bash \
    wget && \
    useradd -u 1000 -m -s /bin/bash stealthy-adm && \
    useradd -u 1001 -m -s /bin/bash stealthy && \
    useradd -u 1002 -m -s /bin/bash stealthy-alt && \
    useradd -u 501 -m -s /bin/bash stealthy-mac && \
    echo "%adm ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers && \
    echo "Set disable_coredump false" >> /etc/sudo.conf && \
    addgroup stealthy-adm adm && \
    addgroup stealthy adm && \
    addgroup stealthy-alt adm && \
    addgroup stealthy-mac adm && \
    mkdir -p /srv/http && \
    echo '#!/bin/bash\n\
    sudo chown -R $(id -u):$(id -g) /srv\n\
    pushd &>/dev/null .boot\n\
    shopt -s globstar \n\
    compgen -G **/*.sh > /dev/null && \
    for f in $(ls **/*.sh)\n\
    do\n\
        tr -d ''"\\015"'' < "$f" > "/var/lib/augmented/$f"\n\
        chmod +x "/var/lib/augmented/$f"\n\
        bash "/var/lib/augmented/$f"\n\
    done\n\
    popd &>/dev/null\n\
    /bin/bash -l -c "$*" \n\
' >> /srv/entrypoint.sh  && \
    chmod +x /srv/entrypoint.sh
   
ENTRYPOINT [ "/srv/entrypoint.sh" ]
WORKDIR /srv/http
USER stealthy:stealthy

CMD ["/bin/bash"]

This template can be extended by installing more packages, and the entryfile can be extended with more logic to run on startup. There is a way to get more scripts in there with the .boot being mounted. This line:

tr -d ''"\\015"'' < "$f" > "/var/lib/augmented/$f"\n\

just removes the CR from CRLF endings when it is mounted in Windows, and therefore it can still run when executed in Linux.

#devops

I thought it would be appropriate to also state when and how to use microservices though.

Toolbelt

Think of an actual toolbelt with which you do construction in the physical world. It should contain some basic tools that are essential in all jobs: – Hammer – Philips head screwdriver – Flat head screwdriver – Saw – Drill machine – Sander or sandpaper

That is basically all you need to do most general jobs. Let us take the hammer for example. You have your basic run of the mill claw hammer. It can hammer in nails and get them out again. Yet there exist a hammer for banging out dents (planishing), a brass hammer that does not create sparks so you can safely use near flammable materials and even a rubber mallet to lay in tiles in the street. If you would present the rubber mallet you would not hammer in the nail with it, but the planishing hammer you might use for it. However it is a specific tool created for a specific job and when you are using it to hammer in nails you might feel uncomfortable using it or feel it is out of place.

Microservices

Microservices are just such an example of a specialist tool that should be used in specific cases. Of course having the tool in your toolbelt is nice, but take care to not overuse it. It is unlike having either a pull saw or a push saw or one that is sawing on both pull and push and you can have a preference on either one of them, as long as they all saw the same material and have a similar size you can compare between them. A microservice is like having a hacksaw vs a circular saw. Yes they both saw but do it very differently and used in very different situations.

I get the sense everyone is treating microservices more as a general tool in the belt rather than a special tool. Work on knowing you general tools first and then move into learning specialist tools. This way you are better equipped with knowing why the tool exist and how it can be applied more effectively.

Imagine giving a jewelers' hammer to someone who never used a hammer and say it is to be used in all cases and it would be just as effective as using a claw hammer.

When to use them

The trick to effectively use microservices I think lies in the fact you should be able to use them in isolation, like small monoliths basically. Take a notification service for example. If it is part of the monolith it can work fine, but you cannot use it when the monolith itself is down. That means that when the big application is down all of the functionalities are down, even ones that generally do not need to be affected by this.

Let us move the notifications out of there and into it's own environment. It should stand to reason that notification code does not really get updated all that often as they generally tend to be the same over a long time, maybe some more platforms to use / support in the future. This also means that the notification service receives fewer updates and also is less prone to be broken due to changes in the core application.

Now the core application goes down, but our notification service stays up and it might even now send notification to us stating the core application is down.

Of course it can still not notify that it itself is down, but that you could monitor somewhere else which itself has the same problem and so on.

#devops

I also made a satirical piece on this a while back already. I wish I was not this prognostic.

I will start with the premise of what the articles sketches out and then the proposed solution and lastly why that is not a good move forward and how it simultaneously looks like a better solution yet is not.

The deal with microservices

The article states that microservices can be tough to get right. There is no clear definition on how big a microservice should be and before you know it, you are actually running multiple monoliths together. There is a lot of complexity in dealing with microservices in general, and indeed one of those is software architecture related boundaries.

Scalability

Another one of those issues is mentioned in the form of scalability. It could be that microservices A and B depend on C, but it seems C is taking a long time all of a sudden. That might result in C scaling up to handle more load but also both A and B as a result of that. So in effect you could have one cascading into the other for scaling.

How would this be in monoliths, well the same. One function might cause the lag, so you scale up the entire application multiple times. In a way you only moved the problem.

Latency

Next is also increased latency. One service calls another, that calls another that calls another. This means lots of added HTTP contexts, database calls etc. This is something to take into consideration.

Deployments

To release a new version can become troublesome if you happen to release a piece of code that had incompatible changes between the latest version and the previous version and spanned two microservices that are dependent on each other. What might happen is that service A will be up but the new version of B not yet and that will result in an error and so the service is stopped and killed and then new version of B is up but the older version of A is still there as healthchecks failed.

Size boundary

Where do you stop a microservice? Personally I think a microservice should be defined by at least one of the following:

• One job to do that is atomic in scale
• All domain related functionality
• One literal function

One atomic job

This means the job it does is maybe comprised of several smaller steps but they all are atomic related. For example reserving an item on a order of a customer and also decreasing the current stock to reflect that one is already taken. This needs to either happen or not at all. If you spread these two steps over two different microservices, it is a lot easier for data inconsistency to happen as maybe the order reservation succeeded but for some reason the inventory stock operation did not. Or the other way around, so the item is not being reserved somewhere but somehow one is gone missing.

Domain related

This lies somewhat in the previous example as well, but all things domain related should be tied together. For example all things related to authorization can be grouped into one, or everything to do with images.

One literal function

This just means it literally has one function entrypoint that might call other functions, but it is really really small. For example only ever giving back the time with the timezone you supplied as a parameter.

In essence, microservices are difficult and complex.

Enter nanoservices

So the solution proposed is to use these nanoservices. They are actually the last variant of the previous size examples. So one function that does one thing. Only getting all users, or only getting a specific user. Take your run of the mill CRUD and each of the letters will be a nanoservice essentially.

Dependencies

In order to make this work all of the individual nanoservices have to depend on the same model that is used to represent the database. That is at least one shared dependency, so will you either get a shared lib or will you make one project that has all the functions but somehow integrate a facsimile of RPC ?

Final solution

The final proposed solution how to use the nanoservices, after choosing a way to deal with the dependency problem outlined above, is to have one microservice handle all those nanoservices. In essence taking the second example (domain related functionality) and then calling those nanoservices.

This means you have to deploy a microservice, and all relevant nanoservices if there is a change to the model. Also deploy a microservice and a relevant nanoservice if you change something in the nanoservice.

This is adding even more moving parts, making it more and more complex. Not solving one of the so called shortcomings of microservices. That it is complex and suffers from scalability issues.

Hybrid solution

There exists the option to have a monolith with some satellite services hovering that are microservices so that you get best of both worlds. It is meant for having the database models in one place, all the functionality pertaining to that in one central place but for example reporting is a separate microservice, or maybe the notifications is one such service.

The nanoservices seems to mimic that. You have the monolith in the form of the microservice, and the nanoservices as the satellite services. The problem is however that you made it one abstraction layer further, meaning there is way more to take into account considering the wiring of this grandiose architecture solution.

In this scenario you could have a monolith calling a microservice calling a nanoservice. That is introducing a lot of latency, communications, network traffic and so on.

So all in all I think the proposed solution by an industry veteran is making things worse in general and not better.

#devops

Workaround

Basically just turned off npm audit for now, or increase the audit level to critical instead of just high.

Update

The issue is resolved now because the frameworks released a minor update addressing this.

Problem

Sometimes you just run into the fact that major frameworks cannot run fast enough because of so many nested dependencies. The tree graph of the dependencies could rival Yggdrasil. Do not be lulled into a false sense of security thinking these frameworks automatically provide the best of the best security.

Because of all these small moving parts working together to create a complex machination it means the attack surface is quite large actually. It also opens up the multitude of possible so-called supply chain attacks.

That is where you do not attack the main framework but a package that is used somewhere along the chain in order to create the bigger framework.

#devops #secops #devsecops

Integrated Windows Auth MSSQL Server

There is always a database involved somewhere and this time the customer is already running MSSQL Server. That is not a big deal, we can all connect to database servers using login credentials, right? Wrong. Well in this case wrong, because the IT department is refusing a separate login. Everything has to be done with named accounts and integrated Windows authorization and authentication. That means I cannot connect to it from Docker as you need a special .dll file that only executes under Windows.

This was a slight setback. Luckily, I say luckily in this case, I was coding it in Java as Python3 does not even support integrated windows auth with their SQL drivers. I might be able to salvage something from a project used to get a sql shell when doing IT security but that is stretching it a bit far.

Java just requires a .dll file to be present. Microsoft has a good page on it. So when that was all setup, I could connect and use it. It did mean running it outside of Docker but it is what it is. I had initially wanted it to be part of the main backend API but thankfully I chose to write a quick CLI app instead.

Docker-compose and volumes

In the Docker-Compose file you have the option to specify volumes. These can use the so called short syntax. If you specify a file, a path or even a named volume it all works the same under Linux. One can map a file to another file, a named volume to a file or path and a path to a path.

In Windows the short syntax only supports path -> path mapping. Whether that is a named volume or path as the source, the target always has to be a path. The long syntax has an option to specify the type and it should be set to bind and then a file can be mapped into a file under Windows.

General issues

Stability

The general issue I have is that the Docker for Windows application does not feel very stable. Once it notified for updates, and I updated it. Then it could not start anymore until a full reboot. I was scared quite considerably.

Disk space usage

The other issue I ran into is the fact the file system does not shrink. Now someone said:

Oh just clear the cache and it will all work

Don't do this unless you are okay with deleting all your data^*. If you want to keep the data from for example your named volumes. Tough luck, it is nigh impossible to get it out as Docker runs on a VM and has weird bindings. So it is not transparent at all how to get the data out. Do I have a solution. Of course.

You stop running all current active containers. Start a new one with mapping the named volume to a path in a new throwaway docker container. Add a mapping to a C:\ path as well.

docker run --rm -it "C:\Users\Public\Data\":/data/c named_volume:/data/named alpine:3.13.2 ash

Then copy all the data from named volume into the C:\ mapped one.

Et voila. You have it.

Linux containers or Windows containers

Just be aware of the fact one can run either only Linux or Windows containers. One cannot run a mix of both. That is something to keep in mind when deciding to run Docker on Windows.

_{* So I did this and it was a not a happy time. I only lost test data and that did not have an impact. I was distraught for a slight moment though.}

#devops

Then I sprinkled some log() statements around and it surfaced that the data import/export I was doing had some unfortunate data that Long.parseLong() did not agree with and threw a NumberFormatException. That however was nowhere in my code to catch and so it failed silently.

This meant I could see the job starting and for all intents and purposes see it do nothing afterwards.

Lesson learned, always have a doOnError on each of your flows.

#code #devops #java