You manage the dependencies or the dependencies will manage you.

So I had a recent discussion on what dependency management is. It turns out there are a lot of dependencies, or in other words you are dependent on a lot of different things. First we defined what a dependency is. If you like it to a machine then any moving part big or small is a dependency. From the operating system to the version of a library to version of tools being used. As you see that is quite the number of moving parts.

The goal therefore is to minimize moving parts. Fewer parts is less to manage. Now I am going into software development but I think you can apply the lesson to any discipline.

Start with using kubernetes or Docker. Then have the entire Docker image be ready to run your application. This might entail getting one for cache, one for a database etc. All that is missing is the source code of the application you are developing and maybe some seed data. When that is the case you vastly reduced the moving parts already. Now all developers need is Docker tooling to get it up and running locally. Also when using Docker wherever you will run it you know it will work if it also works locally. *

This is all and well but what if you do not do this? What happens then? Let us make it so you do not use Docker but install everything locally. A certain runtime and some framework. In a few months a new developer joins. He has to install the runtime and framework but runs into a problem. The version the original developer has running is outdated and no longer available. So a new version is installed and everything still runs fine. Mind you we already have a schism. We add a new framework and a couple of libraries and in another few months two new developers start. At this time another runtime is introduced and some libraries and frameworks for this runtime. Now all developers have to install two runtimes and their respective libraries and frameworks. At this moment some developers cannot install the new runtime and the existing runtime has a new version again and that new version does not play well with what has been developed so far. You can quickly see this getting out of hand.

Therefore if you had two Docker images each with one of the runtimes and its respective libraries and frameworks, all developers only need is Docker tooling.

It might also occur that you never update and then you will become more and more insecure. Furthermore the time , effort and infeasibility all grow exponentially towards infinity.

Try to keep up to date once a month. Do audits, upgrades and all of that goodness to not sway too far from the normal line. That way if correction is needed it is never too invasive. **

* Certain mistakes and errors aside that can occur.

** Certain special situations aside

#devops

My strategy is to have your own images. Built upon a solid base image. For example Alpine Linux Docker image with nodejs and npm installed. Maybe even pnpm and Parcel. Then next image might be Angular, React or Vue. It will install the CLI tooling and make sure the package.json stays fresh. You mount the source code at a consistent and predictable location inside the Docker container. This means any person can now run the frontend as long as they have Docker.

This can be done for any language. PHP I am looking at you with your weird images. Also the community of PHP please embrace this practice. Ditch Vagrant.

Why?!?! Well the following reason. Vagrant gives you a full Virtual Machine. This means that Virtual Machine needs to be the exact same as the server that will be running the code. Which it won't. It lives on your developers' machines. So a MacOSX virtualized it slightly different from a Linux. You get errors. Also drifts in versions of applications. System tooling. You name it.

With Docker it will be run the same everywhere. Added benefit is that Docker is way easier to run for non tech people.

Is there a cool hip term for a non tech person?

#devops

Meaning

The 4 chars are actually two pairs of two chars. The first is CR and the second is LF. CR stands for Carriage Return and LF stands for Line Feed.

Typewriter

Remember those old physical machines where you put whitened wood pulp in and using keys ran into that pulp some iron oxide on a ribbon. Yes, typewriters indeed. Wonderful mechanical machines that made life easier for humans. Now on those machines the sheet of paper would move side to side and up and down. The act of moving back to the beginning is called Carriage Return. The act of moving the sheet of paper is called Line Feeding. See how the terms translate well into modern computers and the terminals they are being run in.

Side note

So on modern Operating Systems there exists a number of terminals or shells. Underwater they are called tty devices, or TeleTypewriter devices. That was because the first type of digital typewriters had a monitor and a typewriter interface. So even more evidence of history and how it all ties together.

CRLF cont.

Ok sorry for the detour but it serves a purpose. Operating Systems interpret those characters differently. On Windows you have to do the same action as on a physical typewriter. Return the carriage to the beginning and feed in a new line. Therefore they use CRLF as the ending. On GNU/Linux and derivatives only feeding a line is necessary. Therefore they have LF only line endings. What does it mean then when you run CRLF file on GNU/Linux and co? Well the shell will do the following. Move the cursor back to the beginning and then execute that line and go to the next but there is no more line endings at the end there remains your command as a garbage input. Garbage in means garbage out. So the scripts usually fail. On Windows the same can happen but the system itself is very good at making sure there never is a singular LF if it can help it.

Docker

What has Docker got to do with this I hear you ask? Ever noticed how most Dockerfile files have got all kinds of RUN commands and not just copy in a shell script and run that? I thought it was much simpler to maintain and read and so I did it. Thinking of how clever I was I suddenly had a team member with Windows and Docker did not work?!?! NANI?!?! After debugging the crap out of it I realized it was CRLF related. Solution is fairly simple though, not use shell scripts unless you copy them in already from a GNU/Linux system or download then from the web. So fellow human software engineers, when using Docker use RUN commands and maybe some staged builds. More on that in a future post/ future posts.

#thoughts #devops

This is where Windows comes in. Windows being the host and it being so different from OSX and GNU/Linux systems it brings it's own quirks with it when running docker on Windows. One of the things I explored in a previous post where you copy in files from Windows into the docker container and the line endings being different resulting in not working scripts. Now I will explore a small problem running a front-end application in docker and Windows.

Mounting

The concept of mounting means you have a device or folder and you want to make it available at a specific location. In most cases you would mount a hard disk or maybe a USB stick to a location you can access the folder and inspect the contents. Well you can also mount files onto a location or a whole filesystem even. When using docker you can also mount directories or files on your host system into the docker application. This creates a nice two way street of manipulating files and directories. Whatever you change on the host system will be reflected inside the docker container and vice versa.

Dependencies

In the front-end world of development dependencies are generally managed through a package.json and a package-lock.json, for you Yarn fanboys a yarn.lock file. Meaning when you run your installer of choice command to install the packages it will use these files as input and put everything in directory node_modules . This is all fine and dandy but what I discovered is that if you mount your application's source code in the docker and therefore have docker create the node_modules directory synced with Windows host it will go awry.

Solution

What seems to be the problem is that Windows cannot handle the vast amount of writes and operations inside the node_modules directory. Solution is to create a volume using docker volume create and then using that named volume as the mount for your node_modules . Actual example for those who are using docker-compose :

volumes:
  node_modules:

services:
  volumes:
     - node_modules:/var/www/html/node_modules

Also place this solution in a file called docker-compose.windows.yaml. Then you only have the override for Windows and not for all other users.

#devops

Social

You talk and speak and neither of those are communication. You can say words but that does not mean you communicate your idea or end result to the other party. The thing that makes it so tricky is two or more parties are involved each with their own central processing unit that has a language center that each has its own interpreter. That means the same message has different results. Take a simple sentence like : I am happy to write this article . People can read it and go well he sure is not happy to write that article or go he wrote this article not being happy first but the act of writing made him happy. Or maybe some other interpretation all together.

The thing is that the receiver dictates how the message is received and thus how the communication will continue.

In software this means pick a correct protocol and a correct message structure and potentially mime type. For protocol not really a lot of choice except UDP and TCP. Then message structure and mine type can be essential. The structure is not whether it is JSON or XML or something else. It is what is inside. How is it ordered. What does each property contain. Do both parties agree on it? That is essential in communication .

Language

The language used can also be a barrier. The Lingua Franca these days is English everywhere. Yet that still poses problems because not everyone is a native speaker and even if you are you still run into dialects or other subtle nuances that allow for miscommunication. I myself worked in a team with 9 different nationalities and that means 9 different cultures and 9 different people all trying to speak English and communicate their thoughts and ideas. Needless to say this caused some confusion and miscommunication was rampant throughout. We had to do even more communication and trying different ways of explaining the same concept because it had the best way of making sure everyone was talking and understanding the same thing. In this case it also helps to have the receiver echo back his understanding to you or the rest of the group to get a consensus.

In this case for software you have to choose how you want to communicate the structure. So will it be JSON, XML or ProtoBuff? This will dictate what will be send and how to process it. I think in general don't go with XML anymore because it is so verbose.

Context

The context matters greatly in communication. Are you communicating about Apple or the fruit? Are you trying to convince someone to go somewhere or just sharing information? The context dictates how the receivers will process the information. Filters will be applied. Information will be enhanced or enriched in certain cases even.

In software you want to make sure the context is agreed upon as well. Is it a REST API ? Or is there a state in the backend or will there be something else going on? How will you authenticate to become a known entity ? How will you authorize accessing information? How do you forward this information? What do you let the end user know about itself ? All these questions you can define from the start and sometimes as you go along as well.

In the end communication is hard and difficult and error prone but important to get right. Try in your conversations with people to make sure they definitely understand you. In software write down how you want to communicate aforehand and eliminate misconceptions.

#thoughts #devops

On AWS we created an account for all our developers to play around and experiment. That goes excellent. People are learning and loving it and everyone knows more and more each time. Like children though they don't clean up after themselves so I made use of a little helper called aws-nuke.

This little helper has an excellent sane default of not actually applying anything. It only does so called dry runs. You give it a config file where you specify what to keep and what to target and so on. Then however if you are satisfied with the results you let it loose and snap your fingers for added dramatical effect and gone are your AWS resources. So you safe costs and be a bad ass while doing it. Every time someone will hear your snap your fingers they will fear that their cloud environment is gone.

One little thing though in the config file please specify for each region that you don't want to delete the default VPC. I forgot and all of a sudden all cloudformation and terraform actions did not work anymore.

#devops

When talking about deploying somewhere , let us say AWS. Then it makes sense to carefully think about autoscaling, initial size of the instances, and seriously think about reserved instances or even spot instances. To lower costs but also to lower used resources.

When programming there is a balance between pre mature optimization and effective resource usage. There is the advice don't optimize too early because you might throw away the piece of code you wrote or you only use it once in the program and then it does not matter either. If you have the code in and even though you call it only once. You still need to make that pretty. Not only to avoid broken window theory but also to not use more resources than necessary. Even if it is once at bootup, you might start or boot the application a considerable amount of times before the lifetime of the software is over. All wasted energy if you optimised it. Large numbers apply of course in general but it is the same for response times. You want the shortest response time possible because the user is waiting too. All wasted energy.

In a cynical way you could say the servers are running anyway so what does it matter. It matters because if we effectively consume resources we can achieve more with the same. Minimal amount of material and maximum results.

#devops

Configuration should be higher priority

So turns out, the tool still read the environment variables first even though we explicitly configured another credentials provider. That is annoying to say the least.

Please remember people, whenever you write a tool or application with a configuration. The configuration takes precedence. If there is nothing in the configuration you are welcome to use so called sane defaults . This means you take the option that makes the most sense or is the best for the developer. Examples are that certain security features are on by default instead of off . Features like encryption, or good random entropy or password protected. Other features can be the need for an explicit parameter to not execute a so called dry run . In a dry run nothing happens but you see what would happen.

Issues

It seems the terraform tool has a lot of issues. I do not mean it does not work well and every other second we run into a problem. I mean their GitHub page has 1500+ issues reported (at the time of writing 1560240633 UTC ). If anyone of HashiCorp reads this, first thank you for reading this and your name reminds me of BlastCorps for the N64 somehow, and secondly please manage those issues. It is very difficult to find stuff that will be fixed, won't be fixed, is being fixed and any other state in between.

#devops

So that had to change and in came CSS. Now you could better style the webpages. HTML for structure and CSS for look and feel. Perfect. However, this all was still static. It would be nice to add something to it that could dynamically update a page without reloading or just fetch data dynamically. In comes JavaScript. So we got the holy trifecta, HTML, CSS and JS.

Nowadays though, they feel everything should be able to be done from within JS ?!?!?!?! That is what React feels like to us.

render( {}, { todos, text} ) {
    return (
         <form onSubmit={this.addTodo} action="javascript:">
             <input value={text} onInput={this.setText} />
            <button type="submit">Add</button>
            <ul>
                { todos.map( todo => ( <li>{todo.text}</li> ) ) }
            </ul>
        </form>
  ); 
}

So we just wrote JS that contains HTML that contains JS that contains HTML. That has got to be the ugliest way to write HTML.

In addition to all this, is the fact that React is a library . It is not a framework . Always a fun question to ask during interviews, what is the difference between them? Well a library is code you use, a framework uses your code. That might sound vague, but in essence this is correct. You would use code from a library that does something specific. For example query DuckDuckGo and return the first result. A framework is something that is built for you to place your specific code in but it handles everything else. Angular is a framework in contrast to React.

What we like about Angular is it splits everything up nicely. You have HTML files for the structure, (S)CSS for the styling and then JS for making it dynamic. Although you only provide your code, Angular makes sure it works. There is a whole bunch of extra layers around it you don't see and know about in order to make it all work. That is beauty.

In the similar line exists the Polymer framework, from Google. It is based around the idea of Web Components and uses the <template> and <slot> tags from the HTML5 spec. Definitely worth a look.

The other issue that arises from using a library in contrast to a framework is that you have no clear same structure. You are left to choose your structure. However you want, as long it works it will be okay. In Angular you don't have that luxury as much and that is why we like it more. It is less gun-ho and cowboy style compared to React. You have to follow the structure they dictate in order to make it work. That means Angular codebases over all are predictable, uniform and that in turn makes it so Angular developers can switch projects relatively easily. In React we don't have that guarantee, so it is wildly different. Unless you had a disciplined dev, the odds are some corners were cut or some breaking away of the mold was done in order to make something work.

So our advice, write HTML, CSS and JS separately and do not try to smoosh them all together. Along a similar vein I will leave you with this.

#devlife #devops #thoughts

Across the board we can look at the frontend eco system with their NPM that just gets the world in packages. Then there is Maven in Java that just seems to go on endlessly, or SBT in Scala. I recently discovered another one, in the golang eco system. Wow... It downloaded so many dependencies and they are all globally stored as well. Fantastic. No virtual environments for golang, oh no. Of course you can set the GOPATH environment variable to sort of install them in different locations, but really come on.

I feel Ruby and PHP also suffer from this but depends on the framework of choice. For example in PHP using Phalcon it is a compiled module and therefore you install it once and that is it. No extra stuff needed. Then in C# you usually have everything in the .NET framework already. In Python the standard library is so vast it most likely is in there and otherwise you have a couple of specific dependencies that are fairly contained.

#devops