Supply and demand
This post will touch on two subjects that I already talked about, Amara's Law and creativity thrives in structure. I will reiterate Amara's Law here though:
We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run.
In the past few weeks I worked on refactoring three Python projects in the field of data processing and OSINT (Open-Source Intelligence) and all those projects turned exactly into the same structure so I thought I would write a piece on how I ( and a friend of mine ) reworked the code into a structure that works for three separate projects. In three different domains but all do data processing and analysis.
The structure we made followed the concepts of Suppliers and Consumers. This is not new by any means, but there a lot of patterns and sometimes people forget them. This pattern has the concept of a class being a Supplier and therefore it supplies data. What that data is, that is arbitrary, numbers, string or other classes. Then there are classes who are Consumers, they consume whatever the data is given by the Supplier and do something with it, or not. The Supplier might get some input to where to get the data, then the input for the Consumer is the output of the Supplier and maybe the output of the Consumer is the input for a new Supplier or Consumer.
There might be special types of Consumers that all do a similar action, like transforming the data in a particular way. These Consumers might collectively be called Transformers. Cue 80s cartoon music... Suppliers that get the data from a specific place might also be named as such, like a Database Supplier getting the data from a database.
I will present the whole code and then in sections refactor it into a nice structure. You can easily find who wrote the original code but it is not a gibe towards that person.
def main():
planet = "mars"
my_earth_weight_kg = 75 # 165lbs / 11.8st
weights = calculate_weights(planet, my_earth_weight_kg)
print(f"A person weighing {my_earth_weight_kg}kg on Earth would weigh:\n")
print(f" {weights['pounds']:>3.2f}lbs")
print(f" {weights['stones']:>2.2f}st")
print(f" {weights['kilograms']:>3.2f}kg")
print(f"\non {planet.capitalize()}.")
def calculate_weights(planet, my_earth_weight_kg):
planet_details = get_planet_details()
G = 6.67408 * 10**-11
Fg = G * ((planet_details[planet]["mass_kg"] * my_earth_weight_kg)
/ (planet_details[planet]["mean_radius_metres"] ** 2))
weights = newtons_to_weights(Fg)
return weights
def get_planet_details():
planet_details = {}
planet_details["mercury"] = {"mass_kg": 3.3011 * 10**23,
"mean_radius_metres": 2439.7 * 1000}
planet_details["venus"] = {"mass_kg": 4.8675 * 10**24,
"mean_radius_metres": 6051.8 * 1000}
planet_details["earth"] = {"mass_kg": 5.97237 * 10**24,
"mean_radius_metres": 6371 * 1000}
planet_details["mars"] = {"mass_kg": 6.4171 * 10**23,
"mean_radius_metres": 3389.5 * 1000}
planet_details["jupiter"] = {"mass_kg": 1.8982 * 10**27,
"mean_radius_metres": 69911 * 1000}
planet_details["saturn"] = {"mass_kg": 5.6834 * 10**26,
"mean_radius_metres": 58232 * 1000}
planet_details["uranus"] = {"mass_kg": 8.681 * 10**25,
"mean_radius_metres": 25362 * 1000}
planet_details["neptune"] = {"mass_kg": 1.02413 * 10**26,
"mean_radius_metres": 24622 * 1000}
return planet_details
def newtons_to_weights(N):
weights = {}
weights["newtons"] = N
weights["pounds"] = N / 4.4482216
weights["stones"] = weights["pounds"] / 14
weights["kilograms"] = N / 9.80665
return weights
main()
Supplier
The method called get_planet_details() looks like a Supplier type to me. It supplies in this case a dictionary of planets with some properties. So first let us create a Planet class that holds data about planets that we need. If you follow along and created an empty dir with nothing in it yet then create a directory now called models . Within that models directory place two files: __init__.py and planet.py . Inside __init__.py place the following code:
class Model(object):
pass
Inside planet.py place the following code:
from typing import Union
from models import Model
class Planet(Model):
def __init__(self,
name: str,
mass: Union[float, str],
radius: Union[float, str],
order_of_mass: Union[int, str],
order_of_radius: Union[int, str]):
self.name = name
self._mass = float(mass)
self._radius = float(radius)
self.order_of_mass = int(order_of_mass)
self.order_of_radius = int(order_of_radius)
@property
def mass(self):
return self._mass * 10 ** self.order_of_mass
@property
def radius(self):
return self._radius * 10 ** self.order_of_radius
I put the order_of_mass and order_of_radius attributes there so we can give the unit like 3.3011 * 10**23 as two separate values the mass base unit 3.3011 and the order of magnitude 23. The @property tells Python the function can actually be accessed like a property and therefore we can give back the calculated properties for mass and radius. The Model is now only there so we can do nice type hinting
Next we need a Supplier to give back all these Planet objects. Make the following directories from the root suppliers/planet/planets . So you have two directories in the root, suppliers and models. Inside suppliers/planet create the file PlanetSupplier.py with the following contents:
import os
from config import Config
from models.planet import Planet
from suppliers.BaseSupplier import BaseSupplier
from typing import Generator
class PlanetSupplier(BaseSupplier):
def __init__(self, config: Config):
self.config = config
self.planets = []
location = self.config.supplier['location']
for ini in sorted(os.listdir(location)):
planet_ini = self.config.parse(open(location + "/" + ini))
self.planets.append(Planet(**planet_ini.defaults()))
def supply(self) -> Generator[Planet, None, None]:
for planet in self.planets:
yield planet
def validate(config):
assert "location" in config.supplier, "location property is needed"
Before we get into the new mentioned classes a quick word about the ideas. The BaseSupplier object is there for type hinting and the static method validate (not annotated here because it comes from BaseSupplier) is there to make sure the configuration is valid.
So we will use .ini files to handle the data representation on all the planets, they are in a location ( I wonder if you can already guess where) and the location comes from a Config object. It will then parse each ini file in that location. Optional improvement here is to use glob to only read in files that have a .ini suffix. Then it creates a Planet instance and appends it to the list.
The **planet.ini_defaults() gives back an OrderedDict so if we make the ini file have the same keys as the Planet constructor ones for the key value pairs it will automatically unpack it for us into the constructor. Regardless of order in the ini file. That is pretty handy. So next first let us take a look a the .ini file.
[DEFAULT]
name=Earth
mass=5.97237
order_of_mass=24
radius=6371
order_of_radius=3
The DEFAULT is a special keyword making the values go into the call for ini_defaults later. You might have wondered at the extra call to float in the Planet __init__ method. This is because ini files handle everything as strings, so therefore we have to convert them once. Naturally all these files will be placed in the suppliers/planet/planets/ folder. This example is called earth.ini.
Next let us take a look at the BaseSupplier. It lives in the file BaseSupplier.py inside the suppliers directory.
import abc
from models import Model
from typing import Union, Generator
from config import Config
class BaseSupplier(object):
@staticmethod
@abc.abstractmethod
def validate(config: Config) -> None:
pass
@abc.abstractmethod
def supply(self) -> Generator[Model, None, None]:
pass
The BaseSupplier sets two methods and there you can see the Model for type hinting. The supply method should be a generator always because that allows for lazy loading and evaluation when dealing with large data sets this means the program will not be limited so much by storage and memory limits.
Next let us take a look at the Config. It lives in the root of the project in the file config.py.
import configparser
class Config(object):
def __init__(self):
parser = configparser.ConfigParser()
assert len(parser.read("config.ini")) != 0, "Could not read config file"
Config._validate(parser)
self.supplier = parser['supplier']
@staticmethod
def _validate(parser: configparser.ConfigParser) -> None:
assert "supplier" in parser, "Supplier section must be there"
assert "type" in parser['supplier'], "Supplier section must have a type"
@staticmethod
def parse(fp) -> configparser.ConfigParser:
parser = configparser.ConfigParser()
parser.read_file(fp)
return parser
As you can see here the Config gets instantiated with a call to a hard coded filename called config.ini. This could be improved by specifying the name in the constructor. Next we do a simple validation check and then also give a static parse method so other classes can parse .ini files without having to import and deal with parsing.
There is one more class I want to add and that is an AgnosticSupplier. I will first show the class here and this class lives in the __init__.py class inside the suppliers directory.
from config import Config
from suppliers.planet.PlanetSupplier import PlanetSupplier
from suppliers.BaseSupplier import BaseSupplier
from typing import Dict
class AgnosticSupplier(object):
def __init__(self, config: Config):
self.config = config
self.provider_map: Dict[BaseSupplier] = {
"planet": PlanetSupplier
}
def get_supplier(self) -> BaseSupplier:
supplier: BaseSupplier = self.provider_map.get(self.config.supplier['type'])
supplier.validate(self.config)
return supplier(self.config)
So the AgnosticSupplier is here for your ease of use. You give it a Config object and on the basis of that it will give you the correct Supplier back based on the type and a simple map, which is just a dictionary of type name and class . As you can see here there is a call to supplier.validate to make sure the config is valid. Next the config.ini file itself, which is placed in the root directory:
[supplier]
type=planet
location=suppliers/planet/planets
There you see all that is needed to make sure the PlanetSupplier has all the information.
Consumer
def calculate_weights(planet, my_earth_weight_kg):
planet_details = get_planet_details()
G = 6.67408 * 10**-11
Fg = G * ((planet_details[planet]["mass_kg"] * my_earth_weight_kg)
/ (planet_details[planet]["mean_radius_metres"] ** 2))
weights = newtons_to_weights(Fg)
return weights
def newtons_to_weights(N):
weights = {}
weights["newtons"] = N
weights["pounds"] = N / 4.4482216
weights["stones"] = weights["pounds"] / 14
weights["kilograms"] = N / 9.80665
return weights
Now then. We have a Supplier, a Model so now all we need is the actual operation of getting the weight and it will be inside a Consumer. A special one, a Transformer. So go ahead and make a directory called transformers/planet in the root of the project. Inside transformers/planet create the file WeightTransformer.py and put the following code in:
from models.planet import Planet
from suppliers.planet.PlanetSupplier import PlanetSupplier
from typing import Generator
class WeightTransformer(object):
class Result(object):
def __init__(self, newtons: float, planet: Planet):
self.newtons = newtons
self.planet = planet
@property
def pounds(self):
return self.newtons / 4.4482216
@property
def stones(self):
return self.pounds / 14
@property
def kilograms(self):
return self.newtons / 9.80665
def __init__(self, supplier: PlanetSupplier, weight: float):
self.supplier = supplier
self.weight = weight
self.G = 6.67408 * 10**-11
def transform(self) -> Generator[Result, None, None]:
for planet in self.supplier.supply():
Fg = self.G * ((planet.mass * self.weight) / (planet.radius ** 2))
yield self.Result(Fg, planet)
So this will take over the piece of calculating the weights and also give an inner class of Result responsible for calculating the different forms of weight. As you can see another generator is used for the case of large data sets. It loops over all the ones from the Supplier not just one. Now all we need is to run the program again. The only thing I did not completely take over again was the formatted printing, I am just going to fully print everything. This next code can be placed inside main.py in your root directory.
from config import Config
from suppliers import AgnosticSupplier
from transformers.planet.WeightTransformer import WeightTransformer
c = Config()
supplier = AgnosticSupplier(c).get_supplier()
wt = WeightTransformer(supplier, 75)
for result in wt.transform():
print(result.planet.name, result.kilograms)
It will print out for every planet in this case. If you just want Mars then you add an if statement.
That is it. We transformed a simple code into a nice structure.
Closing thoughts
Why this structure? Is this not overkill? This seems complicated for such a simple task. Maybe you are having these questions and thoughts right now. Remember the law you read at the beginning. The value in this structure is that I can add properties to every planet ini file and change the Planet constructor and the rest of the code will still work.
We have single point of entry in the AgnosticSupplier, so again if we change something in the PlanetSupplier overall if it will keep the supply method then no worries. If we want to add more calculations for example we add the property to the planets for distance from Earth then a simple CommunicationTimings class can take the PlanetSupplier and give you back how long it will take for a single message to reach that planet using radio, light or something else to communicate. Your WeightTransformer will still work too.
In the future we discover a new planet and it is only one .ini file away from being included in the code. You know exactly where to put it and it will get picked up. Maybe in the future there won't be a need for the ini files. Someone gives you a SQLite database file with all the data. You make a DatabaseSupplier and rewrite the PlanetSupplier a bit to get the data from the DatabaseSupplier and make it into Planets.