I was just working on a small PoC (Proof of Concept) to do some nginx testing. I wanted to have a working nginx, two simple Python APIs and then a client that would send requests to the nginx instance that would be load balancing to either of those Python API instances. For some reason it would not collect the statistics correctly. Hereunder a simplified example but it will contain the core concept of what I was trying to do.

Concurrency

So in Python it is very simple to do concurrency. You get an executor and then it will just be available. In code:

from concurrent.futures import ProcessPoolExecutor

var = 0


def func(var):
    var = var + 1
    print(var)


with ProcessPoolExecutor() as pool:
    for _ in range(10):
        pool.submit(func, var)

print(var)

If you run it you will see something weird. The var will be constantly 1 and the last one will be 0. What is going on here?

Processes

The key indicator is the ProcessPoolExecutor it will create a separate instance/interpreter/process and run the code inside there. So that var will be complete different instance as well. It took some time for me to realize this. So how to fix this? Either switch to ThreadPoolExecutor, or do the following:

from concurrent.futures import ProcessPoolExecutor, wait
from multiprocessing.managers import SyncManager

var = 0


def update_var():
    global var
    var = var + 1


manager = SyncManager(address=('', 5566), authkey=b"secret")
manager.register('update_var', callable=update_var)
manager.register('get_var', callable=lambda: var)
manager.start()


def func():
    m = SyncManager(address=('', 5566), authkey=b"secret")
    m.register("update_var")
    m.connect()
    m.update_var()
    m.shutdown()


futs = []
with ProcessPoolExecutor() as pool:
    for _ in range(10):
        futs.append(pool.submit(func))
wait(futs)
print(manager.get_var())
manager.shutdown()

That is quite the transformation. So in essence what we need extra is a Manager to synchronize between processes. One for updating the var and one to get the var.

Then every function needs to connect to the Manager and then register everything double.

If one wants to get rid of the global var then you could make a simple class that holds state and instantiate one to the manager as well.

Conclusion

Sometimes something simple turns out to be quite complicated. A bonus solution would be to use shared_memory. That could be something for another post.

#devlife #python

The premise

So the premise is that I worked on a project that was written with Graphene Django, for Django obviously, that essentially is the Python framework implementation behind a GraphQL schema. So you have a GraphQL endpoint where you send your queries and mutations and it will validate the schema and execute the queries and mutations. The two problems with the framework are it is not fully async and it is a lot of meta class programming that implies a lot of configuring but not a lot of control over the implementation details.

This meant when we were hitting our performance issues, we tried everything from low-hanging fruit like using Dataloaders, to optimizing certain queries by not relying on the ORM (Graphene Mongo in this case) and just using PyMongo directly.

Still we got stuck again, and the fact we relied heavily on the Promise implementation in Python to match the Promises/A+ from Javascript land did not help actually. Also there tried to improve some things by making things more streamlined in their concurrency, but alas. So the move is to another framework, preferably in Python, to keep the current dev team.

Enter Strawberry

So Strawberry is a nice framework that does both sync and async and it is a lot of configuring as well, but also lets you control the implementation details. For instance through things called FieldExtensions. These can either be run when the schema is first generated (through the apply method) or when the nodes are being resolved (through either sync resolve or async resolve_async functions). This is a wonderful way to, through a middleware type approach, have a way to tweak the implementation details.

One of the things that kind of was lacking in the Graphene Django implementation setup was a nice automatic control of what fields were allowed to be given as filters. Ideally it should just be always the fields exposed on the Node itself. That however was not the case, you had to manually make it so. Trying to make the new stack better and fixing that particular nuisance, I made a simple BaseExtension class:

class BaseExtension(FieldExtension):

    def apply(self, field: StrawberryField) -> None:
        self.filter_fields = []
        resolved_type: Type[WithStrawberryObjectDefinition] = cast(
            Type[WithStrawberryObjectDefinition], field.resolve_type()
        )
        if resolved_type.__strawberry_definition__.specialized_type_var_map:
            node = cast(
                Type[WithStrawberryObjectDefinition],
                resolved_type.__strawberry_definition__.specialized_type_var_map[
                    "NodeType"
                ],
            )

        for f in node.__strawberry_definition__.fields:
            field.arguments.append(
                StrawberryArgument(
                    python_name=f.name,
                    graphql_name=f.name.replace("_", "")
                    if f.name.startswith("_")
                    else None,
                    type_annotation=StrawberryAnnotation(
                        Optional[f.type]
                        if not isinstance(f.type, StrawberryOptional)
                        else Optional[f.type.of_type]
                    ),
                    description="",
                    default=strawberry.UNSET,
                )
            )
            self.filter_fields.append(f.name)

All this really does is go over the fields defined and add them all as Arguments so that you can filter on them and they will be passed along as kwargs in the resolve functions.

Perfect.

Snag time

So I was porting the NodeTypes and this project also uses Relay. It is a certain implementation of GraphQL itself. Not very important, except for me to say now I had not made any connections yet. As in one Node –> another Node. Which is quite common in GraphQL and in Relay.

When I made the first connection, the schema would not even generate. I was so frustrated, and nothing worked. I could go from resolver function -> relay.ListConnection[NodeType] but not from Node -> relay.ListConnection[NodeType]. It kept complaining about it not being a GraphQLInput Type. I did not want it as an input. I struggled and looked deep into the source code of everything, trying to hack it there. Making it dynamically an input or an output depending on properties, and I suddenly stopped. Since there was no mention of this online whatsoever it had to be a problem I caused and created.

I went to bed, late. Woke up. Paced around a bit. In my head I thought, why is it automatically turning into an argu.....oh I am an idiot.

So I revisited my BaseExtension field that powered my dynamic argument adding stuff. I tweaked it here and there and the following is the fixed version:

class BaseExtension(FieldExtension):
    filter_fields = ["project_id", "change_order_id"]

    def apply(self, field: StrawberryField) -> None:
        self.filter_fields = ["project_id", "change_order_id"]
        resolved_type: Type[WithStrawberryObjectDefinition] = cast(
            Type[WithStrawberryObjectDefinition], field.resolve_type()
        )
        if resolved_type.__strawberry_definition__.specialized_type_var_map:
            node = cast(
                Type[WithStrawberryObjectDefinition],
                resolved_type.__strawberry_definition__.specialized_type_var_map[
                    "NodeType"
                ],
            )
        else:
            node = resolved_type

        for f in node.__strawberry_definition__.fields:
            if inspect.isclass(f.type) and issubclass(
                f.type, strawberry.relay.types.ListConnection
            ):
                continue
            if isinstance(f.type, StrawberryOptional):
                if inspect.isclass(f.type.of_type) and issubclass(
                    f.type.of_type, strawberry.relay.types.ListConnection
                ):
                    continue
            field.arguments.append(
                StrawberryArgument(
                    python_name=f.name,
                    graphql_name=f.name.replace("_", "")
                    if f.name.startswith("_")
                    else None,
                    type_annotation=StrawberryAnnotation(
                        Optional[f.type]
                        if not isinstance(f.type, StrawberryOptional)
                        else Optional[f.type.of_type]
                    ),
                    description="",
                    default=strawberry.UNSET,
                )
            )
            self.filter_fields.append(f.name)

Essentially what I needed to do was check if the type or of_type is a class. If it is check if it is a relay.ListConnection type class and then exclude it from the argument generation. All worked right after this.

Conclusion

I really like this framework. It gives me insight into how they operate and why sometimes a particular query is slow, and they give you the space to fix it. For example I already fixed the fact that we can load all the necessary subparts in one go from a node with the Dataloaders. That was not possible before. However it was still as slow as the old stack, because each node on it's own tried to create this new relay.ListConnection for one Edge essentially.

We already have all the instances needed to make all the edges when doing the Dataloader logic, so implement in that particular spot also the creation of all the edges in one go. Then have a simple mapping of node.id -> Edge and you are done. This sped up things by quite a significant margin.

Something the old stack could not really do. It had no real way of giving you the same tools to do the same thing.

#devlife #python #graphql

The goal was to prove that our Python code was horrible and needed to be switched to Golang implementation ASAP.

Rough start

I could not even start our Docker container because some Werkzeug, Flask and Locust combo made it all not work anymore. So I first had to untangle that mess. It turned out that some older code of Flask used a specific call to a function that does not exist anymore at the provided location.

For all who are interested, the actual error is: cannot import name 'BaseResponse' from 'werkzeug.wrappers'.

After that initial rough start I started out by mapping out how Docker actually works. What happens when you do docker pull or docker login for example. Turns out they are all just HTTP calls to a REST API backend. That returns some data and with that data we continue onward to more calls until all the data has been gotten for docker to actually create the containers and/or images.

Docker API

I wrote the simple Python PoC code for the DockerAPI client. In principle I can use that code now to get any image I want, but I do not use that. So I included that whole code into our Locust framework to make sure the test was always set up correctly, and that subsequent images were deleted.

I ran into the second problem. Images cannot be removed from a Docker registry by default. You have to enable that feature. So when I started talking to our devs, they said just forget about it. Do the setup code once, so that the image exists that is needed in a shared test repository and continue onward.

So I scrapped the entire code out of Locust and began again anew.

Concurrent issues

Next up came the problem that I wanted to only get credentials once, and share those credentials amongst the distributed workers. There were several hosts that each run multiple workers as separate processes. I wanted on each of those hosts, that one call got made by the worker process to get a nice token and share that token in memory with the rest. In comes SharedMemory by Python. I got it to finally work after fixing all my concurrent race condition failures, where there was no synchronised flag to make sure everybody waited on each other.

After all that code, the rest of the devs were that is cool but we do not need it. Just call the login at each start of the flow, it will create credentials and if there are already credentials it will return them. So again rip out the code written so far and start anew.

Finally on my way

Started again with the new flow and now I got a nice test up and running. The data returned was a bit baffling and showed our Python code was not the bottleneck as previously thought, hoped for. It was our Nginx reverse proxy setup. Split out the nginx pods unto their own and updated the config to handle things a bit better and give more threads and workers basically.

Okay after fixing the nginx pods, then ran the tests again and it turned out the Docker registry itself was a bottleneck. It just could not cope in terms of memory usage and freeing up stuff. We use Redis as our cache layer and Google Cloud Storage (GCS) as our bucket to actually store the data retrieved by Docker registry.

Breathing room

We had so much services jammed together in one pod it was crazy. Basically one pod ran the following services: – Nginx – Redis – Docker registry – Flask app

Then there was no control of what pod ran what services, so it could be that one pod ran 2 nginx + redis + docker registry + flask, whilst another ran only docker registry + flask. So back to basics, get one service per pod and split off the docker registry unto it's own node. Now we have the following setup:

• Nodepool A:
  • Three nodes
    • running one pod each of Nginx
    • running one pod each of Flask
    • running one pod total of Redis
• Nodepool B:
  • Three nodes
    • running one pod each of Docker Registry

Now that that was cleared up, the next bottleneck seemed to be Redis? So I turned to Redis and it's config and found out we actually were not using the staging Redis but the production Redis ?!?!?!

I quickly changed that config and made it so there was one node running a dedicated Redis. So the full situation becomes:

• Nodepool A:
  • Three nodes
    • running one pod each of Nginx
    • running one pod each of Flask
• Nodepool B:
  • Three nodes
    • running one pod each of Docker Registry
• Nodepool C:
  • One node
    • running one pod total of Redis

Okay, now can we finally move onward to find out that the Python code itself is so slow?

gunicorn

Well not so fast. Turns out that gunicorn was behaving badly and might do with some optimisation. gunicorn uses different worker classes and if we do not feed it the right ones with the right parameters it might actually be blocking. The reason I started looking down this rabbit hole was because of the gunicorn logs stating they ran out of workers.

After much experimenting on what parameters work best, turns out the best one that worked for us was the following:

CONCURRENCY_SETTING=$(python3 -c 'import multiprocessing as mp; print(mp.cpu_count() * 2)')
exec /usr/local/bin/gunicorn -n internal_auth_secret -w${CONCURRENCY_SETTING} -k gevent --worker-connections=1000 -b 0.0.0.0:8000 internal_auth_secret:app -t 180

Meaning use the gevent type worker class, with 1000 worker connections. Also use a total amount of workers to twice the amount of cores available to us in whatever host we are running as. This also meant it is dynamic to the point where if we would ever upgrade the hardware of the node underlying the pod it will grow with it automatically without us having to make sure we also update the amount of workers.

Conclusion

After fixing all the infrastructure setup of correctly allocating memory and CPU to each of the services, coupled with separating them out to make sure each of them gets the appropriate amount needed. Making sure our nginx was configured correctly. Followed by actually configuring the services in staging correctly to point at services in staging rather than production, followed by configuring the gunicorn service and fine-tuning it, there was still a slight bottleneck.

Yeey, finally Python code is slow and dumb and move on to Golang. Hold on, let us first see what is being the bottleneck. I made some call graphs using the following module https://github.com/daneads/pycallgraph2. It showed that the bottleneck was partly in our shared library code that handled authentication and also the way we were determining when to call that particular function. Finally the culprit has been located.

To fix the shared library code was easy, just improved the for loops and small optimizations in terms of what to store so we do not do a constant looking up of the same values. Cache more in Redis, then also use a Redis connection pool rather than starting up a new connection every time for each query.

To fix the problem of knowing when to call the function in the shared code was a literal one if else statement added to the previously declaring of the variable logic. It was a code fix of 44 characters that resulted in an improvement of the total time spent. The longest before this fix was 465ms on the shared library code path. After fixing both it was only around 60ms. So instead of the code being able to handle roughly 2 per second we could now handle roughly 15 per second per worker per worker_connection.

After that roller-coaster of a ride, I made sure we could handle millions of requests coming in rather than just couple of hundred. The next optimisations lie in Network I/O and other factors. Even if we would move towards Golang implementation it might gain us 1ms max in terms of code maybe, that is even highly optimistic and probably not even realistic. The rest lies in the fact that we have a nginx going to a docker registry talking to another service running somewhere else again on the network that talks to Redis. Those round trip times are starting to add up.

However that is for another time. Right now we got enough to make sure we can get through the next years of running our service. If we need more, just scale the entire setup to include more nodes, until the bottleneck is network throughput/bandwith. Then we will revisit this.

#100DaysToOffload #DevOps #python

var greetings = {
 hello: function() { console.info("Hello"); }
};
greetings.hello();
greetings["hello"]();

This is how functions could be shipped around. So the idea is simple, especially the second one. I started to wonder if we can have a Map of String -> Function in other languages like Java, Python, PHP and more?

Java

We sure can. I was working on a JSON-RPC project where you don't send entity bodies but you send what method to invoke and with what arguments. In the backend we made it so internally a consistent object was shipped around (RPCServiceResult) and we had Enum for the methods that were allowed. So I set up an EnumMap of Enum -> Function. This made it so the entrypoint of each service was the same. See if the method key exists, if so execute the corresponding function with the given input otherwise throw a MethodNotFound error. This meant we could extend the Enum and EnumMap and the Function mapping but never had to change how the services called each other.

Python

In Python I was working on a silly hacking simulation game in the Console. I wanted to have a list of commands and execute the corresponding function that was that command in game terms. At this point I just came from PHP and there it is completely normal to execute the string itself. I will come back to this in a bit. This is more difficult to do in Python and so I came up with a dict where command name was the key and function references were the values. This made it super simple to maintain.

PHP

So in PHP this is normal:

function a(){ echo "A"; };
$a = "a";
$a();

However it is really difficult to maintain and see what is going on, let alone insecure to do this if the input to $a is user input. The following makes more sense and is easier to maintain.

function a() { echo "A"; };
$commands = ["a" => function() { call_user_func("a"); }];
if ( array_key_exists("a", $commands) ) { $commands["a"](); }

Granted this is a contrived example but I hope it gets the point across.

Scala

I was working on the Advent of Code 2020 and one of the puzzles had a lot of validation rules. Immediately the structure came to mind of having a Map of Functions, for each field to validate the corresponding function. This worked like a charm and it is very easy to see what each rule does.

Java

Another example was that in a particular flow there was a need to either create a User of type A or type B and then continue the flow. It was first like the following:

if ( type == "UserA" ) {
    return new UserA();
} else {
    return new UserB();
}

Now never mind the bug of always returning type B, I suggested the following:

Map<String, Supplier<User>> userSupplier = new HashMap<>(){{ put("UserA", UserA::new); put("UserB", UserB::new); }};

Now you could replace it with one entrypoint:

return userSupplier.getOrDefault(type, UserB::new).get();

You still have the bug, but it became a feature. So we leave it in. The colleague immediately said, this happens in way more places in the code and I can change all of them.

The nice thing again is, we can extend the map to support more types but it does not change the logic of calling the service.

Conclusion

I hope this concept proves useful to you, I find it quite powerful and easy to construct and it gives lots of nice patterns.

#code #javascript #java #python

First things first, the async structure in essence means a form of cooperative multi-threading. This is in contrast with preemptive multi-threading. The latter means there will be scheduled allotted CPU cycles where for example a function needs 3 cycles, but gets 2 at a time over the course of all work that needs to be done can actually take 6 cycles.

In cooperative model the function gets to take as many cycles with regards to the CPU scheduler though and then hand back control after 3 and hopefully that means in this case the cycles are used in a more direct and efficient way making the program smoother and more efficient.

As a side note, async should only be used as you expect to be I/O bound and also it should be used completely throughout the application and third party libraries.

So this also means it will be running in the background whatever this function is you wanted to make async, and therefore the keyword await was introduced stating you should await the function for when it is done.

Getting started

First something simple.

def f():
    pass

A simple function definition. If we wanted an async function we would do the following:

async def a():
    pass

So far only the only difference, other than the name, is the keyword async. So what happens when you print these defined symbols.

print(f) # <function f at 0x7fa7b50af1f0>
print(a) # <function a at 0x7fa7b4fb9ca0>

So nothing apparently. They are both just function definitions. What happens when you print the result of executing the functions?

print(f()) # None
print(a()) # <coroutine object a at 0x7fa7b5037940>

You will also get a RuntimeWarning of a coroutine not being awaited. We leave that for what it is right now. So interestingly we see they are both functions before and after the execution the one has a result, the other a coroutine object. So we want to see if there is a way to determine if the function is a coroutine aforehand.

Let us write something that does that:

import inspect

print(inspect.iscoroutine(a)) # False
print(inspect.isawaitable(a)) # False

print(inspect.iscoroutine(a())) # True
print(inspect.isawaitable(a())) # True

Hang on, they both state False when operating on the definition. However our RuntimeWarning said we needed to await. When we execute the function we do get the information, but that is still after the fact.

We first have to execute a function in order to find out we need to await. There might be some people out there right now that go well you used a wrong method. There will be two solutions at the end.

Decorators

In Python there exists the following syntactic sugar:

def decorator(func):
    def wrapper():
       func()
    return wrapper

@decorator
async def a():
     pass

a()

This is the same as doing a = decorator(a) and then calling a() will actually execute the wrapper, as a will be now equal to wrapper.

First problem we run into in this example is we gave an async function to a non-async function. You can only await in async functions. That is easily solved:

def decorator(func):
    async def wrapper():
       await func()
    return wrapper

Now the wrapper is also async and we await the function inside. However this decorator might be used for async and non-async functions alike. We still need to determine accurately whether or not a function is async aforehand.

Internals

Looking at the internals of Python there exists the __code__ property on functions. Inside that property there is a co_flags property. That will actually hold a bitmap value of what flags the function itself holds. You can get at this information in the following way:

from dis import pretty_flags

def f():
    pass

async def a():
    pass

print(pretty_flags(f.__code__.co_flags)) # OPTIMIZED, NEWLOCALS, NOFREE
print(pretty_flags(a.__code__.co_flags)) # OPTIMIZED, NEWLOCALS, NOFREE, COROUTINE

Aha, we see now that we can determine if a function is a coroutine or not. This means we can make our decorator correctly now:

from dis import pretty_flags


def decorator(func):
    def wrapper():
        func()
    async def async_wrapper():
        await func()
    
    if "COROUTINE" in pretty_flags(func.__code__.co_flags):
        return async_wrapper
    
    return wrapper

Final Solution

I already mentioned there are two solution. I first wanted to get this internal solution out the way as that is the one I used first. Then I ran into something that made more sense. So the before mentioned decorator can also be written thusly:

import inspect


def decorator(func):
    def wrapper():
        func()
    async def async_wrapper():
        await func()
    
    if inspect.iscoroutinefunction(func):
        return async_wrapper
    
    return wrapper

The importance of using the correct method is abundantly clear in this case. Our definition is a coroutine function, not yet a coroutine and therefor you also cannot await a definition. Only an instance of the executed async function.

Hope this helps out a bit in the future of your async python adventure.

As a final final example the decorator should probably look like this:

import inspect


def decorator(func):
    def wrapper(*args, **kwargs):
        func(*args, **kwargs)
    async def async_wrapper(*args, **kwargs):
        await func(*args, **kwargs)
    
    if inspect.iscoroutinefunction(func):
        return async_wrapper
    
    return wrapper

In order to propagate any and all arguments given to the function you are decorating.

#code #python

So maybe we need a few new ideas.

Point of view

As most quasi intellectuals who want to sound philosophical like to say; It all depends on your point of view. However, in this case I want to make for how to ship Python applications, I want to make sure you embody the different individuals who partake in the process of shipping it or deploying it if you want to be technical. One is the developer who writes the source code, the other is the DevOps engineer tasked with deploying it wherever it is needed.

Ecosystem

The ecosystem for Python applications is great if you are the developer that just has to write the source code to make it work. You activate a virtual environment using any tool that has your preference. You install the necessary libraries outside of the standard library that you need. You write the code until it all works. Job done.

The ecosystem for deploying it sucks. Either you use the same flow as the developer but that is not the context and environment of the DevOps engineer. You really wanted to use the packages from the package managers of the Operating System you deploying it to. You cannot however because the same versions do not sync up. So you have to choose.

Either Or

Either you use the developer tools and flows for the duration of deploying and your Continuous Integration process. Or you use the system packages to pre-built docker images where you run your code and development process. It is either or. There is a big discrepancy between the developer context and the deployment context, or if you will the end user.

Let us assume you made something and you want other people to run it too. How do you get your Python application to the end user? You might get away with making a self running archive. Though one thing needs to be there and that is the C libraries on the host machine. So you cannot get away with prepping the host machine environment.

The one solution is to statically compile everything and not dynamically link it. This means a lot of management done by developers or DevOps engineers themselves. It would solve the problem though that you can ship your archive and anyone else can just run it. Not a feasible thing to accomplish as it introduces a lot of unneeded complexity.

Two layers

There also exists the possibility of having the self running archive and then also putting it inside a docker image. These two layers can have two independent running pipelines. One to produce the archive and one that produces the image. Now you can update the runtime without changing the source code and run the risk of introducing anomalies. You take the archive out and you update the Python runtime and reinstate the archive and run it.

It gives you more fine grained control if that is needed.

#devops #python

Online Encyclopedia Integer Sequences

There exists an online database of integer sequences. It has a lot of cool sequences and one of them is the one with all Narcissistic Numbers. It has number A005188 and you can find it here.

This next code is inspired and based on the code in the OEIS.

Code

from itertools import combinations_with_replacement
from datetime import datetime
import sys

POWERS = [0,1,2,3,4,5,6,7,8,9]
NARCISSISTIC_NUMBERS = []

def recalculate_powers():
    for i,v in enumerate(POWERS):
        POWERS[i] = i * v

def equals(number: int, target: tuple) -> bool:
    digits = []
    while number > 0:
        _, number = digits.append(number % 10), number // 10
    return tuple(sorted(digits)) == target



def find_narcissistic_numbers(desired: int) -> None:
    global POWERS, NARCISSISTIC_NUMBERS
    for k in range(1, sys.maxsize):

        for b in combinations_with_replacement(range(10), k):

            x = sum(map(lambda y:POWERS[y], b))
            if x > 0 and equals(x, b):
                NARCISSISTIC_NUMBERS.append(x)
                if len(NARCISSISTIC_NUMBERS) == desired:
                    return
        recalculate_powers()

start = datetime.utcnow()
find_narcissistic_numbers(28)
print(datetime.utcnow() - start)

The combinations_with_replacement gets numbers 0-9 and how many times to do it. So for example when k = 2 you get:

[0,0] [0,1] ... [1,1] ... [2,2] ... [9,9]

And as you can see you lose one number every time you go up in tens. As 0,1 is equal to 1,0 and 1,2 is equal to 2,1. There are considerable amount of fewer computations to check.

So this code takes this much time to run on my laptop: 0:00:00.204806

That is an insane speedup in time compared to the naive implementation and the optimized implementation.

Caveat

Small thing I found out whilst running this code. I thought it was enough to do sorted((2,1)) but this returns a list!!. So then I needed to wrap that with another tuple or the other one with list and I did not want to do that. So therefore I went with the initial list and turn that into a tuple.

#code #python

import sys
from datetime import datetime

POWERS = [0,1,2,3,4,5,6,7,8,9]
LENGTH = 1

def recalculate_powers() -> None:
    global POWERS
    for i,v in enumerate(POWERS):
        POWERS[i] = i * v

def is_narcissistic(x: int) -> bool:
    global LENGTH, POWERS
    org = x
    digits = []
    total = 0
    while x > 0:
        _, x = digits.append(x % 10), x // 10
    cur_length = len(digits)
    if cur_length != LENGTH:
        LENGTH = cur_length
        recalculate_powers()
    for i in digits:
        total += POWERS[i]
        if total > org:
            return False
    return total == org

def find_narcissistic_numbers(desired: int) -> None:
    for x in range(1,sys.maxsize):
        if is_narcissistic(x):
            desired -= 1
        if desired == 0:
            return

start = datetime.utcnow()
find_narcissistic_numbers(28)
print(datetime.utcnow() - start)

The time for this code to run on my laptop is: 0:06:21.202479

To compare this to the original run of the naive implementation is, which was 0:08:09.315972. Even on my slow hardware, it still resulted in a speed up of about 25%.

#code #python

So we make a lookup table and put all the powers in there in a nice way. If the position is the same as the value contained within then it is really easy to calculate the next power. It is just multiply the value by the index it is located at. This process is called memoization. It is when you store or cache the results of expensive calculations and just look them up the next time you need them.

import sys
from datetime import datetime

POWERS = [0,1,2,3,4,5,6,7,8,9]
LENGTH = 1

def recalculate_powers() -> None:
    global POWERS
    for i,v in enumerate(POWERS):
        POWERS[i] = i * v

def is_narcissistic(x: int) -> bool:
    global LENGTH, POWERS
    org = x
    digits = []
    total = 0
    while x > 0:
        _, x = digits.append(x % 10), x // 10
    cur_length = len(digits)
    if cur_length != LENGTH:
        LENGTH = cur_length
        recalculate_powers()
    for i in digits:
        total += POWERS[i]
    return total == org

def find_narcissistic_numbers(desired: int) -> None:
    for x in range(1,sys.maxsize):
        if is_narcissistic(x):
            desired -= 1
        if desired == 0:
            return

start = datetime.utcnow()
find_narcissistic_numbers(28)
print(datetime.utcnow() - start)

The time for this code to run on my laptop: 0:06:23.868587

#code #python

import sys
from datetime import datetime

def is_narcissistic(x: int) -> bool:
    org = x
    digits = []
    while x > 0:
        _, x = digits.append(x % 10), x // 10
    power = len(digits)
    s = sum([i ** power for i in digits])
    return s == org

def find_narcissistic_numbers(desired: int) -> None:
    for x in range(sys.maxsize):
        if is_narcissistic(x):
            desired -= 1
        if desired == 0:
            return

start = datetime.utcnow()
find_narcissistic_numbers(28)
print(datetime.utcnow() - start)

The time for this code to run on my laptop is: 0:07:14.172538

#code #python