Safety in resource management in Python

Yet another example of the type safety in Python maybe a resources management. We can use typing to ensure that resources were released, so no any resoucres leak. Yes, it's possible. The idea is that we will check in compile time with types that resoucres were released by checking that they were returned as appropriated type after its usage. Look the example:

from typing import *
from abc import *

T = TypeVar('T')
class Hold(ABC): pass
class Free(ABC): pass

class Resources(Generic[T]):
    def __init__(self, fp1:str, fp2:str):
        self.f1 = open(fp1, "r")
        self.f2 = open(fp2, "r")
    def _release(self):
        self.f1.close()
        self.f2.close()
        print("Resources were released")

def get_resources(fp1:str, fp2:str) -> Resources[Hold]:
    return Resources(fp1, fp2)
def release_resources(r: Resources[Hold]) -> Resources[Free]:
    r._release()
    return cast(Resources[Free], r)

def use_resources(r: Resources[Hold]) -> Resources[Free]:
    print(r.f1.read(), r.f2.read())
    return release_resources(r)

r = get_resources("f1.txt", "f2.txt")
use_resources(r)

If we will try to ommit return statement in the use_resources() function, then we will get an error Missing return statement from mypy. OK, if we will try to return None, then the error will be Incompatible return value type (got "None", expected "Resources[Free]"). OK, we will return the resource itself: return r. But an error again, now: Incompatible return value type (got "Resources[Hold]", expected "Resources[Free]").

It's a very good error: it says us that we don't release resources, we complete in the Resources[Hold] state. To fix it we will return release_resources(r) and now all is fine. The function use_resources() is wrapper which wraps the usage of them, tracks correct state of their life-cycle. It's typical for Haskell with phantom types, we tried to create one here, in Python. So, the function gets acquired resources, uses them and then releases them, and we will not forget to call release_resources().

PS. Those 3 resource related function can be implemented as class methods of Resources class.

Yet another example of type safety in Python

Python can be safe (if we need it) too. Another example is: let's say we want to implement a function unique(). It may looks like:

def unique(data):
    res: list = []
    for x in data:
        if res and x == res[-1]: continue
        else: res.append(x)
    return res

Truth is that data should be sorted - we don't want to sort it because data items maybe are sorted already. So, instead to write docstring about this our requirement ("data should be sorted") we can specify this requirement as a type, so it will prevent a client from passing of data which was not sorted, so the check will work at compile time:

from typing import *

class Sorted(list):
    def __init__(self, items):
        super().__init__(sorted(items))

def unique(data: Sorted) -> list:
    res: list = []
    for x in data:
        if res and x == res[-1]: continue
        else: res.append(x)
    return res

items = 11, 20, 11, 20, 1, 20, 11, 5, 1
data = Sorted(items)
print(unique(items))

And if we try to pass something which has not type Sorted then we will get an MYPY error at compile time. It looks like Haskell, ah? :)

Transitions correctness checking in compile-time

One of the accusations against the OOP is state problems. They are divided into several sub-kinds, one of them is problem of transitions correctness, i.e. control when a transition from one state to another state is incorrect and should be avoided.

People often say that no such problem in FP languages. It's not true, but such illusion exists because of explicit arguments flow from function to function. And this problem appears when all arguments collapse to one argument aggregating all others. In this case the situation is reducing to the case in OOP.

There are multiple canonical solutions - working in compile-time with type-checker help or in run-time:

Run-time solutions are:

• FSM - the best solution
• contracts - not so good, but OK
• asserts - reduced version of contracts

For example, function close() makes sense only if something was opened (let's suppose the result of the open in file):

def close(self):
    assert self.file, "File was not opened"
    ...

something like this. The same is possible in an any language.

Compile-time solutions maybe:

• Linear types
• Unification (see Prolog) - more general solution
• Indexed monads (for Haskell, for example)
• Simple data-types (like data Opened = Opened String; data Read = Read String)

So, if some function must be under such control then even if it does not expect arguments at least one argument must be presented always, it may be term without parameters.

Of course, such a solution exists a long time ago in the OOP. Let's simulate it with a simple functionality - open a file, read it, parse it to integer, close it.

So, we see that we have states: {opened, read, parsed, closed} and each state accepts only limited number of transitions:

open -> read | close
read -> parse | close
parse -> close

That is, each function allows only limited number of next functions in our flow: open allows only a call of read or close and so on. Graphically it looks like:

open ------->.
 |           |
 v           |
read ------->.
 |           |
 v           |
parse ------>.
             |
             v
           close

close is allowed after any state.

We can turn it upside down:

       close
         ^
         |
         +----- parse
         |        ^
         +----- read
         |        ^
         +----- open

and to treat it as classes hierarchy: classes parse, read, open includes common feature - close, they know how to do it, so they inherit the close. Each class has some methods (one or more) and methods expect arguments of these classes types. This will guard us from wrong transitions in compile-time. Actually, this solution is possible even in Python:

class Open:
    def open(self, path: str) -> "Open":
        self.file = open(path, "r")
        return self

class Close:
    def close(self, state: Open) -> "Close":
        state.file.close()
        return self

class Read(Close):
    def read(self, state: Open) -> "Read":
        self.buf = state.file.read()
        return self

class Parse(Close):
    def parse(self, state: Read) -> "Parse":
        self.n = int(state.buf)
        return self

o = Open().open("z")
r = Read().read(o)
p = Parse().parse(r)
print(p.n)
p.close(o)

All classes implement methods - what they can do. And each of them additionally can close opened file, so they inherit Close class (except Open sure). Their methods are typed with explicit argument state which restricts allowed transition, also it keeps data of the state. Pay attention that call p.close(r) for example, leads to "compile-time" (mypy checking, to be more accurate) error, i.e. transitions are explicit: Parse does not allow a parsing immediately after an opening of the file, without reading of it: p = Parse().parse(o):

error: Argument 1 to "parse" of "Parse" has incompatible type "Open"; expected "Read"
Found 1 error in 1 file (checked 1 source file)

A little improvement maybe to keep Open as instance attribute .opened to allow to close the file from any state, keeping the result of the opening (or something similar).

Classical pattern helps us even better than modern FP languages like Haskell where developers use indexed monads very rarely (and do miss linear types).

But there are not OOP languages where this is solving on type-level too: ATS, Clean, F*...

Reduce points density

Sometimes charts have so many points that drawing of them is very highweight. The solution is to reduce points density (to thinning them). Idea is simple: if there are too many samples per linear (of the output device) resolution.

There are different possible approaches: a filtering - it's a bad solution because peaks will be lost or some kind of approximation - thinning, which is better because peaks are saved:

Original chart:

Chart with reduced points density:

The algorithm in Python language:

def reduce_density(k, points):
    x0 = y0 = u0 = None
    last = None
    for i, (x, y, u) in enumerate(points):
        if i == 0:
            yield (x, y, u)
            x0, y0, u0 = x, y, u
        else:
            if abs(y - y0) < k:
                last = (x, y, u)
                continue
            else:
                x0, y0, u0 = x, y, u
                if last is not None:
                    yield last
                yield (x, y, u)
                last = None
    if last is not None:
        yield last

points is the list/iterator of triplet: (x, y, physical_value). x, y - are coordinates of the point, physical_value is the 0y value in some physical units which can be used for a printing value near the chart point or something else. The idea is simple: enumerate over the points and looking for a change of the value (y) greater than some coefficient k (number of pixels of the output device) - while values change is less than k treat them as a line (skip them with continue). But otherwise - treat it as a valuable point and return it (with yield).

This algorithm allows to minimize drawing points number which will improve output performance and will decrease the size of the files with the points without to loose valuable points.