Home


python notes

See also:

~/notes/pip.md
~/notes/pyenv.md
~/notes/pynamo.md
~/notes/numpy.md
~/notes/pandas.md

How to get interactive help in a repl

  1. Append a ? to the fn or method I’m trying to understand

    open?

  2. Append ?? to see docstring and source!

    import requests
    requests??

  3. Use the help built-in (I prefer ??):

    help(sorted)

  4. Print docstring directly:

    print(open.__doc__)

  5. Use dir, the old standby:

    x = “abc”
    dir(x)

How to memoize

Before 3.9:

from functools import lru_cache  
@lru_cache(maxsize=None)  
def fn...

After 3.9:

from functools import cache  
@cache  
def fn...

Timing functions

If in ipython or jupyter notebook, use magic %timeit

For a single pass, use perf_counter, which is guaranteed to be monotonic:

import time  

t = time.perf_counter()  
# do work  
print(time.perf_counter() - t)

For repeated runs and summary statistics, use timeit:

import timeit, statistics  

def work():  
    pass  

times = timeit.repeat(work, number=1000, repeat=5)  

print("runs:", times)  
print("min:", min(times))  
print("max:", max(times))  
print("mean:", statistics.mean(times))  
print("stdev:", statistics.stdev(times))

Can get a ruby-ish feel with a context manager:

import time  
from contextlib import contextmanager  

@contextmanager  
def timed_block(label: str = "timed_block"):  
    start = time.perf_counter()  
    try:  
        yield  
    finally:  
        end = time.perf_counter()  
        print(f"{label}: {(end - start) * 1000:.3f} ms")  

with timed_block():  
    total = sum(i * i for i in range(10_000_000))

Get multiple indices of a list

from operator import itemgetter  
x = ['a', 'b', 'c']  
indices = [0, 2]  
itemgetter(*indices)(x) # => ('a', 'c')

Or, if I can’t remember all that

x = ['a', 'b', 'c']  
indices = [0, 2]  
[x[i] for i in indices]

Or

x = ['a', 'b', 'c']  
[y for i,y in enumerate(x) if i in [0,2]]

Use StrEnum if I need rawvalues of strings

from enum import StrEnum  

class Role(StrEnum):  
    SYSTEM = "system"  
    USER = "user"  

Role.SYSTEM.value # => "system"

Use auto to skip labeling enums

from enum import Enum, auto  

class Role(Enum):  
    SYSTEM = auto()  
    USER = auto()

Use classmethod decorator for class methods

class C:  
    @classmethod  
    def y(cls):  
        return cls  

C is C.y() # => True

Use array for homogenous primitives

import array  
array.array('I', [1,2])

asyncio run main

Basic setup to get an async context in python:

import asyncio  

async def main() -> None:  
    print("sleeping for one second...")  
    await asyncio.sleep(1)  
    print("done!")  

asyncio.run(main())

Pydantic basic usage

Pydantic doesn’t use positional arguments:

from pydantic import BaseModel  

class User(BaseModel):  
    name: str  

# Create a user from safe input  
user = User(name="a")  

# Or  
user_data = {'name': 'a'}  
user = User(**user_data)  

# Create a user from unknown user input  
User.model_validate({'name': 1})  

# Or   
User.model_validate_json('{"name": "lou"}')  

# Get json:  
user.model_dump_json()

Python’s GC

Primary mechanism is ref counting:

In CPython, the primary algorithm for garbage collection is reference  
counting. Essentially, each object keeps count of how many references point  
to it. As soon as that refcount reaches zero, the object is immediately  
destroyed: CPython calls the __del__ method on the object (if defined) and  
then frees the memory allocated to the object. In CPython 2.0, a  
generational garbage collection algorithm was added to detect groups of  
objects involved in reference cycles

Fluent Python pg 219

tuples are immutable

x = (1,)  
id(x)  
x += (2,)  
id(x) # => different ID

But be careful when a tuple holds a mutable instance:

x = ["a"]  
y = (x,)  
x.append("b")  
y  
(['a', 'b'],)

types with value semantics

Get bytecode

def fn():  
    return "hello".count("l")

Raw bytecode:

fn.__code__.co_code

Readable bytecode:

import dis  
dis.dis(fn)

Or:

list(dis.get_instructions(fn))

ruff linter can fix in place

I’ve been using flake8, but starting to use ruff instead. E.g.

ruff check  
ruff format

assert is a statement

If I use parens, it’s just grouping, not a function call:

assert(x == 1)

Is actually

assert (x == 1)

It’s as if I were writing:

if(true):

Which I would not do.

Python visualizer

https://pythontutor.com/render.html#mode=display

Disallow named params

Use the /

def fn(a, /):  
    pass  

fn(a="1")  # Raises TypeError

Remove from dictionary without raising

d = {'a': 'b'}  
d.pop('c', None)

Iterate over a dict, python gotcha

This does not do what I expect:

for k, v in { 'ab': 12 }:  
    print(k)  
    print(v)  
# => prints 'a' followed by 'b', because I'm actually unpacking the key only

To destructure the keys and values from a tuple at each iteration, use items():

for k, v in {'ab': 12 }.items():  
    print(f'k: {k}, v: {v}')

Pydantic immutable record

I always liked NameTuple. Pydantic went with mutable by default, which makes
sense for the domain. I can use this where I want immutability:

from pydantic import BaseModel, ConfigDict  
  
class User(BaseModel):  
    id: int  
    name: str  
    model_config = ConfigDict(frozen=True)

Then:

 u = User(id=1, name="foo")  
 u.id=2  
 # => Raises a ValidationError

Type hints in 3.9+

Importing typing is no longer necessary.
But, I still need mypy as the static checker.
There is nothing built into python to static check your types.

Type hints refresher

a: tuple[int, str] = (1, "hi")  
b: list[str] = ["a", "b"]  
c: dict[str, str] = {"a": "b"}  
d: set[int] = {1, 2, 3}  

from collections.abc import Callable  
e: Callable[[int], int] = lambda x: x + 1  
f: Callable[[int, int], int] = lambda x, y: x + y  
g: Callable[[int | None], int] = lambda x: x if x else 0

For optionals, I can either use:

#  3.10+ only  
h: int | None = None  
i: int | None = 1

or:

from typing import Optional  
j: Optional[int] = None  
k: Optional[int] = 1

or:

from typing import Union  
l: Union[int, None] = None  
m: Union[int, None] = 1

Set refresher

s = set()  
s.add(1)  
1 in s # => True

NamedTuple refresher

NamedTuples are immutable:

from typing import NamedTuple  
class Point(NamedTuple):  
    x: int  
    y: int  
p = Point(x=1, y=0)  
p.y = 1  
# => Raises AttributeError

Dataclass refresher

Dataclasses are mutable:

from dataclasses import dataclass  

@dataclass  
class Point:  
    x: int  
    y: int  
p = Point(x=1, y=0)  
p.y = 1  
# => Does not throw

Frozen dataclasses and a gotcha

from dataclasses import dataclass  

@dataclass(frozen=True)  
class D:  
    field = 1  

d = D()  
d.field = 2 # => raises FrozenInstanceError, as expected  
d.field # => 1  
D.field = 99  
d.field # => 99

Dataclasses with field factory for list

Don’t do this:

@dataclass  
class D:  
    l = []  

D().l.append('a')  
D().l  
# => ['a']

Do this:

from dataclasses import dataclass, field  
@dataclass  
class D:  
    l: list[int] = field(default_factory=list)  

D().l.append('a')  
D().l  
# => []

Immutable dict

It’s not hashable, but it does guarantee elements can’t chance:

from types import MappingProxyType  

x = MappingProxyType({'a': 1})  
x['a'] = 2 # => raises 'TypeError: 'mappingproxy' object does not support item assignment'

Note a mutable type still allows mutation in the collection:

x = MappingProxyType({'a': [1]})  
x['a'].append(2) # Fine  
x['a'] # => [1, 2]

Ternary gotcha

I always assume this is valid python, it’s not: x ? x : 0
I want: x if x else 0

Check MRO on a type

Foo.__mro__

Debugging

pdbpp is an absolute must. I don’t use the following any more. I just stick to pdb and pdbpp with this in ~/.pdbrc.py

from pdb import DefaultConfig  

class Config(DefaultConfig):  
    sticky_by_default = True

No longer used, because ipdb doesn’t play super nicely with pdbpp:

`ipdb` is also handy, because I can get it show additional lines around a breakpoint.  
  
Add this to `~/.bash_profile`  
  
    # Use ipdb when python hits a `breakpoint()`  
    export PYTHONBREAKPOINT="ipdb.set_trace"  
  
    # Show additional context around ipdb breakpoints  
    # Change this by typing 'context NUM' when a breakpoint is hit. Then 'bt' to redraw the source.  
    export IPDB_CONTEXT_SIZE=5  

With that config, I don't have to explicitly breakpoint with `import ipdb; ipdb.set_trace(context=5)`.  
I can use `breakpoint()` and then either `sticky` if I want to use pdbpp or `context NUM` followed by `bt` if I just want to see more.

Start repl with async support

python -m asyncio

Install a modern version on AL2023

https://github.com/amazonlinux/amazon-linux-2023/issues/483#issuecomment-1928446605
^ Don’t do this. Do this:

sudo dnf install python3.12

Walrus

Available in python 3.8+

if (n := "world"):  
    print(f"{n} hello")

Define all public exports

I prefer explicit imports, but I see this often. __all__ defines what should be imported on import * from xyz

__all__ = ["MyClass"]  

class MyClass():  
    pass

Defaultdict usage

from collections import defaultdict  
x = defaultdict(str)  
x['y'] # => ''

Or commonly with a list

x = defaultdict(list)  
x['y'].append(1)

sorted always returns a list

sorted(['b', 'a']) # => ['a', 'b']  
sorted('ba') #=> ['a', 'b']  
sorted(('b', 'a')) # => ['a', 'b']

Lists are not hashable

Only immutable objects can be hashed and added to sets / dicts.
The trick is to use a tuple instead:

s = set()  
x = ['a', 'b']  
s.add(x) # => raises TypeError  
s.add(tuple(x))

Split a string into list

"a b".split(" ") # => ["a", "b"]  
list("ab") # => ["a", "b"]

Join a list into str

''.join(['a', 'b'])

Modern typing for callable

Old:

from typing import Callable

New:

from collections.abc import Callable

Protocol usage

from typing import Protocol  
from typing import TypeVar  

class ExampleProtocol(Protocol):  
    """  
    A common interface for types that provide a `foo` property, which  
    can be helpful to satisfy mypy when duck typing / using generics.  
    """  
    @property  
    def foo(self) -> float:  
        ...  


# Later  
# from whatever.example_protocol import ExampleProtocol  
T = TypeVar('T', bound=ExampleProtocol)  

def requires_a_foo(x: T):  
    print(x.foo)

Lists can be assigned by slice

x = [0, 1, 2, 3]  
s = slice(None, None, 2)  
x[s] = ['a', 'b']  
x  
# =>  ['a', 1, 'b', 3]

Get the next element from an iterator

it = iter(range(10))  
next(it) # => 0

Get the first elements from an iterator or generator

from itertools import islice  
gen = (x * x for x in range(1, 10))  
list(islice(gen, 0, 3))  
# => [1, 4, 9]

Get first elements from generator

import itertools
first5 = itertools.islice(gen, 5)
print(list(first5))

Decode bytes to utf8 string

my_bytes.decode('utf8')

It’s ok to define a var in a catch block

# Do not need to define x here  
try:  
    raise ValueError("broken precondition")  
except:  
    x = 1  
x += 1  # => 2

Huh, same as ruby

begin  
  raise 'bad'  
rescue  
  x = 1  
end  
x + 1  # => 2

With js I need to define x ahead of time:

let x;  
try {  
  throw Error("Bad");  
} catch {  
  let x = 1;  
}  
x + 1;  // => 2

Run pytest with verbose output

pytest -svv

Set a breakpoint

In modern versions of python breakpoint() will suffice.
No need to import ‘pdb’ explicitly.

TODO: This file needs to be formatted.

Also see ~/notes/pip.md
Also see ~/notes/pyenv.md

(python, inspiration, live coding, David Beazley, performance, concurrency, GIL, coroutines)
https://www.youtube.com/watch?v=MCs5OvhV9S4
Source: https://news.ycombinator.com/item?id=36785005

(python simple http server, SimpleHTTPServer, python 3)
Serve current directory at http://localhost:8000
python -m http.server

(python, repl, save file)
After prototyping in the repl, dump all code I’ve entered into the repl:
%save my_file.py
So awesome!

(python, debugging)
Examine MRO with:
class Foo
pass
class Bar:
pass
class Baz(Foo, Bar):
pass
Baz.__mro__

(python, gotchas)
https://github.com/satwikkansal/wtfpython

(pdb, which file am I in)
p file

(pdb, command list)
https://docs.python.org/3.6/library/pdb.html#pdbcommand-step
unt is helpful to get out of a loop
Also use up and down

ipython turn off autocompletions

ipython --TerminalInteractiveShell.autosuggestions_provider=None

(ipython, mypy)
Check mypy types in the repl with:
pip install mypy_ipython
ipython
%load_ext mypy_ipython
… do some prototyping
%mypy

(pdb, pytest)
run tests with –pdb to drop you into a repl on test failure

(pytest, mocking)
Use mock.mock_calls to get a list of all calls sent to a mock.
Or mock.call_args_list. I’m not sure what the difference is.

(print fstring, debug)
x = ‘hello’
print(f’{x = }’)

Also see alternatives ‘ice cream’ and ‘q’: https://github.com/zestyping/q

(pdbpp)
Use bt to print frames, then
f [number]
to jump to one

(python modify path)
PYTHONPATH=“./:$PYTHONPATH” python path/to/script.py

(read unit.xml files)
Install https://github.com/lukejpreston/xunit-viewer
Then
pytest –junit-xml=build/unit.xml
xunit-viewer -r build/unit.xml

(python, pypi, list versions, pip list versions, hack)
pip install my-package==

(python zen of python)
import this

(python testing sibling library)
cd ~/dev/my_app
source .env/bin/activate
pip install -e ~/dev/my_lib

(python path, pythonpath, modify path, run from vim)
:!PYTHONPATH=“app:$PYTHONPATH” python %

(files, naming, modules)
.py files can be imported as a module, but only if they do not contain hyphens in the name
bad: from bad-thing import foo
good: from good_thing import foo

(bdist, sdist)
See ~/dev/snippets/python/example_package/README.md and the associated
package layout

sdist is a source distribution, build with:
python setup.py sdist
Build wheel distribution with:
python setup.py sdist bdist_wheel

Note that .whl files can be unzipped with unzip

The distributions are saved in the relative dist/ directory. Untar the source
distribution or unzip the .whl distribution to debug.

To include a non-python file in the wheel, the setting include_package_data
must be True in setup.py AND an include statement must be present in
MANIFEST.in

Wheel is the more modern form of egg: https://packaging.python.org/discussions/wheel-vs-egg/
> Wheel is currently considered the standard for built and binary packaging for Python.

Good slides here: https://blog.ionelmc.ro/presentations/packaging/#slide:1

(debugging, python, repl)
Pickle is fully specified to reconstruct types
Insert breakpoint in problematic process, pickle.dump to file
pickle.load in script with debugging aids

(mypy, readme)
http://calpaterson.com/mypy-hints.html

(python get size of object, memory, ram, size in bytes)
import sys
sys.getsizeof(full_dataset)
See ~/notes/random_notes_worth_keeping.txt search ‘memory usage’

(module not found)
Seeing: ModuleNotFoundError: No module named ‘.’
Fix: Add an init.py file into

(pip install git branch)
pip install git+https://github.com//@

(python and &, numpy, vectorized and)
https://stackoverflow.com/questions/22646463/and-boolean-vs-bitwise-why-difference-in-behavior-with-lists-vs-nump/22647006#22647006

(python2 to python3, migration, equivalence)
http://diveinto.org/python3/porting-code-to-python-3-with-2to3.html#next

(python mocking)
Most articles I’ve found on python mocking have been junk. Collection of easy to follow tips:
https://wesmckinney.com/blog/spying-with-python-mocks/
https://stackoverflow.com/a/25424012/143447

(environment variables, env variables, python, virtual env)
- In a virtual environment, the path to env is stored at $VIRTUAL_ENV
- To add additional env variables, add to the bottom of .env/bin/activate, and
also unset inside the deactivate function

(find where package is located, installed)
$ python -v
> import

(python debugging, pdb)
Install pdb++ https://pypi.org/project/pdbpp/
pip install pdbpp
Then at the debugger prompt, type ‘sticky’ for a much better debugging experience

(python debugging, pdb, multiline)
type ‘interact’ at the pdb prompt

(virtual environment, virtual env, python2)
$ /usr/local/Cellar/python@2/2.7.15_2/bin/pip2.7 install virtualenv
$ python2.7 /usr/local/lib/python2.7/site-packages/virtualenv.py .env_2_7
$ source .env_2_7/bin/activate

(virtualenv, system python, python 2.7)
sudo /usr/bin/easy_install-2.7 pip
/usr/bin/python2.7 -m pip install virtualenv
cd
/usr/bin/python2.7 -m virtualenv .env
source .env/bin/activate
pip install -r requirements.txt

(pdb, list, display, source, return to breakpoint)
list .

(ipython enter newline)
It’s tricky to add code to a previous code block after up-arrowing to it. To insert a newline, use:
ctlr+o+n or ctrl+q+j

(ipython read file, ipython load, ipython load script)
%load filename.py

(ipython, pdb)
Use pp to pretty print objects. Not built in anymore?

from pprint import pprint as pp

(pytest hides prints, show print, pytest)
pytest -s to show print statements when you run tests
very confusing that the default setting eats print statements
pytest -s -vv to show verbose difference

(pytest stop after first failure)
pytest -x

(pytest run single test)
pytest -k ‘test_the_thing’

(pytest run single file)
pytest path/to/file.py

(ipython reload, autoreload, repl)
%load_ext autoreload
%autoreload 2
Or add this to ~/.ipython/profile_default/ipython_config.py to automatically
reload on every session:
c.InteractiveShellApp.exec_lines = [‘%load_ext autoreload’, ‘%autoreload 2’]

(ipython profile, python profile)
%prun some_function

(virtual environment, python2)
$ virtualenv .env

(python create environment, python virtual environment, python3)
python -m venv .env
(add env to gitignore)
source .env/bin/activate
[optional]
see ~/notes/jupyter.txt
deactivate

(venv, virtual env, requirements, freeze)
pip freeze > requirements.txt

(python env, git)
Add .env to .gitignoremy_environment
python -m venv .env
source .env/bin/activate
pip install -r requirements.txt

(readme)
https://access.redhat.com/blogs/766093/posts/2592591

(ipython, matplotlib, plots, graph, math)
ipython –matplotlib
See ~/dev/snippets/python/matplotlib_experiment.py for an example

Within the plot window, use control to pan (drag along the x or y axis), or use
the zoom rect feature on the toolbar

(ipdb, single variable names)
Say you have a variable ‘n’. Use exclamation point in front:
ipdb> !n

(string match, methods matching, array select, list includes)
Find all methods that have pid in it:
[x for x in dir(os) if re.search(‘pid’, x, re.I)] # => re.I for ignore case

(getting help, help text, helptext, docstring)
> help(s.listen)
or
> print(s.listen.__doc__)

(print, lambda)
from future import print_function

Getting list of methods:
> dir(StringIO.StringIO)

Empty class implementation:
class Foo:
pass

Debugging

  Get class name:  
  ipdb> x.__class__  
  or  
  ipdb> x.__class__.__name__

Packages

  $ pip install numpy  
  $ pip install matplotlib  
  $ pip install ipdb   # => debugger

Autocompletion:

              $ pip install jedi  
~/.vim/bundle $ git clone --recursive https://github.com/davidhalter/jedi-vim.git

Interactive python:

$ python -i  
Better:  
$ ipython

System python installs to
pip installs things to: /usr/local/lib/python2.7/site-packages

Pretty printing:

from pprint import pprint as pp

Longer pretty printing:

import pprint  
params = {  
    'latitude': 37.775818,  
    'longitude': -122.418028,  
    'server_token': 'snip'  
}  
pp = pprint.PrettyPrinter(indent=4, width=1).pprint  
pp(params)

Pretty printing something that resembles a dictionary

pp(dict(response.headers))   # => note the width of 1 above is important

Patches
(python debugger, ipdb, list, patch):
In /home/lou/dragon/lib/python2.7/site-packages/IPython/core/debugger.py,
Changed context from 3 to 10:

def print_stack_entry(self,frame_lineno,prompt_prefix='\n-> ',  
                      context = 10):

Configuring iPython

Create default profile:  
$ ipython profile create  
$ ipython locate profile

iPython tricks

Run a script:  
> %run workbench.py

Insert an enter (newline) in repl:

ctrl+v + ctlr+j

Also try %edit

Exec from command line

python -c "print('sup')"

Sys

import sys  
sys.path

Entry detection

if __name__ == '__main__':  
    print("yes, I am main")

Trick to assign characters to fixed 0 to 25 slots

    slots = [0] * 26  
    c = 'x'  
    slot = ord(c) - ord('a')  
    slots[slot] += 1

Opposite of ord

    chr(97) # => 'a'

How to apply typing to generators

The arguments are:

Using a python generator as a coroutine (before async def)

from typing import Generator  

def fn() -> Generator[str, str, None]:  
    name: str = yield "I am now primed"  
    yield f"{name} world"  

g = fn()  

# Prime the generator. The argument needs to be None, otherwise this will raise:  
# TypeError: can't send non-None value to a just-started generator"  
print(g.send(None))  

# Now I can get the coroutine to spit out "hello world"  
print(g.send("hello"))

The above is much better written as:

import asyncio  

async def fn(name: str) -> str:  
    return f"{name} world"  

async def main():  
    result = await fn("hello")  
    print(result)  

asyncio.run(main())

Python gotcha, no implicit self

Bad:

class C:  
    def __init__(self):  
        _x = "x"

Good:

class C:  
    def __init__(self):  
        self._x = "x"

Python get context switch interval

sys.getswitchinterval()

Change it with:

sys.setswitchinterval(1E-2)

Python delegate pattern

Not sure how many python devs use this, but a translation of obj-c/swift delegate pattern could be:

class C:  
    def __init__(self, delegate):  
        self.delegate = delegate  
    def dowork(self):  
        self.delegate.did_finish()  

class D:  
    def did_finish(self):  
        print("work is done")  

c = C(D())  
c.dowork() # => "work is done"

Or, with typing:

from typing import Protocol  

class P(Protocol):  
    def did_finish(self) -> None:  
        ...  

class C:  
    def __init__(self, delegate: P):  
        self.delegate = delegate  
    def dowork(self) -> None:  
        self.delegate.did_finish()  

class D(P):  
    def did_finish(self) -> None:  
        print("work is done")  

c = C(D())  
c.dowork()

Python callback pattern

Using callbacks with pluggable closures

from collections.abc import Callable  

class C:  
    def __init__(self, did_finish: Callable[[], None]):  
        self.did_finish = did_finish  
    def dowork(self) -> None:  
        self.did_finish()  

def done() -> None:  
    print("work is done")  

c = C(done)  
c.dowork()

Or

c = C(lambda: print("work is done"))  
c.dowork()

Or

class X:  
    def __call__(self) -> None:  
        print("work is done")  

c = C(X())  
c.dowork()

Cartesian product

from itertools import product  
list(product(('a', 'b'), (1, 2)))  
# => [('a', 1), ('a', 2), ('b', 1), ('b', 2)]

Or

list(product(('a', 'b'), repeat=2))  
[('a', 'a'), ('a', 'b'), ('b', 'a'), ('b', 'b')]

Random numbers

Ints

from random import randint  
randint(0, 10) # => element in [0, 10]

Or

from random import randrange  
randrange(0, 10) # => element in (0, 10]

Or

from random import choice  
choice(range(10))

Use listcomps instead of maps

Say I want a list of random numbers in [1, 5]. I naturally think:

map(lambda _: randint(1,5), range(10))

but it’s more idiomatic to:

[randint(1,5) for _ in range(10)]

Or, better:

from random import choices  
choices(range(1, 6), k=10)

Or, by far the fastest for large N:

import numpy as np  
np.random.randint(0, 11, size=100000)

Python gotcha, mistaken use of class variable

Example from: https://docs.python.org/3/tutorial/classes.html

class Dog:  
    tricks = []             # mistaken use of a class variable  

    def __init__(self, name):  
        self.name = name  

    def add_trick(self, trick):  
        self.tricks.append(trick)  

d = Dog('Fido')  
e = Dog('Buddy')  
d.add_trick('roll over')  
e.add_trick('play dead')  
d.tricks  
['roll over', 'play dead']

Python gotcha, pop(0) is O(n)

[1,2,3].pop()  # => 3 in O(1)  
[1,2,3].pop(0) # => 1 in O(n)

Use deque instead (double ended queue):

from collections import deque  
d = deque([1,2,3])  
d.pop()     # => 3 in O(1)  
d.popleft() # => 1 in O(1)

deque also has automatic eviction:

d = deque([1,2,3], maxlen=3)  
d.append(4)  
d  
# => [2,3,4]

Python gotcha, methods that mutate return None

x = [1,2].append(3)  
x # => none

Python insort

from bisect import bisect, bisect_left  

x = [0,1,1,2]  
bisect(x, 1) # => 3  
bisect_left(x, 1) # => 1  

from bisect import insort  
insert(x, 1.5)  
x # => [0,1,1,1.5,2]

Get binary of a base 10 number

 bin(255)  
 # => '0b11111111'

How to create a generic type (for mypy)

from typing import TypeVar  
from typing import Generic  

T = TypeVar('T')  

class Node(Generic[T]):  
    def __init__(self, value: T):  
        self.value = value  

node1: Node[int] = Node('1')  # This will throw a mypy error  
node2: Node[int] = Node(1)    # mypy happy

How to create a generic fn

I don’t need to use Generic for this. It’s just:

from typing import TypeVar  
T = TypeVar("T")  
def rotate(list: list[T]) -> list[T]:  
    ...

How to read a file in chunks

In this example I’m using rb and then decoding as utf-8.
It’s not necessary to do this, but I want to remember how to read as binary:

with open('read_in_chunks.txt', 'rb') as f:  
while binary := f.read(2):  
    print(binary.decode('utf-8'))

How to read from an IO stream in chunks

from io import BytesIO  

with BytesIO(b'hello world') as stream:  
    print(stream.read(3))

Range in descending order

Note that just like an ascending range, the range is not inclusive of the second argument:

list(range(3, 0, -1))  # => [3, 2, 1]

Check truthiness, truthy

Use string formatters

a = f'{15:x}'                # => 'f'  
b = format(15, 'x')          # => 'f'  
c = '{:x}'.format(15)        # => 'f'  
d = '{num:x}'.format(num=15) # => 'f'

Benefit of __slots__

More efficient, sure. But what I really like is this:

...with the default __dict__, a misspelled variable name results in the  
creation of a new variable, but with __slots__ it raises in an  
AttributeError.   

Source: https://wiki.python.org/moin/UsingSlots

For example:

class C:  
    def __init__(self):  
        self._x = None  

    def run(self):  
        self.x = 1  

c = C()  
c.run() # => Nothing raised  
c.__dict__ # => contains `_x` and `x`

versus:

class C:  
    __slots__ = ('_x',)  

    def __init__(self):  
        self._x = None  

    def run(self):  
        self.x = 1  

c = C()  
c.run() # => Raises "AttributeError: 'C' object has no attribute 'x'"

Getters and setters

The setter syntax is a little funky, and relies on the existance of the getter.

 class C:  
     @property  
     def p(self):  
         return self._p  

     @p.setter  
     def p(self, value):  
         self._p = value

Frozenset usage

frozenset is helpful to make things hashable, but you can’t use it blindly.
Say I was trying to use [‘a’, ‘b’] as a key to a dict. I can’t do this:

{['a', 'b']: 'x'}  # => Raises "TypeError: unhashable type: 'list'"

but I could do this:

fs1 = frozenset(['a', 'b'])  
d = {fs1: 'x'}

but beware:

fs2 = frozenset(['b', 'a'])  
d[fs2] # => 'x'

How to put a set inside a set:

Wrong:

set(frozenset())   # => Not what I expect, but it makes sense. `frozenset()` is an iterable that the `set(...)` initializer iterates over to set the initial contents of the set.

Right:

set([frozenset()])

Flatten

There is nothing like ruby’s built in [1, [2,3]].flatten # => [1,2,3].

Using itertools with type inspection is the best I’ve found:

from itertools import chain  
l = [1,[2,3]]  
chain.from_iterable(x if isinstance(x, list) else [x] for x in l)

Regex extracting groups

x = re.match(r'a(b)', 'ab')  
x.group(0)  
# => 'ab'  
x.group(1)  
# => 'b'

Counters support + overload

c = Counter()  
c['a'] += 1  
d = Counter()  
d['a'] += 1  
d['b'] = 99  
c + d  
# => Counter({'b': 99, 'a': 2})