Advanced Python 3.13 on Windows: typed package design, pyproject/src layout, asyncio and cancellation, threads vs processes, robust subprocesses, pytest, ruff/mypy, profiling, C/PyO3 extensions, and Windows-specific failures. Use after the python-advanced basics when code must be typed, packaged, concurrent, testable, or fast.
Python Advanced
Reference interpreter: Python 3.13.5; use python or py -3.13. Use python-advanced for interpreter/venv basics; this skill is the design and failure-mode layer.
1. Project layout
project/
pyproject.toml
README.md
LICENSE
src/
package_name/
__init__.py
cli.py
core.py
tests/
test_core.py
.venv/
- Use a src layout; it prevents importing the working tree instead of the installed package.
- One project, one distribution name, one import package.
- Do not commit
.venv,__pycache__,dist, or.egg-info. - Pin the interpreter in
requires-python.
2. pyproject template
[build-system]
requires = ["hatchling>=1.27"]
build-backend = "hatchling.build"
[project]
name = "lygo-tool"
version = "0.1.0"
description = "Small typed tool"
readme = "README.md"
requires-python = ">=3.11"
license = "MIT"
dependencies = [
"httpx>=0.27",
"pydantic>=2.8",
]
[project.optional-dependencies]
dev = ["pytest>=8", "pytest-cov>=5", "ruff>=0.8", "mypy>=1.13"]
[project.scripts]
lygo-tool = "lygo_tool.cli:main"
[tool.ruff]
line-length = 100
target-version = "py311"
[tool.mypy]
strict = true
python_version = "3.11"
Build:
python -m pip install -U build
python -m build
The wheel goes to dist/; install with the same interpreter.
3. Typing that pays for itself
from __future__ import annotations
from collections.abc import Iterable, Sequence
from dataclasses import dataclass
from typing import Literal, Protocol, TypeAlias
Mode: TypeAlias = Literal["fast", "safe"]
@dataclass(frozen=True, slots=True)
class Job:
name: str
retries: int = 0
class Store(Protocol):
def put(self, key: str, value: bytes) -> None: ...
def get(self, key: str) -> bytes | None: ...
def total(xs: Iterable[int]) -> int:
return sum(xs)
def first(xs: Sequence[int]) -> int:
if not xs:
raise ValueError("empty")
return xs[0]
Rules:
Protocolfor interfaces you do not own.TypedDictfor JSON-like records;dataclassfor domain objects.Literalfor finite modes;Annotatedfor metadata.- Avoid
Any; isolate untyped third-party boundaries in one adapter. from __future__ import annotationsmakes annotations strings; good for startup and forward refs.- Run
mypy --strictorpyright; fix errors at boundaries first.
4. asyncio
import asyncio
async def fetch(i: int) -> int:
await asyncio.sleep(0.1)
return i * 2
async def main() -> None:
async with asyncio.TaskGroup() as tg:
tasks = [tg.create_task(fetch(i)) for i in range(5)]
print([t.result() for t in tasks])
asyncio.run(main())
asyncio.run()is the entrypoint; do not call it inside a running loop.- Use
TaskGroup(3.11+) for structured concurrency; it cancels siblings on failure. - Wrap network calls in
asyncio.timeout(10). - Use
asyncio.to_thread()for blocking libraries. - Do not call
time.sleep,requests, or CPU-heavy code directly in a coroutine. - Cancel tasks deliberately and await cancellation.
async withfor clients/sessions; close them.
5. Threads vs processes
| Workload | Choice |
|---|---|
| Network/file I/O, waiting | threads or asyncio |
| CPU-bound pure Python | ProcessPoolExecutor |
| C library releasing the GIL | threads may help |
| Shared mutable state | avoid; queue/message passing |
| Windows multiprocessing | guard with if __name__ == "__main__": |
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
with ThreadPoolExecutor(max_workers=8) as ex:
results = list(ex.map(fetch_url, urls))
with ProcessPoolExecutor() as ex:
results = list(ex.map(cpu_job, items))
- Windows uses spawn; the entry module is re-imported. Guard the entrypoint.
- Do not pass unpicklable objects to processes.
- Threads share memory; processes do not.
- The GIL limits pure-Python CPU parallelism; use processes or native code.
6. Robust subprocesses
import subprocess, sys
def run(cmd: list[str], timeout: int = 30) -> subprocess.CompletedProcess[str]:
return subprocess.run(
cmd,
capture_output=True,
text=True,
encoding="utf-8",
errors="replace",
timeout=timeout,
check=False,
shell=False,
)
Rules:
shell=Falseunless you need shell syntax; then never interpolate untrusted input.- Pass a list, not a string with spaces.
- Set
timeout; handlesubprocess.TimeoutExpired. check=TrueraisesCalledProcessError;check=Falsemeans inspectreturncode.- Use
encoding+errorson Windows; console code pages are not UTF-8. - Capture both streams; a full pipe can deadlock if the child writes a lot.
- For long-lived children, use
Popenwith explicit pipes/files and a reader thread.
7. Files, JSON, encoding
from pathlib import Path
import json
p = Path("data") / "config.json"
data = json.loads(p.read_text(encoding="utf-8"))
p.write_text(json.dumps(data, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
- Always specify
encoding="utf-8"for text. Pathover string concatenation.json.dumps(..., ensure_ascii=False)for Japanese/Unicode.- Atomic write: write a temp file in the same directory, then
os.replace. - Use
newline=""for CSV.
8. Testing and quality
python -m pytest -q
python -m pytest --cov=package_name --cov-report=term-missing
python -m ruff check .
python -m ruff format --check .
python -m mypy src
Pytest patterns:
import pytest
from package_name.core import normalize
def test_normalize():
assert normalize(" Hi ") == "hi"
def test_empty():
with pytest.raises(ValueError):
normalize(" ")
@pytest.fixture
def tmp_config(tmp_path):
p = tmp_path / "config.json"
p.write_text('{"ok": true}', encoding="utf-8")
return p
monkeypatchfor env vars and network.- Parametrize cases; do not duplicate test bodies.
- Test the public API, not private helpers.
- Use
tmp_path, not the repo directory. - Avoid sleeps; use events/fakes.
9. Profiling and performance
python -m cProfile -o out.prof .\script.py
python -m pstats out.prof
python -m timeit -s "x=list(range(1000))" "sum(x)"
cProfilefor call counts;py-spyfor live processes (if installed).tracemallocfor allocations.- Measure before changing; vectorize with NumPy only when the dependency is justified.
- Cache expensive pure functions with
functools.lru_cache, but not functions with side effects. - Use generators for large streams instead of building lists.
10. Native extensions
ctypesfor simple C ABI calls; defineargtypes/restype.cffifor a safer C API.PyO3+maturinfor a Rust extension:
python -m pip install maturin
maturin develop --release
- Match the extension ABI to the interpreter; do not mix 32/64-bit or debug/release runtimes.
- Build wheels per platform; pure-Python wheels are not enough for C extensions.
11. Troubleshooting table
| Symptom | Cause | Fix |
|---|---|---|
python opens Store | WindowsApps alias | use python or py |
ModuleNotFoundError after install | wrong interpreter/venv | run sys.executable; use -m pip |
pip installs elsewhere | pip tied to another Python | python -m pip, not bare pip |
asyncio.run() error | already inside a loop | make the caller async; do not nest |
| “coroutine was never awaited” | forgot await | await or schedule it |
| Task hangs | blocking call in loop | to_thread / async library / timeout |
TimeoutExpired leaves child | subprocess.run kills only direct child | use process group/job object; inspect |
Windows spawn re-runs main | no guard | if __name__ == "__main__": |
| UnicodeDecodeError | default cp1252 | encoding="utf-8", errors="replace" |
| JSON looks escaped | ensure_ascii=True | ensure_ascii=False |
| package imports from cwd | flat layout | src layout; install editable |
| slow tests | real sleeps/network | fixtures, fakes, monkeypatch |
mypy errors in third-party | no stubs | ignore_missing_imports for that module only |
| C extension import fails | ABI/arch mismatch | rebuild for this interpreter/arch |
| long path failure | Win32 limit | enable long paths; use \\?\; shorten root |
| stdlib import breaks | local file named subprocess.py/asyncio.py | rename the local module |
12. Templates
templates/pyproject.tomltemplates/cli_app.pytemplates/async_tasks.pytemplates/run_subprocess.pytemplates/test_example.py
Sources
- Python docs: asyncio, typing, pathlib, subprocess, venv, packaging.
- PyPA packaging tutorial.
- pytest, ruff, mypy.
- PyO3/maturin.
See references/SOURCES.md.
Related skills
- Downloads
- 278All time
- Security audit
- Pass
- Last updated
- 1mo ago
- Current version
- v1.0.0
- License
- MIT-0
