ohlc-toolkit
OHLC Toolkit
A Polars-native toolkit for reading, validating, and aggregating OHLC (Open, High, Low, Close) market data.
Each stage is a separate, composable step: fetch a published dataset, read and validate it, aggregate windows over it, add returns, apply a quality policy. Nothing sorts, fills, interpolates, or de-duplicates your data on the way past. When the data is not what you expected, you get a value describing that — or an exception — rather than a tidied frame and a log line.
Installation
pip install ohlc-toolkit
Python 3.11 or newer. Seven declared runtime dependencies: polars,
requests, loguru and orjson, which this package imports, plus
urllib3, idna and certifi, which it does not. Those three are
declared because requests carries them into every install and its own
ranges admit versions with published advisories against them; a floor
here is the only place a consumer’s resolve can be closed.
1.0 is a break
1.0 is a rewrite, not an upgrade. Every name 0.4 exported is gone:
| 0.4 | 1.0 |
|---|---|
read_ohlc_csv |
source.read_source_csv |
transform_ohlc |
windows.compute_windows |
DatasetDownloader |
snapshot.fetch_snapshot |
parse_timeframe, format_timeframe |
temporal.Duration |
validate_timeframe, validate_timeframe_format |
temporal.validate_window_duration, temporal.validate_cadence |
calculate_percentage_return |
returns.add_forward_returns |
There are no aliases, no deprecation warnings, and no migration path. The replacements are not renames — they take different arguments, return different types, and mean different things. A shim would have had to guess which of those differences you wanted, and guessing about a price series is how a wrong number reaches a model.
pandas is no longer a dependency of any kind.
0.4.x stays on PyPI and keeps working. If you depend on it, pin it:
pip install "ohlc-toolkit<1"
Quickstart
Against the real published BTC/USD minute history (ff137/bitstamp-btcusd-minute-data). The release is about 260 MB across three assets; it is fetched once and re-used on later runs. The toolkit logs each fetch and verification step as it runs; those log lines are omitted from the pasted output below.
from ohlc_toolkit.returns import (
ReturnMethod,
add_backward_returns,
add_forward_returns,
)
from ohlc_toolkit.snapshot import (
BITSTAMP_BTCUSD_1M_REPOSITORY,
SnapshotRelease,
fetch_snapshot,
read_snapshot_frame,
verify_snapshot_continuity,
)
from ohlc_toolkit.source import BITSTAMP_BTCUSD_1M
from ohlc_toolkit.windows import compute_windows
# Fetch the published minute history. Nothing lands until its SHA-256 and
# size match what the release's own manifest declared.
release = SnapshotRelease(
repository=BITSTAMP_BTCUSD_1M_REPOSITORY,
tag="bitstamp-btcusd-1m-2026-08",
)
result = fetch_snapshot(release, "data/bitstamp")
print("snapshot identity:", result.manifest_sha256)
# Read it under strict validation, then check the frame against what the
# manifest said it would be.
frame = read_snapshot_frame(result)
report = verify_snapshot_continuity(frame, result.manifest)
print("rows:", report.rows_checked, "| seam mismatches:", report.seam_mismatches)
# One-hour windows, emitted every fifteen minutes.
hourly = compute_windows(
frame,
BITSTAMP_BTCUSD_1M,
window="1h",
emit_every="15m",
materialization="skip_warmup",
)
print(hourly.tail(3))
# The causal four-hour log return, and the non-causal one beside the
# instant it becomes readable.
labelled = add_forward_returns(
add_backward_returns(
hourly, horizon="4h", cadence="15m", method=ReturnMethod.LOG
),
horizon="4h",
cadence="15m",
method=ReturnMethod.LOG,
)
print(
labelled.select(
"close_time",
"backward_return_log_4h",
"forward_return_log_4h",
"forward_return_log_4h_available_at",
).tail(3)
)
snapshot identity: 96e96cc32b313e4985a3d2d105e40ee528f8243bd2d1146a38f1b600f0bd3de1
rows: 7714079 | seam mismatches: ()
shape: (3, 10)
┌────────────┬───────────┬──────────┬──────────┬───┬───────────┬───────────┬───────────┬───────────┐
│ open_time ┆ close_tim ┆ open ┆ high ┆ … ┆ volume ┆ src_count ┆ coverage_ ┆ traded_se │
│ --- ┆ e ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ seconds ┆ conds │
│ i64 ┆ --- ┆ f64 ┆ f64 ┆ ┆ f64 ┆ u32 ┆ --- ┆ --- │
│ ┆ i64 ┆ ┆ ┆ ┆ ┆ ┆ i64 ┆ i64 │
╞════════════╪═══════════╪══════════╪══════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡
│ 1788215400 ┆ 178821900 ┆ 78734.71 ┆ 78742.28 ┆ … ┆ 37.490404 ┆ 60 ┆ 3600 ┆ 3600 │
│ ┆ 0 ┆ ┆ ┆ ┆ ┆ ┆ ┆ │
│ 1788216300 ┆ 178821990 ┆ 78642.61 ┆ 78642.61 ┆ … ┆ 34.778154 ┆ 60 ┆ 3600 ┆ 3600 │
│ ┆ 0 ┆ ┆ ┆ ┆ ┆ ┆ ┆ │
│ 1788217200 ┆ 178822080 ┆ 78572.31 ┆ 78576.84 ┆ … ┆ 36.012591 ┆ 60 ┆ 3600 ┆ 3600 │
│ ┆ 0 ┆ ┆ ┆ ┆ ┆ ┆ ┆ │
└────────────┴───────────┴──────────┴──────────┴───┴───────────┴───────────┴───────────┴───────────┘
shape: (3, 4)
┌────────────┬────────────────────────┬───────────────────────┬─────────────────────────────────┐
│ close_time ┆ backward_return_log_4h ┆ forward_return_log_4h ┆ forward_return_log_4h_availabl… │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ f64 ┆ f64 ┆ i64 │
╞════════════╪════════════════════════╪═══════════════════════╪═════════════════════════════════╡
│ 1788219000 ┆ -0.006804 ┆ null ┆ 1788233400 │
│ 1788219900 ┆ -0.005415 ┆ null ┆ 1788234300 │
│ 1788220800 ┆ -0.003928 ┆ null ┆ 1788235200 │
└────────────┴────────────────────────┴───────────────────────┴─────────────────────────────────┘
Those three forward returns are null because their counterparts lie past
the end of the data — and their available_at is stated anyway, because
when a value would arrive is a property of the horizon, not of whether it
happened to be found.
Timings from that run, on a 16-core desktop: 2.8 s to read and strictly validate all 7,714,079 rows, 0.8 s to aggregate the 514,268 windows, and under 0.1 s for both return columns. polars uses the whole thread pool.
What’s in it
import ohlc_toolkit reaches all seven subpackages. Names are not
flattened into the top level — spell them ohlc_toolkit.windows.X, or
import from the subpackage.
temporal — durations that carry their unit
Duration holds exact whole seconds and parses one compact grammar:
components in strictly descending order, each unit at most once, no
separators or signs.
from ohlc_toolkit.temporal import Duration
Duration.parse("1h15m").total_seconds == 4500
str(Duration(4500)) == "1h15m"
Anywhere a duration is accepted, a Duration or its string spelling both
work; a bare integer whose unit you have to infer does not. ConfigError,
DataValidationError and CoverageError are the package’s error
taxonomy, and every message that quotes untrusted input goes through one
bounded echo.
source — reads that report instead of repair
A SourceProfile states a source’s cadence, phase, timestamp column, and
raw column kinds. read_source_csv reads against one without sorting,
filling, dropping or de-duplicating anything, and validate_source_frame
returns findings as data:
import polars as pl
from ohlc_toolkit.source import (
BITSTAMP_BTCUSD_1M,
ValidationMode,
validate_source_frame,
)
frame = pl.DataFrame(
{
"timestamp": [1786924800, 1786924860, 1786924980], # 1786924920 missing
"open": [1.0, 2.0, 3.0],
"high": [1.0, 2.0, 3.0],
"low": [1.0, 2.0, 3.0],
"close": [1.0, 2.0, 3.0],
"volume": [1.0, 1.0, 1.0],
}
)
report = validate_source_frame(frame, BITSTAMP_BTCUSD_1M, mode=ValidationMode.REPORT)
for finding in report.findings:
print(finding)
Finding(kind=<FindingKind.GAP: 'gap'>, message='1 missing candle(s) expected in [1786924920, 1786924980)', count=1, sample_timestamps=(1786924920, 1786924980))
The eight finding kinds are schema, nulls, non-finite values, non-increasing
timestamps, overlapping intervals, off-phase timestamps, irregular intervals,
and gaps. A non-finite value is a NaN or an infinity in a declared price or
volume column; it is kept distinct from a null because a null is an absent
cell and a NaN is a present cell that is not a number, and neither is coerced
into the other. ValidationMode.STRICT raises SourceValidationError on any
of them instead.
windows — aggregation with an independent oracle
compute_windows emits one row per tick of an epoch-anchored emit grid,
each row aggregating the candles whose intervals fall inside the window
ending at that tick. Membership is decided by close time, never by
counting rows, so a gap in the source changes the window’s reported
coverage rather than silently changing what it spans. Ten columns come
back: open_time, close_time, OHLCV, src_count, coverage_seconds,
traded_seconds. The last two measure different things: coverage is how
much of the window the source had rows for, traded_seconds how much of
it those rows traded in. On a complete grid coverage is full everywhere
and a dead window looks exactly like a busy one without the second
number.
compute_reference_windows computes the same thing the plainest possible
way — quadratic, on purpose. On valid input it is the specification and
the engine is what you run; neither validates what it is handed, and on a
NaN or a null price the two do not agree, which both docstrings state. The suite holds the two to the same rows, in the same order,
with the same dtypes, across a synthetic matrix, property tests,
committed goldens, and a real 14-day slice — exactly equal on every
integer column and every selected price, and within a tolerance derived
from the oracle’s own fold on volume, the one column either
implementation sums. Both resolve schedules through the same module, so a
configuration one refuses is refused by the other in the same words.
apply_quality_policy is a separate, later step over the output, never
inside it. PASS_THROUGH records a deliberate no-op, FILTER drops rows
below a coverage threshold, GATE raises (or reports) without dropping.
Its threshold is exact rational arithmetic, not a float product.
annotate_windows is another later step: it joins a sparse sidecar of
half-open [start, end) intervals – an outage log, a maintenance
calendar – onto the output as two appended columns, the distinct flags
overlapping each window and the seconds of the window inside the union of
those intervals. Overlap is half-open on both sides, so an interval that
ends exactly at a window’s open touches nothing; flags are opaque strings
the transform reports and never interprets; no OHLCV or coverage value is
read or changed. read_annotations reads such a sidecar from CSV, typed
and checked, in file order and under an optional row cap, with the
Bitstamp provenance file’s column names as defaults; a broken sidecar
raises AnnotationValidationError, a misconfigured call ConfigError.
schedules — schedules that record what they are
Generators (log_spaced, metallic_recurrence, explicit) produce a
WindowSchedule; cadence rules (w_over_k, explicit_pairs) produce a
CadenceRule. There is no default schedule, coefficient, bound or
divisor anywhere: a caller states what it wants and gets back something
that records exactly what was asked for, named by a content hash over
that record.
from ohlc_toolkit.schedules import WindowSchedule, log_spaced
schedule = log_spaced(count=5, minimum="15m", maximum="1d", grain="15m")
print([str(window) for window in schedule.windows])
print(schedule.schedule_id)
print(WindowSchedule.from_dict(schedule.to_dict()) == schedule)
['15m', '45m', '2h30m', '7h45m', '1d']
0abc992caafb27a7ce8ba1ba5edca1118366b9ec84190744e2a641c0972d3e07
True
A lookback schedule is the count-valued twin: its members are period
counts, not durations, so the same 21 is twenty-one minutes on a 1m
frame and twenty-one weeks on a 1w one. It is a separate type rather than
a setting, because a lookback borrowed from a window schedule would be
silently coupled to it. The arithmetic is shared — the same recurrence,
the same log-spaced placement, the same quantize/bound/dedup rule — and
only the unit differs.
import math
from ohlc_toolkit.schedules import metallic_lookback
lookback = metallic_lookback(
coefficient=math.sqrt(math.e + math.sqrt(5)),
seed=1,
grain=1,
minimum=3,
maximum=21,
)
print(lookback.periods)
print(lookback.schedule_id)
(3, 8, 21)
2b7c4fa642bfd96d9727e10ea28438ffcbb7756b9a5809e8e57a56339cc9b0b4
The two identities cannot collide. A lookback records its members under
periods and a window schedule under windows, and 3 is not "3m",
so {1, 3, 8} and {1m, 3m, 8m} hash differently and each reader
refuses the other’s payload rather than reading it as its own.
A payload whose recorded schedule_id does not match its content is
refused rather than repaired, so a schedule read back from disk is the
one that was written.
returns — causal and non-causal, told apart
add_backward_returns relates a row’s close to the close exactly H
earlier; both were known at the row’s own close time, so it is a feature.
add_forward_returns relates the close H later to the row’s own, and
is not. The distinction is carried in the data twice — the forward_
prefix travels with the column name into any file or plot, and every
forward value has an available_at column stating the instant it may
first be read.
Counterparts are located by exact close-time equality, never by shifting
rows, so a gap yields a null instead of a wrong pairing. method is
required — ReturnMethod.SIMPLE or ReturnMethod.LOG — and is recorded
in the column name. Any value that is not a finite float comes back null.
snapshot — fail-closed fetching
A release is named by repository and immutable tag; asset URLs are
composed from that identity rather than discovered, so there is no API
call, no token, and no listing step to disagree with the manifest.
Downloads land on a temporary name and are renamed into place only after
size and SHA-256 both match what the manifest declared. An already
present file is reused only when its digest matches, and a mismatch is
refused rather than overwritten unless you pass
ExistingAssetPolicy.REPLACE.
fetch_snapshot also returns the snapshot identity — the digest over the
manifest bytes — and accepts it back as expected_manifest_sha256, which
refuses both a wholesale manifest swap and a release re-cut under the
same tag.
verify_snapshot_on_disk answers the other question. Fetching asks did
these bytes arrive intact; this asks are these still the bytes, which
is what a caller reading a snapshot it downloaded last week needs — a
directory verified in August is not thereby a directory verified now, and
nothing about a directory stops something else writing to it. It touches
no network, checks presence, size and digest for every declared asset,
and returns the same result type read_snapshot_frame consumes.
What it proves is that the directory is internally consistent: these bytes are the bytes this manifest describes. It does not prove the manifest is the one you meant — a different, self-consistent release verifies clean and is reported as itself, which is the right behaviour, since the identity it returns is what you record.
indicators — the phased lookback every indicator reads through
At each tick of the emit grid, a phased indicator consumes the L
non-overlapping windows of duration W ending at t, t − W, t − 2W,
and so on. phased_lookback is the fast path; phased_lookback_reference
is the brute-force oracle it is tested against.
The inputs are read from the window’s source-cadence materialization,
not from its own E-cadence frame. {t − kW} lies on the emit grid only
when E divides W, and it frequently does not: at W = 2h26m against
E = 3m, the emit frame holds none of the phases at all.
Lookups are exact equality on close_time — never a shift, a nearest
match or an as-of join. A window whose traded_seconds falls below the
recipe’s threshold is a null input, applied here whatever quality mode
wrote the frame, because report mode removes nothing. Any null among the
L inputs nulls the whole output row rather than leaving a list with a
hole in it, so no indicator downstream has to remember the rule. The
effective history L × W comes back with the result rather than being
left to callers to multiply.
A feature column’s name is derived from its identity, never passed beside
it: {indicator}_{family}{period}_w{window} — rsi_p14_w21m,
logvolratio_p14_w2h26m — with the window spelled the same way parquet
filenames and manifest fields spell it. What varies within a frame is in
the name; what is constant across the artifact is in the manifest, so the
two cannot disagree. The name parses back to the record it came from.
Every feature reports two counts. Effective history is L × W, the span
one value is computed from. Effective-N counts independent blocks —
whole non-overlapping windows in the range, divided by the lookback — and
is deliberately not a statistical effective sample size, which accounts
for autocorrelation and is smaller. They have different names because
reading one as the other overstates the evidence.
Development
git clone https://github.com/visikai/ohlc-toolkit.git
cd ohlc-toolkit
uv sync --all-groups
uv run pytest
uv run mypy .
uv run ruff check .
pytest deselects the network-marked tests by default; run them with
uv run pytest -m network to fetch and verify the real published
release. benchmarks/window_engine.py measures the window engine over a
full minute history against independently computed references.
Support
If you need any help or have any questions, please feel free to open an issue or contact me directly.
We hope this repo makes your life easier! If it does, please give us a star! ⭐