datapyground.compute.pagination

Support limiting or skipping data in a query plan.

Implements nodes whose purpose is to slice the data emitted by a query plan. Discarding the rows that are not part of the selected slice of data.

Classes

class datapyground.compute.pagination.PaginateNode(offset: int | None, length: int | None, child: QueryPlanNode)

Emit only one page of the received data.

Given a starting index and a length, only emit length rows after the starting index is reached.

For example if offset=1 and length=1 onlt the second row will be emitted:

0: skip because < offset
1: emit
2: skip because > length=1 and one row was already emitted.

Parameters:

• offset – From which row to take data, first row is 0.

• length – How many rows to take after offset was reached.

• child – the node from which to consume the rows.

batches() → Generator[RecordBatch, None, None]

Apply the pagination to the child node and emit the rows.

Consume rows from the child node skipping those until we reach offset. Once offset is reached start yielding rows until length is reached.

Subsequent rows are never consumed, so the child might not get exhausted. This requires special attention in resources management, because any resource open by the child might remain unclosed if the child waits for all the data to be consumed before closing it.