This is the last release before R Polars 2.0. It doesn't remove any existing API and keeps the behaviour of Polars 1.x, except where noted below. However, it comes with a number of deprecation warnings and suggestions that can be applied before moving to R Polars 2.0, which will break code or silently produce wrong results otherwise.
Note that some changes described below cannot warn, meaning that you should manually check your code when upgrading.
engine = "auto" used to select the in-memory engine for most queries in
Polars 1.x, but it will select the streaming engine in Polars 2.0.
The streaming engine may change the row order of the output, so if you rely on
it you should make this row order explicit with $sort() or an operation-specific
ordering option (e.g. with the argument .maintain_order in some functions).
Pass engine = "in-memory" to keep the behavior from Polars 1.x.
column_1 to column_0.schema is matched by position regardless of names, while
the file's column order is preserved. In Polars 2.0, schema for CSV files
with has_header = TRUE is matched by name, and schema field names must
match the header names. With has_header = FALSE, schema remains
matched by position and its length must match the input width. Empty string
names are valid; NA names are invalid.schema_overrides elements are matched by name, while
unnamed elements are treated as empty string names. In Polars 2.0, an
unnamed list is matched by position and must include one override for every
input column; a list with names is matched by name. Empty string names are
valid; NA names are invalid. The file's column order is preserved.UInt64 changes from Float64 to
Int128.is_in(), <expr>$list$contains(), and
<expr>$arr$contains() becomes an error. Cast the operands explicitly to
compatible numeric dtypes.Struct dtype reject mismatched fields.$std() and $ewm_std() on Duration columns become errors.pl$datetime() and pl$repeat_() change. Use
$alias() if your code depends on a particular output name.shuffle value for <dataframe>$sample(), <expr>$sample(),
and <expr>$list$sample() changes from FALSE in Polars 1.16 to NULL in
Polars 2.0, so output order is not guaranteed. Pass shuffle = FALSE to
preserve the Polars 1.16 ordering semantics.with_replacement = TRUE and shuffle = FALSE
may return values in a different order, even with a fixed seed.List and Array values remain outer nulls when converted with $to_struct().(3, 0). Selecting
no columns remains a (0, 0) result. In Polars 2.0, an empty pl$DataFrame()
has a fixed height of 0, so adding a longer column with $with_columns()
raises instead of adopting the new column's length.Unless noted otherwise, the deprecated forms below retain their Polars 1.x behavior in R Polars 1.16 but no longer retain that behavior in R Polars 2.0.
... interfaces of pl$col(), cs$by_name(), and cs$by_dtype() are
deprecated. Pass column names or data types as one vector or list instead, for
example pl$col(c("a", "b")), cs$by_name(c("a", "b")), or
cs$by_dtype(c(pl$Int32, pl$Float64)) (#1857).pl$col() as the right-hand operand of selector &, |, or
$xor() is deprecated. Use cs$by_name() for set operations on columns, or
<selector>$as_expr() for element-wise operations (#1861).... to pl$all(), pl$any(), pl$max(),
pl$min(), pl$sum(), and pl$cum_sum() is deprecated, including an empty
splice. A single positional scalar, vector, or list remains supported. Pass
multiple column names or data types in one names vector or list instead.pl$any(), pl$max(), pl$min(), pl$sum(), and
pl$cum_sum() are deprecated; use names = character(). Calling
pl$all() without arguments remains supported as an all-column selector.<expr>$str$contains_any() and
<expr>$str$replace_many() will be interpreted as column names in Polars 2.0.
Use pl$lit(...)$implode() for literal patterns or pl$col() for column patterns.
A shared literal vector can also be passed as list(c(...)) (#1855).replace_with to
<expr>$str$replace_many() is deprecated. Use list(c(...)) for a literal
scalar or vector replacement; Polars 2.0 requires the list form.<expr>$list$to_struct() and <series>$list$to_struct(), omitting fields
or passing a function as fields is deprecated. The n_field_strategy argument
is also deprecated in all forms. Pass an explicit character vector of field
names instead (#1863).upper_bound is deprecated for <expr>$list$to_struct(). Passing a function to
fields is deprecated for <expr>$arr$to_struct() and <series>$arr$to_struct()
(#1863).schema values passed to CSV readers with
has_header = TRUE are deprecated because schema fields are matched by
position in Polars 1.16 but by header name in Polars 2.0. Name all schema
fields using the corresponding header names. This warning does not apply to
headerless CSV files, empty schemas, fully named schemas, or empty string
names.NA names in CSV schema or schema_overrides are deprecated because they
become invalid in Polars 2.0. Replace them with the corresponding input
column names. With has_header = FALSE, unnamed schemas remain positional
and must follow the positional and length rules described above; empty string
names remain valid.infer_schema_files in CSV readers now warns because its default
changes from NULL to 10 in Polars 2.0. Pass infer_schema_files = 10 to
opt into the new default, or infer_schema_files = NULL to continue using all
files (#1870).raise_if_empty when reading a CSV with has_header = FALSE and a
non-NULL schema now warns because its default becomes FALSE in Polars
2.0. In other cases, the effective default remains TRUE without a warning.
Pass an explicit value to select the desired behavior.compression in $write_ipc() and $sink_ipc() now warns because
the default changes from "zstd" to "uncompressed" in Polars 2.0. Pass
compression = "zstd" to preserve the current behavior, or compression = "uncompressed"
to opt into the new default (#1856).cache argument of CSV and Arrow IPC File Format readers is deprecated.
Polars 2.0 streaming readers do not use the file cache, and there is no direct
replacement. Remove the argument (#1865).file_cache_ttl argument of CSV, Arrow IPC File Format,
and NDJSON readers is no longer translated into storage_options and can be
removed (#1854, #1865).n and fraction in <expr>$list$sample() uses fraction = 1 in
Polars 1.16 and will use n = 1 in Polars 2.0. Pass fraction = 1 to keep
the current behavior or n = 1 to opt into the new behavior.min_samples in <expr>$rolling_sum_by() uses 1 in Polars 1.16
and will use 0 in Polars 2.0. Pass an explicit value to select the desired
behavior.signed in <expr>$reinterpret() uses signed = TRUE in Polars
1.16; Polars 2.0 requires exactly one of signed or dtype. Pass
signed = TRUE to preserve the current behavior or specify a target dtype.<expr>$search_sorted(element = ...) or
<expr>$shift(fill_value = ...) are interpreted as columns in Polars 1.16
and as literals in Polars 2.0. Use pl$col(...) for a column or pl$lit(...)
for a literal.List
dtype is deprecated. Use pl$list(expr) to construct a list expression
instead.seed_1, seed_2, and seed_3 arguments of <expr>$hash() and
<dataframe>$hash_rows() are deprecated. Polars 2.0 removes them and retains
only seed. Hash values are not guaranteed to remain the same across Polars
versions (#1860).<expr>$agg_groups() is deprecated. Use the row-index aggregation pattern
documented in its help page instead (#1859).<series>$cat$is_local() and <series>$cat$uses_lexical_ordering() are
deprecated. Categoricals no longer have a local scope and are always ordered
lexically (#1859).<Enum>$union() is deprecated. Construct an Enum explicitly from the combined
categories instead (#1859).The following APIs were already deprecated before R Polars 1.16. They are repeated here because their Polars 2.0 migration path was previously incomplete, incorrect, or easy to miss.
<expr>$flatten() (deprecated in 1.9.0): use $list$explode(empty_as_null = FALSE, keep_nulls = FALSE)
for Polars 2.0-compatible behavior. To preserve the legacy behavior instead,
set both arguments to TRUE (#1866).<expr>$str$concat() (deprecated before 1.0.0): use $str$join("-") when
delimiter is omitted, or pass the same delimiter explicitly to $str$join()
(#1866).allow_missing_columns in Parquet readers (deprecated in 1.7.0): use
missing_columns = "insert" for TRUE or missing_columns = "raise"
for FALSE (#1866).<lazyframe>$profile() (deprecated in 1.14.0): this will be entirely removed
because Polars will use the streaming engine by default, which makes the
profiling information reported by this method misleading (#1866).<expr>$dt$with_time_unit() (deprecated before 1.0.0): cast to Int64, then
cast to the desired Datetime or Duration dtype and time unit (#1866).<expr>$cat$get_categories() (deprecated in 1.14.0): this will be entirely
removed. Use $unique() for the distinct values present in a Categorical
column, or dtype$categories for the fixed category list of an Enum.strict argument of pl$concat() (deprecated in 1.13.0): in Polars 2.0,
how = "horizontal" requires all frames to have the same height instead of
padding shorter frames with null. Use how = "horizontal_extend" to preserve
padding, or pass strict = TRUE in R Polars 1.16 to opt into the Polars 2.0
behavior early.pivot(aggregate_function = "len") now counts null values consistently with
pl$element()$len().collapse_joins = FALSE was previously ignored, while
no_optimization = TRUE left simplify_expression and fast_projection
enabled (#1864).This is an update that corresponds to Python Polars 1.44.1.
rechunk argument of pl$read_csv(), pl$scan_csv(),
pl$read_parquet(), pl$scan_parquet(), pl$read_ndjson(),
pl$scan_ndjson(), pl$read_ipc(), pl$scan_ipc() and
pl$read_ipc_stream() is deprecated. Call $rechunk() on the output
instead
(#1842, pola-rs/polars#28063).<expr>$rechunk() is deprecated. Rechunking within a query is not
well-defined; call $rechunk() on the DataFrame after collecting the
results instead
(#1842, pola-rs/polars#28692).<expr>$struct$rename_fields() now warns when the number of names passed
doesn't match the number of fields of the struct. This will become an error
in Polars 2.0. Use the new <expr>$struct$drop() to drop the trailing
fields first
(#1842, pola-rs/polars#28672).<expr>$struct$drop() to drop one or more fields from a struct
(#1842, pola-rs/polars#28666).<expr>$arr$dot() to compute the row-wise dot product of two Array
columns of numeric type
(#1842, pola-rs/polars#28504,
pola-rs/polars#28829).<lazyframe>$join_where() and <dataframe>$join_where() gain a how
argument, which accepts "inner" (default), "left" and "right"
(#1842, pola-rs/polars#28880).pl$read_csv() and pl$scan_csv() gain the experimental
infer_schema_files argument to control how many files are used to infer
the schema when reading several files at once
(#1842, pola-rs/polars#28809).pl$when()$then()$otherwise() could return incorrect results in some cases
involving broadcasting or a non-scalar null mask
(#1842, pola-rs/polars#28970,
pola-rs/polars#28946).pl$min_horizontal() and pl$max_horizontal() now ignore NaN values, as
pl$min() and pl$max() already do
(#1842, pola-rs/polars#28710).<expr>$rolling_*_by() now returns null for rows where the by column is
null instead of producing incorrect results
(#1842, pola-rs/polars#27367).<expr>$dt$add_business_days() now propagates null values in its input
correctly
(#1842, pola-rs/polars#28703).<expr>$sum() on a Decimal column now raises on overflow instead of
silently wrapping around
(#1842, pola-rs/polars#28688).<expr>$is_nan(), <expr>$is_not_nan(), <expr>$is_finite(), and
<expr>$is_infinite() now return null for null entries
(#1842, pola-rs/polars#28883).pl$lit(x, dtype = pl$Unknown) now behaves identically to pl$lit(x)
(#1842, pola-rs/polars#28830).null
values in Map arrays, buffer offsets for String and Binary, and
nested LargeList values
(#1842, pola-rs/polars#28680,
pola-rs/polars#28662,
pola-rs/polars#28632).mirai integration no longer hangs indefinitely when a worker fails to
deserialize a Polars object (for example because the worker loaded a
different Polars build); the error is now reported immediately (#1842).This is an update that corresponds to Python Polars 1.43.2.
missing_utf8_is_empty_string argument of pl$read_csv() and
pl$scan_csv() is deprecated in favor of empty_string_is_null, whose
meaning is inverted
(pola-rs/polars#28173).<expr>$cat$get_categories() is deprecated. To get the distinct values
present in a Categorical column, use $unique(). For the fixed category
list of an Enum, use its dtype$categories
(pola-rs/polars#28299).<series>$cat$to_local() is deprecated; Categoricals no longer have a local
scope
(pola-rs/polars#28299).<lazyframe>$profile() is deprecated. Starting with Polars 2.0,
engine = "auto" will use the streaming engine by default, and the
profiling information from this method would be misleading
(pola-rs/polars#28275).pl$list() to gather several elements into a list column. Contrary to
pl$concat_list(), pl$list() doesn't merge elements into a single list,
i.e. merging a List(Float64) and a String will give
List(List(Float64), String). (#1825)<expr>$cat$to() and <expr>$cat$physical() to convert between a Categorical
or Enum column and its physical representation (#1826).This is an update that corresponds to Python Polars 1.42.1.
empty_as_null argument in $explode() will
change from TRUE to FALSE in Polars 2.0. Affected functions:
<expr>$explode(), <expr>$list$explode(), <expr>$arr$explode(),
<lazyframe>$explode(), <dataframe>$explode(). A deprecation warning is
now emitted when empty_as_null is not explicitly set
(pola-rs/polars#28040, #1804).strict argument of pl$concat() is deprecated. Use how = "horizontal_extend" (pad with null) to keep the current behavior.
how = "horizontal" will require equal heights by default in the next
breaking release
(pola-rs/polars#27965, #1812).polars_warning class and polars_deprecation_warning class (#1812).pl$concat() gains how = "horizontal_extend", which stacks DataFrames
horizontally and pads shorter frames with null
(pola-rs/polars#27965, #1812).This is an update that corresponds to Python Polars 1.41.2.
<expr>$gather() gains an argument null_on_oob (#1789).This is an update that corresponds to Python Polars 1.40.1.
pl$row_index(), a shortcut for pl$int_range(pl$len()) (#1770).polars_code_completion_activate() and polars_code_completion_deactivate() to
enable Polars-specific code completion. This only works in RStudio for now (#1768).<expr>$arr$any(), <expr>$arr$all(), <expr>$list$any(), and <expr>$list$all()
gain an argument ignore_nulls (#1778).<dataframe>$merge_sorted() and <lazyframe>$merge_sorted() gain an argument
maintain_order (#1778).rlang dependency to be >= 1.2.0.This is an update that corresponds to Python Polars 1.39.3.
<expr>$implode() gains the maintain_order argument to control whether
the order of elements within each group is preserved (#1751).<expr>$set_sorted() gains the nulls_last argument to specify the
position of null values in the sorted order (#1751).<expr>$dt$add_business_days() now accepts a Polars expression for
the holidays argument, in addition to R Date vectors (#1751).$pivot() gains the column_naming argument to control how pivoted
column names are constructed (#1751).<expr>$truncate() (#1755).<expr>$round() can now take mode = "to_zero" (#1755).pl$scan_csv() and pl$read_csv() gain the missing_columns argument to
control behavior when some CSV files have missing columns compared to the
expected schema (#1754).$unpivot() now properly checks for duplicate column names when using
custom value_name or variable_name
(pola-rs/polars#26606, #1751).This is an update that corresponds to Python Polars 1.38.1.
retries argument in scan/read and sink/write functions is deprecated (#1726).
Use max_retries in storage_options instead.file_cache_ttl argument in pl$scan_csv(), pl$scan_ipc(), pl$scan_ndjson(),
and their read_* counterparts is deprecated (#1726). The previous
recommendation to use file_cache_ttl in storage_options has since been
superseded: the file cache is no longer supported and has no direct
replacement in Polars 2.0.<expr>$flatten() is deprecated. Use
<expr>$list$explode(empty_as_null = FALSE, keep_nulls = FALSE) for
Polars 2.0-compatible behavior, or set both arguments to TRUE to preserve
the legacy behavior (#1726).cs$by_name() gains the expand_patterns argument. When set to TRUE, regex patterns
(^...$) and wildcards (*) in column names are expanded
(pola-rs/polars#26437, #1726).<expr>$bin$get() to extract a specific byte from a binary value (#1731).<expr>$str$split() has two new arguments literal and strict (#1730).pl$scan_lines() and pl$read_lines() to read one or several files into
a single column (#1732).$sink_ndjson(), $write_ndjson(), $sink_csv(), and $write_csv() can now export compressed files (#1733, #1735).<expr>$rolling_rank_by() now requires the closed argument to be "right" or "both".
Previously, "left" and "none" were silently accepted but could produce incorrect results
(pola-rs/polars#26287, #1726).per_partition_sort_by argument of the deprecated partition classes
(pl$PartitionByKey(), pl$PartitionMaxSize(), pl$PartitionParted()) has been removed.
This feature was removed from upstream Polars
(pola-rs/polars#26130, #1726).This is an update that corresponds to Python Polars 1.37.1.
pl$PartitionByKey(), pl$PartitionMaxSize(), or pl$PartitionParted())
are deprecated in favor of the new experimental partition by class, constructed with
pl$PartitionBy() (#1716).nanoarrow::as_nanoarrow_array_stream(<lazyframe>) to export a LazyFrame via the Arrow C stream interface
(pola-rs/polars#25918, #1709).<expr>$min_by() and <expr>$max_by() to get the value of a column ordered by another column (#1715).This is an update that corresponds to Python Polars 1.36.1, which includes significant internal changes.
allow_missing_columns argument of pl$scan_parquet() and pl$read_parquet() is deprecated (#1662).
Use the new missing_columns argument instead.<expr>$index_of() now strictly checks types. Therefore, operations like $index_of(NA)
that worked previously will error if NA is converted to Bool type and the target column cannot be converted from Bool type.
Use NULL or vctrs::unspecified(1) instead of NA for such operations (#1662).Float16 (#1673, pola-rs/polars#25185).<lazyframe>$pivot() and <dataframe>$pivot()'s new argument on_columns
(#1662, pola-rs/polars#25016).<lazyframe>$unique() and <dataframe>$unique()' now allow polars expressions in ...
(#1662, pola-rs/polars#25099).<expr>$item() to strictly extract a single value from an expression (#1652).<expr>$arr$eval() to run any Polars expression on all subarrays of an Array column (#1653).<expr>$name$replace() to replace expression names using regular expressions (#1654).<expr>$dt$days_in_month() (#1659).<expr>$rolling_rank() and <expr>$rolling_rank_by() (#1656).<expr>$arr$agg() and <expr>$list$agg(), similar to their $eval() counterparts
but automatically explode the column if all elements return a scalar (#1655).<expr>$rolling_kurtosis() (#1665).pl$explain_all() to show a single optimized query plan from several input LazyFrames (#1666).<expr>$bin$reinterpret() (#1664).<expr>$mode() gains the maintain_order argument (#1662).empty_as_null and keep_nulls (#1662).
<expr>$explode()<expr>$arr$explode()<expr>$list$explode()<lazyframe>$explode()is_list_of_polars_expr() (#1662).<groupby>$having(), <rolling_groupby>$having(), <dynamic_groupby>$having()
(and their lazy implementations) to filter groups before applying aggregations (#1671).<expr>$str$extract_many(), <expr>$str$find_many(), and <expr>$str$replace_many()
gain the leftmost argument (#1673, pola-rs/polars#25398).pl$concat() gains the strict argument (#1673, pola-rs/polars#25452).<expr>$list$eval() now properly errors (as documented) if the input is not a Polars expression (#1655).<lazyframe>$sink_parquet() and <dataframe>$write_parquet()'s compression argument
should not be "lzo", which does not work correctly
(#1673, pola-rs/polars#25522).pl$PartitionByKey(), pl$PartitionMaxSize(), or pl$PartitionParted())
is rewritten as S7 class (#1662).pl$element() is rewritten in Rust (#1662, pola-rs/polars#24885).This is an update that corresponds to Python Polars 1.35.2.
As of this version, this package depends on {S7}.
The newly added QueryOptFlags object is an S7 object.
The following arguments of certain LazyFrame methods, which were previously used for query optimization,
are deprecated in favor of the new optimizations argument (#1635).
Some arguments that were intended for internal use have been removed without deprecation.
type_coercionpredicate_pushdownprojection_pushdownsimplify_expressionslice_pushdowncomm_subplan_elimcomm_subexpr_elimcluster_with_columnsno_optimization_type_check (removed)_check_order (removed)_eager (removed)Functions affected are those that gained the optimizations argument.
See the next new features section for details.
For the experimental <lazyframe>$lazy_sink_* methods,
the above arguments and the collapse_joins argument (deprecated as of polars 1.4.0) are removed.
<lazyframe>$to_dot()s ignored ... (dots) argument is deprecated (#1635).
In future versions, an error will be raised if dots are not empty.
optimizations argument
taking a QueryOptFlags object (#1633, #1634, #1635).
<lazyframe>$collect()<lazyframe>$explain()<lazyframe>$profile()<lazyframe>$to_dot()<lazyframe>$sink_batches()<lazyframe>$sink_csv()<lazyframe>$sink_ipc()<lazyframe>$sink_parquet()<lazyframe>$sink_ndjson()as_polars_df(<lazyframe>)engine argument (#1635).
<lazyframe>$explain()<lazyframe>$profile()pl$collect_all() to efficiently collect a list of LazyFrames (#1598, #1635).<lazyframe>$remove() and <dataframe>$remove() as a complement to
$filter() (#1632).<expr>$is_close() (#1637).<group_by>$len() and <lazy_group_by>$len() (#1638).This is an update that corresponds to Python Polars 1.35.1.
<lazyframe>$unnest() and <dataframe>$unnest() gain the separator argument.polars_info() shows the corresponding Python Polars version and the supported
Polars CompatLevel (#1591).{reticulate} integration.
Series, DataFrame, and LazyFrame can be exchanged between R Polars and Python Polars
using reticulate::r_to_py() and as_polars_* functions (#1607).
Conversion of Series and DataFrame relies on {nanoarrow}
(apache/arrow-nanoarrow#817).<expr>$pct_change() preserves null values
(#1603, pola-rs/polars#24952).This is an update that corresponds to Python Polars 1.34.0.
pl$Decimal()'s arguments should not be NULL (#1553).
Since the automatic inference feature has been removed, precision and scale must always be specified.collapse_joins argument of some LazyFrame methods is deprecated (#1553).
Use predicate_pushdown instead.<lazyframe>$sink_batches() to apply a function to each reading batch (#1557).<lazyframe>$lazy_sink_* methods, variants of <lazyframe>$sink_* methods, that return a LazyFrame
instead of executing immediately (#1562).<lazyframe>$sink_* methods gain the engine argument (#1562).compat_level or polars_compat_level arguments, which specifies the compatibility level with Apache Arrow format,
can be overridden by the polars.compat_level option if not specified (#1565).
This can be useful especially when overriding the behavior of nanoarrow::as_nanoarrow_array_stream() used in external packages' functions.<dataframe>$write_ipc_stream() to write Arrow IPC stream format (.arrows file) (#1570).<expr>$dt$total_* methods gain the new fractional argument
(pola-rs/polars#24598, #1573).polars_envvars() to show all environment variables available
in polars, for instance to customize the number of rows displayed when printing
a DataFrame. This was available in polars < 1.0.0 but not in the rewritten
version until now (#1580).<expr>$reshape() does not allow -1 in dimensions other than the first dimension
(pola-rs/polars#24591, #1564).This is an update that corresponds to Python Polars 1.33.1.
Only updates Rust dependencies (#1540, #1542).
This is an update that corresponds to Python Polars 1.33.0, which includes significant internal changes.
Some entire expr methods or arguments of expr methods have been deprecated. They still work the same way on series (pola-rs/polars#24027, #1507, #1531, #1534).
As a workaround of these deprecations, the new <dataframe>$map_columns() function can be used
to apply functions for Series to each column (#1533).
df <- pl$DataFrame(n = list(c(0, 1), c(0, 1, 2)))
# `df$with_columns(pl$col("n")$list$to_struct(n_field_strategy = "max_width"))` no longer works identically.
df$map_columns("n", \(s) s$list$to_struct(n_field_strategy = "max_width"))
#> shape: (2, 1)
#> ┌────────────────┐
#> │ n │
#> │ --- │
#> │ struct[3] │
#> ╞════════════════╡
#> │ {0.0,1.0,null} │
#> │ {0.0,1.0,2.0} │
#> └────────────────┘
<expr>$shrink_dtype() (#1507).<expr>$list$to_struct()'s first argument n_field_strategy (#1507).<expr>$list$to_struct()'s upper_bound argument must be specified when used with
non-vector fields specification (#1534).<expr>$str$json_decode()'s first argument dtype must be specified (#1507).<expr>$str$json_decode()'s infer_schema_length (#1507).<expr>$str$to_datetime()'s first argument format or time_zone must be specified
for time zone aware datetime (#1507).
<expr>$str$strptime(), if the string to be parsed contains a time zone,
the time zone must be specified.<expr>$str$to_decimal()'s inference_length (#1507).<expr>$str$to_decimal()'s new scale argument must be specified (#1507).<expr>$log(), argument base can now take an expression (#1523).<dataframe>$map_columns() (#1533).<expr>$index_of() (#1519).mkdir in <DataFrame>$write_parquet() (#1525).This is an update that corresponds to Python Polars 1.32.3.
Only updates Rust dependencies (#1502).
This is an update that corresponds to Python Polars 1.32.2.
<expr>$arr$len() (#1478).<expr>$dt$millennium() (#1485).<expr>$dt$replace() (#1491).<expr>$meta$is_literal() (#1483).<expr>$str$escape_regex() (#1486).<expr>$str$find_many() (#1484).<expr>$str$normalize() (#1479).This is an update that corresponds to Python Polars 1.32.0, which includes significant internal changes.
pl$Categorical()'s first argument ordering is deprecated
(pola-rs/polars#23016, #1452, #1468).
In this version, global categories are always used, and the behavior matches the previous ordering = "lexical".as_polars_expr()'s argument structify.POLARS_AUTO_STRUCTIFY environment variable to 1.<lazyframe>$unique() and <dataframe>$unique()'s first argument is replaced from subset to ...
(dynamic dots) (#1463).
Because of this change, it is also deprecated to pass the following objects as the first argument of these functions:
NULL: Use cs$all() or pass nothing to select all columns.
If you want to pass column selections as a variable, you can use the %||% (base R >= 4.4.0, or {rlang}'s op-null-default)
operator to replace NULL with cs$all():
subset <- nullable_selection %||% cs$all()
lf$unique(!!!c(subset))
A list of column names or selectors: Use !!! to expand the list to the dynamic-dots.
subset <- list("col1", "col2")
lf$unique(!!!c(subset))
New experimental polars selectors have been added (pola-rs/polars#23351, #1452).
cs$empty() to avoid matching any column.cs$enum() for Enum data types.cs$list() for List data types.cs$array() for Array data types.cs$struct() for Struct data types.cs$nested() for List, Array, or Struct data types.polars selectors can now be used in place of column names in more locations (#1452).
... (dynamic dots) of these functions.
<dataframe>$to_dummies()<dataframe>$partition_by()<lazyframe>$drop_nulls() and <dataframe>$drop_nulls()<lazyframe>$drop_nans() and <dataframe>$drop_nans()<lazyframe>$unique() and <dataframe>$unique()<lazyframe>$drop() and <dataframe>$drop()<lazyframe>$explode() and <dataframe>$explode()<lazyframe>$unnest() and <dataframe>$unnest()<dataframe>$pivot()'s on, index, and values.<lazyframe>$join() and <dataframe>$join()'s on and index.This change also fixes the odd behavior of some functions that had the semantics
of selecting all columns by default
($drop_nulls(), $drop_nans(), and $unique() of lazyframe or dataframe).
In the previous version, passing c() (NULL) would result in strange behavior
doesn't match either of "select nothing" or "select all columns".
And, expanding an empty vector with !!! would select all columns.
### OLD
df <- pl$DataFrame(a = c(NA, TRUE), b = 1:2)
df$drop_nulls(c())
#> shape: (0, 2)
#> ┌──────┬─────┐
#> │ a ┆ b │
#> │ --- ┆ --- │
#> │ bool ┆ i32 │
#> ╞══════╪═════╡
#> └──────┴─────┘
df$drop_nulls(!!!c())
#> shape: (1, 2)
#> ┌──────┬─────┐
#> │ a ┆ b │
#> │ --- ┆ --- │
#> │ bool ┆ i32 │
#> ╞══════╪═════╡
#> │ true ┆ 2 │
#> └──────┴─────┘
In the new version, passing c() (NULL) will cause an error,
and expanding an empty vector with !!! will select no columns.
### NEW
df <- pl$DataFrame(a = c(NA, TRUE), b = 1:2)
df$drop_nulls(c())
#> Error:
#> ! Evaluation failed in `$drop_nulls()`.
#> Caused by error:
#> ! Evaluation failed in `$drop_nulls()`.
#> Caused by error:
#> ! `...` can only contain single strings or polars selectors.
df$drop_nulls(!!!c())
#> shape: (2, 2)
#> ┌──────┬─────┐
#> │ a ┆ b │
#> │ --- ┆ --- │
#> │ bool ┆ i32 │
#> ╞══════╪═════╡
#> │ null ┆ 1 │
#> │ true ┆ 2 │
#> └──────┴─────┘
pl$nth() gains the strict argument (#1452).
<expr>$str$pad_end() and <expr>$str$pad_start()'s length argument accepts a polars expression (#1452).
<expr>$str$to_integer() gains the dtype argument to specify the output data type (#1452).
<lazyframe>$sink_csv() and <dataframe>$write_csv() gains the decimal_commna argument (#1452).
This is a small patch release that includes minor improvements discovered right after the 1.0.0 release.
data.frame-like objects,
the [[ operator can now be used to extract a column from a polars DataFrame as a Series (#1442).This is a completely rewritten new version of the polars R package. It improves the internal structure of the package and catches up with Python Polars' API. This version of R Polars matches Python Polars 1.31.0.
Therefore it contains many breaking changes compared to the previous R Polars implementation. Some of those breaking changes are explained below, but many others are due to modifications of function names, argument names, or argument positions. There are too many to list here, so you should refer to the Python Polars API docs.
For compatibility, the old version (polars 0.22.4) is now available as a separate package named "polars0". We can install both polars and polars0 at the same time. See the polars0 documentation for details.
The class names of polars objects have changed:
RPolarsLazyFrame -> polars_lazy_frameRPolarsDataFrame -> polars_data_frameRPolarsSeries -> polars_seriesRPolarsExpr -> polars_exprConversion from unknown classes to Polars objects now fails. Developers can
specify how those objects should be handled by polars by creating a method
for as_polars_series.my_class.
### OLD
a <- 1
class(a) <- "foo"
as_polars_series(a)
#> polars Series: shape: (1,)
#> Series: '' [f64]
#> [
#> 1.0
#> ]
### NEW
a <- 1
class(a) <- "foo"
as_polars_series(a)
#> Error:
#> a <foo> object can't be converted to a polars Series.
#> Run `rlang::last_trace()` to see where the error occurred.
Conversion from polars objects to R vectors has been revamped: <series>$to_r(),
<series>$to_list() and <dataframe>$to_data_frame() no longer exist. Instead, you must use
as.data.frame(<dataframe>), as.list(<dataframe>), as.vector(<series>), or <series>$to_r_vector().
as.vector(<series>) will remove attributes that might be useful, for instance to
convert Int64 values using the bit64 package or to convert Time values using
the hms package. It is therefore recommended to use <series>$to_r_vector() instead for usual conversions.
s_time <- as_polars_series(c("00:00", "12:00"))$str$to_time()
as.vector(s_time)
#> ℹ `as.vector()` on a Polars Series of type time may drop some useful attributes.
#> ℹ Use `$to_r_vector()` instead for finer control of the conversion from Polars to R.
#> [1] 0 43200
s_time$to_r_vector()
#> 00:00:00
#> 12:00:00
In general, polars now uses dots (...) in two scenarios:
to pass an unlimited number of inputs (for instance in <lazyframe>$select(), <lazyframe>$cast(),
or <lazyframe>$group_by()), using dynamic-dots.
For example, if you used to pass a vector of column names or a list of
expressions, you now need to expand it with !!!:
### OLD
dat <- as_polars_df(head(mtcars, 3))
my_exprs <- list(pl$col("drat") + 1, "mpg", "cyl")
dat$select(my_exprs)
#> shape: (6, 3)
#> ┌──────┬──────┬─────┐
#> │ drat ┆ mpg ┆ cyl │
#> │ --- ┆ --- ┆ --- │
#> │ f64 ┆ f64 ┆ f64 │
#> ╞══════╪══════╪═════╡
#> │ 4.9 ┆ 21.0 ┆ 6.0 │
#> │ 4.9 ┆ 21.0 ┆ 6.0 │
#> │ 4.85 ┆ 22.8 ┆ 4.0 │
#> └──────┴──────┴─────┘
### NEW
dat <- as_polars_df(head(mtcars, 3))
my_exprs <- list(pl$col("drat") + 1, "mpg", "cyl")
dat$select(!!!my_exprs)
#> shape: (3, 3)
#> ┌──────┬──────┬─────┐
#> │ drat ┆ mpg ┆ cyl │
#> │ --- ┆ --- ┆ --- │
#> │ f64 ┆ f64 ┆ f64 │
#> ╞══════╪══════╪═════╡
#> │ 4.9 ┆ 21.0 ┆ 6.0 │
#> │ 4.9 ┆ 21.0 ┆ 6.0 │
#> │ 4.85 ┆ 22.8 ┆ 4.0 │
#> └──────┴──────┴─────┘
This also affects pl$col():
### OLD
pl$col(c("foo", "bar"), "baz")
#> polars Expr: cols(["foo", "bar", "baz"])
### NEW
pl$col(c("foo", "bar"), "baz")
#> Error in `pl$col()`:
#> ! Evaluation failed in `$col()`.
#> Caused by error in `pl$col()`:
#> ! Invalid input for `pl$col()`.
#> • `pl$col()` accepts either single strings or Polars data types.
pl$col(!!!c("foo", "bar"), "baz")
#> cols(["foo", "bar", "baz"])
Another important change in functions that accept dynamic dots is that
additional arguments are prefixed with .. For example, <lazyframe>$group_by() now
takes dynamic dots, meaning that the argument maintain_order is renamed
.maintain_order (for now, we add a warning if we detect an argument named
maintain_order in the dots).
to force some arguments to be named. We now throw an error if an argument is not named while it should be, for example:
df <- pl$DataFrame(a = 1:4)
df$with_columns(pl$col("a")$shift(1, 3))
#> Error in `df$with_columns()`:
#> ! Evaluation failed in `$with_columns()`.
#> Caused by error:
#> ! Evaluation failed in `$with_columns()`.
#> Caused by error:
#> ! Evaluation failed in `$shift()`.
#> Caused by error:
#> ! `...` must be empty.
#> ✖ Problematic argument:
#> • ..1 = 3
#> ℹ Did you forget to name an argument?
df$with_columns(pl$col("a")$shift(1, fill_value = 3))
#> shape: (4, 1)
#> ┌─────┐
#> │ a │
#> │ --- │
#> │ f64 │
#> ╞═════╡
#> │ 3.0 │
#> │ 1.0 │
#> │ 2.0 │
#> │ 3.0 │
#> └─────┘
Related to the extended use of dynamic dots, pl$DataFrame() and
pl$LazyFrame() more accurately convert input to the correct datatype, for
instance when the input is an R data.frame:
### OLD
pl$DataFrame(data.frame(x = 1, y = "a"))
#> shape: (1, 2)
#> ┌─────┬─────┐
#> │ x ┆ y │
#> │ --- ┆ --- │
#> │ f64 ┆ str │
#> ╞═════╪═════╡
#> │ 1.0 ┆ a │
#> └─────┴─────┘
### NEW
pl$DataFrame(data.frame(x = 1, y = "a"))
#> shape: (1, 1)
#> ┌───────────┐
#> │ │
#> │ --- │
#> │ struct[2] │
#> ╞═══════════╡
#> │ {1.0,"a"} │
#> └───────────┘
pl$DataFrame(!!!data.frame(x = 1, y = "a"))
#> shape: (1, 2)
#> ┌─────┬─────┐
#> │ x ┆ y │
#> │ --- ┆ --- │
#> │ f64 ┆ str │
#> ╞═════╪═════╡
#> │ 1.0 ┆ a │
#> └─────┴─────┘
Use as_polars_df() and as_polars_lf() to convert existing R data.frames
to their polars equivalents.
The class names PTime and rpolars_raw_list (used to handle time and binary
variables) are removed. One should use the classes provided in packages
hms and blob instead.
### OLD
r_df <- tibble::tibble(
time = hms::as_hms(c("12:00:00", NA, "14:00:00")),
binary = blob::as_blob(c(1L, NA, 2L)),
)
# R to Polars
pl_df <- as_polars_df(r_df)
pl_df
#> shape: (3, 2)
#> ┌─────────┬──────────────┐
#> │ time ┆ binary │
#> │ --- ┆ --- │
#> │ f64 ┆ list[binary] │
#> ╞═════════╪══════════════╡
#> │ 43200.0 ┆ [b"\x01"] │
#> │ null ┆ [] │
#> │ 50400.0 ┆ [b"\x02"] │
#> └─────────┴──────────────┘
# Polars to R
tibble::as_tibble(pl_df)
#> # A tibble: 3 × 2
#> time binary
#> <dbl> <list>
#> 1 43200 <rplrs_r_ [1]>
#> 2 NA <rplrs_r_ [0]>
#> 3 50400 <rplrs_r_ [1]>
### NEW
r_df <- tibble::tibble(
time = hms::as_hms(c("12:00:00", NA, "14:00:00")),
binary = blob::as_blob(c(1L, NA, 2L)),
)
## R to Polars
pl_df <- as_polars_df(r_df)
pl_df
#> shape: (3, 2)
#> ┌──────────┬─────────┐
#> │ time ┆ binary │
#> │ --- ┆ --- │
#> │ time ┆ binary │
#> ╞══════════╪═════════╡
#> │ 12:00:00 ┆ b"\x01" │
#> │ null ┆ null │
#> │ 14:00:00 ┆ b"\x02" │
#> └──────────┴─────────┘
## Polars to R
tibble::as_tibble(pl_df)
#> # A tibble: 3 × 2
#> time binary
#> <time> <blob>
#> 1 12:00 <raw 1 B>
#> 2 NA NA
#> 3 14:00 <raw 1 B>
R objects that convert to a Series of length 1 are now treated like scalar values when converting to polars expressions:
### OLD
series <- pl$Series("foo", 1)
pl$DataFrame(bar = 1:2)$with_columns(series)
#> [...truncated...]
#> Encountered the following error in Rust-Polars:
#> Series foo, length 1 doesn't match the DataFrame height of 2
#>
#> If you want expression: Series[foo] to be broadcasted, ensure it is a
#> scalar (for instance by adding '.first()').
### NEW
series <- pl$Series("foo", 1)
pl$DataFrame(bar = 1:2)$with_columns(series)
#> shape: (2, 2)
#> ┌─────┬─────┐
#> │ bar ┆ foo │
#> │ --- ┆ --- │
#> │ i32 ┆ f64 │
#> ╞═════╪═════╡
#> │ 1 ┆ 1.0 │
#> │ 2 ┆ 1.0 │
#> └─────┴─────┘
<expr>$map_batches() still exists but its usage is discouraged. This function is
not guaranteed to interact correctly with the streaming engine. To apply
functions from external packages or custom functions that cannot be translated
to polars syntax, we now recommend converting the data to a data.frame and
using purrr (note that as of 1.1.0, purrr enables parallel computation).
The vignette "Using custom functions" contains more details about this.