Skip to contents

Changelog

Source: NEWS.md

kit 0.0.24 (development version)

Bug Fixes

  • Fix vswitch()/nswitch() with checkEnc = TRUE when the first string is already UTF-8 but later elements are latin1 or NA: enc2UTF8() now checks every element and preserves NA_STRING, and value-major nswitch() compares each pair against its own converted value instead of the first one.

  • Fix funique()/fduplicated()/uniqLen() hashing -0 and differing NaN payloads as distinct doubles. Vector, matrix and data-frame paths now canonicalise doubles the same way REQUAL compares them, matching base R.

Notes

  • Speed up vswitch() and nswitch() when there are many value/output pairs. For more than 8 pairs the implementation builds a value-to-last-pair hash map once and scans x in cache-sized blocks instead of comparing x against every value. Matching semantics are unchanged: integer/logical NA still matches NA, doubles/complex canonicalise -0 and never match NA/NaN, strings compare CHARSXP identity, duplicate values keep the last pair, and threaded results stay deterministic. On 2m elements, 20 values are about 2-2.5x faster and 100 values 4.5-5.7x faster; the value-major path is kept for 8 or fewer pairs.

  • Speed up funique()/fduplicated()/uniqLen() on data frames and matrices. Row hashes now use a 64-bit rolling mix instead of mixing every cell and mixing again, and a bounded sample estimates cardinality so high-cardinality inputs allocate the final table once while low-cardinality inputs keep the small grow-on-demand table. On 10m rows, 2-column integer data frames improve about 2.5x and mixed integer/double frames about 2.3-2.5x; 2m frames improve 1.5-1.8x. The scalar vector path was left unchanged: it is already faster than data.table::uniqueN() and dedicated hash variants stayed under the 20% threshold.

  • Add tools/benchmark-core.R, a manual benchmark covering 2m/10m elements, 5/20/100 switch values, integer/string/factor/NA inputs, data frames, and 1/2/4 threads. It is not run in CI.

kit 0.0.23 (2026-11-02)

Bug Fixes

  • Fix nswitch() with duplicate values and nThread > 1: the outer loop over value/output pairs is now serial and atomic-output pairs are parallelised over the elements of x, so the last matching pair wins deterministically. Also fix the list-input/logical-output path reading output element i instead of j (#63 follow-up).

  • Fix fduplicated() on logical vectors longer than INT_MAX: the generic path used an int loop index and stored i + 1 in an int table slot, so a multi-gigabyte logical vector could overflow and read out of bounds. It now raises the standard “Long vector not supported yet” error (#61 follow-up).

  • Apply xtfrm to classed vectors before the topn() long-vector heap path, so custom classes follow order() semantics there too; classed long vectors whose xtfrm is not integer/double now error clearly instead of falling back to the integer-index order() path (#83 follow-up).

  • Fix nswitch() segfaulting when called without value/output pairs (e.g. nswitch(1:3) or nswitch(1:3, default = 0L)): the argument count is now validated before any pair is read, and the call errors with a clear message (#87).

  • Evaluate the internal order()/sort() calls in topn, psort, and charToFact in the base environment. A caller-defined order/sort could previously shadow them and make topn return out-of-bounds indices or psort/charToFact error or sort incorrectly (#74).

  • Fix shared-memory lifetime and error-path safety in shareData(): POSIX mapping names are now owned copies, long-vector lengths, allocation sizes, and existing mapping sizes are checked, finalizer verbosity is per mapping, failed mappings release descriptors, mappings, and temporary state, reader cleanup is unwind-safe, and finalizers verify mapping identity before unlinking owned names (#55).

  • Fix segfault in vswitch/nswitch on length-0 character input: the character encoding path dereferenced element 0 without checking length. The encoding check is now skipped for empty x and isMixEnc/enc2UTF8 handle empty input safely, with no overhead on the fast path (#66).

  • Fix .onAttach/.onLoad crash on bad kit.nThread (NULL/NA/length != 1): the option is now sanitized to 1L, the default is restored via is.null(getOption()), and the startup message only prints in interactive sessions (#56).

  • Keep nswitch’s per-output recycle mask on the heap instead of the C stack, matching vswitch, so a very large number of value/output pairs cannot overflow the stack (#74).

  • Fix shareData/getData/clearData input validation: bad map_name (""/NA/length != 1/non-character) and verbose (NA/non-logical/length != 1) now fail fast in R with a clear message instead of reaching shm_open and its cryptic error; connections are closed via on.exit even when serialize/unserialize fails (#59).

  • Audit compiled length handling for 64-bit long vectors: IS_BOOL/IS_LOGICAL and the character psort/charToFact paths now use XLENGTH() (long character vectors are refused with the standard “Long vector not supported yet” error before any data is read), funique/uniqLen handle > 2^31 elements for factors and logicals, topn on long inputs returns double indices like base::order, subSetRowMatrix row indexing no longer overflows for matrices with >= 2^31 cells, and byte-size casts that truncated above 4 GiB are fixed (#61).

  • Fix two segfaults found while auditing length handling (#61): funique/uniqLen on factors containing NA read the count table out of bounds (the NA code is now kept in its own slot and missing elements are returned in the same order as base::unique/length(unique())), and dfToMatrix refused to bound-check a flat int index, so funique/uniqLen on data.frames with more than INT_MAX cells (e.g. 50k columns) wrote out of bounds; such data.frames now raise the standard “Long vector not supported yet” error. Factor codes outside 1..nlevels (e.g. after attr(x, "levels") surgery) no longer index the count table out of bounds either and are returned as NA like base; distinct invalid codes collapse into one missing value.

  • Guard the mixed-type data.frame paths in funique/fduplicated (dupDataFrameR), uniqLen (dupLenDataFrameR), and countOccur (countOccurDataFrameR): with more than INT_MAX rows the row count was truncated to int inside allocMatrix(), so 231-232 rows errored with R’s confusing “negative extents to matrix” message (and above 2^32 rows the mixed funique/uniqLen logical-column case could write out of bounds, which the per-column 2^30 guard only prevents for other types). These paths now stop up front with the standard “Long vector not supported yet” error, before any allocation, when the rows exceed INT_MAX or the matrix would exceed INT_MAX cells — the same bound the all-same-type branch has applied to dfToMatrix() since the length audit (#61), so the same data.frame now fails the same way regardless of whether its columns share one type (#82).

Notes

  • Mix hashed keys with a 32-bit avalanche before indexing the growable duplicate tables. The previous top-bits-only multiplicative hash mapped sequential integer IDs to only ~16% of the slots in a power-of-two table, so funique()/fduplicated()/uniqLen() fell back to long linear-probe scans on common ID-like data. The tables still start at 256 slots and grow on demand; adding tools/benchmark-unique.R keeps this case visible.

  • Document ?kit with a package page that explains OpenMP threads, platform support, encoding behavior, and the main functions. Add “OpenMP and threads” and “Platforms” sections to the README, and add library(kit) to the first README example (#90).

  • Reduce peak memory in funique, fduplicated, uniqLen, psort, and charToFact by starting duplicate hash tables at 256 slots and growing them on demand (4x, capped at the previous full-size table) instead of allocating the full table up front. Rehashing re-reads the source values, so results are unchanged; inputs with few unique values use much less memory, while peak memory can be higher at growth boundaries for high-cardinality inputs (#74).

  • Speed up vswitch for string and list outputs by scanning x once instead of once per value and fusing default/na writes into the match pass; single-value lookups also avoid a separate initialization pass. Atomic outputs keep the value-major vectorized pass, with the first value fused into initialization. Last-match-wins behavior for duplicate values is preserved, mixed output item types now fail with a clear error, and list-output scratch arrays are heap allocated so large outputs no longer exhaust the C stack (#74).

  • Bound the topn linear candidate scan: input ordered opposite to decreasing made every element replace the current worst and rescan all n candidates, so n = 1500 on 2e7 elements took ~14.5 s. The scan now falls back to the existing radix order() path once its rescan work exceeds a few passes over the input, reducing that case to ~0.11 s while leaving random and small-n inputs on the fast path. The fallback allocates a full-length order index, so peak memory is higher on adversarial inputs. Classed numeric vectors have xtfrm applied once and then use the same fast selector, so a custom class ordering matches order() without forcing a full radix sort for identity classes such as Date, POSIXct, and factors (#74).

  • Serialize shareData payloads directly to a raw vector instead of through a raw connection, removing one full-payload copy on write and connection overhead on read; the shared-memory format is unchanged (#74).

  • Reduce topn memory use on long vectors (more than INT_MAX elements): the long path allocated a full-length radix order() index (~8 x length bytes, ~17 GB for a 2^31-element input, only to return n indices) and is replaced by a bounded heap of the n best (value, index) entries, so memory is now proportional to n; unmaterialized ALTREP inputs are read element-wise without being materialized. The result is unchanged — double indices like order(vec, method="radix", na.last=TRUE)[1:n], ties and missing values keeping input order regardless of hasna — and decreasing=TRUE now works on such inputs, where the order() path errored with base R’s “long vectors not supported” (#83).

  • Document the shareData/getData/clearData contract in ?shareData: deep-copy cost (object serialized once, then copied into and out of shared memory, not a pointer), single-read getData, lifetime and cleanup semantics, platform differences, and a two-session \dontrun example; cross-links the zero-copy request (#43) (#60).

  • Clean up DESCRIPTION metadata: bump Depends to R (>= 3.5.0) to match the DATAPTR_RO fallback in src/kit.h (...length(), R_RegisterCCallable/STRING_PTR_RO need newer than 3.1), use ByteCompile: yes per WRE, drop Repository: CRAN from the dev tree, and add Config/Needs/website: pkgdown (Suggests already covers data.table/tibble/tinytest used in tests and vignette) (#58).

  • The test suite was migrated to tinytest: the legacy custom check() only printed failures without stopping, so regressions went unnoticed. Tests now fail loudly (1271 expectations) with zero additional dependencies.

  • Replace the remaining Rboolean scalars in the C sources with bool (callToOrder, dupMatrixR, topnR, and the hasFactor flags in psum), keeping asLogical() as the R-value boundary. Rboolean is only intended for the TRUE/FALSE enum and misbehaves for logical vector elements under stdbool.h and C23, as reported by @aitap in #42. The LGLSXP/INTSXP branches in subSetColMatrix are merged since LOGICAL() and INTEGER() are both int* (#62).

  • Reduce a redundant initialization pass in multi-argument psum(..., na.rm = TRUE) for integer, double, and complex inputs while preserving missing-value semantics (#74).

  • New test coverage for fpmin/fpmax/prange (NA/NaN + na.rm, logical < integer < double promotion, prange double-only rule, single list/data.frame path), kit.nThread = 1 vs 2 determinism (iif/vswitch/nswitch/charToFact/psort), and mixed-encoding checkEnc, funique data.table/tibble attributes, Date/factor preservation and empty inputs (#63, tests-only, no C changes). Coverage now requires > 90% via Codecov.

  • Add a root .gitignore for build artifacts (src/*.o, src/kit.so, the src/Makevars that configure generates, *.tar.gz, *.Rcheck/, cobertura.xml, docs/, and R session junk), so an in-tree R CMD INSTALL no longer leaves 13 untracked files that a stray git add -A would commit. The src/Makevars rule is narrow so the tracked src/Makevars.in and src/Makevars.win stay visible, and the tracked cleanup script is deliberately not ignored: it is what keeps the generated Makevars out of the distribution tarball (#78).

kit 0.0.22 (2026-08-25)

Bug Fixes

  • Fix shareData and getData on platforms where shm_open requires names to start with a slash, e.g. FreeBSD. Shared memory object names are now normalized on all POSIX platforms as recommended by POSIX. Thanks to @nunotexbsd for raising an issue (#40).

  • Fix a mapping leak in getData where the wrong region was unmapped during cleanup.

Notes

  • File descriptors from shm_open are now closed after the mappings are established.

  • The test suite now skips shareData checks gracefully when POSIX shared memory is unavailable instead of aborting the whole run.

kit 0.0.21 (2026-01-17)

CRAN release: 2026-01-23

New Features

  • Functions fpmin, fpmax, and prange are added as parallel equivalents of base R pmin, pmax, and range operations. These functions compute element-wise minimum, maximum, and range (max - min) across multiple vectors. They support integer, logical, and double types, with automatic type promotion. Implemented by @SebKrantz.

Notes

  • The GitHub repo move to fastverse/kit, a new website exists at fastverse.org/kit, and Sebastian Krantz and the fastverse Team have taken responsibility to maintain the package. Morgan Jacob has stepped down, but remains available for consultation.

kit 0.0.20 (2025-04-17)

CRAN release: 2025-04-17

Notes

  • Update copyright date in c files

  • Fix note on CRAN regarding Rf_isFrame

kit 0.0.19 (2024-09-07)

CRAN release: 2024-09-07

Bug Fixes

  • Fix multiple warnings in C code.

kit 0.0.18 (2024-06-06)

CRAN release: 2024-06-10

Bug Fixes

  • Fix iif tests for new version of R.

kit 0.0.17 (2024-05-03)

CRAN release: 2024-05-04

Bug Fixes

  • Fix nswitch. Thanks to Sebastian Krantz for raising an issue.

Notes

  • Update copyright date in c files

  • Fix note on CRAN regarding SETLENGTH

kit 0.0.16 (2024-03-01)

CRAN release: 2024-02-29

Notes

  • Check if "kit.nThread" is defined before setting it to 1L

kit 0.0.15 (2023-10-01)

CRAN release: 2023-10-01

Notes

  • Correct typo in configure file

kit 0.0.14 (2023-08-12)

CRAN release: 2023-08-12

Notes

  • Update configure file to extend support for GCC

  • Correct warnings in NEWS.Rd (strong)

  • Correct typo in funique.Rd thanks to @davidbudzynski

kit 0.0.13 (2023-02-24)

CRAN release: 2023-02-24

Notes

  • Function pprod now returns double output even if inputs are integer - in line with base::prod - to avoid integer overflows.

  • Update configure file

kit 0.0.12 (2022-10-26)

CRAN release: 2022-10-26

New Features

  • Function pcountNA is equivalent to pcount(..., value = NA).

  • Function pcountNA and pcount(..., value = NA) allow NA counting with mixed data type (including data.frame). pcountNA also supports list-vectors as inputs and counts empty or NULL elements as NA.

  • Functions panyv, panyNA, pallv and pallNA are added as efficient wrappers around pcount and pcountNA. They are parallel equivalents of scalar functions base::anyNA and anyv, allv and allNA in the ‘collapse’ R package.

  • Functions pfirst and plast are added to efficiently obtain the row-wise first and last non-missing value or non-empty element of lists. They are parallel equivalents to the (column-wise) ffirst and flast functions in the ‘collapse’ R package. Implemented by @SebKrantz.

  • Functions psum/pprod/pmean also support logical vectors as input. Implemented by @SebKrantz.

Bug Fixes

  • Function charToFact was not returning proper results. Thanks to @alex-raw for raising an issue.

Notes

  • Function pprod now returns double output even if inputs are integer - in line with base::prod - to avoid integer overflows.

  • C compiler warnings on CRAN R-devel caused by compilation with -Wstrict-prototypes are now fixed. Declaration of functions without prototypes is depreciated in all versions of C. Thanks to Sebastian Krantz for the PR.

kit 0.0.11 (2022-03-19)

CRAN release: 2022-03-27

New Features

  • Function pcount now supports data.frame.

Bug Fixes

  • Function pcount now works with specific NA values, i.e. NA_real_, NA_character_ etc…

kit 0.0.10 (2021-11-28)

CRAN release: 2021-11-28

New Features

  • Function psum, pmean, pprod, pany and pall now support lists. Thanks to Sebastian Krantz for the request and code suggestion.

Bug Fixes

  • Function topn should now work for ALTREP object. Thanks to @ben-schwen for raising an issue.

kit 0.0.9 (2021-09-12)

CRAN release: 2021-09-12

Notes

  • Re-organise header to prevent compilation errors with new version of Clang due to conflicts between R C headers and OpenMP.

kit 0.0.8 (2021-08-21)

CRAN release: 2021-08-17

New Features

  • Function funique now preserves the attributes if the input is a data.table, tibble or similar objects. Thanks to Sebastian Krantz for the request.

  • Function topn now defaults to base R order for large value of n. Please see updated documentation for more information ?kit::topn.

  • Function charToFact gains a new argument addNA=TRUE to be used to include (or not) NA in levels of the output.

  • Function shareData, getData and clearData implemented to share data objects between R sessions. These functions are experimental and might change in the future. Feedback is welcome. Please see ?kit::shareData for more information.

Notes

  • Few calloc functions at C level have been replaced by R C API function Calloc to avoid valgrind errors/warnings in Travis CI.

  • Errors reported by rchk on CRAN have been fixed.

kit 0.0.7 (2021-03-07)

CRAN release: 2021-03-09

New Features

  • Function charToFact gains a new argument decreasing=FALSE to be used to order levels of the output in decreasing or increasing order.

  • Function topn gains a new argument index=TRUE to be used return index (TRUE) or values (FALSE) of input vector.

Bug Fixes

  • Some tests of memory access errors using valgrind and AddressSanitizer were reported by CRAN. An attempt to fix these errors has been submitted as part of this package version. It also seems that these same errors were causing some tests to fail for funique and psort on some platforms.

Notes

  • Functions pmean, pprod and psum will result in error if used with factors. Documentation has been updated.

kit 0.0.6 (2021-02-21)

CRAN release: 2021-02-21

New Features

  • Function funique and fduplicated gain an additional argument fromLast=FALSE to indicate whether the search should start from the end or beginning PR#11.

  • Functions pall, pany, pmean, pprod and psum accept data.frame as input PR#15. Please see documentation for more information.

  • Function charToFact is equivalent to to base R as.factor but is much quicker and only converts character vector to factor. Note that it is parallelised. For more details and benchmark please see ?kit::charToFact.

  • Function psort is experimental and equivalent to to base R sort but is only for character vector. It can sort by “C locale” or by “R session locale”. For more details and benchmark please see ?kit::psort.

Notes

  • A few OpenMP directives were missing for functions vswitch and nswitch for character vectors. These have been added in PR#12.

  • Function funique was not preserving attributes for character, logical and complex vectors/data.frames. Thanks to Sebastian Krantz (@SebKrantz) for bringing that to my attention. This has been fixed in PR#13.

  • Functions funique and uniqLen should now be faster for factor and logical vectors PR#14.

kit 0.0.5 (2020-11-21)

CRAN release: 2020-11-21

New Features

  • Function uniqLen(x) is equivalent to base R length(unique(x)) and uniqueN in package data.table. Function uniqLen, implemented in C, supports vectors, data.frame and matrix. It should be faster than these functions. For more details and benchmark please see ?kit::uniqLen.

  • Function vswitch now supports mixed encoding and gains an additional argument checkEnc=TRUE. Thanks to Xianying Tan (@shrektan) for the request and review PR#7.

  • Function nswitch is a nested version of function vswitch and also supports mixed encoding. Please see please see ?kit::nswitch for further details. Thanks to Xianying Tan (@shrektan) for the request and review PR#10.

Notes

  • Small algorithmic improvement for functions fduplicated, funique and countOccur for vectors, data.frame and matrix.

  • A tests folder has been added to the source package to track coverage and bugs.

C-Level Facilities

  • Function nif has been split into two distinctive functions at C level, one has its arguments evaluated in a lazy way and is for R users and the other one (nifInternalR) is not lazy and is intended for usage at C level.

kit 0.0.4 (2020-07-21)

CRAN release: 2020-07-21

New Features

  • Function countOccur(x), implemented in C, is comparable to base R function table. It returns a data.frame and is between 3 to 50 times faster. For more details, please see ?kit::countOccur.

  • Functions funique and fduplicated now support matrices. Additionally, these two functions should also have better performance compare to previous release.

  • Functions topn has an additional argument hasna=TRUE to indicates whether data contains NA value or not. If the data does not contain NA values, the function should be faster.

C-Level Facilities

  • A few C functions have been added to subset data.frame and matrix as well as do other operations. These functions are not exported or visible to the user but might become available and callable at C level in the future.

Bug Fixes

  • Function fpos was not properly handling NaN and NA for complex and double. This should now be fixed. The function has also been changed in case the ‘needle’ and ‘haysatck’ are vectors so that a vector is returned.

  • Functions funique and fduplicated were not properly handling data containing POSIX data. This has now been fixed.

kit 0.0.3 (2020-06-21)

CRAN release: 2020-06-26

New Features

  • Functions fduplicated(x) and funique(x), implemented in C, are comparable to base R functions duplicated and unique. For more details, please see ?kit::funique.

  • Functions psum and pprod have now better performance for type double and complex.

Bug Fixes

  • Function count(x, y) now checks that x and y have the same class and levels. So does pcount.

  • Function pmean was not callable at C level because of a typo. This is now fixed.

kit 0.0.2 (2020-05-22)

CRAN release: 2020-05-24

New Features

  • Function count(x, value), implemented in C, to simply count the number of times an element value occurs in a vector or in a list x. For more details, please see ?kit::count.

  • Function pmean(..., na.rm=FALSE), pall(..., na.rm=FALSE), pany(..., na.rm=FALSE) and pcount(..., value), implemented in C, are similar to already available function psum and pprod. These functions respectively apply base R functions mean, all and any element-wise. For more details, benchmarks and help, please see ?kit::pmean.

Bug Fixes

  • Fix Solaris Unicode warnings for NEWS file. Benchmarks have been moved from the NEWS file to each function Rd file.

  • Fix some NA edge cases for pprod and psum so these functions behave more like base R function prod and sum.

  • Fix installation errors for version of R (<3.5.0).

kit 0.0.1 (2020-05-03)

CRAN release: 2020-05-08

Initial Release

  • Function fpos(needle, haystack, all=TRUE, overlap=TRUE), implemented in C, is inspired by base function which when used in the following form which(x == y, arr.ind =TRUE). Function fpos returns the index(es) or position(s) of a matrix/vector within a larger matrix/vector. Please see ?kit::fpos for more details.

  • Function iif(test, yes, no, na=NULL, tprom=FALSE, nThread=getOption("kit.nThread")), originally contributed as fifelse in package data.table, was moved to package kit to be developed independently. Unlike the current version of fifelse, iif allows type promotion like base function ifelse. For further details about the differences with fifelse, as well as hutils::if_else and dplyr::if_else, please see ?kit::iif.

  • Function nif(..., default=NULL), implemented in C, is inspired by SQL CASE WHEN. It is comparable to dplyr function case_when however it evaluates it arguments in a lazy way (i.e only when needed). Function nif was originally contributed as function fcase in the data.table package but then moved to package kit so its development may resume independently. Please see ?kit::nif for more details.

  • Function pprod(..., na.rm=FALSE) and psum(..., na.rm=FALSE), implemented in C, are inspired by base function pmin and pmax. These new functions work only for integer, double and complex types and do not recycle vectors. Please see ?kit::psum for more details.

  • Function setlevels(x, old, new, skip_absent=FALSE), implemented in C, may be used to set levels of a factor object. Please see ?kit::setlevels for more details.

  • Function topn(vec, n=6L, decreasing=TRUE), implemented in C, returns the top largest or smallest n values for a given numeric vector vec. It is inspired by dplyr::top_n and equivalent to base functions order and sort in specific cases as shown in the documentation. Please see ?kit::topn for more details.

  • Function vswitch(x, values, outputs, default=NULL, nThread=getOption("kit.nThread")), implemented in C, is a vectorised version of base R function switch. This function can also be seen as a particular case of function nif. Please see ?kit::switch for more details.