<?xml version="1.0" encoding="UTF-8"?>
<rss  xmlns:atom="http://www.w3.org/2005/Atom" 
      xmlns:media="http://search.yahoo.com/mrss/" 
      xmlns:content="http://purl.org/rss/1.0/modules/content/" 
      xmlns:dc="http://purl.org/dc/elements/1.1/" 
      version="2.0">
<channel>
<title>The R Project Blog</title>
<link>https://blog.R-project.org/</link>
<atom:link href="https://blog.R-project.org/index.xml" rel="self" type="application/rss+xml"/>
<description>The R Project blog - by members of the R Core Team and the R Foundation, along with other contributors</description>
<image>
<url>https://blog.R-project.org/Rlogo.png</url>
<title>The R Project Blog</title>
<link>https://blog.R-project.org/</link>
<height>112</height>
<width>144</width>
</image>
<generator>quarto-1.11.9</generator>
<lastBuildDate>Sat, 07 Nov 2026 00:00:00 GMT</lastBuildDate>
<item>
  <title>Distribution-Free Analysis for Stratified Oneway Layouts</title>
  <dc:creator>Torsten Hothorn, Kurt Hornik</dc:creator>
  <link>https://blog.R-project.org/posts/free1way/</link>
  <description><![CDATA[ 





<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">Introduction</h2>
<p>Since more than two decades, the <strong>stats</strong> package serves as a home for well-established statistical analysis tools, including linear and generalized linear models or a versatile collection of statistical test procedures featuring <code>*.test()</code> user interfaces. In some cases, such as <code>t.test()</code>, <code>var.test()</code>, or <code>binom.test()</code>, the null and alternative hypotheses are formulated in terms of a parametric model. Model diagnostics and power analysis is relatively simple in this situation and corresponding functionality is provided by <strong>stats</strong>, for example as <code>qqnorm()</code>, <code>power.t.test()</code> etc.</p>
<p>Other procedures are fully nonparametric. The Smirnov two-sample test, for example, allows testing the null of equal distributions against arbitrary alternatives in the two-sample oneway layout via <code>ks.test()</code>.</p>
<p>For yet another group of test procedures it is harder to understand the underlying model. This makes model diagnostics and also power analysis much more challenging, which explains the absence of, for example, <code>power.wilcox.test()</code> or <code>power.friedman.test()</code> from the <strong>stats</strong> package.</p>
<p>Beginning with R version 4.6.0, the novel <code>free1way()</code> function provides a unified interface to semiparametrically efficient population and permutation inference in distribution-free stratified K-sample oneway layouts. This post explains the rationale of this model class and discusses how existing test procedures, for example <code>kruskal.test()</code>, are enhanced by this new tool.</p>
</section>
<section id="distribution-free-oneway-layouts" class="level2">
<h2 class="anchored" data-anchor-id="distribution-free-oneway-layouts">Distribution-free oneway layouts</h2>
<p>When comparing the distributions of some outcome in certain experimental groups, we might refrain from making assumptions regarding the distribution function in one of these groups (typically a control group corresponding to the first factor level, say <img src="https://latex.codecogs.com/png.latex?F_0">) while, at the same time, assuming that the distribution functions in the remaining groups are simple transformations of this control distribution characterized by a single parameter. Under this model, we assume that the distribution function in the second group <img src="https://latex.codecogs.com/png.latex?F_1"> is a function of distribution function in the control group <img src="https://latex.codecogs.com/png.latex?F_0"> and a parameter <img src="https://latex.codecogs.com/png.latex?%5Ctau">. For example <img src="https://latex.codecogs.com/png.latex?F_1%20=%20F_0%5E%7B%5Cexp(-%5Ctau)%7D">, with <img src="https://latex.codecogs.com/png.latex?%5Ctau"> being our unknown treatment effect parameter to be estimated.</p>
<p>This idea, going back to Erich Lehmann’s 1953 paper on <a href="https://doi.org/10.1214/aoms/1177729080">The Power of Rank Tests</a>, allows the formulation of many statistical tests for the composite null <img src="https://latex.codecogs.com/png.latex?%5Ctau%20=%200"> or one-sided alternatives. As an example, consider the Wilcoxon-Mann-Whitney rank sum test. It turns out that this classical test is the score test under the proportional-odds model <img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7BF_1%7D%7B1%20-%20F_1%7D%20=%20%5Cexp(-%5Ctau)%20%5Cfrac%7BF_0%7D%7B1%20-%20F_0%7D%0A"> whose parameters <img src="https://latex.codecogs.com/png.latex?F_0"> and <img src="https://latex.codecogs.com/png.latex?%5Ctau"> can be estimated by maximising the empirical log-likelihood. If more than two samples are to be compared, the model corresponds to the Kruskal-Wallis test.</p>
<p>As an illustration, consider the example presented in the manual page of <code>kruskal.test()</code>:</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## Hollander &amp; Wolfe (1973), 116.</span></span>
<span id="cb1-2"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">## Mucociliary efficiency from the rate of removal of dust in normal</span></span>
<span id="cb1-3"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">##  subjects, subjects with obstructive airway disease, and subjects</span></span>
<span id="cb1-4"><span class="do" style="color: #5E5E5E;
background-color: null;
font-style: italic;">##  with asbestosis.</span></span>
<span id="cb1-5"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">example</span>(kruskal.test, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">echo =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>)</span></code></pre></div></div>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">levels</span>(g) <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Normal"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"COPD"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Asbestosis"</span>)</span>
<span id="cb2-2">(kt <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">kruskal.test</span>(x <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> g))</span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>
    Kruskal-Wallis rank sum test

data:  x by g
Kruskal-Wallis chi-squared = 0.77143, df = 2, p-value = 0.68</code></pre>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1">(ft <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">free1way</span>(x <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> g))</span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>
    3-sample Kruskal-Wallis test against proportional odds alternatives

data:  x by g (Normal, COPD, Asbestosis)
Perm chi-squared = 0.77143, df = 2, p-value = 0.68
alternative hypothesis: true log-odds ratio is not equal to 0</code></pre>
</div>
</div>
<p>The test statistics and p-values are identical, but the print method of the latter procedure informs about the alternative this test is powerful (in fact, the locally most powerful test) against.</p>
<p>In contrast to the classical implementation in <code>kruskal.test()</code> (where <code>coef(kt)</code> comes back empty handed), the novel <code>free1way()</code> is aware of the semiparametric nature of the model and allows to access the estimated log-odds ratios along with a confidence interval (a Rao score interval is presented here):</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb6" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb6-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">cbind</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">LogOR =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">coef</span>(ft),</span>
<span id="cb6-2">      <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">confint</span>(ft, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">test =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Rao"</span>))</span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>                 LogOR     2.5 %   97.5 %
gCOPD        1.0350087 -1.397616 3.475640
gAsbestosis -0.3706949 -2.482256 1.748293</code></pre>
</div>
</div>
<p>The data analysis could also question the sample size needed to demonstrate an overall effect when the true log-odds ratios are close to the one estimated from this dataset</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb8-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">power.free1way.test</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">delta =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">coef</span>(ft), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">power =</span> .<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">8</span>)</span></code></pre></div></div>
<div class="cell-output cell-output-stdout">
<pre><code>
      3-sample Kruskal-Wallis test against proportional odds alternatives 

                n = 30
Total sample size = 30 (Control) + 30 (gCOPD) + 30 (gAsbestosis) = 90
            power = 0.8002641
        sig.level = 0.05
   log-odds ratio = 1.0350087, -0.3706949

NOTE: 'n' is sample size in control group</code></pre>
</div>
</div>
</section>
<section id="features" class="level2">
<h2 class="anchored" data-anchor-id="features">Features</h2>
<p>Binary, ordered categorical, numeric and possibly right-censored outcomes can be analyzed using <code>free1way()</code>. In stratified designs, strata can be specified as blocks. Wald, likelihood-ratio and Rao score tests and confidence intervals are available. In addition, permutation tests (asymptotic and resampling-based, exact for two-sample proportional odds models) can be performed to test hypotheses. Multiple comparisons can be conducted via an interface to the <a href="https://doi.org/10.32614/CRAN.package.multcomp">multcomp package</a>. Similar to Firth’s method in binary GLMs, penalisation via Jeffreys’ prior is implemented.</p>
</section>
<section id="resources" class="level2">
<h2 class="anchored" data-anchor-id="resources">Resources</h2>
<p>In addition to the documentation, examples, and tests provided in the <strong>stats</strong> package, a dedicated <a href="https://doi.org/10.32614/CRAN.package.free1way.docreg">free1way.docreg package</a> features two vignettes explaining the implementation and statistical framework in depths, along additional regression tests comparing the implementation against benchmarks such as <code>rms::orm()</code> (<a href="https://doi.org/10.32614/CRAN.package.rms">rms package</a>), <code>MASS::polr()</code> (<a href="https://doi.org/10.32614/CRAN.package.MASS">MASS package</a>), or <code>tram::Polr()</code> (<a href="https://doi.org/10.32614/CRAN.package.tram">tram package</a>).</p>


</section>

 ]]></description>
  <category>R Core</category>
  <category>stats</category>
  <guid>https://blog.R-project.org/posts/free1way/</guid>
  <pubDate>Sat, 07 Nov 2026 00:00:00 GMT</pubDate>
  <media:content url="https://blog.R-project.org/posts/free1way/free1way.png" medium="image" type="image/png" height="100" width="144"/>
</item>
<item>
  <title>Sovereign Tech Fund Project: Milestone 2</title>
  <dc:creator>Šimon Urbánek, Tomáš Kalibera</dc:creator>
  <link>https://blog.R-project.org/posts/stf-project-milestone-2/</link>
  <description><![CDATA[ 





<section id="sovereign-tech-fund-stf-project" class="level2">
<h2 class="anchored" data-anchor-id="sovereign-tech-fund-stf-project">Sovereign Tech Fund (STF) Project</h2>
<p>Starting in mid 2025, the R Project has been working on a series of work packages to improve the long-term stability of the R ecosystem. The project is supported by an investment from the <a href="https://www.sovereign.tech/">Sovereign Tech Agency</a>, under their <a href="https://www.sovereign.tech/tech/r-project">Sovereign Tech Fund</a>. In a series of blog posts, some details about the achieved results will be shared with the public, with this post covering Milestone 2. These blog posts are a tribute to the late <a href="https://www.R-project.org/doc/obit/tomas.html">Tomáš Kalibera</a>, who continued working on this project during his last months.</p>
</section>
<section id="challenges-for-binary-r-packages" class="level2">
<h2 class="anchored" data-anchor-id="challenges-for-binary-r-packages">Challenges for Binary R Packages</h2>
<p>R packages are fundamental to the success of R since they provide a way for large number of people to contribute to the R ecosystem without having to rely on changes in R itself. The contributions can range from cutting edge statistical methods to visualisation tools or bindings to other useful systems. R packages are created and distributed in <em>source</em> form, requiring additional processing before they can be used by the user. This processing can be quite complex for packages that contain native code or rely on additional dependent libraries or tools - making them harder for users to use.</p>
<p>One of the key aspects of solving that problem is that the Comprehensive R Archive Network (<a href="https://CRAN.R-project.org/">CRAN</a>) distributes ready-to-use binaries for the two most popular operating systems Windows and macOS, which makes it easy for users to download and use both R and all packages present on CRAN. The complexities of dealing with tools and dependencies are thus handled by the team producing the binaries, here the CRAN team. The goal is that all parts necessary for the function of the package are included in the binary package such they do not require the user to download or install anything else. This means that any required dependent parts such as libraries have to be included as well.</p>
<p>Historically, there are two challenges related to binary packages. Firstly, the R package system is entirely based on full identification by package <em>name</em> and <em>version</em>. However, the same source package (name + version) may result in more than one binary package file in cases when a dependency requires the package to be re-built. For example, a library used by the package may have a serious vulnerability and thus a newer version of the library must be used and included in a newer version of the binary package, even though the source package did not change. Another case is that a package may make a change that affects those package that use it in an internal way. For example, the methods may be cached from the a package that is used and if they change the cached version must be updated. This poses a problem since tools like <code>update.packages()</code> only see the name and version and thus have no way of finding out that the binary package must be re-installed.</p>
<p>Secondly, the support for binary packages has been added for each operating system individually. Historically, Windows packages use a different archive format and entirely different way to be processed than macOS packages. In addition, there is no direct support for binary packages for other operating systems. Both are an issue for sustainability since it is hard to maintain or support future changes, such as different architectures or operating systems.</p>
<p>Addressing both of those major points was the goal for the Milestone 2 of the <a href="https://www.sovereign.tech/tech/r-project">Sovereign Tech Fund</a>: to provide a way for binary packages to signal that they need to be updated even if the source package has been changed, and to unify the support for binary packages extensible beyond Windows and macOS.</p>
</section>
<section id="package-freshness" class="level2">
<h2 class="anchored" data-anchor-id="package-freshness">Package Freshness</h2>
<p>The way package repositories work is by providing a well-known location of an index file <code>PACKAGES</code> which lists all packages in a given repository and their metadata which is mostly obtained from the <code>DESCRIPTION</code> files of the packages. The most important ones are the package name and version, but there are others. This information is obtained using the <code>available.packages()</code> function. What distinguished a source package from a binary package is the presence of the <code>Built</code> field:</p>
<pre><code>&gt; packageDescription("tools")$Built
[1] "R 4.6.1; x86_64-apple-darwin20; 2026-06-25 22:07:05 UTC; unix"</code></pre>
<p>It allows R to determine if the package is compatible with the currently-used R by looking at the R version and the platform string (here <code>x86_64-apple-darwin20</code> which identifies Intel-based Mac with macOS 11). Another useful information is the timestamp of the build.</p>
<p>This provides R with a mechanism to determine that a given package has been re-built even if the version remained the same. To implement this feature, this information is propagated into the <code>PACKAGES</code> file and provide it to <code>available.packages()</code>. One user-visible change is that the <code>Built</code> field as returned from <code>available.packages()</code> only contained the R version, but now is provided in full. This allows to check the freshness as well as determine compatibility with the R build used.</p>
<p>As a side note, it also means that the <code>checkBuilt=</code> argument in <code>update.packages()</code> now becomes a bit of a misnomer since it has been defined to only check the R version and was used to upgrade packages from one annual R release to another, which was a rare event. However, regular updates are now considering the full <code>Built</code> field and this is crucial to detect new binaries and therefore the <code>checkBuilt=</code> argument retains its original meaning (only comparing R versions) even though the field has been expanded.</p>
<p>One additional complication is that functions like <code>install.packages()</code> and <code>update.packages()</code> support a virtual convenience installation type <code>"both"</code> which is intended to augment the binary installation with source packages where the binary does not exist or has an older version. The main problem is that, historically, this has been performed by checking the availability solely using the source repository. Therefore tools that assessed package freshness like <code>old.packages()</code> only ever looked at the source packages. In order to support <code>"both"</code> they now need to fetch the availability from both the source and binary repositories, which means that they can no longer operate on one package database alone.</p>
<p>Finally, there are cases where a change in one package may require a re-build of another package that depends on it without the latter needing to be changed. This can happen either if the package changes its ABI inadvertently or in cases where some information from the package is cached in the packages that import it. It means that an update of a package may require its reverse-dependencies to be rebuilt - and this may be true for both source and binary packages (e.g., if the package cached some information from the dependency).</p>
<p>The key here is that an update of the source package needs a way to communicate this information, and that this information is specific to repositories as it is caused by the relative relationship of packages and updates (it does not occur if simply a snapshot is installed). We already have a field that can be used for this purpose in repositories: <code>Published</code>. Therefore the new scheme is that if a package is re-published with the same version in the repository then it counts as being more recent. Since this is a repository-level metadata it can be set by the repository independently, thus allowing all reverse-dependencies to be tagged accordingly. This can be done either by the repository maintainer triggering such updates, or by a package author declaring the list of packages that are known to be affected (if known explicitly) in the <code>Invalidates:</code> field (which is really only informative for the repository maintainers). Note that the update of the <code>Published:</code> field should be done only once when such a compatibility-breaking package is published since regular version updates will make it obsolete.</p>
<p>Theses two mechanisms now allow repositories to determine when a package needs to be re-installed, even if the source version of the package did not change. R will now automatically install updated binaries, making sure the user’s library is up to date.</p>
</section>
<section id="custom-binary-packages" class="level2">
<h2 class="anchored" data-anchor-id="custom-binary-packages">Custom Binary Packages</h2>
<p>The second challenge related to binary R packages is that the support for binary packages has been added independently for Windows and macOS, based on the fact that users of those operating systems do not expect to install software from sources. Each operating system uses a different format (Windows binary packages use ZIP archives, while for macOS gzip-compressed tar balls with the <code>.tgz</code> extension are used) and code path within R. The idea of a binary package is not limited to just Windows and macOS. The goal of the second update was to unify the support of binary packages and to make it extensible to support arbitrary operating systems.</p>
<p>There are two components to this: the format of the archives and location of files in R package repositories. The <code>tar</code> format is used everywhere but on Windows, therefore it makes sense as a basis. The gzip compression is fairly dated and newer compression formats yield better compression even at comparable speeds, thus reducing the size of repositories. Therefore the decision was made to support newer compression and thus formats including <code>.tar.bz2</code>, <code>.tar.xz</code> and <code>.tar.zstd</code>. The last one is optional since R does not require the presence of the Zstandard library.</p>
<p>The other component is the ability to declare and locate files for a specific build of R which matches the binaries it produces. The macOS binaries already provide a way of distinguishing different binary builds by using a suffix to <code>pkgType</code>, so for example the binaries for x86_86 (Intel) architecture for macOS 11 (Big Sur) use the type <code>"mac.binary.big-sur-x86_64"</code>. The <code>contrib.url()</code> function transforms this information to a path that determines where in a repository the binary package files are located, for the above case that is <code>"/bin/macosx/big-sur-x86_64/contrib/4.6"</code>.</p>
<p>To generalize this concept, package types of the form <code>&lt;system&gt;.binary.&lt;build&gt;</code> are now mapped to</p>
<p><code>/bin/&lt;system&gt;/&lt;build&gt;/contrib/&lt;x.y&gt;</code></p>
<p>where <code>&lt;system&gt;</code> is the lowercase name of the operating system (e.g., <code>linux</code>), <code>&lt;build&gt;</code> is the designation of the specific build of R which generated this binary and <code>&lt;x.y&gt;</code> is the R version without patch level. The build name may contain alphanumeric characters, <code>_</code> and <code>-</code>. Currently, there are no specific restrictions on the build names, but it is a good idea to include at least the architecture and possibly toolchain name if multiple incompatible toolchain exist for the platform.</p>
<p>Note that the values <code>mac</code> and <code>win</code> for <code>&lt;system&gt;</code> are reserved for compatibility with old R versions as those will revert back to the legacy package code. In order to use the new functionality on Windows use <code>windows</code> and on macOS use <code>macos</code>. For example, the current (at the time of writing) R-devel builds for Windows ARM64 use <code>windows.binary.clang-aarch64</code> and on macOS arm64 use <code>macos.binary.arm64</code>.</p>
<p>The <code>pkgType</code> of the current R build can be seen in <code>.Platform$pkgType</code>. A build of R can declare the kind of binaries it produces by setting the <code>R_PLATFORM_PKGTYPE</code> environment variable which will be in turn reflected in <code>.Platform$pkgType</code>.</p>
<p>At user level, specifying <code>type="binary"</code> in <code>install.packages()</code> and friends will match the binary corresponding to the currently used R build. Similarly, <code>type="both"</code> is the logical equivalent of using <code>type="source"</code> and <code>type="binary"</code>. Legacy names like <code>"win.binary"</code> or <code>"mac.binary"</code> should no longer be used since they are no longer just two binary types.</p>
<p>All tools have been updated to deal with the new binaries, so, for example using <code>R CMD INSTALL --build</code> in R built with <code>pkgType="macos.binary.arm64"</code> produces a correspondingly named <code>.tar.xz</code> binary package:</p>
<pre><code>$ R CMD INSTALL --build base64enc_0.1-6.tar.gz 
[...]
packaged installation of ‘base64enc’ as ‘base64enc_0.1-6_R_macos-arm64.tar.xz’
* DONE (base64enc)</code></pre>


</section>

 ]]></description>
  <category>packages</category>
  <category>windows</category>
  <guid>https://blog.R-project.org/posts/stf-project-milestone-2/</guid>
  <pubDate>Tue, 29 Sep 2026 00:00:00 GMT</pubDate>
  <media:content url="https://blog.R-project.org/posts/stf-project-milestone-2/stf.png" medium="image" type="image/png" height="92" width="144"/>
</item>
<item>
  <title>Zulip Chat for the R Project</title>
  <dc:creator>Ella Kaye, Heather Turner</dc:creator>
  <link>https://blog.R-project.org/posts/zulip-chat-for-the-r-project/</link>
  <description><![CDATA[ 





<p>The R Project now has a <a href="https://zulip.com">Zulip</a> instance for informal chat about developing, maintaining and supporting R. The Zulip (pronounced “zoo-lip”) will replace the R Contributors Slack group, with a slightly broader scope, also covering wider R Foundation activities.</p>
<p>The <a href="https://www.r-project.org/mail.html#r-devel">R-devel mailing list</a> remains the official channel for technical discussions about R’s development and <a href="https://bugs.r-project.org/">R’s Bugzilla</a> remains the official place to report bugs and open feature requests. However, the Zulip provides a supportive space to get initial feedback on ideas and work in progress, especially for novice contributors. It also provides a useful communication channel for working groups and collaborative events such as <a href="https://contributor.r-project.org/events/r-dev-days/">R Dev Days</a>.</p>
<section id="motivation" class="level2">
<h2 class="anchored" data-anchor-id="motivation">Motivation</h2>
<p>When the <a href="https://contributor.r-project.org/working-group.html">R Contribution Working Group</a> set up the R Contributors Slack in November 2020, the free tier gave access to the most recent 10,000 messages, which would have covered several years of messages. Since September 2020, posts older than 90 days are no longer visible, making it difficult to collaborate on tasks that span several months, which is particularly common when people are contributing in their spare time. Although we have been making manual exports of our Slack data and sharing them as JSON files on GitHub, this is clearly far from optimal.</p>
<p>There are other limitations of Slack that have affected collaboration and communication. The free tier does not support email addresses for channels, so the #bugzilla-inbox channel that relayed new comments on <a href="https://bugs.r-project.org/">bugs.r-project.org</a> relied on a third-party automation with a limit of 100 messages per month - a ceiling we regularly hit during busy periods, such as a month with one or more R Dev Days. Some users found the email notifications unreliable, so would miss when people were trying to contact them. Moreover, as a proprietary platform, Slack was never popular among the R Core Team, so it provided little opportunity for interaction with core developers.</p>
</section>
<section id="why-zulip" class="level2">
<h2 class="anchored" data-anchor-id="why-zulip">Why Zulip</h2>
<p>Zulip is a better fit for the R Project: not only is it open source, but as a company, <a href="https://zulip.com/for/open-source/">Zulip supports open source</a>. They are sponsoring the R Project by providing a free standard cloud plan, enabling us to keep our full chat history.</p>
<p>A number of other open source projects are using Zulip, including Python, Julia and Rust. Bioconductor and NumFOCUS moved from Slack to Zulip last year. Given that members of the R community are often involved in other open source communities, we know there will be some experience with Zulip, and we can hopefully minimise the number of platforms our community needs to use.</p>
<p>Zulip also has some attractive features that Slack does not, including:</p>
<ul>
<li><p><a href="https://zulip.com/help/introduction-to-topics">Topics</a>, that make it easier to hold both sychronous and asynchonous conversations.</p></li>
<li><p>The ability to move messages to keep related chat together.</p></li>
<li><p>Markdown and KaTeX rendering.</p></li>
<li><p>An in-built feature to run polls.</p></li>
</ul>
<p>See the full <a href="https://zulip.com/help/">documentation</a> for more.</p>
</section>
<section id="timeline" class="level2">
<h2 class="anchored" data-anchor-id="timeline">Timeline</h2>
<p>We are launching the Zulip according to the following timeline:</p>
<ul>
<li><p>August 19th - September 22nd: members invited from the R Foundation, <a href="https://contributor.r-project.org/working-group.html">R Contribution Working Group</a>, <a href="../../2025/12/17/rsmf-enabling-the-next-generation-of-contributors-to-r/index.html">RSMF project</a>, and <a href="https://contributor.r-project.org/r-project-sprint-2026/">R Project Sprint 2026</a> participants, to test the set-up and give feedback.</p></li>
<li><p>September 22nd: sign-up page opened to all contributors and supporters of R. From this date the R Contributors Slack will no longer be open to new sign-ups. Existing members will be encouraged to sign up to the new Zulip.</p></li>
<li><p>October 27th: planned date for archiving all R Contributor Slack channels.</p></li>
</ul>
</section>
<section id="joining-the-r-project-zulip" class="level2">
<h2 class="anchored" data-anchor-id="joining-the-r-project-zulip">Joining the R Project Zulip</h2>
<p>You can join the Zulip as follows:</p>
<ul>
<li><p>Sign-up at <a href="https://r-project.zulipchat.com" class="uri">https://r-project.zulipchat.com</a>. Note that we will not be migrating members from Slack to Zulip automatically.</p></li>
<li><p>Read our <a href="https://contributor.r-project.org/zulip">Zulip guide for R Contributors</a>.</p></li>
<li><p>A welcome bot will greet you when you first join with further pointers on effective participation.</p></li>
</ul>
<p>We encourage you to read Zulip’s <a href="https://zulip.com/help/getting-started-with-zulip">getting started guide</a> to learn more.</p>
</section>
<section id="acknowledgements" class="level2">
<h2 class="anchored" data-anchor-id="acknowledgements">Acknowledgements</h2>
<ul>
<li><p>This work was supported by the Research Software Maintenance Fund, managed by the Software Sustainability Institute and funded by UKRI grant reference AH/Z000114/1.</p></li>
<li><p>We’re grateful to Josh Triplett from the Rust Community and Maria Doyle from Bioconductor, for their advice on Zulip and conversations about the transition process.</p></li>
</ul>


</section>

 ]]></description>
  <category>R Foundation</category>
  <category>R Core</category>
  <category>community</category>
  <guid>https://blog.R-project.org/posts/zulip-chat-for-the-r-project/</guid>
  <pubDate>Tue, 22 Sep 2026 00:00:00 GMT</pubDate>
  <media:content url="https://blog.R-project.org/posts/zulip-chat-for-the-r-project/zulip.png" medium="image" type="image/png" height="75" width="144"/>
</item>
<item>
  <title>New Tricks for the R Sweave Drivers</title>
  <dc:creator>Vincent Goulet, Kurt Hornik</dc:creator>
  <link>https://blog.R-project.org/posts/new-tricks-sweave-drivers/</link>
  <description><![CDATA[ 





<section id="introduction" class="level1">
<h1>Introduction</h1>
<p>The concept of literate programming was pioneered by Donald Knuth in 1984 to improve the documentation of computer programs, as this famous quote explains:</p>
<blockquote class="blockquote">
<p>I believe that the time is ripe for significantly better documentation of programs, and that we can best achieve this by considering programs to be <em>works of literature</em>. Hence, my title: “Literate Programming.”</p>
</blockquote>
<p>The aim of literate programming is to create programs that are suitable literature for <em>human beings</em> rather than a set of instructions to a computer. A program becomes a web of two languages: a document formatting language, and a programming language. In Knuth’s original presentation, these were respectively TeX and Pascal. A program, or a set of programs, then process this web of code. Knuth developed the <a href="https://en.wikipedia.org/wiki/WEB">WEB</a> system and coined the terms <em>weaving</em> for the process of creating the documentation, and <em>tangling</em> for the process of producing the executable program.</p>
<p>Friedrich Leisch brought literate programming to the R ecosystem with the <a href="https://en.wikipedia.org/wiki/Sweave">Sweave</a> framework. Based on the system <a href="https://en.wikipedia.org/wiki/Noweb">noweb</a>, Sweave allowed to combine text in LaTeX and R code for literate statistical practice and reporting. This sparkled the use of <em>vignettes</em> for improved documentation within R and contributed packages, and it certainly contributed to the rise of reproducible research practices in statistical computing. Sweave was also of course the inspiration for <a href="https://en.wikipedia.org/wiki/Knitr">knitr</a>, by far the most widely used literate programming system in the R community nowadays, if only as the basis of the document format R Markdown.</p>
<p>Although knitr provides more bells and whistles, Sweave remains simpler and, perhaps most importantly for the forthcoming discussion, it continues to give equal treatment to the lesser known of the two literate programming procedures: tangling.</p>
<p>Sweave was designed from the outset in a modular way to allow different drivers for the weaving and tangling procedures. The standard drivers are <code>RweaveLatex</code> to transform <code>.Rnw</code> files with LaTeX documentation chunks and R code into <code>.tex</code> files, and <code>Rtangle</code> to extract R code from a <code>.Rnw</code> file into <code>.R</code> scripts. A number of packages propose additional drivers for Sweave, for example <a href="https://cran.r-project.org/package=ascii">ascii</a>, <a href="https://cran.r-project.org/package=highlight">highlight</a> or <a href="https://cran.r-project.org/package=R2HTML">R2HTML</a>, just to name a few; see also the <a href="https://CRAN.R-project.org/view=ReproducibleResearch"><em>Reproducible Research</em></a> CRAN Task View.</p>
<p>The feature set of Sweave has remained remarkably stable since its inception. This blog introduces new tricks for the standard drivers that we added in version 4.6.0 of R. Many of these first came to life in the now deprecated package <a href="https://cran.r-project.org/package=RweaveExtra">RweaveExtra</a>.</p>
</section>
<section id="motivation" class="level1">
<h1>Motivation</h1>
<p>The standard Sweave drivers of package utils used to enforce some conventions that, if sensible in a statistical analysis context, could prove limiting in more general uses of literate programming. We will discuss the following conventions in the sequel:</p>
<ol type="1">
<li>The values of chunk options have to be given explicitly, with no possibility to reuse values computed in earlier code chunks.</li>
<li>Expressions in code chunks are always parsed on weaving, whether the option <code>eval</code> is <code>TRUE</code> or <code>FALSE</code>.</li>
<li>Code chunks with <code>eval=FALSE</code> are automatically and irrevocably commented out on tangling.</li>
<li>Tangling allows to control whether evaluated code chunks are extracted to a single file or to separate files via the option <code>split</code>, but not whether or not a given code chunk is extracted in the first place (the option <code>drop.evalFALSE=TRUE</code> only omits unevaluated chunks).</li>
<li>Code chunks collected in the same file on tangling end up always separated by two blank lines, and these are also present at the end of the file.</li>
<li>The names of tangled files always end with a <code>.R</code> extension (omitting the other supported default of <code>.S</code> that is probably not much used nowadays).</li>
</ol>
<p>The first item above limits the amount of logic one can build into a literate programming document. The combination of the second and third items makes it impossible to include in a code chunk invalid R code that should appear uncommented in the tangled script. The fourth and fifth items illustrate that authors have little control over what is extracted on tangling, and in what format. Finally, the sixth item becomes a limitation once the parsing requirement of item 2 is lifted and code chunks may contain code in languages other than R entirely.</p>
</section>
<section id="objects-as-chunk-options" class="level1">
<h1>Objects as chunk options</h1>
<p>In Sweave, code chunks start with <code>&lt;&lt;</code><em>options</em><code>&gt;&gt;=</code> at the beginning of a line, where the optional <em>options</em> have the form <code>key</code>=<code>value</code> and are separated by commas. All options must take a value that used to be either logical, numerical, or character.</p>
<p>The <code>RweaveLatex</code> driver now allows the value of logical and numerical chunk options (only) to be the name of an object defined in earlier, evaluated code chunks. This is useful to pass computed values to options. The example below uses this feature to evaluate a chunk only if a package is available, and to create a plot with computed dimensions (this example and all others in the sequel are stripped of all non-essential text content).</p>
<pre><code>&lt;&lt;&gt;&gt;=
hasfoo &lt;- requireNamespace("foo", quietly = TRUE)
ht &lt;- 5
wd &lt;- (1 + sqrt(5))/2 * ht
@

&lt;&lt;eval=hasfoo&gt;&gt;=
foo::foo(42)
@

&lt;&lt;echo=FALSE, fig=TRUE, width=wd, height=ht&gt;&gt;=
data(airquality, package="datasets")
library("graphics")
boxplot(Ozone ~ Month, data = airquality)
@</code></pre>
<p>This type of option evaluation allows Sweave package vignettes to conveniently deal with the situation that suggested dependencies (that is: mentioned in the <code>Suggests</code> field of the <code>DESCRIPTION</code> file) are used conditionally when checking.</p>
<p>Why limit the feature to logical and numerical options? All the values of options first reach the Sweave driver as character strings. The driver then tries to coerce them to the expected type. When that fails, it is fairly safe for logical and numerical options to let the driver check if an object of the correct mode exists in the environment, and use its value when that is the case. Trying to do the same for character options would prove impossible, or at least very risky, since the driver could not tell apart intended values of options from object names. Given that the vast majority of the options of the <code>RweaveLatex</code> driver are either logical or numerical, we did not deem necessary to devise a scheme to extend the feature to character options.</p>
</section>
<section id="unparsable-code-in-code-chunks" class="level1">
<h1>Unparsable code in code chunks</h1>
<p>Automatic parsing of code chunks is a nifty Sweave feature: it ensures that all code in a document is at least syntactically valid. However, sometimes unparsable code is a feature, not a bug. For example, an instructor may wish to include erroneous R code in a chunk for didactic purposes, yet distribute the tangled code uncommented such that learners may nevertheless evaluate it.</p>
<p>To cope with such situations, the <code>RweaveLatex</code> driver has gained options to completely ignore code chunks on weaving. Chunks using <code>ignore.on.weave=TRUE</code> are not parsed on weaving, yet remain written out verbatim on tangling. A shorter alias <code>weave</code> for <code>!ignore.on.weave</code> is also supported. An option <code>ignore</code> sets at once <code>ignore.on.weave</code> and the equivalent option for the <code>Rtangle</code> driver (see below).</p>
<p>The following example illustrates the use of the new option to maintain a document that evaluates without error on weaving, yet produces an uncommented script on tangling.</p>
<pre><code>&lt;&lt;&gt;&gt;=
## Some R comparison operators.
486 &lt; 521                  # smaller than
486 &gt;= 521                 # greater or equal
486 != 521                 # different from
@
&lt;&lt;ignore.on.weave=TRUE&gt;&gt;=
## Watch out! Equality is not tested with '='.
5 = 2                      # syntax error
@
&lt;&lt;&gt;&gt;=
## The equality comparison operator is '=='.
486 == 486                 # equality
@</code></pre>
<p>We will cover another interesting use of this option in the following section.</p>
</section>
<section id="finer-control-of-the-tangling-process" class="level1">
<h1>Finer control of the tangling process</h1>
<p>New options for the <code>Rtangle</code> driver address the last four motivations mentioned earlier, thereby giving users finer control on the tangling process.</p>
<p>First, <code>Rtangle</code> also gains an option <code>ignore.on.tangle</code> to completely omit a code chunk on tangling, irrespective of the value of option <code>eval</code>. The driver also supports the shorter alias <code>tangle</code> for <code>!ignore.on.tangle</code>, and the option <code>ignore</code> to set both <code>ignore.on.weave</code> and <code>ignore.on.tangle</code> at once. The possibility to skip a chunk on tangling is convenient to avoid cluttering the tangled script with unimportant code or, when using <code>split=TRUE</code>, the file system with unneeded scripts.</p>
<p>Second, the new option <code>chunk.sep</code> allows to specify the separator between code chunks when they are collected in a single file. The historic default of two blank lines is maintained. One may also use <code>chunk.sep=FALSE</code> to completely omit the separator, either throughout a script when used as an argument to the <code>Rtangle</code> call, or between two chunks when used as a chunk option. It is also worth noting that the chunk separator is now added <em>before</em> all chunks, except the first one. Therefore, there is no longer a chunk separator at the end of tangled scripts.</p>
<p>Finally, <code>Rtangle</code> now provides an option <code>extension</code> to specify the extension, without the leading dot, for the file name of a tangled code chunk when splitting is selected. If used as <code>extension=TRUE</code>, the default extension (usually <code>R</code>) is used. If the option is <code>FALSE</code>, no extension is added to the file name. The option <code>extension</code> is specially useful with <code>ignore.on.weave=TRUE</code> of <code>RweaveLatex</code> to include code or text that the engine would not be able to parse on weaving, yet that needs to be tangled to a file with a specific extension, such as <code>.sh</code> or <code>.txt</code>.</p>
<p>Consider a file <code>example.Rnw</code> that contains the following code chunks:</p>
<pre><code>&lt;&lt;script, echo=TRUE&gt;&gt;=
1:10
@

&lt;&lt;echo=FALSE, ignore.on.tangle=TRUE&gt;&gt;=
x &lt;- 42
@

&lt;&lt;script&gt;&gt;=
gamma(1:5)                 # gamma function
factorial(0:4)             # factorial
@
&lt;&lt;script, ignore.on.weave=TRUE, chunk.sep=FALSE&gt;&gt;=
?gamma                     # all related functions
@

&lt;&lt;hello, extension=sh, ignore.on.weave=TRUE&gt;&gt;=
#!/bin/sh

echo "Hello, World!"

exit 0
@

&lt;&lt;README, extension=FALSE, ignore.on.weave=TRUE&gt;&gt;=
Hello, World!
@</code></pre>
<p>We use <code>ignore.on.weave=TRUE</code> in the fourth chunk to hide the help system call from the weaving process. Now, tangling the file with the following expression will create three files: <code>script.R</code>, <code>hello.sh</code>, and <code>README</code>.</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Stangle</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"example.Rnw"</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">split =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">prefix =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>,</span>
<span id="cb4-2">         <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">annotate =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">chunk.sep =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">\n</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"</span>)</span></code></pre></div></div>
</div>
<p>The file <code>script.R</code> is shown below. Notice that: the expression <code>x &lt;- 42</code> is omitted from the script; the help system call is present and uncommented; the two main blocks of code are separated by a single blank line; what were originally two code chunks became only one block; there are no blank lines at the end of the file.</p>
<pre><code>--- script.R ---
1:10

gamma(1:5)                 # gamma function
factorial(0:4)             # factorial
?gamma                     # all related functions</code></pre>
<p>As for <code>hello.sh</code> and <code>README</code>, shown below, the fact that the files exist with an extension other than <code>.R</code>, and that their contents is <em>not</em> R code, illustrates how one may use the options <code>extension</code> and <code>ignore.on.weave</code> to maintain various pieces of code or text inside a single source file.</p>
<pre><code>--- hello.sh ---
#!/bin/sh

echo "Hello, World!"

exit 0</code></pre>
<pre><code>--- README ---
Hello, World!</code></pre>
</section>
<section id="obtaining-the-name-of-the-processed-file" class="level1">
<h1>Obtaining the name of the processed file</h1>
<p>Weaving and tangling are often seen as separate, somewhat orthogonal procedures. However, one may devise various schemes where both weaving <em>and</em> tangling are needed to create a document. Here are a few examples:</p>
<ul>
<li>maintaining documentation and code together even though the code is not directly used to create the text;</li>
<li>code that needs to be included verbatim in a document, but not necessarily as or where it appears in the sources;</li>
<li>pieces of code that rely on other pieces being saved as files.</li>
</ul>
<p>Vincent (the first author) proposes in a <a href="https://doi.org/10.47397/tb/47-1/tb145goulet-stangle">TUGboat article</a> a nice strategy to achieve this: embedding a tangling procedure <em>inside</em> a weaving procedure by calling <code>Stangle</code> in a code chunk. Now, this requires the name of the file to process… which is the file being processed. To avoid hard coding the file name in the <code>Stangle</code> call, we also introduced in the Sweave framework the function <code>SweaveGetSourceName</code> that returns the name of the file being woven by an <code>Sweave</code> process launched from the command line. This allows to start <code>Stangle</code> inside <code>Sweave</code> with the a code chunk such as:</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb8-1"><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span><span class="er" style="color: #AD0000;
background-color: null;
font-style: inherit;">&lt;</span>echo<span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span><span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>, results<span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span>hide, ignore.on.tangle<span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">=</span><span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&gt;</span><span class="er" style="color: #AD0000;
background-color: null;
font-style: inherit;">&gt;=</span></span>
<span id="cb8-2">FILE <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">SweaveGetSourceName</span>()</span>
<span id="cb8-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Stangle</span>(FILE)</span>
<span id="cb8-4"><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">@</span></span></code></pre></div></div>
</div>
<p>The function works by parsing the output of <code>commandArgs</code>, and supports weaving processes started by means of either <code>R CMD Sweave</code>, <code>R -e</code>, or <code>Rscript -e</code>.</p>


</section>

 ]]></description>
  <category>documentation</category>
  <category>Sweave</category>
  <guid>https://blog.R-project.org/posts/new-tricks-sweave-drivers/</guid>
  <pubDate>Fri, 18 Sep 2026 00:00:00 GMT</pubDate>
  <media:content url="https://blog.R-project.org/posts/new-tricks-sweave-drivers/sweave-tricks.png" medium="image" type="image/png" height="107" width="144"/>
</item>
<item>
  <title>Rousseeuw Prize 2026 Awarded to the R Project</title>
  <dc:creator>Martyn Plummer</dc:creator>
  <link>https://blog.R-project.org/posts/rousseeuw-prize/</link>
  <description><![CDATA[ 





<p>The <a href="https://rousseeuwprize.org">Rousseeuw Prize for Statistics</a> is a biennial prize that awards pioneering work in statistical methodology. The prize recognizes a statistical innovation, that is, an outstanding contribution or tool that has had significant impact and found wide application in statistical practice.</p>
<p>Five members of the R Core Team have been awarded the Rousseeuw Prize for Statistics for their decades of work building and maintaining the R Project. The 2026 laureates are:</p>
<ul>
<li>Prof.&nbsp;Brian Ripley, University of Oxford, United Kingdom</li>
<li>Prof.&nbsp;Martin Maechler, ETH Zurich, Switzerland</li>
<li>Prof.&nbsp;Kurt Hornik, Vienna University of Economics and Business, Austria</li>
<li>Prof.&nbsp;Peter Dalgaard, Copenhagen Business School, Denmark</li>
<li>Prof.&nbsp;Luke Tierney, University of Iowa, United States</li>
</ul>
<p>Half of the prize money goes to the five laureates because they are deemed to have made the longest sustained contributions, and half goes to the other members of the R Core Team.</p>
<p>The <a href="https://rousseeuwprize.org/2026">prize announcement</a> explains the choice of the R Project:</p>
<blockquote class="blockquote">
<p>It is through the combination of providing a high quality base system, and mechanisms and infrastructure for extending this base system, that R has become the common language of statistics and data science. Methodological innovations in modern statistics are typically obtained using R and freely provided as R extension packages, making them accessible to everyone. By keeping R free and open-source under the GNU General Public License, the R Core Team removed many of the financial barriers that have historically limited access to advanced analytics software. Due to this increased accessibility, millions of users including researchers, students, hospitals, public health organizations, and governments around the world are able to utilize the same statistical tools regardless of institutional resources, also in developing countries. Their work has transformed statistics and data science from many isolated workers writing long programs into an enthusiastic collaborative community where ideas and code are easily shared and built upon.</p>
</blockquote>
<blockquote class="blockquote">
<p>The efforts of the R Core Team members have been driven by the vision that everybody should have free access to the state-of-the-art of statistical methods, and therefore be able to perform better data analysis and decision making. This involves spending huge amounts of time on the ongoing work of adapting R to evolving hardware and software environments. The fact that the main R Core Team reference has been cited over 343,000 times on Semantic Scholar, and the quantity and quality of available extension packages shows that the time investment of the R Core Team has clearly paid off, with huge benefits for the wide community of users.</p>
</blockquote>



 ]]></description>
  <category>R Foundation</category>
  <category>R Core</category>
  <guid>https://blog.R-project.org/posts/rousseeuw-prize/</guid>
  <pubDate>Wed, 17 Jun 2026 00:00:00 GMT</pubDate>
  <media:content url="https://blog.R-project.org/posts/rousseeuw-prize/rousseeuwprize.png" medium="image" type="image/png" height="144" width="144"/>
</item>
<item>
  <title>Sovereign Tech Fund Project: Milestone 1</title>
  <dc:creator>Simon Urbanek, Tomáš Kalibera</dc:creator>
  <link>https://blog.R-project.org/posts/stf-project-milestone-1/</link>
  <description><![CDATA[ 





<section id="sovereign-tech-fund-stf-project" class="level2">
<h2 class="anchored" data-anchor-id="sovereign-tech-fund-stf-project">Sovereign Tech Fund (STF) Project</h2>
<p>Starting in mid 2025, the R Project has been working on a series of work packages to improve the long-term stability of the R ecosystem. The project is supported by an investment from the <a href="https://www.sovereign.tech/">Sovereign Tech Agency</a>, under their <a href="https://www.sovereign.tech/tech/r-project">Sovereign Tech Fund</a>. In a series of blog posts, some details about the achieved results will be shared with the public, starting with Milestone 1. These blog posts are a tribute to the late <a href="https://www.r-project.org/doc/obit/tomas.html">Tomáš Kalibera</a>, who continued working on this project during his last months.</p>
</section>
<section id="milestone-1-triage-and-reduce-backlog-of-bugs" class="level2">
<h2 class="anchored" data-anchor-id="milestone-1-triage-and-reduce-backlog-of-bugs">Milestone 1: Triage and reduce backlog of bugs</h2>
<p>At the beginning of the project, base R had over <img src="https://latex.codecogs.com/png.latex?600"> open bugs in its issue tracking system, a Bugzilla instance hosted at <a href="https://bugs.r-project.org" class="uri">https://bugs.r-project.org</a>. The work for this milestone involved systematically analysing the backlog of bug reports: identifying those that were not relevant and could be closed; researching solutions for significant bugs, and implementing fixes that resolved some of those issues. The quality of the codebase was thus improved, and the visibility of significant bugs in the bug tracker was increased, easing future bug management for the maintainers of R.</p>
</section>
<section id="milestone-results" class="level2">
<h2 class="anchored" data-anchor-id="milestone-results">Milestone Results</h2>
<p>On July 31, 2025, there were <img src="https://latex.codecogs.com/png.latex?624"> open bug reports in the R Project Bugzilla. This number was reduced to <img src="https://latex.codecogs.com/png.latex?467"> open bug reports on October 1, 2025, which is a reduction of about <img src="https://latex.codecogs.com/png.latex?25%5C%25">. Bugs no longer relevant or not reproducible/actionable were closed with a short explanation. Invalid bugs and issues not feasible to fix/not worth fixing, as well as wishlist items not worth adding, were closed with a detailed explanation, so that the people reporting the bugs would not be discouraged from further contribution, but instead learn how to contribute better. Some remaining relevant bugs were fixed, including some bugs that were many years old. In addition, comments were added to some (still) open bugs, in an attempt to get further information from the reporters or expert comments, increasing the chances they could be resolved, soon. This effort has been noticed by package developers who reported some of the bugs and positively reflected on the R-devel mailing list:</p>
<blockquote class="blockquote">
<p><em>“Maybe others will share in my astonishment seeing that Tomas Kalibera has since Aug 1 closed ~80 bugs on R’s Bugzilla, in all cases providing quite thoughtful and instructive explanation.”</em><br>
- Mikael Jagan, full post: <a href="https://stat.ethz.ch/pipermail/r-devel/2025-August/084155.html" class="uri">https://stat.ethz.ch/pipermail/r-devel/2025-August/084155.html</a></p>
</blockquote>
<blockquote class="blockquote">
<p><em>“I too share the positive feelings about this effort. The detailed explanations clearly helps me learn, regardless of the fact that sometimes the answer is not the desired one.”</em><br>
- Lluís Revilla, full post <a href="https://stat.ethz.ch/pipermail/r-devel/2025-August/084157.html" class="uri">https://stat.ethz.ch/pipermail/r-devel/2025-August/084157.html</a></p>
</blockquote>
<p>The results of the work are publicly available on <a href="https://bugs.r-project.org" class="uri">https://bugs.r-project.org</a>, where the advanced search feature allows to focus on specific activity in the two months allocated for Milestone 1, the result is available as <a href="https://bugs.r-project.org/buglist.cgi?chfield=resolution&amp;chfieldfrom=2025-08-01&amp;chfieldto=2025-09-30&amp;order=Last%20Changed&amp;product=R&amp;query_format=advanced" class="uri">https://bugs.r-project.org/buglist.cgi?chfield=resolution&amp;chfieldfrom=2025-08-01&amp;chfieldto=2025-09-30&amp;order=Last%20Changed&amp;product=R&amp;query_format=advanced</a>.</p>


</section>

 ]]></description>
  <category>R Core</category>
  <category>community</category>
  <category>testing</category>
  <guid>https://blog.R-project.org/posts/stf-project-milestone-1/</guid>
  <pubDate>Tue, 02 Jun 2026 00:00:00 GMT</pubDate>
  <media:content url="https://blog.R-project.org/posts/stf-project-milestone-1/stf.png" medium="image" type="image/png" height="92" width="144"/>
</item>
<item>
  <title>S at 50</title>
  <dc:creator>Martyn Plummer</dc:creator>
  <link>https://blog.R-project.org/posts/s-at-50/</link>
  <description><![CDATA[ 





<p>Today marks 50 years since Rick Becker and John Chambers presented a new project to a few of their colleagues in Statistics and Data Analysis research at Bell Labs. John writes:</p>
<blockquote class="blockquote">
<p>It had no name and we were unsure what it was, exactly. A few months later it was named S, initially called an interactive environment. Only a revised version was referred to as a language. (Lately I’ve been labelling it R Version 0.)</p>
</blockquote>
<p>A poster by Balasubramanian Narasimhan summarizes 50 years of S and R. You can <a href="s_at_50_poster.jpg">download</a> the full size poster or open up the image below in a new tab to see it full size.</p>
<p><img src="https://blog.R-project.org/posts/s-at-50/s_at_50_poster.jpg" width="540" height="720"></p>



 ]]></description>
  <category>R Foundation</category>
  <category>R Core</category>
  <guid>https://blog.R-project.org/posts/s-at-50/</guid>
  <pubDate>Tue, 05 May 2026 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Debugging Sensitivity to C Math Library on Linux</title>
  <dc:creator>Tomáš Kalibera</dc:creator>
  <link>https://blog.R-project.org/posts/mfuzz/</link>
  <description><![CDATA[ 





<p>C math library functions, such as <code>exp</code> or <code>sin</code>, are not guaranteed to be “precise”. The results might be slightly different on different platforms. A recent change in mingw-w64 v12, which is a core dependency of compilers we use on Windows (both GCC and LLVM), resulted in failures in about 20 CRAN packages. Rtools45 uses mingw-w64 v11, the same version as Rtools44, to allow time for resolving the issues. The change in mingw-w64 v12 switched about 90 math functions from internal implementations to UCRT, the Windows C runtime provided by Microsoft and shipped with the OS. The goal of the change was improving performance and reducing maintenance cost of mingw-w64.</p>
<p>In principle, this is not only a problem on Windows. It is not hard to come across R package tests, which test that numerical results of some computation is within bounds, when the bounds were apparently established experimentally on several platforms at hand at the time the tests were written. These tests then commonly start failing on a new CPU or with a new compiler, and they may be difficult to debug, especially to a person who is not a maintainer of the package.</p>
<p>Section <a href="https://cran.r-project.org/doc/manuals/r-devel/R-exts.html#Writing-portable-packages">1.6 Writing portable packages</a> of <a href="https://cran.r-project.org/doc/manuals/r-devel/R-exts.html">Writing R Extensions</a> has more details in th paragraph stating with:</p>
<blockquote class="blockquote">
Only test the accuracy of results if you have done a formal error analysis. Things such as checking that probabilities numerically sum to one are silly: numerical tests should always have a tolerance. That the tests on your platform achieve a particular tolerance says little about other platforms. R is configured by default to make use
</blockquote>
<p>This blog post presents the <a href="https://www.github.com/kalibera/mfuzz">mfuzz</a> tool for Linux, which fuzzes the results from C Math functions. So if establishing tolerance intervals empirically, despite the suggestion cited above, one might use the tool to get somewhat wider intervals. As of this writing, the tool fuzzes the results deterministically, adding 1 ULP (unit of least precision) towards positive infinity, but if that were to change the sign of the result, then away from zero.</p>
<p>The tool tries to be reasonable about fuzzing and tries to preserve certain properties which would likely hold on real C Math library implementations and which hold in Mathematics, such as that <code>sin(0) == 0, cos(0) == 1</code> (exactly), etc. In principle, rounding functions are also C Math library functions, but they are indeed not fuzzed - one would normally only fuzz transcendental functions (via environment variable <code>MFUZZ_FUZZ=trans</code>). Making the tool sufficiently reasonable in what is fuzzed is joint work with Martin Maechler from R Core.</p>
<p>In principle, the tool could be extended to make the fuzzing somewhat random, or bigger for some functions and some intervals. The 1 ULP fuzzing has been motivated by that all over-sensitivity bugs found in R CRAN packages when debugging issues with mingw-w64 v12 were caused by differences in numerical results of only 1 ULP.</p>
<p>Interested readers are referred to previous blog posts on the topic from<a href="../../2026/01/06/debugging-sensitivity-to-c-math-library-and-mingw-w64-v12/">January</a> from <a href="../../2025/04/24/sensitivity-to-c-math-library-and-mingw-w64-v12">April</a> and <a href="../../2025/05/21/sensitivity-to-c-math-library-and-mingw-w64-v12-part-2">May</a>.</p>
<section id="wrapping-math-library-calls" class="level2">
<h2 class="anchored" data-anchor-id="wrapping-math-library-calls">Wrapping Math library calls</h2>
<p>The tool (<a href="https://www.github.com/kalibera/mfuzz">mfuzz</a>) implements configurable wrappers most of the C Math library functions. These are built into a shared library, which is then provided to the dynamic linker via <code>LD_PRELOAD</code> and is used in precedence of the system C Math library. The wrappers in <code>wrapppers.c</code> are generated by an R script <code>genwrappers.r</code>, so it is best to modify the script when changes are needed.</p>
<p>Unlike the <a href="https://www.github.com/kalibera/mdebug">mdebug</a> tool, which is for Windows, one doesn’t have to rebuild R to use <a href="https://www.github.com/kalibera/mfuzz">mfuzz</a>, which is for Linux.</p>
</section>
<section id="example" class="level2">
<h2 class="anchored" data-anchor-id="example">Example</h2>
<p>Let’s have this R test for a demonstration how the tool can be used:</p>
<pre><code>x &lt;- -1e4:1e4
y &lt;- sin(x)^2 + cos(x)^2
stopifnot(max(1-y) &lt;= .Machine$double.eps)
</code></pre>
<p>This is an example of an unreasonable test to be in an R package, because it tests the R math functions only without any relevant added value/computation, and because the tolerance limit is arbitrary. Still, the test passes on my Linux, Windows 11 and macOS systems, but it fails on Linux with <code>mfuzz</code>:</p>
<pre><code>env LD_PRELOAD=~/mfuzz/libmfuzz.so MFUZZ_FUZZ=trans ./bin/R
[...]
&gt; x &lt;- -1e4:1e4
y &lt;- sin(x)^2 + cos(x)^2
stopifnot(max(1-y) &lt;= .Machine$double.eps)
Error: max(1 - y) &lt;= .Machine$double.eps is not TRUE
&gt; </code></pre>
<p>Lets check which Math functions were called and fuzzed in the execution:</p>
<pre><code>env LD_PRELOAD=~/mfuzz/libmfuzz.so MFUZZ_FUZZ=trans MFUZZ_LOG=trans MFUZZ_LOGFILE=mylog.log ./bin/R</code></pre>
<p>The log file looks as follows:</p>
<pre><code>$ head mylog.log
log10 ( 0.000000(0x1p-1022) ) -&gt; -307.652656(-0x1.33a7146f72a41p+8) [originally -307.652656(-0x1.33a7146f72a42p+8)]
log10 ( 60.000000(0x1.ep+5) ) -&gt; 1.778151(0x1.c734eb9bbd4p+0) [originally 1.778151(0x1.c734eb9bbd3ffp+0)]
log ( 6.000000(0x1.8p+2) ) -&gt; 1.791759(0x1.cab0bfa2a2003p+0) [originally 1.791759(0x1.cab0bfa2a2002p+0)]
log10 ( 3.500000(0x1.cp+1) ) -&gt; 0.544068(0x1.1690163290f41p-1) [originally 0.544068(0x1.1690163290f4p-1)]
log10 ( 22.500000(0x1.68p+4) ) -&gt; 1.352183(0x1.5a28a22d82ep+0) [originally 1.352183(0x1.5a28a22d82dffp+0)]
sin ( 10000.000000(0x1.388p+13) ) -&gt; -0.305614(-0x1.38f2fa75d9288p-2) [originally -0.305614(-0x1.38f2fa75d9289p-2)]
sin ( 9999.000000(0x1.3878p+13) ) -&gt; 0.636087(0x1.45ad3086449bep-1) [originally 0.636087(0x1.45ad3086449bdp-1)]
sin ( 9998.000000(0x1.387p+13) ) -&gt; 0.992973(0x1.fc66f13ab2e0dp-1) [originally 0.992973(0x1.fc66f13ab2e0cp-1)]
sin ( 9997.000000(0x1.3868p+13) ) -&gt; 0.436924(0x1.bf6909b0793f7p-2) [originally 0.436924(0x1.bf6909b0793f6p-2)]
sin ( 9996.000000(0x1.386p+13) ) -&gt; -0.520831(-0x1.0aaa510fa1ffep-1) [originally -0.520831(-0x1.0aaa510fa1fffp-1)]</code></pre>
<p>So lets check which Math functions were called and how many times:</p>
<pre><code>$ cut -d' ' -f1 mylog.log | sort | uniq -c | sort -n
      1 log
      4 pow
      6 log10
  20001 cos
  20001 sin</code></pre>
<p>The output is not very surprising. One can easily check via <code>MFUZZ_FUZZ=sin</code> and <code>MFUZZ_FUZZ=cos</code> that fuzzing only one of the functions is indeed enough to make the test fail.</p>
</section>
<section id="summary" class="level2">
<h2 class="anchored" data-anchor-id="summary">Summary</h2>
<p>This tool should help package authors who empirically set tolerances in their tests do some preventative maintenance. If the package normally passes its tests, but fails with <code>mfuzz</code>, the tolerances should be increased correspondingly. It is not guaranteed that doing this is enough to prevent any numerical issues with a new C Math library on a new platform, because the differences may be bigger than what <code>mfuzz</code> tries. Likewise, it is not guaranteed it would be enough for the same code to work on Windows with mingw-w64 v12 or newer.</p>
<p>To test and debug a package with tests of numerical results, it is better to use <a href="https://www.github.com/kalibera/mdebug">mdebug</a>. One can also install an <a href="https://www.r-project.org/nosvn/winutf8/tmp/mw13/">experimental build of Rtools45</a>, which uses mingw-w64 v13, and test with it, but this has to be done with care to avoid polluting the package library on the system with code generated by an experimenral build of Rtools. One would have to rebuild R from source and all dependencies of the package under test. One would have to do that in a fresh library, and then delete the library after the experiment, and uninstall this Rtools45. Still, this experimental build confirmed that most CRAN packages that help back the upgrade to mingw-w64 v12 in Rtools45 due to over-sensitivity to C Math function accurracy, have been fixed.</p>
<p>Probably Rtools46 will use mingw-w64 v13, so the testing will be possible after Rtools46 release, but the testing will also become mandatory, as R-devel (and then R 4.6) will likely use Rtools46.</p>
<p>The mfuzz tool also helped to discover some tolerances in base R that needed to be adjusted.</p>
<p>This work also helped to make it clear that if R wants to provide more reasonable behavior of R math functions than no guarantees on any properties, it needs to implement such properties on its own.</p>
<p>This work has been possible thanks to an investment of the <a href="https://www.sovereign.tech/tech/r-project">Sovereign Tech Fund</a>.</p>


</section>

 ]]></description>
  <category>testing</category>
  <category>numerical</category>
  <guid>https://blog.R-project.org/posts/mfuzz/</guid>
  <pubDate>Fri, 30 Jan 2026 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Debugging Sensitivity to C Math Library and mingw-w64 v12</title>
  <dc:creator>Tomáš Kalibera</dc:creator>
  <link>https://blog.R-project.org/posts/math-sensitivity-mdebug/</link>
  <description><![CDATA[ 





<p>C math library functions, such as <code>exp</code> or <code>sin</code>, are not guaranteed to be “precise”. The results might be slightly different on different platforms. A recent change in mingw-w64 v12, which is a core dependency of compilers we use on Windows (both GCC and LLVM), resulted in failures in about 20 CRAN packages. Rtools45 uses mingw-w64 v11, the same version as Rtools44, to allow time for resolving the issues. The change in mingw-w64 v12 switched about 90 math functions from internal implementations to UCRT, the Windows C runtime provided by Microsoft and shipped with the OS. The goal of the change was improving performance and reducing maintenance cost of mingw-w64.</p>
<p>Interested readers should first read the previous blog posts on the topic from <a href="../../2025/04/24/sensitivity-to-c-math-library-and-mingw-w64-v12">April</a> and <a href="../../2025/05/21/sensitivity-to-c-math-library-and-mingw-w64-v12-part-2">May</a>. This post presents a tool which allows interested package authors to debug any potential issues in their package.</p>
<section id="wrapping-math-library-calls" class="level2">
<h2 class="anchored" data-anchor-id="wrapping-math-library-calls">Wrapping Math library calls</h2>
<p>The tool (<a href="https://www.github.com/kalibera/mdebug">mdebug</a>) implements configurable wrappers for the C Math library functions, which were switched from an internal representation to UCRT in ming-w64 v12. With the tool comes a patch for R, which causes R and R packages to be linked dynamically (not statically) to wrappers of the corresponding Math library functions. The configurable wrappers allow to unconditionally use UCRT (and hence mimic mingw-w64 v12), to always use internal implementations (and hence mimic mingw-w64 v11 - the wrappers themselves are to be built with mingw-w64 v11), or to do something more (some diagnostic, loggging, etc). In principle, one might also provide an implementation that would give correctly rounded results using mpfr (or some other library), but I’ve not done that.</p>
<p>In the previous posts I have mentioned problems found in packages, I wrote to precisions of which Math function they were overly sensitive to, and whether inaccuracies within 1 ULP were enough to trigger the problem. All of this can be found using the tool.</p>
</section>
<section id="limitations" class="level2">
<h2 class="anchored" data-anchor-id="limitations">Limitations</h2>
<p>The tool isn’t very user friendly - it is suited to people who are used to building R from source on Windows and who know the C programming language. In addition to the configuration exposed via environment variables, I’ve sometimes modified the generated C code of the concrete wrappers (e.g.&nbsp;to confirm that the problem was caused only by inaccurate results within some range - I would modify the wrapper to use UCRT when the result was within 1 ULP of the internal code result, or use the internal code, instead).</p>
<p>One should keep in mind, as reported in one of the earlier posts, that UCRT results are OS-specific, they differ between Windows versions.</p>
<p>The tool is only usable on Windows and would not discover over-sensitivity to C Math functions where it doesn’t make the program break with mingw-w64-v12. Also, one might argue - why not simply test with an experimental version of Rtools that would already use mingw-w64 v12. Such testing, would, however require a lot of re-compilations to isolate the potential problems (one would have to keep re-compiling package dependencies, R itself, possibly also some external libraries, etc). Still, it is inevitable that at some point some such testing will have to be done, and then this tool may help narrowing down the causes of any numerical problems found.</p>


</section>

 ]]></description>
  <category>testing</category>
  <category>numerical</category>
  <guid>https://blog.R-project.org/posts/math-sensitivity-mdebug/</guid>
  <pubDate>Tue, 06 Jan 2026 00:00:00 GMT</pubDate>
</item>
<item>
  <title>RSMF: Enabling the Next Generation of Contributors to R</title>
  <dc:creator>Heather Turner</dc:creator>
  <link>https://blog.R-project.org/posts/rsmf-enabling-contributors/</link>
  <description><![CDATA[ 





<p>The Software Sustainability Institute (UK) has awarded a grant of £500k (~650k USD) under the <a href="https://www.software.ac.uk/rsmf-round-1-projects">Research Software Maintenance Fund</a>, to the project <em>Enabling the Next Generation of Contributors to R</em>.</p>
<p>The project is co-led by research software engineers at the University of Birmingham (<a href="https://www.linkedin.com/in/heathrturnr">Heather Turner</a> and <a href="https://ellakaye.co.uk/about.html">Ella Kaye</a>), members of the R Core Team (<a href="https://profiles.auckland.ac.nz/s-urbanek">Simon Urbanek</a> and <a href="https://www.cbs.dk/en/research/department-finance/peter-dalgaard">Peter Dalgaard</a>), and a team of academic/freelance R developers (<a href="https://github.com/gmbecker">Gabe Becker</a>, <a href="https://www.khoury.northeastern.edu/people/kylie-ariel-bemis/">Kylie Bemis</a>, <a href="https://github.com/jaganmn">Mikael Jagan</a>, <a href="https://github.com/jeroen">Jeroen Ooms</a>). The project will receive further support from the <a href="https://r-consortium.org/">R Consortium</a> (developer events and travel support), the <a href="https://www.r-project.org/foundation/">R Foundation</a> (travel support for core developers), and several companies dedicating staff time: <a href="https://posit.co">Posit</a>, <a href="https://a2-ai.com/">A2-Ai</a>, and <a href="https://about.google/">Google</a>.</p>
<p>The overall aim is to address the challenge of sustaining the R Project. This will be achieved through work on four areas described below.</p>
<section id="establishing-a-cohort-of-expert-contributors" class="level2">
<h2 class="anchored" data-anchor-id="establishing-a-cohort-of-expert-contributors">Establishing a Cohort of Expert Contributors</h2>
<p>A cohort of expert developers (project co-leads: Gabe Becker, Kylie Bemis, Mikael Jagan, Jeroen Ooms; partner-sponsored developers: <a href="https://www.linkedin.com/in/kevin-ushey-33a35542/">Kevin Ushey</a> and the tidyverse team [Posit], <a href="https://www.linkedin.com/in/cgmossa">Mossa Reimert</a> [A2-Ai], <a href="https://www.linkedin.com/in/michael-chirico-9b6b0527/">Mike Chirico</a> [Google]) will be mentored by members of the R Core Team (including Simon Urbanek, Peter Dalgaard, <a href="https://www.stat.auckland.ac.nz/~paul/">Paul Murrell</a>, <a href="https://www.isid.ac.in/~deepayan/">Deepayan Sarkar</a>, <a href="https://stat.uiowa.edu/people/luke-tierney">Luke Tierney</a> and <a href="https://www.linkedin.com/in/michael-lawrence-74a9b482/">Michael Lawrence</a>). They will work on projects supporting the development or maintenance of R, as well as helping to triage and fix bugs.</p>
<p>Example projects include broader use of GitHub actions, using generative AI to enhance the CRAN checks, and improved testing of R-devel.</p>
</section>
<section id="supporting-broader-engagement" class="level2">
<h2 class="anchored" data-anchor-id="supporting-broader-engagement">Supporting Broader Engagement</h2>
<p>Heather Turner and Ella Kaye will lead activities to support broader engagement from both new and experienced contributors. This includes developing a code of conduct for the R Project, exploring alternative platforms for code development that provide an easier on-ramp for contributors, and piloting a process for discussion of enhancement proposals. The team will sustain recent initiatives of the R Contribution Working Group: organizing outreach events such as <a href="https://contributor.r-project.org/events/r-dev-days/">R Dev Days</a> and maintaining documentation and infrastructure created to facilitate contribution.</p>
<p>The Brazil-based consultancy, <a href="https://www.rseis.com.br/">R6 Estatística e Treinamentos</a>, will work to improve the integration of translations contributed via Weblate, mentored by the current Weblate admins Mike Chirico and <a href="https://economics.ceu.edu/people/gergely-daroczi">Gergely Daróczi</a>.</p>
</section>
<section id="improving-communication" class="level2">
<h2 class="anchored" data-anchor-id="improving-communication">Improving Communication</h2>
<p>The third area of focus will be to improve communication, especially between the R Core Team and the wider community. This will involve adopting modern open source tools to facilitate discussions related to R contribution and expanding the use of the R Blog and R Foundation social media accounts to engage R users in the process of R development.</p>
</section>
<section id="sustaining-the-r-foundation" class="level2">
<h2 class="anchored" data-anchor-id="sustaining-the-r-foundation">Sustaining the R Foundation</h2>
<p>Finally, the project will address sustainability of the R Foundation, the non-profit behind R. On the governance side, this will involve proposing a new organisational structure and modernising election procedures to encourage wider participation. On the financial side, work will focus on diversifying income streams. This is important because the RSMF grant, similar to the <a href="https://www.sovereign.tech/tech/r-project">recent investment in the R Project by the Sovereign Tech Agency</a> recently received by the R Foundation, is only intended for short-term funding so there is a continued need to raise funds for long-term sustainability.</p>
</section>
<section id="acknowledgements" class="level2">
<h2 class="anchored" data-anchor-id="acknowledgements">Acknowledgements</h2>
<p>The University of Birmingham is providing institutional support for this project. <a href="https://www.birmingham.ac.uk/staff/profiles/computer-science/academic-staff/van-moorsel-aad">Aad van Moorsel</a>, Head of the School of Computer Science, is providing overall leadership, while <a href="https://www.birmingham.ac.uk/research/arc/rsg/staff/adrian-garcia">Adrian Garcia</a>, Research Software Engineering Service Lead, is providing managerial support.</p>
<p>The Software Sustainability Institute is supported by UK Research and Innovation (UKRI), a non-departmental public body sponsored by the UK government.</p>
<p>Recent initiatives of the R Contribution Working Group have been largely supported by Heather Turner’s research software engineering fellowship from the UK Engineering and Physical Sciences Research Council, EP/V052128/1.</p>
</section>
<section id="links" class="level2">
<h2 class="anchored" data-anchor-id="links">Links</h2>
<ul>
<li>Research Software Maintenance Fund Round 1 Results: <a href="https://www.software.ac.uk/ssi-awards-funding-13-critical-projects-through-research-software-maintenance-fund-round-1">https://www.software.ac.uk/ssi-awards-funding-13-critical-projects-through-research-software-maintenance-fund-round-1</a><br>
</li>
<li>R Contributors Subsite: <a href="https://contributor.r-project.org/">https://contributor.r-project.org/</a><br>
</li>
<li>R Foundation Donations: <a href="https://www.r-project.org/foundation/donations.html">https://www.r-project.org/foundation/donations.html</a></li>
</ul>


</section>

 ]]></description>
  <category>R Foundation</category>
  <category>R Core</category>
  <category>community</category>
  <guid>https://blog.R-project.org/posts/rsmf-enabling-contributors/</guid>
  <pubDate>Wed, 17 Dec 2025 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Reducing the Backlog of Open Bug Reports</title>
  <dc:creator>Tomas Kalibera</dc:creator>
  <link>https://blog.R-project.org/posts/reducing-bug-backlog/</link>
  <description><![CDATA[ 





<p>The number of open bug reports in the <a href="https://bugs.r-project.org/">R bug tracker</a> has been reduced by about 25% during August and September this year. This work has been possible thanks to an investment of the <a href="https://www.sovereign.tech/tech/r-project">Sovereign Tech Fund</a>.</p>
<p><img src="https://blog.R-project.org/posts/reducing-bug-backlog/open.png" class="img-fluid"></p>
<p>Urgent bug reports that are easily reproducible, ideally coming with a small reproducible example, are typically resolved quickly. Other bugs, however, can remain open for a long time. As the backlog grows in size, it becomes hard to navigate, even though it still contains some real bugs experienced by users. Larger is the backlog, harder it is for core developers to find the real bugs. Also, all genuine (non-spam) reports probably deserve a response, even if a negative response, and particularly those that involved non-trivial effort from the reporter.</p>
<p>The primary goal of this work was a significant reduction of the backlog, so initially it focused on reports that were easiest to resolve. This included real, reproducible bugs that were easy to fix or came with patches easy to review.</p>
<p>Also, some reports were closed without any fix: reports not reproducible and no longer relevant, invalid reports, requests for changes not desirable (or not in line with the documented behavior, breaking back compatibility, etc). The goal was to always provide a clear explanation why a report was closed, also to help reporters focusing their energy in future reports. While no bug report was closed simply because of being old or inactive, it was a factor that contributed to the decisions (here I mean inactivity of years or many years; some projects close bug reports automatically due to inactivity for much less than that).</p>
<p>The secondary goal was to also fix or at least analyze some real bugs not necessarily easy to fix. In some cases this lead to new comments in bugzilla, in some cases I’ve alerted core developers who have better expertise for the reports.</p>
<p>Six years ago, there was a <a href="../../2019/10/09/r-can-use-your-help-reviewing-bug-reports">call for external help</a> with reviewing bug reports. The advice provided to potential external volunteers still applies. I would add that it also helps reviewing comments in the bug reports: when a bug report discussion ends with a technically invalid comment, it can mislead people trying to navigate the bug backlog. It helps when that is corrected.</p>
<p>The plot (thanks to Sebastian Meyer) shows the number of open bugs and number of comments in the R bug tracker from October 1, 2019 to September 30, 2025. It is a variant of the plot in <a href="https://journal.r-project.org/news/RJ-2024-2-rcore/">Kalibera &amp; Meyer, “Changes in R”, The R Journal, 2024</a> and such summaries provided in the previous years.</p>
<p>The plot hence shows the number of open bug reports during the last 6 years, starting with increased bug-fixing activity supported by help of external volunteers in 2019 and ending with this work in 2025.</p>
<p>One of my observations based on this work is that it is surprisingly difficult to find a report one can help with. It is hard to tell just from a quick look at each report, so one can easily end up spending more time reading and analyzing reports one cannot help with in the end. This is another consequence of having a large backlog which is harder to navigate, and this work reduced the size of the problem, yet over 400 open reports still seems more than ideal.</p>
<p>Perhaps unsurprisingly, some of the re-discovered very old bug reports were about bugs still present and reproducible in the current version of R.</p>
<p>During this work, I experienced again that on Windows, where CRAN provides binary distributions of R, and where back-compatibility is regarded as very important by the OS, it is possible today to still run an R version more than 10 years old (that is, R using base and recommended packages only).</p>
<p>Another observation was that the classification of the bug reports often seemed wrong, while at the same time it was clear that in some cases, it is often a matter of opinion how to classify. Certainly the classification wasn’t very helpful for guiding the analysis of the bug backlog. It is not surprising that the reporter and R core developers tend to have different views on importance of a bug, but some wishlist items were classified as bugs, and I’ve even seen a real, obvious bug being classified as a wishlist item.</p>
<p>This work is recorded in the <a href="https://bugs.r-project.org/">R bug tracker</a>. It can be viewed via the web interface, for example via “Advanced search”, and for computing on the bugs, one can use the Bugzilla API.</p>



 ]]></description>
  <category>R Core</category>
  <category>community</category>
  <category>testing</category>
  <guid>https://blog.R-project.org/posts/reducing-bug-backlog/</guid>
  <pubDate>Mon, 13 Oct 2025 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Sensitivity to C Math Library and mingw-w64 v12 - Part 2</title>
  <dc:creator>Tomas Kalibera</dc:creator>
  <link>https://blog.R-project.org/posts/math-sensitivity-2/</link>
  <description><![CDATA[ 





<p>C math library functions, such as <code>exp</code> or <code>sin</code>, are not guaranteed to be “precise”. The results might be slightly different on different platforms. A recent change in mingw-w64 v12, which is a core dependency of compilers we use on Windows (both GCC and LLVM), resulted in failures in about 20 CRAN packages. Rtools45 uses mingw-w64 v11, the same version as Rtools44, to allow time for resolving the issues. The change in mingw-w64 v12 switched about 90 math functions from internal implementations to UCRT, the Windows C runtime provided by Microsoft and shipped with the OS. The goal of the change was improving performance and reducing maintenance cost of mingw-w64.</p>
<p>Interested readers should first read the previous <a href="../../2025/04/24/sensitivity-to-c-math-library-and-mingw-w64-v12">blog post on the topic</a>. This post presents additional observations.</p>
<section id="results-differ-between-windows-versions" class="level2">
<h2 class="anchored" data-anchor-id="results-differ-between-windows-versions">Results differ between Windows versions</h2>
<p>Four CRAN packages failed on (two independent) CRAN servers, both running the latest version of Windows Server 2022, but they passed their tests on a Windows 10 system. It turned out that the failures were due to math functions returning different values on Windows Server 2022 compared to Windows 10.</p>
<p>For example, <code>exp(-0x1.7411ffffffff6p+9)</code> was exact zero on a Windows Server 2022 machine, but <code>0x0.0000000000001p-1022</code> on Windows 10. The latter is a correctly rounded result according to mpfr</p>
<pre><code>&gt; library("Rmpfr")
&gt; sprintf("%a", as.double(exp(mpfr(-0x1.7411ffffffff6p+9, precBits=53))))
[1] "0x1p-1074</code></pre>
<p>and was returned with mingw-w64 v11 (the R function exp() in this case simply calls the corresponding C function). The result is a subnormal number very close to zero, so tests/algorithms shouldn’t be sensitive to that difference. They were in case of three of the failing packages, and the issues were reported to the authors.</p>
<p>At least in two cases it was via assumptions about what is the smallest <code>x</code> such that <code>exp(x) &gt; 0</code>. In one case, the result of such numerical computation was about <code>-744.03</code> on Windows Server 2022 and <code>-745.13</code> on Windows 10, which is already very visible difference. The computation corresponds to what <code>uniroot</code> does in base R (see <code>?uniroot</code>, look at the examples) and is very sensitive to the accuracy of <code>exp()</code>. Focusing on the individual computations done when iteratively approximating this number on the two systems, on Windows Server 2022 the smallest subnormal returned by <code>exp</code> was surprisingly <code>0x0.0000000000002p-1022</code>. On Windows 10 it was unsurprisingly <code>0x0.0000000000001p-1022</code>.</p>
<p>I’ve confirmed this difference on systems running Windows natively to rule out any influence of virtualization.</p>
<p>I’ve seen some influence of virtualization. Interestingly, a Windows 10 VM behaving as above started behaving more like Windows Server 2022, returning plain zero in the <code>exp</code> example, in a newer version of virtualbox, and the four packages seen failing on Windows Server 2022 were also failing there. Downgrading back to older virtualbox restored the previous behavior. It might be interesting to follow this hint to find out the true cause for the difference, but for the purpose of testing R packages and preparing them for next mingw-w64, the true cause doesn’t matter: R and packages need to work reliably also on (at least native) systems set up like these.</p>
</section>
<section id="windows-server-2022-ucrt-wasnt-always-less-precise" class="level2">
<h2 class="anchored" data-anchor-id="windows-server-2022-ucrt-wasnt-always-less-precise">Windows Server 2022 UCRT wasn’t always less precise</h2>
<p>My experiments did not suggest that more packages would have been failing with mingw-w64 v12 on Windows Server 2022 than on Windows 10. The full CRAN checks were ran on the server, and then the failing packages were debugged on Windows 10 (when the problem could be reproduced) or on Windows Server 2022 (when it could not be). Running the full CRAN checks on a desktop system would take too long.</p>
<p>As a concrete example to illustrate this, the result of <code>sin(0x1.921f154161812p-1)</code> was <code>0x1.6a097542c0b21p-1</code> on a Windows Server 2022 system, but <code>0x1.6a097542c0b22p-1</code> on a Windows 10 system. The former result is correctly rounded (“precise”) according to mpfr and is also what I get on Linux and with mingw-w64 v11. So the Windows 10 system was (very slightly) imprecise.</p>
</section>
<section id="ucrt-is-not-always-less-precise" class="level2">
<h2 class="anchored" data-anchor-id="ucrt-is-not-always-less-precise">UCRT is not always less precise</h2>
<p>My experiments did not suggest that UCRT would be always less precise than the internal implementations in mingw-w64. I’ve been looking only at newly failing packages with mingw-w64 v12, I’ve been looking at due to which math functions they were failing and how much those functions changed their results. I looked at whether the trivial changes (within 1 ULP, or the smallest possible absolute difference with the subnormals above) were the ones to cause the failure. But I wasn’t looking at possible improvements in precision: I would only find them if they broke some tests, but they didn’t and it would be surprising if they did.</p>
<p>As a concrete example to illustrate this, <code>sin(-0x1.ffffffff36f02p+32)</code> is <code>-0x1.64b688aec8cb4p-3</code> according to mpfr, which is exactly the same value I got with mingw-w64 v12 on Windows 10. But, the internal implementation in mingw-w64 v11 yields <code>-0x1.64b688af2894dp-3</code>, which is 392345 ulps away. This and other extreme differences between <code>sin()</code> in v11 and v12 have been found, but didn’t cause any test failures. Unsurprisingly, as tests are probably typically based also on results observed on Windows at some point. But, yet still, had someone found about this earlier, such big inaccuracy would certainly be worth reporting upstream.</p>
</section>
<section id="summary" class="level2">
<h2 class="anchored" data-anchor-id="summary">Summary</h2>
<p>Readers interested in testing computed results are advised to read also “Writing portable packages” in <a href="https://cran.r-project.org/doc/manuals/r-devel/R-exts.html">Writing R Extensions</a>. Look for paragraph starting with: “Only test the accuracy of results if you have done a formal error analysis.”</p>
<p>When tolerances are established empirically, one should probably at least test on different Windows systems (once Rtools use mingw-w64 newer than v11, assuming the use of UCRT functions remains unconditional there). One should also be prepared to see differences due to virtualization.</p>
<p>UCRT is linked dynamically to R and R packages (when using GCC or LLVM). Hence, any system update in principle could change UCRT and the accuracy of math functions implemented there. This is a new thing: previously (up to v11, so including Rtools45), the internal implementations of math functions in mingw-w64 would be linked statically and be used on all Windows systems.</p>


</section>

 ]]></description>
  <category>numerical</category>
  <category>windows</category>
  <guid>https://blog.R-project.org/posts/math-sensitivity-2/</guid>
  <pubDate>Wed, 21 May 2025 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Sensitivity to C Math Library and mingw-w64 v12</title>
  <dc:creator>Tomas Kalibera</dc:creator>
  <link>https://blog.R-project.org/posts/math-sensitivity/</link>
  <description><![CDATA[ 





<p>C math library functions, such as <code>exp</code> or <code>sin</code>, are heavily used by R and packages. The C standard doesn’t require these functions to be “precise”. Instead, there is room for performance optimizations causing a reasonable amount of inaccuracy. The results differ between platforms and may change even on a single platform. This happened with mingw-w64 v12, the SDK/runtime used by GCC and LLVM on Windows, which reduced accuracy of many math functions (for performance and maintenance reasons).</p>
<p>Around 20 R CRAN packages ended up failing their tests as a result of this and as these issues are hard to debug, Rtools45 ended up using an older version of mingw-w64. Rtools will have to switch to current mingw-w64 eventually, but staying with v11 for now should allow time to resolve the issues.</p>
<p>This text reports on the initial work on debugging these issues and presents several types of problems encountered in packages. While this work is on Windows, the problems in packages are general and potentially affect all platforms: some packages are clearly too sensitive to even small inaccuracies in math library functions.</p>
<section id="debugging-numerical-issues" class="level2">
<h2 class="anchored" data-anchor-id="debugging-numerical-issues">Debugging numerical issues</h2>
<p>This is not the first time I’ve ran into R packages failing their tests due to unexpected numerical results. Debugging these issues is much harder than good old correctness bugs, where one can relatively easily tell what is correct and what is wrong, and accordingly narrow down the test to a case that is small enough to find the cause and to decide which software component is to blame and needs to be fixed.</p>
<p>With the numerical issues, I often end up being told by the package authors that the compiler (or math library, etc) is wrong and that I should report it as a bug. But such a bug report would have to come with solid argumentation that the behavior is wrong, ideally that it violates a standard the code should follow.</p>
<p>Neither the C standard nor any other linked standard require “precise” (correctly rounded) results of math functions. They don’t even require any accuracy limits for them. Therefore, it is not realistic one would succeed reporting inaccuracies to the provider of math functions, unless the inaccuracies are very big (and this observation is based on experience, discussions with mingw-w64 developers).</p>
<p>Reporting a small inaccuracy as a “non-reproducibility” issue is not working, either: outside the R community, the priorities are set differently and performance improvements within what is allowed by the standards are welcome. From a testing (computer science) viewpoint, it may be actually beneficial if systems change in ways that are allowed within the standards, because it increases the chances true bugs are found in the software.</p>
<p>The common accuracy metric for the math library is ULP (“units in the last place”, “units of least precision”), and one usually looks for the maximum difference from the correctly rounded results across the whole domain of a function. The general understanding is that several ULPs difference is fine, but some functions (and perhaps in some domains) are more expensive to get right than other, so the expectations may also depend on the function. <a href="https://members.loria.fr/PZimmermann/papers/accuracy.pdf">[Gladman et al, 2025]</a> has more details, references and measurements.</p>
<p>If a big inaccuracy was found in some math function, one could report it, but indeed one needs to know which function, how inaccurate, and for which inputs. So, the first goal of my debugging was to find out which math functions cause failures of which packages, and how inaccurate these functions really are for the inputs used.</p>
<p>Mingw-w64 provides a static library, <code>mingwex</code>, with internal implementations of some math functions. The remaining ones are used from UCRT, the Windows C runtime. Mingw-w64 v12 simply reduced the amount of functions with internal implementations, about 90 functions were switched to UCRT.</p>
<p>While testing a release candidate of Rtools45, which were to use mingw-w64 v12, 21 CRAN packages were identified as failing their tests because of mingw-w64 v12, and it seemed likely that most of these because of numerical differences (from the outputs of the failed tests, and also from the commit logs of mingw-w64 v12). These 21 CRAN packages required 430 dependent packages (from CRAN and Bioconductor) with a total library size of about 1G when installed.</p>
<p>I’ve made some initial experiments manually, using debuggers, profilers, reading the code of the failing tests. Also I’ve built R and packages with <a href="https://www.netlib.org/fdlibm/">fdlibm</a>, which provides more accurate implementations of some of the functions. I’ve built R selectively with some math functions from fdlibm, then also with some math functions from <code>mingwex</code>. This way I was able to debug some of the packages, but it was taking way too long.</p>
<p>To avoid frequent re-building of that 1G of code and simplify the debugging, I’ve wrapped the math functions (ones switched in v12 to UCRT) by a dynamic library and made all of R and packages use that library. I’ve then experimented with different implementations of that dynamic library, switching between math function implementations and reporting differences, sometimes making custom changes to the dynamic library e.g.&nbsp;to enter a debugger on some special input value.</p>
<p>So far I’ve been able to confirm that 14 of those 21 packages were failing because of the new math functions in v12 and almost always because of their reduced accuracy. On the other hand, looking into the causes revealed that the inaccuracies involved were never above 1 ULP. In 13 cases, the problems were in the packages (not in the library). One case is debatable (perhaps in the compiler, more below).</p>
<p>The rest of the text covers the problems found in packages. I’m using simplified code examples/pseudocode and intentionally not naming the individual packages - that could have been be unfair given that likely many other have the same problems, but just haven’t been discovered by this exercise. Also, the original code would be too long for presentation.</p>
</section>
<section id="requiring-an-exact-numerical-result" class="level2">
<h2 class="anchored" data-anchor-id="requiring-an-exact-numerical-result">Requiring an exact numerical result</h2>
<p>One of the packages made an assumption that could be narrowed down to:</p>
<pre><code>x1 &lt;- -log(-expm1(-0.5))
x2 &lt;- -log1p(-exp(-x1))/0.5
x2 == 1</code></pre>
<p>This used to hold with mingw-w64 v11, but no longer with v12 (due to reduced accuracy of <code>expm1</code>). The difference was in the very last bit:</p>
<pre><code>&gt; sprintf("%a", x2)  # v11
[1] "0x1p+0"
&gt;  sprintf("%a", x2) # v12
[1] "0x1.0000000000001p+0"</code></pre>
<p>And the error in the involved <code>expm1</code> result is 1 ULP. The correct result can be computed using mpfr library, from R using package Rmpfr.</p>
<p>The author of the package knows very well that checks for numerical results should come with a reasonable tolerance limit, but has just forgotten in this case, and the package used to be working for many years on three different platforms. This change in mingw-w64 helped to find this bug.</p>
<p>While this input to <code>expm1</code> lead to 1 ULP error, by random search I found input for which it was 3 ULP. <a href="https://members.loria.fr/PZimmermann/papers/accuracy.pdf">[Gladman et al, 2025]</a> reports 3.06. I learned this is still something regarded as acceptable by mingw-w64 developers. The paper reports inaccuracies also of other UCRT functions (my understanding is that “MSVC” in the paper boils down to UCRT).</p>
<p>Another package used tolerance limits for comparisons of equality, but forgot to use them for less-than/greater-than comparisons of computed results, which has lead to a test failure (due to differences up to 1 ULP in a math function).</p>
</section>
<section id="requiring-a-too-precise-result" class="level2">
<h2 class="anchored" data-anchor-id="requiring-a-too-precise-result">Requiring a too precise result</h2>
<p>Using a too low tolerance in tests was a common problem and not necessarily surprising. A number of packages were failing tests due to unexpected numerical results, caused by changes in <code>atan2</code>, <code>exp</code>, <code>log</code>, <code>pow</code>, <code>sin</code> and <code>sqrt</code>. But, all these changes compared to v11 were within 1 ULP.</p>
<p>Such problem could be caused either by an unrealistically narrow tolerance limit (the tests should be updated), or by a numerically unsound algorithm/implementation (the code is wrong and should be fixed).</p>
<p>At least two of the packages provided different tests (different expectations and/or tolerances) for different platforms, apparently based on what has been observed on those platforms (e.g.&nbsp;differentiating Windows, macOS, Linux, once even looking at which Linux distribution). I think that instead one should have a single set of expectations and tolerances, which should be used on all platforms. If this makes the testing too weak (the limits are too wide), perhaps the algorithms should be re-visited. Or, put the other way, testing on different platforms is a kind of sensitivity analysis that reveals a lower bound on realistic tolerances.</p>
<p>One package allowed very different results on different platforms - not just differences in some low-order digits, but one test have accepted that the same computation on one platform gave ~600 and on another ~1200. A sign of that the algorithm/implementation is wrong.</p>
</section>
<section id="branching-based-on-numerical-result" class="level2">
<h2 class="anchored" data-anchor-id="branching-based-on-numerical-result">Branching based on numerical result</h2>
<p>One package included code with conditions based on comparisons of already computed numerical result. While the package carefully rounded final results before comparing them to expected values to allow for some implementation-specific differences, the branches were simple/exact comparisons. The code with this problem could look like</p>
<pre><code>  x &lt;- sin(y)
  if (x &gt; 0.543) z &lt;- 12.123
  else z &lt;- 157.51</code></pre>
<p>Due to small differences, again within 1 ULP from mingw-w64 v11, different branches were takes with v12, providing results that did not satisfy the tests.</p>
<p>In some cases, but not all, the differences were an <code>NA</code> (as a default value) vs <code>NaN</code> as a computed value. One should be aware that <code>NA</code> could become <code>NaN</code> in computations (more in <code>?is.nan</code>) and be prepared for that in the tests. But, the problem here seems bigger as some differences involved finite values: the algorithm will have to be adjusted.</p>
</section>
<section id="failures-due-to-constant-folding" class="level2">
<h2 class="anchored" data-anchor-id="failures-due-to-constant-folding">Failures due to constant folding</h2>
<p>This is perhaps the most interesting and surprising problem I’ve ran into, and one where it seems questionable whether the problem is actually in the package. And one that I’ve debugged fully, having to read the x86_64 disassembly of the involved code.</p>
<p>Imagine code like:</p>
<pre><code>double f(double x, double y)
{
  if (x == y)
    z = sqrt(sin(x) - sin(y));
  else
    z = 0;
  return z;
}</code></pre>
<p>The code was indeed more complicated and it took long to debug, but the core problem is visible above. One would expect the result would be 0, but it was <code>NaN</code>.</p>
<p>Function <code>sin()</code> in mingw-w64 v12 is now less precise than it was, but for the argument observed, the difference was within 1 ULP. The function was part of a program where the optimizer was able to figure out the value of <code>y</code>, but not the value of <code>x</code>. At runtime, the values were exactly identical, so the branch computing <code>sqrt()</code> was taken. The value of <code>sin(x)</code> is computed at runtime using UCRT, but as the value of <code>y</code> was known at compile time, the compiler also computed (constant-folded) the value of <code>sin(y)</code>.</p>
<p>The compiler (GCC, which we use on Windows/x86_64 in Rtools) used a correctly rounded result it computed using mpfr library. But, the UCRT implementation is less precise and the result happened to be smaller, so the argument of <code>sqrt()</code> became negative, and hence <code>NaN</code> was returned. In the package, this has later lead to a runtime exception being thrown.</p>
<p>Constant-folding using correctly-rounded values makes it possible to constant-fold also when cross-compiling, and hence also to generate the same code when cross-compiling and when compiling natively. Also, even when compiling natively, it could be that the version of the math library would be different from the time when executing (and we would have the same problem). Some also say that programmers probably wouldn’t object to getting more precise values from the compiler than from the runtime library, in principle, but, in this case it matters. One can disable constant folding of the <code>sin</code> function (<code>-fno-builtin-sin</code>), but, how can one know in advance. In principle, one could also disable constant folding of all builtin functions. See the <a href="https://discourse.llvm.org/t/fp-constant-folding-of-floating-point-operations/73138">LLVM discussion</a> for more on the difficulties with constant-folding of math functions.</p>
<p>It may not help in all cases, but sometimes a useful approach is to call a function and then diagnose the result, rather than upfront checking the arguments and then assuming the function will succeed. This is not necessarily only the case with math functions, but it can avoid race conditions with some other, say file system operations.</p>
</section>
<section id="depending-on-unspecified-behavior" class="level2">
<h2 class="anchored" data-anchor-id="depending-on-unspecified-behavior">Depending on unspecified behavior</h2>
<p>It turned out not to be a numerical problem, but still involved a change in a math function. Several packages ran into seemingly infinite computation with v12. It turned out this was because of a single dependency, a package that included a loop similar to this one:</p>
<pre><code>uint64_t rnd() {
  uint64_t res = 0;
  while (res &lt;= 2) {
    double u = uniform() * UINT64_MAX;
    res = lround(u);
  }
  return res;
}</code></pre>
<p>In the code above, one problem is that math function <code>lround()</code> gives a long value, which is 32-bit on Windows. When the value doesn’t fit, the result is unspecified. So, a double value between 0 and 1 is multiplied by the maximum for <em>unsigned</em> 64-bit integer. When that doesn’t fit to a <em>signed</em> 32-bit integer, which is common, <code>res</code> gets the unspecified value. With v11, the unspecified value happened to be <code>LONG_MIN</code> (which came from the Intel hardware, not even controlled by the internal <code>lround</code> implementation), so <code>res</code> ended up something very large, yet not very random, and the loop finished. With v12, the value returned from <code>lround</code> happened to be 0 (probably controlled by UCRT), so the loop kept running.</p>
</section>
<section id="summary" class="level2">
<h2 class="anchored" data-anchor-id="summary">Summary</h2>
<p>This is still work in progress. Some package failures with mingw-w64 v12 haven’t been analyzed, yet, and some of those may still be caused by numerical differences. Some cases I’ve analyzed weren’t yet confirmed/dealt with by the package authors.</p>
<p>The packages that were analyzed were reported to the authors, with patches making the “easy” fixes or work-arounds, such as extending tolerance limits. Some of the package authors have already fixed their packages, but some would have to spend more time when it needs algorithmic improvements.</p>
<p>Some of the issues were reported to <a href="https://github.com/mingw-w64/mingw-w64/issues">mingw-w64</a>, and I’ve used advice from Martin Storsjo and other mingw-w64 developers in this work.</p>
<p>So far my interpretation is that this work revealed an important problem in R packages: that (some of them) are overly sensitive to even very small variations in inputs, such as (surprisingly) even the math library. And it would be worth pursuing methods and tools to help detecting this more widely - after all, it is true that many tolerances in tests are unreasonable, but the question is how to choose reasonable tolerances. At least some fuzzing of the inputs should be possible and perhaps could be implemented in later stages of this work.</p>
<p>R packages in CRAN and Bioconductor are used as a test suite for changes in R itself and in Rtools. The current practice is that such changes are postponed or given up on if they break too many packages. When they only break a little, the people making these changes (to R, Rtools) provide advice or even patches to package authors to fix their packages.</p>
<p>This requires also debugging all problems found by package checks: without debugging those, one cannot know if the problem is in some of the packages (and which) or if in R, or in Rtools (hence say in the compiler, binutils, some library). It is not an exception when a bug is woken up by an innocuous change somewhere else, which can further extend the scope of the debugging.</p>
<p>This practice makes the R ecosystem much more stable than it would have been otherwise. It helped to find real bugs in external software, including in GCC, which have been reported and fixed upstream. On the other hand, given this practice, problems in tests in packages, where they accidentally or even intentionally capture unspecified behavior (including tests using too narrow tolerances) complicate the maintenance of R itself. In this case, it prevented an upgrade of a core Rtools component.</p>


</section>

 ]]></description>
  <category>numerical</category>
  <category>windows</category>
  <guid>https://blog.R-project.org/posts/math-sensitivity/</guid>
  <pubDate>Thu, 24 Apr 2025 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Alternative Toolchains on Windows</title>
  <dc:creator>Tomas Kalibera</dc:creator>
  <link>https://blog.R-project.org/posts/windows-toolchains/</link>
  <description><![CDATA[ 





<p>R and R packages on Windows, if they include native code, are built using compiler toolchain and libraries from <a href="https://cran.r-project.org/bin/windows/Rtools/">Rtools</a>. There is always a specific version of Rtools for a given version of R. Rtools44 is used for R 4.4.x, and hence R 4.4.x and packages for that version of R are always built with GCC 13.3 for x86_64 systems and with LLVM 17 for aarch64 systems. This is the case at least since Rtools42 (R 4.2 was released in April, 2022).</p>
<p>Having a fixed toolchain is normally a benefit, but it is sometimes useful to be able to test the code of R and packages with other toolchains.</p>
<p>This blog post is about recent improvements in R-devel (to become R 4.5) which allow this without changing the code and allow the use of sanitizers. It may be of interest to maintainers of packages that include native code (C, Fortran or C++), and to people working the the source code of base R.</p>
<section id="benefits-of-a-fixed-toolchain" class="level1">
<h1>Benefits of a fixed toolchain</h1>
<p>Having a fixed toolchain has certain benefits. When all users of a package use that package built via the same toolchain, it is very unlikely they would run into compilation problems not seen by the developers such as unsupported options, unsupported language features or incompatibility with system headers. It doesn’t matter whether users download package binaries say from CRAN or Bioconductor or build them from source: all of these use the same Rtools. To some extent, this also reduces the risks users would run into runtime bugs caused by compiler differences, even though this depends on the quality of testing.</p>
<p>Unlike Linux, there is no established software distribution of open-source software that could be expected to exist on Windows users’ machines. R itself and R packages hence link such needed external dependencies statically: the R executable, R shared libraries and shared libraries of R packages already include all the needed code, which avoids maintenance problems that would be otherwise imposed on the users, say if dynamic linking of external dependencies was used.</p>
<p>Particularly on Windows, it is not safe to link code compiled by different toolchains statically due to possible differences in runtime (may include C runtime, compiler runtime, C++ library, etc). All code linked statically should be compiled by exactly the same toolchain.</p>
<p>Having a fixed toolchain then makes this easy to achieve without rebuilding always everything from scratch: the toolchain is distributed together with static libraries compiled by that toolchain. R itself is built by the same toolchain. This works as long as packages only use either pre-compiled code from Rtools or code that is always compiled using the toolchain from Rtools at package installation time (i.e.&nbsp;source code embedded in the package). On <a href="https://cran.r-project.org/web/packages/policies.html">CRAN</a> this is a requirement.</p>
<p>Some problems may arise in a single address space even with dynamic linking, when different dynamically linked parts are built by different toolchains. These problems are less common, sometimes may be avoided by careful interface design, some have been avoided by switching to UCRT as the Windows C runtime (in R 4.2).</p>
<p>With R, dynamic linking is used primarily between R, R packages, and wrapper libraries provided by R, which are also built by Rtools. Indeed, some dynamic libraries linked are built by different toolchains: system libraries and in rare cases some external dependencies of R packages.</p>
<p>Rtools used for a given minor version of R, e.g.&nbsp;Rtools44, are usually updated several times and frozen some time before the next R minor release, e.g.&nbsp;R 4.5.0. This could mean that some dynamically linked parts are built by a slightly different version of Rtools44, e.g.&nbsp;R release built with a different version than an already installed package or previously built and not-yet-rebuilt binary version of the package. To minimize these risks, these small updates in Rtools do not change the toolchain: typically they use the same version of GCC or only increase a patch version, the same version of binutils and the mingw-w64 runtime.</p>
</section>
<section id="drawbacks-of-a-fixed-toolchain" class="level1">
<h1>Drawbacks of a fixed toolchain</h1>
<p>Some of the benefits mentioned above are also drawbacks. A fixed environment may be good at hiding errors, but then, they wouldn’t be found by testing.</p>
<p>These errors then may show up when the compiler toolchain is actually updated in a new version of Rtools. Problems may also show up when a new architecture is added and uses a different toolchain, such as LLVM toolchain on aarch64.</p>
<p>These problems in practice are usually found in Windows-specific code and Windows-specific configuration (e.g.&nbsp;make files), because non-portabilities in code used also on Unix would be likely quickly found on Linux, where different distributions and installations use different toolchains.</p>
</section>
<section id="testing-with-alternative-toolchains" class="level1">
<h1>Testing with alternative toolchains</h1>
<p>As described above, it is key that all code in packages and in R is built by the same toolchain. So all of this only applies to testing, when one builds all of R and R packages completely from source. Binaries built by different/alternative toolchains should never be distributed to users. It is only recommended for experienced package authors to experiment with this testing and it requires knowledge of Rtools as well as Msys2.</p>
<p>Testing base R with alternative toolchain in the past required changes in R source code (make files). It was still done time to time, but with the current R-devel, one can use at least some alternative toolchains without modifying the code.</p>
<p>This is useful for better testing of R itself, because new compilers may introduce stricter checks that discover bugs in the code that could result potentially in incorrect results or crashes even with other (or older) compilers. So, this is useful even though for Windows, we use a fixed compiler, which is known before the corresponding release of R.</p>
<p>There are additional benefits of testing with additional toolchains described below.</p>
<p>R-devel now by default uses <code>pkg-config</code> for figuring out which external libraries to link and which C preprocessor directives to use for those libraries from which library/include directories. R 4.4 and earlier used a fixed setup tuned for the corresponding version of Rtools.</p>
<p><code>pkg-config</code> is supported both by Rtools and by toolchains in <a href="https://www.msys2.org/">Msys2</a>. This made it easier to encapsulate the differences between toolchains and compiled libraries, even though it did not completely erase the need for tweaks to support Msys2 toolchains. Not all Msys2 toolchains would work at the moment with R. Not all are necessary, but the interesting ones for testing are “ucrt64” and “clang64”.</p>
<p>One can use the “ucrt64” toolchain from Msys2 to test a recent GCC on x86_64. Especially around the end of the year, Msys2 will have a newer version of GCC than Rtools issued the year before. At this time, it is 14.2, which could easily be the toolchain version for the next Rtools. There is no special configuration needed. R make files with find the compiler toolchain on PATH. So, package authors who would want to prepare their packages a bit even before Rtools45 is released, can do it using “ucrt64”.</p>
<p>The “clangarm64” Msys2 toolchain on aarch64 would serve the same purpose. Package authors can test their packages with a newer version of LLVM than Rtools has, around this time of the year. However, this requires access to aarch64 hardware.</p>
<p>The “clang64” toolchain from Msys2 can be used to test R and packages with the LLVM compiler toolchain, including the flang-new Fortran compiler. This allows to test compatibility of a package with LLVM for those who do not have access to a Windows/aarch64. Many of the issues found in R packages on Windows/aarch64 are in fact because of using non-portable GCC features or options that are not supported by LLVM. Only one bit of special configuration is needed, set <code>USE_LLVM = 1</code> in <code>MkRules.local</code>, as is done when building on Windows/aarch64.</p>
<p>To be able to test R package with these alternative toolchains, it is necessary to use only static libraries already available on the system of the user (so, in real use static libraries only from Rtools, but in these tests, they will be from the corresponding toolchain from Msys2). One also needs to use <code>pkg-config</code> so that the proper library dependencies and C preprocessor directives apply to the toolchain.</p>
<p>One needs to specify all pkg-config packages directly used by the R package when querying <code>pkg-config</code> for libraries and flags. This may not always be the case normally when Rtools is used, because Rtools uses static linking, so <code>pkg-config</code> with Rtools would include to the link list all static dependencies of the libraries, some of which may be also directly used by the R package, but may belong to other pkg-config packages.</p>
<p>Msys2 uses dynamic linking. Dynamic libraries remember their dependencies on their own, so <code>pkg-config</code> will not include those in the list of libraries to link. Building with an alternative Msys2 toolchain is hence a good test of whether the specification is complete.</p>
<p>Now, it should be said that even with Rtools, packages get tested by multiple toolchains when they are tested with multiple versions of R. CRAN currently tests R-devel, the current release and the previous release. Also, a new release of Rtools is usually available some time before an R release, so that packages using R-devel may be tested with it.</p>
</section>
<section id="testing-with-sanitizers" class="level1">
<h1>Testing with sanitizers</h1>
<p>Sanitizers are useful tools for finding different bugs in programs, including memory access/allocation errors and relying on undefined behavior. In some cases, sanitizers have discovered R memory protection (“PROTECT”) errors.</p>
<p>Unfortunately, GCC currently does not support sanitizers on Windows. LLVM supports sanitizers on x86_64, but not on aarch64. Rtools hence doesn’t support sanitizers on any platform.</p>
<p>With the Msys2 “clang64” toolchain, one may now use sanitizers by setting <code>SANOPTS</code> in <code>MkRules.local</code> to a suitable value, such as <code>-fsanitize=address,undefined,bounds</code>, which is then passed to the compiler and linker where needed during compilation of R itself and then of packages.</p>
<p>The new sanitizer support has been used with base R and the discovered issues, where possible, were fixed. These issues included code that is only used on Windows, but also common code where differences between Unix and Windows mattered (e.g.&nbsp;size of integer, long vs size of pointer).</p>
<p>Testing the sanitizer support with packages only extended to packages distributed with base R. Problems may appear when checking other packages.</p>


</section>

 ]]></description>
  <category>packages</category>
  <category>windows</category>
  <guid>https://blog.R-project.org/posts/windows-toolchains/</guid>
  <pubDate>Tue, 28 Jan 2025 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Problems with iconv on macOS</title>
  <dc:creator>Tomas Kalibera</dc:creator>
  <link>https://blog.R-project.org/posts/iconv/</link>
  <description><![CDATA[ 





<p>For conversion of strings from a given character encoding to another, R uses <code>iconv</code>, a function defined by POSIX. It is available on Linux and macOS with the operating system and for Windows, R ships with a slightly customized version of <code>win_iconv</code>, which implements the same functionality on top of Windows API.</p>
<p>The differences between iconv implementations, partially allowed by a rather permissive definition of the interface in POSIX, pose a challenge for maintaining R and cause differences between platforms observed by users.</p>
<p>A recent significant challenge has been new iconv implementation that came with macOS 14.0. It not only changed the behavior with characters not representable in the target encoding, but also caused crashes and incorrect conversions. This post focuses on work-arounds in R, some of which were already in R 4.4, but have been extended and improved in R-devel, the development version of R to become R 4.5.0. The work-arounds were part of a bigger effort dealing with libiconv changes on macOS, otherwise mostly by Brian Ripley.</p>
<p>This text includes technical details. The higher-level message to users and package authors is that converting characters to an encoding where they are not representable is platform-dependent and the outcomes can change over time; while R documents what it does with such characters, it won’t happen when the system silently transliterates the characters without telling R. It is best to avoid such conversions, so e.g.&nbsp;to only use characters in plot labels that are representable in the given encoding (more in <code>?pdf</code>). It is good to use UTF-8 whenever possible. A message specific to R package authors and developers on macOS: when R is built from source, by default the system libiconv will be used, which may behave strangely and change its behavior on any system update. This is currently not the case with R CRAN builds.</p>
<section id="non-representable-characters" class="level1">
<h1>Non-representable characters</h1>
<p>What should ideally happen when converting a string where some characters cannot be represented in the target encoding? It probably depends on what we need the string for. If it is say a file name of a file to be saved, we would probably want to throw an error. But, already in the error message for such an error, we would want to see exactly what was the non-representable character (e.g.&nbsp;Unicode <code>U+0161</code> or bytes <code>&lt;C5&gt;&lt;A1&gt;</code> for Unicode Small Letter S with Caron). If it is a file name of a file to be selected say from a user dialog, some might prefer to see a replacement character (e.g.&nbsp;a question mark, possibly in a diamond). If it is a plot label, we might prefer, for some characters, to get a similarly looking replacement character (e.g.&nbsp;<code>s</code> so simply drop the caron). In other words, the application, in this case R on an R package, should have full control.</p>
<p>Unfortunately, this cannot be efficiently implemented using iconv, relying only on the POSIX specification. According to POSIX, non-representable characters are subject to implementation-defined conversion, so the implementation is free to say replace all such characters by <code>*</code> (or any other character, and the behavior may depend on the encoding), but POSIX doesn’t specify a way how non-representable characters could be reported to the user.</p>
<p>While it probably could be claimed as a violation of POSIX, some implementations report valid but non-representable characters the same way as invalid characters (bytes). The conversion stops at the first byte of the non-representable character and <code>EILSEQ</code> is returned. R handles this behavior and implements its own handling of non-representable characters.</p>
<p>On Linux with glibc, this allows R to have full control, because glibc’s iconv implementation reports <em>all</em> non-representable characters this way. It also used to be the case on macOS (before 14.0), where the system iconv was GNU libiconv.</p>
<p>On Windows, transliteration (called “best fit”) historically has been used and is the default behavior in non-standard Windows API for character conversion. Some non-representable characters are replaced by similarly looking ones, while other are reported as error. This is what Windows users are used to and expect, but it means that R doesn’t know about transliterated characters in the output and handles them as if they were unique representations.</p>
<p>In plots transliteration may be fine and expected, in handling of file names highly undesirable. R’s customized version of win_iconv has been set to report non-representable characters only when converting to ASCII (so disabled transliteration) as a compromise between what is common in Windows and R’s ability to customize the outcome based on what the string is needed for. Windows API, however, allows to disable transliteration completely.</p>
<p>Before R 4.2, the problem with transliteration on Windows has been bigger than in later R versions, because the native encoding wasn’t a Unicode one (it is UTF-8 from R 4.2 on recent Windows systems). Before R 4.2, one could run into transliteration e.g.&nbsp;when working with R symbols (names), where say the letter alpha would silently become letter a. The bugs in R (or non-bugs, but behavior that surprised users) due to non-representable characters on Windows really almost disappeared with R 4.2.</p>
<p>On Unix, UTF-8 has been normally the native encoding much earlier, so one would have thought that we won’t run again into many problems with non-representable characters, or even other problems in character conversion.</p>
<p>This turned out not completely true. There is musl, a C library implementation used by some less common Linux distributions, which comes with an iconv implementation which replaces non-representable characters all by an asterisk (so no transliteration, no reporting of non-representable characters). This doesn’t affect typical platforms used these days.</p>
<p>But then, there was macOS 14.0.</p>
</section>
<section id="iconv-on-macos" class="level1">
<h1>Iconv on macOS</h1>
<p>The source code for the system libiconv on macOS can be found <a href="https://opensource.apple.com/releases/">here</a>. In macOS before 14.0, it used to be GNU libiconv. In macOS 13.5, it was still version 1.11 of GNU libiconv (named libiconv-64 on the website above). It was a rather old version, but still provided almost the same behavior in R as the implementation of iconv in glibc on Linux and worked fine. GNU libiconv 1.12 has been released in 2007, 16 years before macOS 13.5, and has changed the license for the tool from GPL version 2 to version 3, yet the library itself remained at LGPL.</p>
<p>Instead of updating its GNU libiconv, macOS 14.0 came with iconv based on an implementation from Citrus/FreeBSD (named libiconv-80.1.1 on the website above), which has been modified to identify itself as GNU libiconv 1.11. All the versions in macOS up to 15.0 (latest I’ve checked) report libiconv version this way. See how the author of GNU libiconv <a href="https://lists.gnu.org/archive/html/bug-gnulib/2024-02/msg00123.html">described</a> this decision. The source code of libiconv provided with macOS claims compatibility with GNU libiconv 1.11, but as it turned out, there were changes in the handling of non-representable characters as well as newly introduced bugs.</p>
<p>Note that when R is built from source on macOS, by default it dynamically links to the system libiconv, which can be changed by any OS update. So, existing R installations broke by a system update bringing the new libiconv, while <code>extSoftVersion()</code> would still report the same iconv (<code>"GNU libiconv 1.11"</code> or <code>"Apple or GNU libiconv 1.11"</code>, depending on the version of R, not the library).</p>
<p>The version reporting has been extended in R-devel to show also the name of the shared library, e.g.&nbsp;<code>"Apple or GNU libiconv 1.11 /usr/lib/libiconv.2.dylib"</code>.</p>
<p>CRAN builds of R provided by Simon Urbanek currently use a static build of libiconv-64, so the last version of iconv on macOS which matched GNU libiconv 1.11. Hence, the CRAN builds of R aren’t affected by the problems of the new libiconv. In the current R-devel CRAN build, <code>extSoftVersion()</code> would report iconv to be <code>"Apple or GNU libiconv 1.11 /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/lib/libR.dylib"</code>.</p>
<p>With <code>extSoftVersion()</code> one can hence see whether a system libiconv is in use, and then one can check the version of macOS e.g.&nbsp;via <code>sessionInfo()</code>, and then consult the <a href="https://opensource.apple.com/releases/">Apple website</a> whether it already has source code for the corresponding libiconv.</p>
</section>
<section id="non-representable-characters-on-macos" class="level1">
<h1>Non-representable characters on macOS</h1>
<p>In the new libiconv on macOS, many characters are transliterated or replaced. This is something that the POSIX specification allows, but it changes the previous behavior on the system and hence also in R, and R package tests depending on the previous outcome would show differences.</p>
<p>Even though R on macOS (as well as on other current systems) uses UTF-8 as the native encoding, conversion to 8-bit encodings is done when plotting (see <code>?pdf</code> for more). Normally, the 8-bit encoding is Latin-1. R detects and reports when a character cannot be represented, but, it cannot do that when it doesn’t know - e.g.&nbsp;when iconv silently transliterates or replaces some of the characters.</p>
<p>To reduce the platform differences and alert users to the problem, thanks to Brian Ripley R now transliterates some of the characters commonly used in plots, with a warning, and the warnings and replacement of non-transliteraded characters have also been improved. One can then see when testing e.g.&nbsp;on Linux or CRAN macOS builds that the transliteration would happen or that a character is not representable, and ideally avoid using that character to avoid platform differences and possible future problems. Also, if one keeps the non-representable characters, the outputs will still have fewer differences between platforms thanks to the transliteration done in R. See the <a href="https://cran.r-project.org/doc/manuals/r-release/NEWS.html">NEWS file</a> for more details.</p>
<p>I’ve written a simple <a href="https://github.com/kalibera/iconvtest">program</a> to detect what iconv does with non-representable characters during conversion, whether they are replaced, reported via <code>EILSEQ</code>, or discarded. The program also allows to experiment with non-standard features of iconv documented on macOS that should allow to disable transliteration or/and enable reporting of non-representable characters via <code>EILSEQ</code>.</p>
<p>When converting to ASCII, the differences between libiconv-64 and libiconv-86 (macOS 14.1) are small and probably wouldn’t cause much trouble. The normally used characters are not transliterated, but are still reported via <code>EILSEQ</code>. It is possible to enable transliteration (or disable <code>EILSEQ</code>) - we don’t want to change the defaults, but changing it works.</p>
<p>But, when converting to say CP1252, libiconv-86 introduces a lot of transliteration/replacing of non-representable characters. It is not possible to disable the transliteration nor to enable <code>EILSEQ</code>. Actually, transliteration appears disabled already, but is still happening. A cursory look at the source suggests that there are different places where transliteration can be triggered, but some may be harder to handle than other.</p>
<p>The observed behavior seems contradicts the documentation on macOS, which states that transliteration is always enabled and cannot be turned off, that doing so would fail with an error. If the behavior is ever changed to allow disabling transliteration, R could do that to regain control over the non-representable characters, to make sure that e.g.&nbsp;in plots they are reported to users.</p>
<p>In principle, there is always a rather inefficient way to detect transliteration/replacement of characters via double conversion (convert back and compare), and there is some experimental code in R that allows that, but it may be easier to detect these problems with Linux or current CRAN builds of R, and avoid using non-representable characters. That approach would also avoid problems say with musl.</p>
</section>
<section id="bugs-on-macos" class="level1">
<h1>Bugs on macOS</h1>
<p>The following describes other problems found in the new libiconv on macOS, which now have work-arounds in R-devel, so should be invisible to users (and packages).</p>
<section id="crash-after-invalid-byte" class="level2">
<h2 class="anchored" data-anchor-id="crash-after-invalid-byte">Crash after invalid byte</h2>
<p>At the time of macOS 14.1 (so libiconv-86), before the release of R 4.4.0, I’ve been debugging sudden crashes of R on macOS narrowed down by Brian Ripley to be due to libiconv. I found that libiconv, and hence R, crashes some time after encountering an invalid byte on input. Invalid input is sometimes provided intentionally in tests, sometimes by accident, but certainly shouldn’t crash R.</p>
<p>I found that after correctly reporting the byte as invalid, iconv will then erroneously report the following bytes as invalid as well, and eventually crash in subsequent calls. I found that when we re-set the iconv conversion state after the first invalid byte is encountered, the problem disappears. It sounded as a good thing to do, anyway. R isn’t tested with stateful encodings, but in principle, one should not unnecessarily assume a stateless encoding, so one should re-set before emitting escapes for invalid bytes (e.g.&nbsp;<code>&lt;FC&gt;</code>). I’ve thus modified R code itself to do that, and also I’ve extended <code>Riconv()</code> to do that automatically on macOS for stateless encodings, so that even packages are covered. The idea was that for a stateless encoding, the state re-set should have no impact (in a correct implementation), so it is safe to do and will cause no harm even once this is fixed in libiconv.</p>
<p>Unfortunately, this caused a regression in the CRAN builds of R 4.4.0 which still used libiconv-64. That version of libiconv has another bug (described next), which interfered with this work-around.</p>
</section>
<section id="bom-forgotten-after-reset" class="level2">
<h2 class="anchored" data-anchor-id="bom-forgotten-after-reset">BOM forgotten after reset</h2>
<p>To correctly decode UTF-16 or UTF-32 input, the decoder needs to know the byte-order (little or big endian). One can specify the byte order as part of the encoding name to iconv, e.g.&nbsp;“UTF-16LE”, or one can specify the encoding without it (“UTF-16”) and then include a byte-order mark at the beginning of input. The byte-order mark (BOM) is Unicode character U+FEFF (zero-width no-break space), and from how its bytes are laid out the decoder infers the byte-order. If the BOM is not present, the decoder uses the default order.</p>
<p>My reading of the POSIX standard is that the byte-order learned from BOM is <em>not</em> part of the encoding state, because UTF-16 and UTF-32 are stateless encodings. Instead, the byte-order learned from BOM should be immutable and stay that way until the conversion stream is closed. This is also explicitly stated by Ulrich Drepper in a response to a <a href="https://bugzilla.redhat.com/show_bug.cgi?id=165368">bug report</a>.</p>
<p>Unfortunately, some iconv implementations forget the byte-order learned from BOM on reset, as if it was part of the shift/conversion state, as if UTF-16/-32 were a stateful encoding. This problem is not present in iconv in Linux/glibc, but it is present both in GNU libiconv 1.11 as well as in later versions of libiconv on macOS. Also, it doesn’t help that the default byte-order with libiconv on macOS is big-endian (while UTF-16 is mostly used on Windows where it is encoded as little-endian).</p>
<p>Conversion state reset is a relatively rare operation, after all it should ever only be needed with stateful encodings; but, not so rare with the work-around for the crash after an invalid byte. This problem of the work-around was found after R 4.4.0 has been out and when the present libiconv on macOS already has been fixed (libiconv-92 no longer had that problem). So, as a minimal hot-fix, R 4.4.2 comes without the work-around for the crash after an invalid byte, and this is intended to remain the behavior with R 4.4.x.</p>
<p>In R-devel, there is now a work-around also for the case that iconv forgets the byte-order after reset. <code>Riconv</code> would listen to the input and after seeing the BOM, it would continue decoding with byte-order specified to iconv via the encoding name, e.g.&nbsp;“UTF-16LE”.</p>
<p>In addition to that, all work-arounds for iconv in R-devel are now conditional on runtime tests. So, specifically with a system libiconv no longer getting confused after an invalid byte, R would not be issuing state resets anymore.</p>
<p>The problem of forgetting BOM after reset is still present at least in libiconv-107 (macOS 15.0).</p>
</section>
<section id="bom-forgotten-on-incomplete-character" class="level2">
<h2 class="anchored" data-anchor-id="bom-forgotten-on-incomplete-character">BOM forgotten on incomplete character</h2>
<p>The new libiconv on macOS (from libiconv-86 to libiconv-107 at least) also has forgets the byte-order learned from BOM when it is given an incomplete character to decode. This is a quite serious bug, because an incomplete character on input is a common situation during iterative conversion, when one reads in some part of input, gives it to iconv to convert, then reads some more, etc. This specific pattern is heavily used in R and other applications, and there is probably no other way using iconv. It is also on of the examples in <a href="https://www.gnu.org/software/libc/manual/html_node/iconv-Examples.html">Glibc documentation</a>.</p>
<p>R-devel checks at runtime whether this problem is present, and if it is the case, it uses the same work-around as for BOM forgotten after re-set, by falling back to conversion using the byte-order specified in the encoding name.</p>


</section>
</section>

 ]]></description>
  <category>encoding</category>
  <category>macOS</category>
  <guid>https://blog.R-project.org/posts/iconv/</guid>
  <pubDate>Wed, 11 Dec 2024 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Faster Downloads</title>
  <dc:creator>Tomas Kalibera</dc:creator>
  <link>https://blog.R-project.org/posts/faster-downloads/</link>
  <description><![CDATA[ 





<p>Most R users would sometimes install or update R packages and hence are impacted by how long this takes. The parts of package installation that take potentially longest have already been addressed by support for binary packages and parallel installation. A remaining overhead that may be rather surprising, but is easy to reduce, is package download.</p>
<p>The overhead may be noticeable when installing many mostly small packages in parallel, because so far the package download has been sequential even with parallel installation. This post reports on recent work in R-devel, the development version of R (to become R 4.5.0), on improving the support for simultaneous download. As any work in the development version of R, this should be regarded as experimental and may be updated or changed before release.</p>
<p>The text includes technical details. The short story for R users not necessarily interested in those is that R 4.5 will probably download packages for installation several times faster than R 4.4. In certain specific situations, hopefully rare, users might have to increase the internet timeout (see <code>?download.file</code>, look for <code>timeout</code>), which applies to downloading individual files. For best performance of their users, maintainers of package repository mirrors might consider enabling <code>HTTP 2</code>. With <code>HTTP 1.1</code>, they might experience more concurrent connections than with previous versions of R. In either case, they would experience more concurrent transfers to the same client.</p>
<section id="rsync" class="level1">
<h1>Rsync</h1>
<p>Some, very small, group of R users would often download and install all or almost all packages from a repository, typically CRAN or Bioconductor, for R or R package testing. The work presented here doesn’t impact such use. For such use, the best practice is to have a local mirror of each package and install packages as files from the mirror (one would use <code>rsync</code> to create and update the mirror). This will save network traffic and resources on the repository servers. The improved download in R presented here wasn’t meant to compete with <code>rsync</code>.</p>
</section>
<section id="progress-bar" class="level1">
<h1>Progress bar</h1>
<p>In already released versions of R (R 4.4 and older), a simple way to speed up downloads of very many small files is to disable the progress bar by passing <code>quiet=TRUE</code> to <code>download.packages()</code> or <code>install.packages()</code>. The progress bar is only displayed when downloading a single file, but even when installing several packages at once, they are downloaded sequentially. Disabling the progress bar helps particularly on Windows.</p>
</section>
<section id="simultaneous-download" class="level1">
<h1>Simultaneous download</h1>
<p>R has support for simultaneous download since version 3.2.0 (so for over 9 years, see <code>?download.file</code>) using download method <code>libcurl</code>: with the download method explicitly specified to <code>download.file</code>, one can provide a vector of URLs to download from and a vector of destination file names.</p>
<p>At the low level, the implementation creates a single curl “multi-handle” for the whole operation and a curl “easy-handle” for each URL. Transfers would be initiated and performed concurrently in a single thread (R main thread). This offers a substantial speedup. The key is that R doesn’t have to wait for the transfer of each file individually to be set up, but while setting it up, it can keep setting up other transfers and downloading.</p>
</section>
<section id="package-download-and-installation" class="level1">
<h1>Package download and installation</h1>
<p>The simultaneous download, however, hasn’t been used by <code>download.packages()</code> and <code>install.packages()</code>, so couldn’t be easily used to speed up package downloading.</p>
<p>These days, R requires the curl library and the <code>libcurl</code> download method is the default on all platforms, so it makes sense to enable simultaneous download for package installation. It has been done in R-devel for <code>download.packages()</code> and for <code>install.packages()</code> when the default download method is used (or when <code>libcurl</code> is requested explicitly).</p>
<p><code>install.packages()</code> needs to know download of which files has succeeded, so this has been added to the <code>download.file()</code> API.</p>
<p>With the existing version of simultaneous download from R 3.2.0, I could get about 2-3x speedup on downloading 100 CRAN packages via <code>download.packages()</code>, but see below for the disclaimer on reproducibility of such measurements in setups one does not control. The simultaneous download doesn’t use progress bars, so I have disabled them with sequential download as well to have more relevant comparisons.</p>
<p>The experiments also exposed a limitation that prevented downloading of a large number of files (on my system over 500, but, it could be much less on other systems). The existing implementation creates and opens all files upfront, but, this can run into the limit on the number of open files imposed by the C library: the current implementation uses C library standard I/O streams with the default curl callback.</p>
</section>
<section id="reducing-the-concurrency-and-resource-usage" class="level1">
<h1>Reducing the concurrency and resource usage</h1>
<p>The code has hence been refactored to create and open the output files on the fly. At most a pre-defined number of files (and hence curl easy-handles) is used at a time, currently 15, but it is adaptively less when opening files fails.</p>
<p>As a byproduct, this also reduces the number of concurrent connections to remote servers held at a time (even though curl can do this part for the user). In practice, when installing packages, a number of connection could be to the same server, and then it makes even more sense to limit the number of connections in order not to overload it.</p>
<p>As now not all transfers happen concurrently, but only up to a defined limit, the connection re-use done by curl becomes more visible and saves some resources. Established connections to the same server are re-used for additional transfers. With HTTP 2 connections, multiple transfers from the same server can be multi-plexed over the same connection, further reducing the number of concurrent connections to the same server to one.</p>
<p>An original version of HTTP 1.1 protocol asked for at most 2 concurrent connections to the same server. This restriction/suggestion has been removed in a later version (RFC7230 vs RFC2616). Most browsers seem to use 6, some a bit more. In the current implementation in R, at most 6 concurrent connections are used to the same server (via a curl option). This means that at most 6 files from the same server are being downloaded at a time, compared to at most 15 using HTTP 2.</p>
</section>
<section id="timeout" class="level1">
<h1>Timeout</h1>
<p>In R, downloading files is subject to a timeout (by default 60s), documented in <code>?download.file</code>, <code>?options</code>, <code>?connections</code>. The timeout serves as a limit for blocking, low-level network operations and ensures that R would not get blocked due to network problems. But, higher-level R operations typically do more than a single low-level network operation, so may take longer than the value of the timeout. In practice, most users probably experience internet timeout only when the remote server is down, and at implementation level this can result in the connection timing out.</p>
<p>Users downloading large files may have to increase the timeout, because the transfer itself is protected by it as well. In some sense, absolute time timeout also gives some level of guarantee that operations would finish and do so in reasonable time.</p>
<p>Some imprecision in the definition of R timeout is probably necessary to be able to provide implementations with different download methods and libraries, which may have their views on how reasonable timeout should work.</p>
<p>With libcurl, the R timeout has been mapped to curl connection timeout and overall transfer timeout of each file being downloaded (which also includes the connection time).</p>
<p>With simultaneous download, this becomes more complicated, and yet more with the changes described above.</p>
<p>The curl overall timeout includes also the time a transfer has been paused by curl e.g.&nbsp;in case there are already too many concurrent connections to a given server or another connection is being established which might (or might not) allow multiplexing. The first of these pauses could in theory be very long, and caused by transfers of different files, but would cause a timeout. Therefore, a custom implementation of the timeout has been provided in R-devel, instead, on top of curl callbacks. As the callbacks only allow to cover the transfer part of the timeout, the curl connection timeout is still used in addition, and this gives a bit more time for each of the transfer.</p>
<p>In principle, if the local bandwidth was the bottleneck rather than network problems, it is possible that the R timeout with simultaneous download would not be sufficient for a download when in sequential download it was. Similar situations could be caused by other applications on the same machine or network, or temporary issues on the way to the server, which were situations users already had to cope with (e.g.&nbsp;by increasing the timeout). It has to be seen whether such issue would appear in practice and whether the current implementation should be adjusted.</p>
<p>An alternative to absolute-time timeouts for file transfer as in R is aborting the transfer when it seems it has stopped (no data transferred within an interval, available in wget) or when it became too slow (average speed below a specific limit during an interval, available in libcurl). An obvious advantage is that these are independent on the file size. On the other hand, these limits might keep slow downloads too long for other reasons (e.g.&nbsp;by accident downloading a too large file), and anecdotically some proxies batch transfers so much to cause an undesirable abort. At the implementation level, currently in R-devel, R falls back to the speed limit with older versions of curl to cover a phase between a request is ready to be sent to the server and the first data is transferred: that phase doesn’t seem to be otherwise easily covered by a custom timeout implementation.</p>
</section>
<section id="speedup" class="level1">
<h1>Speedup</h1>
<p>I’ve measured the speedup in several scenarios. First, on fast academic networks (client is a server machine on a well-connected academic network), downloading 500 CRAN packages from the CRAN cloud mirror (CDN). Second, on laptops connected via a decent home internet connection on different operating systems (on the client), downloading 100 CRAN packages from the CRAN cloud mirror. Then, to test the impact of connection speed, I tried also downloading from fixed CRAN mirrors at selected locations (New Zealand, UK, Europe) to a macOS laptop in Europe.</p>
<p>I’m not reporting the numbers, because they are not repeatable: the download times are too dynamic and the involved systems out of my control. Measuring the same thing the next day can give a very different speedup. The usual speedups were 2-5x (that is, twice to five times shorter download time), but sometimes it was much more (even around 30x speedup) and sometimes it was about the same speed.</p>
<p>One could measure on a lab setup (own CRAN mirror, emulated network of different characteristics), but that is probably not worth the effort, and it would be hard to know what are realistic setups.</p>
<p>It still seems from the experiments that big speedups can be achieved even on slower connections (I expect because of longer latency). Also, the speedup can be bigger when HTTP 2 is enabled on the server. On Windows, R doesn’t yet support HTTP 2 because Rtools 4.4 use a version of curl without HTTP 2 support, but the next version of Rtools and R is expected to support it. I’ve also tested that the speedup is bigger compared to just a sequential download with re-using connections.</p>
<p>The actual speedup seen by users would be different, as they wouldn’t be just downloading the packages, but also installing them. I find the speedup noticeable when installing a large number of packages in parallel on many cores, even source packages. It is also visible when installing binary packages on Windows, not necessarily in parallel, where the installation is very fast. It certainly wouldn’t be visible when installing just several packages from source which use C++, where the compile time would dominate.</p>
</section>
<section id="possible-further-improvements" class="level1">
<h1>Possible further improvements</h1>
<p>In principle, it would be possible to push things further by using more concurrent transfers, but that in turn would put bigger stress on the servers - the actual performance is a balance between the download time on the client and peak load on the server.</p>
<p>Should the number of parallel transfers be still increased, say with HTTP 2, one could solve the problem of limited number of I/O streams (or file handles) by partially buffering the downloads in memory and closing/re-opening files during the download as needed.</p>


</section>

 ]]></description>
  <category>performance</category>
  <category>packages</category>
  <guid>https://blog.R-project.org/posts/faster-downloads/</guid>
  <pubDate>Mon, 02 Dec 2024 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Long Input Lines</title>
  <dc:creator>Tomas Kalibera</dc:creator>
  <link>https://blog.R-project.org/posts/long-input-lines/</link>
  <description><![CDATA[ 





<p>When using R interactively via a console, one edits a line of input, confirms it by pressing ENTER, then R parses the line, evaluates it, prints the output and lets the user enter another line. This is also known as REPL (Read-Eval-Print-Loop).</p>
<p>The maximum length of the input line is sometimes limited. It is essentially impossible that one would run into a limit when typing the commands, but there was a report from a user who pasted generated content to the console and have ran into a limit of 4096 bytes. The input has been truncated. Other bug reports have indicated that users paste generated code into R console.</p>
<p>This text explains how R processes long input lines and reports on recent improvements in R-devel, the development version of R. In short, it is now possible to paste lines of unlimited length into R consoles that are part of the R distribution (so running R from the Linux/macOS terminal, running Rterm on Windows, with some limitations also running Rgui on Windows).</p>
<p>Despite these improvements, a recommendation for defensive users would be to instead save generated code to a file and then let software (R or any other) read it from there. Consoles, terminals or even clipboards have been designed for interactive work and may have not been tested with such long inputs or may have input length limitations.</p>
<section id="how-input-gets-to-r" class="level1">
<h1>How input gets to R</h1>
<p>This describes (my interpretation) of the current design of R components interacting with the input, and how they deal with lines of unlimited length.</p>
<section id="parser" class="level2">
<h2 class="anchored" data-anchor-id="parser">Parser</h2>
<p>The parser is not to blame. The parser must be and is able to to work with input of unlimited length, spanning multiple lines. Think of a large piece of R code placed inside a branch of the <code>if</code> statement, the parser needs to see the whole <code>if</code> statement. The parser is used to process not only input provided by the user interactively, but also non-interactive input including that read from files, which can be very large.</p>
<p>The parser is always given a piece of input of certain length by the REPL and tries to parse some of it. When it cannot make progress parsing any part of it without seeing more, it reports this back to the REPL and the REPL instead gives it a bigger chunk of input. So, the parser can work with input of unlimited length. There are some limits in the parser/lexer, but not in principle for the total length of input (but e.g.&nbsp;to the number of digits in a number, which indeed is also limited semantically). The parser can accept for example arbitrarily long string literals, or arbitrary number of commands on a line, so things that would be most likely seen in generated code.</p>
<p>This mechanism is not only to support very long lines of input, but also to support commands spanning multiple lines. When one enters say</p>
<pre><code>if (TRUE) {</code></pre>
<p>the parser needs to ask for additional input to make any progress.</p>
</section>
<section id="repl" class="level2">
<h2 class="anchored" data-anchor-id="repl">REPL</h2>
<p>The REPL gets the input from the console via R console API, function <code>R_ReadConsole</code>. The interface is modelled after the C function <code>fgets()</code>. The caller provides a fixed-size buffer and the size of the buffer in bytes. The console fills that buffer with a line of input or a segment of it. If the whole line fits, including the newline (<code>\n</code>) and NUL (<code>\0</code>), it is placed into the buffer. If it doesn’t fit, only as much as fits is added and terminated by the NUL byte. The caller (REPL) hence knows whether it received the full line or not, and this interface allows to receive lines of unlimited length via repeated calls.</p>
<p>The size of the console buffer used by the REPL for this is a constant defined in R source code, which has been 4096 since year 2008, before it was 1024. The REPL copies the input into a growable buffer, which is then presented to the parser. When the parser asks for more input to be able to make any progress, the REPL gets more data from the console, adds that to the growable buffer, and runs the parser again. So, the REPL can process lines of unlimited length.</p>
</section>
<section id="console" class="level2">
<h2 class="anchored" data-anchor-id="console">Console</h2>
<p>When the REPL asks the console for input, it also provides it with a prompt for the user to edit the line. Normally it is “<code>&gt;</code>”, but it can be something else, e.g.&nbsp;when debugging (in a browser). When the REPL asks for a continuation of a line, the prompt is normally “<code>+</code>”.</p>
<p>This can be easily seen when one enters</p>
<pre><code>if (TRUE) {</code></pre>
<p>to the console. The next console prompt asks for a continuation using “<code>+</code>”, because the parser could not have made any progress parsing just the previous command; it didn’t see the end of the <code>if</code> statement.</p>
<p>This design also allows R to get lines of unlimited length from a console that itself provides lines of <em>limited</em> length. Normally the user would enter a line that fits into the limit within the normal prompt (“<code>&gt;</code>”). But if the user tries to add one more character than could fit, the console returns the segment of the line to the REPL. The REPL then asks the console for the continuation. The console allows the user then to edit another segment of the line using the continuation prompt (“<code>+</code>”), and there may be multiple continuation prompts of a line. The user can only edit the current segment of the line, it is not possible to edit the previously entered segments.</p>
<p>The standard “console” used non-interactively in R works this way. It reads the input using C <code>fgets()</code>. Passing an R script with very long lines to <code>R --vanilla</code> provides an output with the continuation prompts (“<code>+</code>”).</p>
<p>This is similar to when these are shown to indicate that more lines are needed to parser the input, where running <code>R --vanilla</code> on</p>
<pre><code>if (TRUE) {
  cat("Hello.\n")
}
</code></pre>
<p>yields</p>
<pre><code>&gt; if (TRUE) {
+   cat("Hello.\n")
+ }
Hello.
&gt;</code></pre>
</section>
</section>
<section id="improvements" class="level1">
<h1>Improvements</h1>
<p>Unfortunately, there is a limit on the input line length when R is used interactively, with some console implementations in R. The situation differs for each console.</p>
<section id="rgui" class="level2">
<h2 class="anchored" data-anchor-id="rgui">Rgui</h2>
<p>On Windows, in Rgui, the console implementation is part of base R and has support for the continuation mechanism described above. So, the user is provided with a continuation prompt to enter an additional segment of the line.</p>
<p>While testing this in more detail, however, several issues were discovered (R was compiled with a small console buffer size so that the problems were more visible). I’ve observed that a newline <code>\n</code> character gets introduced into the text at the continuation segments, so e.g.&nbsp;long string literals are corrupted. Also, the allowed length of the line with multi-byte characters varied and pasting characters sometimes introduced garbage into the continuation segments (some other characters than the overflow character appeared on the continuation line).</p>
<p>A conceptual problem with the continuation mechanism based on <code>fgets()</code> is that the limit is given in bytes, not characters. It may be that several bytes of a character would fit, but the additional ones not. Also, an editor based on lines of <em>limited</em> length would probably want to limit the number of characters (or perhaps width with a fixed-width font): the user shouldn’t be exposed to that how many bytes are needed to represent different characters.</p>
<p>This has been fixed by an intermediate interface, which is prepared to that Rgui provides segments of lines of arbitrary length (note that <code>fgets()</code> would always fill up the buffer when not returning the last segment of the line). With this interface, Rgui can limit the line by the number of characters (as I think was the original design), while entering lines of arbitrary length to that intermediate interface (e.g.&nbsp;less than the limit when some bytes of the overflow character fits but others don’t, or much more than the limit when many multi-byte characters have been entered).</p>
<p>The problem of incorrect character or garbage on the continuation line was related to incorrect implementation of pushback (internal to Rgui). This pushback is only used when a line overflows, so this understandably went unnoticed and has now been fixed.</p>
</section>
<section id="rterm" class="level2">
<h2 class="anchored" data-anchor-id="rterm">Rterm</h2>
<p>On Windows, in Rterm, the console is implemented using a significantly rewritten version of the getline library, which is part of R source code. The original as well as the rewritten version has a limit on the number of bytes per line. It is not possible to enter more than that - so essentially the input is truncated, with a beep signalling this has happened. The limit is the same as the size of the buffer provided by the REPL, so R’s console buffer size. Getline, unlike Rgui, unfortunately doesn’t support the continuation mechanism.</p>
<p>As part of this work, getline in R has been extended to support lines of arbitrary length via re-allocation. To work via R’s console interface, an intermediate interface was implemented which presents segments of the already edited line to the REPL. So, the user first edits the whole line, which is then provided in segments to the REPL. The continuation prompt is never displayed to ask for continuation of a long line, because such request is always fulfilled by the intermediate interface, which remembers the rest of the line.</p>
</section>
<section id="unixreadline" class="level2">
<h2 class="anchored" data-anchor-id="unixreadline">Unix/readline</h2>
<p>R on the Linux/macOS (Unix) command line in interactive mode uses the readline library. The library itself provides a line editor supporting lines of unlimited length, dynamically allocated, but R truncates those lines.</p>
<p>This was fixed in R-devel, this console implementation has been extended so that it returns the additional segments of the line to the REPL. The user experience would be the same as newly with Rterm: the continuation prompt would never be displayed to ask for the rest of a line (it could be displayed to ask for the rest of an R command spanning multiple lines) and the user would be able to fully edit the whole line.</p>
</section>
<section id="repl-parser" class="level2">
<h2 class="anchored" data-anchor-id="repl-parser">REPL, parser</h2>
<p>Some corner cases issues related to continuation of long lines were fixed also in the REPL and the parser (lexer). Some of these involved multi-byte characters: the lexer didn’t correctly handle when the input for the parser ended within such a character. Some were general programming errors.</p>
</section>
</section>
<section id="limitations" class="level1">
<h1>Limitations</h1>
<p>One may run into limitations imposed by other software. For example, on Linux, the line editor of the terminal (so called cannonical input mode) has a limit of 4096 bytes. More characters can be entered, but they are not stored, so it appears as if the input is truncated. This limit of the terminal does not affect the normal use of R with readline.</p>
<p>On macOS, there is a similar limit of around 1K, and at least observed on one system, one cannot even enter more than that, the input is stopped. Again, this does not affect the normal use of R with readline.</p>
<p>The continuation support visible to the user, as presented in Rgui, is less flexible than support for fully editing the line, as now present in Rterm and on Unix console with readline. With Rgui, one cannot go to previous segments of the line and overflow is only handled at the end of the line. Therefore, e.g.&nbsp;pasting some very long generated text <em>inside</em> an existing line wouldn’t work. For any new console implementation/front-ends, it would probably make sense to implement full editing of arbitrarily long lines.</p>


</section>

 ]]></description>
  <category>R Core</category>
  <category>language</category>
  <category>internals</category>
  <guid>https://blog.R-project.org/posts/long-input-lines/</guid>
  <pubDate>Fri, 30 Aug 2024 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Generalizing Support for Functional OOP in R</title>
  <dc:creator>Tomasz Kalinowski, Michael Lawrence, Martin Maechler, Davis Vaughan, Hadley Wickham, Luke Tierney</dc:creator>
  <link>https://blog.R-project.org/posts/general-support-oop/</link>
  <description><![CDATA[ 





<section id="generalizing-support-for-functional-oop-in-r" class="level2">
<h2 class="anchored" data-anchor-id="generalizing-support-for-functional-oop-in-r">Generalizing Support for Functional OOP in R</h2>
<p>R has built-in support for two functional Object Oriented Programming (OOP) systems: S3 and S4, corresponding to the third and fourth version of the S language, respectively. The two systems are largely compatible; however, they are two fundamentally distinct systems, two systems for a user to understand, two systems for a developer to contend with when making their package interoperate with another, two systems for R core to maintain.</p>
<p><a href="https://rconsortium.github.io/S7/index.html">S7</a> is a new OOP system being developed as a collaboration between representatives from R-Core, Bioconductor, tidyverse/Posit, ROpenSci, and the wider R community, with the goal of unifying S3 and S4 and promoting interoperability. The long-term hope is to incorporate S7 into base R, following a period during which the concepts of S7 have been validated in a standalone R package.</p>
<p>The development of S7 has prompted its authors to reflect on how to generally enable functional OOP systems like S3, S4 and S7, as their remains a lot of room for innovation in this area. Considering just the three mentioned systems, a number of commonalities become evident, including:</p>
<ol type="1">
<li><p>Reifying classes, generics and methods as objects (in S4 and S7),</p></li>
<li><p>Achieving polymorphism through generics dispatching to implementations (methods) based on one or more of their inputs, and</p></li>
<li><p>Modeling objects as collections of components, such as slots or properties, that can be accessed and modified.</p></li>
</ol>
<p>This blog post outlines four patches that have been integrated into base R to make it easier for an S3-based package like S7 to implement the above features.</p>
<p>Each of these four patches introduces a new S3 generic:</p>
<ul>
<li><code>chooseOpsMethod()</code></li>
<li><code>%*%</code></li>
<li><code>nameOfClass()</code></li>
<li><code>@</code></li>
</ul>
<p>This blog post describes these new generics, the features in S7 that motivated their implementation, and their utility to the broader R community outside of S7. These enhancements are generally beneficial for packages integrating or implementing alternative OOP systems. To illustrate these additional benefits, we use the reticulate package as an example. Reticulate bridges Python’s OOP semantics with R’s S3 system.</p>
<section id="chooseopsmethod" class="level3">
<h3 class="anchored" data-anchor-id="chooseopsmethod"><code>chooseOpsMethod()</code></h3>
<p>S3 generics only support dispatching on their first argument, while S4 generics can dispatch on an arbitrary number of arguments. S7 implements double dispatch (i.e.&nbsp;multiple dispatch on exactly two arguments) on top of S3 dispatch. Dispatch in S7 is performed in two stages: S3 generics first dispatch on the first argument to an S7 aware method, which then invokes S7-internal dispatch routines that support multiple dispatch. These are implementation details which, when everything works, are hidden from S7 users. A general complication with implementing any sort of dispatch is that primitives and other functions with internal implementations must dispatch internally, in C code. Internal dispatch and multiple dispatch intersect at the Ops group generics.</p>
<p>Unlike other S3 generics, the Ops group generics, consisting of infix binary operators like <code>+</code>, <code>-</code>, <code>*</code>, and so on, support dispatching to S3 methods on either the first or second argument. Before R 4.3.0, if the two arguments would cause single dispatch to different methods, R would signal a warning and use neither. Starting with R 4.3.0, if different methods are found for the first and second argument to the Ops generic, R will invoke <code>chooseOpsMethod()</code>, giving an opportunity for the object types to declare that their Ops method implementation can handle the combination. If the <code>chooseOpsMethod()</code> method for an object returns <code>TRUE</code>, then the method found for that object is used by the Ops generic.</p>
<p>Because S7 implements its own generic dispatch mechanism, the <code>chooseOpsMethod()</code> method for an S7 object always returns <code>TRUE</code>. This permits, for example, an S7 object type to define a <code>+</code> method that works with other S3 objects, like <code>Sys.Date()</code>.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"S7"</span>)</span>
<span id="cb1-2"></span>
<span id="cb1-3">ClassX <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">new_class</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ClassX"</span>)</span>
<span id="cb1-4"></span>
<span id="cb1-5"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">method</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">`</span><span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">+</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">`</span>, <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">list</span>(ClassX, class_any)) <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">function</span>(e1, e2) {</span>
<span id="cb1-6">  <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"method: X + class_any"</span></span>
<span id="cb1-7">}</span>
<span id="cb1-8"></span>
<span id="cb1-9">x <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ClassX</span>()</span>
<span id="cb1-10"></span>
<span id="cb1-11">x <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Sys.Date</span>()</span></code></pre></div></div>
<pre><code>## [1] "method: X + class_any"</code></pre>
<p>This also has utility for other packages that implement alternative OO systems. For example, reticulate is an R package that embeds Python in an R session, and allows R users to operate with Python objects directly. Python has its own implementation of method dispatch for infix operators like <code>+</code>. The addition of <code>chooseOpsMethod()</code> enables reticulate to implement a default fallback <code>+</code> method that dispatches to the corresponding Python routines, while still allowing R users to define a custom <code>+</code> method for specific Python object types.</p>
<p>For example, the <code>+</code> operator can now select the appropriate method when passed a TensorFlow Tensor and some other Python object. Previously, dispatch failed, because both arguments are instances of an S3 class but would cause dispatch to different methods.</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb3-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"reticulate"</span>)</span>
<span id="cb3-2"></span>
<span id="cb3-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Adding a NumPy array and an R array dispatches to the default</span></span>
<span id="cb3-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># `+` method for "python.builtin.object"</span></span>
<span id="cb3-5"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">np_array</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">array</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>)</span></code></pre></div></div>
<pre><code>## array([2, 4, 6], dtype=int32)</code></pre>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Adding a TensorFlow Tensor and an R array invokes</span></span>
<span id="cb5-2"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># invokes the specific `+` method for "tensorflow.tensor"</span></span>
<span id="cb5-3"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">array</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> tensorflow<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as_tensor</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>)</span></code></pre></div></div>
<pre><code>## tf.Tensor([2 4 6], shape=(3), dtype=int32)</code></pre>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb7-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Adding a NumPy array with a TensorFlow Tensor.</span></span>
<span id="cb7-2"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Prior to R 4.3.0, this would signal an error and warn:</span></span>
<span id="cb7-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">#   Incompatible methods ("+.tensorflow.tensor", "+.python.builtin.object")</span></span>
<span id="cb7-4"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Beginning with R 4.3.0, chooseOpsMethod() is called to choose the appropriate</span></span>
<span id="cb7-5"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Ops method, and this now works</span></span>
<span id="cb7-6"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">np_array</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> tensorflow<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">as_tensor</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>)</span></code></pre></div></div>
<pre><code>## tf.Tensor([2 4 6], shape=(3), dtype=int32)</code></pre>
</section>
<section id="section" class="level3">
<h3 class="anchored" data-anchor-id="section"><code>%*%</code></h3>
<p>Base R defines additional operators that dispatch internally and support some form of multiple dispatch, but are not in the Ops group. The matrix multiplication operator, <code>%*%</code>, is a prime example.</p>
<p>Prior to R 4.3.0, <code>%*%</code> only worked with S4 objects or base R objects. Now, S3 methods for this operator can be defined as well. Just like the Ops group generics, <code>%*%</code> will dispatch on the second argument, and use <code>chooseOpsMethod()</code> to resolve conflicts.</p>
<p>To enable this new behavior, a new generics group was created: <code>matrixOps</code>. Presently, the sole member of the group is <code>%*%</code>, though in the future, <code>crossprod()</code> and <code>tcrossprod()</code> are expected to join the group.</p>
<p>The addition of the <code>"matrixOps"</code> group is a narrow backwards-compatible change, intended to enable existing operators to work with S3 and S7. Future functions that aim to support double dispatch via S3 will be able to make use of this general mechanism, rather than requiring further changes to R’s S3 semantics.</p>
</section>
<section id="nameofclass" class="level3">
<h3 class="anchored" data-anchor-id="nameofclass"><code>nameOfClass()</code></h3>
<p>Though S7 uses S3 dispatch, S7 reifies classes as objects, and its user interface is based on those objects, instead of class names. The class name stored in the class vector for S3 dispatch is considered an implementation detail (it is non-syntatic and mangled to minimize the potential for conflicts, and not intended to be user-facing). Additionally, it can be subject to change if the generic definition migrates between packages.</p>
<p>For these reasons, it is desirable to be able to check for inheritance without directly using the S3 class name. To enable this for S7 and potentially other packages with their own class representations, beginning with R 4.3.0 <code>base::inherits()</code> now accepts arbitrary objects for the second argument: if the second argument is an S3 object, the new <code>nameOfClass()</code> S3 generic is invoked to resolve the appropriate class name.</p>
<p>This permits usage in S7 like this:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb9" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb9-1">ClassX <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> S7<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">new_class</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"ClassX"</span>)</span>
<span id="cb9-2"></span>
<span id="cb9-3">x <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ClassX</span>()</span>
<span id="cb9-4"></span>
<span id="cb9-5"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">inherits</span>(x, ClassX)</span></code></pre></div></div>
<pre><code>## [1] TRUE</code></pre>
<p>This also has utility for other R packages, in particular, packages where an external class hierarchy is being mirrored in R. For example, in reticulate, the S3 class vector for Python object is generated from the Python module where the class is defined. However, in sophisticated codebases like Keras and TensorFlow, the location where a class is defined in the Python sources is considered an internal implementation detail, and often changes between library versions (i.e., the module where a class is defined is decoupled from the module where users are meant to access the class). To ensure compatibility between and across Python library versions, reticulate now implements a <code>nameOfClass()</code> method which enables R usage like this:</p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb11" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb11-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"tensorflow"</span>)</span>
<span id="cb11-2">x <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> tf<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Variable</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">3</span>)</span>
<span id="cb11-3"></span>
<span id="cb11-4"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">inherits</span>(x, tf<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>Variable)</span></code></pre></div></div>
<pre><code>## [1] TRUE</code></pre>
</section>
<section id="section-1" class="level3">
<h3 class="anchored" data-anchor-id="section-1"><code>@</code></h3>
<p>Prior to R 4.3.0, <code>@</code> could be used to access slots of S4 objects. Beginning with R 4.3.0, this operator can now perform S3 dispatch as well. This bring <code>@</code> to parity with <code>@&lt;-</code> (which has long been able to do S3 dispatch), and makes <code>@</code> a peer (in dispatch-ability) of <code>$</code>. This is a strictly backwards compatible change, as S3 dispatch is only performed <em>after</em> S4 routines have had an opportunity to dispatch.</p>
<p>The ability to have formally defined slots or properties is one of the benefits of S4 and S7; it enables collaborative work at a greater scale than the ad-hoc conventions of S3 classes, and having a dedicated operator for property access in S7 is desirable. We expect other OOP systems will have similar concepts and also benefit from this generalization. Other than performing S3 dispatch, there are no other user-facing changes with <code>@</code> in base R—the default behavior of <code>@</code> in the absence of an S3 method is unchanged.</p>


</section>
</section>

 ]]></description>
  <category>classes/methods</category>
  <category>packages</category>
  <guid>https://blog.R-project.org/posts/general-support-oop/</guid>
  <pubDate>Fri, 17 May 2024 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Statement on CVE-2024-27322</title>
  <link>https://blog.R-project.org/posts/cve-2024-27322/</link>
  <description><![CDATA[ 





<p><em>This is a brief statement on behalf or the R Core Team on the serialization bug recently reported by the cybersecurity form HiddenLayer. The bug has been reported as a vulnerability with id <a href="https://www.cve.org/CVERecord?id=CVE-2024-27322">CVE-2024-27322</a>.</em></p>
<p>R is a full-featured language that includes functions to interface with both the host operating system and the Internet. As with any such language, there are many ways to write code with malicious intent. Users should ensure that they only use R code and data from trusted sources and that the privileges of the account running R are appropriately limited. These basic security principles are not limited to the R language.</p>
<p>The ability to write malicious code in R does not imply that the language itself is insecure. The base R language is maintained by a select group of high-trust individuals in The R Core Development Team, most of whom have been working on the language for decades.</p>
<p>Like any open source project, we rely on our wide user base to identify bugs in the source code. We welcome all bug reports, including those with security implications. The bug report by HiddenLayer is one such example. Briefly, promises are language objects used to implement lazy evaluation. Promises constructed in R are always bound to an environment, but HiddenLayer found a way to make an unbound promise by deserializing a hacked file. This bug has been fixed in R 4.4.0 and any attack vector associated with it has been removed.</p>
<p>We reject the idea that there are wider security implications associated with promises or serialization, both of which are core features of the language.</p>



 ]]></description>
  <category>security</category>
  <category>internals</category>
  <guid>https://blog.R-project.org/posts/cve-2024-27322/</guid>
  <pubDate>Fri, 10 May 2024 00:00:00 GMT</pubDate>
</item>
<item>
  <title>R on 64-bit ARM Windows</title>
  <dc:creator>Tomas Kalibera</dc:creator>
  <link>https://blog.R-project.org/posts/windows-aarch64-2/</link>
  <description><![CDATA[ 





<p>R 4.4.0, to be released tomorrow, comes with experimental native support for 64-bit ARM Windows machines (aarch64, arm64). Rtools44 with native support for the platform has been released at the beginning of March.</p>
<p>The effort to add Windows/aarch64 support to R has been announced in <a href="../../2023/08/23/will-r-work-on-64-bit-arm-windows">August, 2023</a>.</p>
<section id="installation" class="level1">
<h1>Installation</h1>
<p>R for Windows/aarch64 is distributed separately from the primary distribution for x86_64 (64-bit Intel machines). There is a special installer of R for Windows/aarch64. From the user perspective it is almost identical to the installer for x86_64. Windows 11 is required by R on aarch64.</p>
<p>The R installer builds are expected to appear on CRAN pages, but for now can be downloaded from a <a href="https://www.r-project.org/nosvn/winutf8/aarch64/">temporary location</a>, currently including R 4.4 pre-release builds and daily builds of R-devel.</p>
<p>The installer is itself an x86_64 application, which runs on Windows/aarch64, emulated by the OS transparently to the user. The directory layout is slightly different, because R “sub-architectures” are not used. Hence, e.g.&nbsp;<code>Rgui.exe</code> is located directly in <code>bin</code>, while it is in <code>bin/x64</code> on x86_64.</p>
<p>The installer can be built natively on Windows/aarch64 from R sources the same way that an installer for x86_64 can be build on x86_64. The installers provided via the URL above are cross-compiled on existing x86_64 infrastructure: the native code is cross-compiled using Rtools44 on Linux/x86_64, the rest including the installer is built on Windows/x86_64.</p>
<p>R is by default installed to <code>Program Files/R-aarch64</code> (unlike <code>Program Files/R</code> with an x86_64 build) and the default library directory is named <code>aarch64-library</code> (unlike <code>win-library</code> with an x86_64 build). This is to allow side-by-side installation with the x86_64 build, which can run on Windows/aarch64 via emulation.</p>
<p>The x86_64 version, at the time of this writing, didn’t pass its installation tests when running in the emulator due to numerical differences. The aarch64 did and in addition to that it is indeed expected to have better performance and lower power consumption than running in the emulator.</p>
<p>Using R on Windows/aarch64 should be considered experimental, both the native version and the emulated x86_64 version. The native support for Windows/aarch64 is still limited in R packages (more below): some packages and external software still need to be adapted. R isn’t tested with CRAN nor Bioconductor packages on Windows/aarch64, neither natively nor via x86_64 emulation. This is unlike x86_64, where regular and frequent testing of R packages by CRAN also indirectly tests R itself.</p>
<p>Building of R on Windows/aarch64 is documented in <a href="https://cran.r-project.org/bin/windows/base/howto-R-4.4.html">Howto: Building R 4.4 and packages on Windows</a>.</p>
</section>
<section id="rtools" class="level1">
<h1>Rtools</h1>
<p>Rtools44 includes an installer for Windows/aarch64 and toolchain/libraries tarballs (full, base, cross-compiler, Tcl/Tk bundle), mirroring the usual components provided for Windows/x86_64. They are available from <a href="https://cran.r-project.org/bin/windows/Rtools/rtools44/rtools.html">CRAN</a>. Later versions of Rtools43 already included unreleased temporary builds for aarch64 for testing with the development version of R, which were available from a temporary location.</p>
<p>Rtools44 is based on LLVM 17 and has almost all the libraries provided in the Intel version. It uses LLVM linker (lld), resource compiler and C++ library (libc++) instead of binutils and libstdc++. LLVM is needed because GCC does not support Windows/aarch64.</p>
<p>The Rtools44 installer is an x86_64 application, which runs on Windows/aarch64 in the emulator. Rtools44 is installed by default to <code>rtools44-aarch64</code> (unlike <code>rtools44</code> with an x86_64 build) as to allow the two installations to coexist on Windows/aarch64. Having an x86_64 version may be useful for package developers with aarch64 as their main platform. The R installer build will use the correct Rtools automatically: R for aarch64 installed via the installer would use Rtools for aarch64, R for x86_64 will use Rtools for x86_64.</p>
<p>Note that in a side-by-side installation of Rtools from the installers, one would have even the Msys2 part of Rtools twice, because the Rtools applications are fully independent. The Msys2 part of Rtools consists of build tools, which are currently built for x86_64 and run in the emulator. As is the case since Rtools42, users can also install Rtools from a toolchain+libraries tarball and maintain a separate, single Msys2 installation with the build tools for use with different versions of Rtools toolchain+libraries (say for x86_64 Rtools43, x86_64 Rtools44 and aarch64 Rtools44).</p>
</section>
<section id="installing-packages" class="level1">
<h1>Installing packages</h1>
<p>At the time of this writing, CRAN does not distribute binary packages for aarch64. All packages have to be installed from source as they are on Linux. That is also the default when R is built on macOS from source (CRAN builds of R use binary packages from CRAN).</p>
<p>This makes little difference for packages implemented solely in R: only installation takes slightly longer, because packages are byte-compiled and prepared for lazy loading. A binary package only has to be decompressed, which would typically be faster.</p>
<p>Users however now have to install Rtools whenever they need to install any package that “needs compilation” (so includes C, Fortran or C++ code). Installation of these packages will take much longer than when installing binary versions. Still, it should not make the use any harder: Rtools can be installed accepting the defaults, with no configuration necessary. Base and recommended packages are already part of the R binary distribution, so one does not need Rtools for those. Parallel installation provides a significant performance boost on multi-core systems.</p>
<p>Expect, though, that some packages will fail to install from source either because they need to be adapted or they require some external software not available yet for the platform.</p>
</section>
<section id="porting-and-testing-packages" class="level1">
<h1>Porting and testing packages</h1>
<p>In an ideal case, even packages needing compilation would not need any changes: portable, standards-compliant C, Fortran and C++ code should just work and in many cases, CRAN and Bioconductor packages just do. Also, CRAN packages are already tested with LLVM clang on macOS and Linux and with LLVM flang-new on Linux, so code accidentally making assumptions about the platform/compiler should has mostly been detected. A number of GCC compiler extensions is supported by LLVM as well.</p>
<p>In an ideal case, packages carefully written to follow <a href="https://cran.r-project.org/doc/manuals/R-exts.html">Writing R Extensions</a> and the <a href="https://cran.r-project.org/web/packages/policies.html">CRAN repository policy</a> in linking to external libraries should also work without changes even to the linking/make files, and many do. Such packages use libraries from Rtools (rather than downloading pre-compiled versions), and the names of these libraries are almost always the same as for x86_64. The names differ where R already provides macros (BLAS, LAPACK, OpenMP, Fortran runtime) abstracting out the differences. The macros are documented and recommended for use in Writing R Extensions, so portable packages have already been using those. Packages can use <code>pkg-config</code> in Rtools, which should reduce the maintenance cost for package authors on Rtools updates, but it is not related to nor needed by the aarch64 support.</p>
<p>In practice, it is easy to overlook some portability issues when testing on a single architecture - and some package code is Windows-specific, so only got tested with GCC and for Intel CPUs so far. Close to 100 patches have been provided to CRAN and Bioconductor package maintainers to fix their packages to build and pass checks on Windows/aarch64. Many of these were easy-to-fix issues of the kinds described above (C++ features removed from current/corresponding standards, GCC-specific compiler options, explicitly naming OpenMP runtime used with GCC, naming conflicts due to using R headers before system headers, the C/C++ compilers being stricter than GCC and spotting issues hence not found before). Some of the cases required more work.</p>
<p>A long-lasting problem of the R ecosystem is R packages downloading pre-compiled code (static libraries) at build time (R CMD INSTALL). Some packages are doing this despite Rtools has the required libraries, some even when Rtools has newer versions. On CRAN, this is not allowed (it is a violation of the CRAN repository policy). Clearly, such code definitely won’t work on aarch64 when pre-compiled for x86_64. Some of the provided patches have been just emergency hot-fixes disabling the downloads on non-Intel platforms and falling back to compilation from source or to using libraries from Rtools. The recommended way is to use libraries from Rtools, and if they are not there, provide build configurations via upstream MXE.</p>
<p>A number of packages don’t work on Windows/aarch64 because they link to libraries provided by externally installed software, and such software sometimes isn’t yet readily available (ported or packaged). At the time of this writing, this includes important software like Java, MPI (Microsoft MPI doesn’t support aarch64) and JAGS. This is likely to improve as the platform becomes more popular, and indeed R is not alone, other open-source projects are being adapted to support Windows/aarch64 as well.</p>
<p>This progress also sometimes drives the need to upgrade, some Rtools libraries had to be updated to newer versions that support Windows/aarch64. One case that will require cooperation from CRAN package authors is Intel TBB (Threading Building Blocks). It is part of Rtools (both x86_64 and aarch64), but a number of CRAN packages use an older version that is embedded in another CRAN package. That older version, however, cannot be built for Windows/aarch64. The new version made several breaking changes in the API and packages will have to be updated.</p>
<p>Another case was MinGW-W64: the experimentally built Rtools43 versions for aarch64 conditionally used a newer version of MinGW-W64 to avoid some issues on aarch64, but this required debugging and fixing naming conflicts in packages. These conflicts would have had to be fixed regardless of aarch64. Rtools44 now uses the newer version of MinGW-W64 on both platforms.</p>
<p>Luckily it is extremely uncommon for CRAN and Bioconductor R packages to include (and unconditionally use) Intel assembly code. Such cases should have an automatic fallback to portable C code for platforms that are not supported. A number of Rtools libraries themselves are built for aarch64 thanks to opting-out from hand-written assembly code, not available for aarch64.</p>
<p>Some CRAN packages cannot be compiled for Windows/aarch64 because of limitations of the flang-new compiler from LLVM.</p>
</section>
<section id="hardware-availability" class="level1">
<h1>Hardware availability</h1>
<p>There are no powerful 64-bit ARM laptops to run Windows natively, but in October 2023, Qualcomm announced their new 64-bit ARM “Snapdragon Elite X” platform with integrated “Oryon”, with an expectation of PCs to be available in mid-2024. The company is to provide a further statement on April 24 (the day when R 4.4 is to be released). It is not hard to find articles speculating about upcoming systems based on this platform. It is not hard to find benchmark results measured on test systems against Apple M3, generally claiming comparable performance, so there seems to be some progress.</p>
<p>64-bit ARM processors for server systems have been available for 3 years (Ampere Altra from Ampere Computing). One can also buy developer/prototyping workstations based on these processors (AADP). Still, while there is some server hardware available to run Windows on ARM, Microsoft has not, yet, released a Windows server OS. Currently, one would have to use Windows 11 (and probably figure out how to do that in line with the license in case of a server system).</p>
<p>R package developers who work on Apple Silicon machines can already test their packages on Windows 11 running in UTM/QEMU (or Parallels). Such developers probably want to test and debug their packages on Windows, and using the aarch64 version of Windows on their machines would be natural (the OS emulator, at the time of this writing, doesn’t support full virtual machines, and, running natively would be preferred, anyway). Regular macOS R users would naturally run R natively on macOS, but if for some reason they were forced to use Windows, it would again be Windows on aarch64.</p>
</section>
<section id="ready-for-package-developers" class="level1">
<h1>Ready for package developers</h1>
<p>It is common and necessary that software for new platforms is developed before the platforms actually become generally available, because, the development simply takes too long. This is why this work on R and Rtools has been started as soon as compilers for the platform became available, particularly when the LLVM flang-new compiler became stable enough to build base R and recommended packages. R is not alone, there is clearly an effort to make open-source software support Windows/aarch64, e.g.&nbsp;there is already Python for the platform.</p>
<p>R for Windows/aarch64 and Rtools is now stable enough to allow for development/porting of R packages to the platform and this is what needs to happen if R should be a viable option also on this platform. Most of the fixes would entail making the packages more portable and standards-compliant, which is a good thing on its own and would simplify maintenance and potential support for other future platforms or compilers. This is particularly important for maintainers of packages with a large number of reverse dependencies.</p>
<p>The easiest way of testing packages for Windows/aarch64 is probably still to run Windows 11 in a virtual machine on Apple Silicon hardware. This is even recommended by Microsoft, it is documented to work in Parallels and QEMU (UTM). The work on R/Rtools has been done via UTM/QEMU. Using Windows natively this way should be easy for R package developers: the OS and R works almost identically to what they have been used for. Performance is comparable to doing similar work natively on an Intel laptop.</p>
<p>An indirect alternative to cover some cases, but more technically involved, is to use Rtools44 cross-compilers to test compilation on Linux/x86_64. This will only work for packages with a simple installation process, but the approach has been used before when looking for issues in code detected by LTO. The cross-compilers in Rtools themselves should be sufficiently tested, because Rtools libraries themselves are cross-compiled. Also now R installers for aarch64 are cross-compiled, even though minimally tested. However, the limitation is in R packages that were not designed for cross-compilation. Also, this would only test compilation of native code.</p>
<p>The aarch64 support has been partially integrated into the infrastructure used for testing development versions of Rtools (“ucrt3”) and package build outputs from this are <a href="https://www.r-project.org/nosvn/winutf8/ucrt3/">available</a> for CRAN and some Bioconductor packages. These tests patch packages using patches available also at that site, which have been provided to package maintainers and will be removed from the site as they are incorporated into packages. The installation tests are only ran occasionally and may be well behind the current CRAN source package repository.</p>
<p>At the time of this writing, there is no GitHub Actions runner support for Windows/aarch64, which can be tracked <a href="https://github.com/actions/runner-images/issues/768">here</a>.</p>


</section>

 ]]></description>
  <category>R Core</category>
  <category>platform support</category>
  <category>windows</category>
  <guid>https://blog.R-project.org/posts/windows-aarch64-2/</guid>
  <pubDate>Tue, 23 Apr 2024 00:00:00 GMT</pubDate>
</item>
</channel>
</rss>
