Interested in Org but not enough to monitor the mailing list? In this blog I digest developments, and each month regurgitate what I consider the highlights, along with anything else Org-related I consider nifty.

Posts are published in .html, .org, .org.html, .txt, and .pdf forms — because why not 😛

November 2021 2021-11-30

With a 9.5 release highlight post last month, and the month before skipped, it’s now three months since the last regular instalment of TMIO. Let’s get back up to date on some of the latest happenings with Org.

Org as markup

Looking at the wider ecosystem, it certainly appears that there is a growing appetite for Org markup outside org-mode. More projects like Hugo and Logseq seem to be interested in supporting Org markup, and there has been a recent growth in editor extensions like Neovim’s orgmode.nvim (started in March this year) and Sublime Text’s OrgExtended (started in June this year).

Interest in Org as a general-usage markup format can also be seen within the Org project. Primarily lead by Nicolas Goaziou, there is an ongoing attempt to codify the Org syntax in a formal specification in the Worg document Org Syntax (draft). Other members of the Org mailing list have directed their effort to creating non-elisp parsers for Org, both to help Org tools be created in other languages, and as put in the README for Tom Gillespie’s laundry parser

The long term goal of this work is to provide a reference that can be used to standardize Org syntax and behavior and to specify various levels of compliance for an implementation of Org mode.

Earlier this week Karl Voit, the author of the rather well-known document Org Mode Is One of the Most Reasonable Markup Languages to Use for Text, surprised the mailing list by announcing his independent creation of a multi-leveled standard for Org syntax subsets called “Orgdown” (the name is a blend of “Org-mode” and “markdown”, but the standard is only a subset of Org). Each level defines a compliance score given by a mix of parsing and editing support, with example compliance scores for the first (and currently only) level of the standard given for common tools.

At this stage, it isn’t clear exactly how the Org-outside-Emacs landscape will evolve, but the swelling interest is very encouraging.

An Org parser in Julia

Speaking of parsers, I may be somewhat biased but I’m quite happy that a Org parser for Julia now exists 🎉.

OrgMode.jl is a parser, but also intended as a general-purpose Org library for Julia. It’s only been a week since development started, but it currently supports most of the Org Syntax draft specification, along with the rendering of a parsed Org AST to a TTY or back to Org text. A few utility functions are also included, such as filtermap which operates similarly to org-element-map.

Autoloading citation backends

One small but impactful change is autoloading of citation backends. Until recently before say using the csl backend, one needed to (require 'oc-csl) or face error messages.

Now, if you have a line like:

#
#+cite_export: FORMAT ...

org-mode will try to load the file oc-FORMAT before trying to process citations.

This should make getting started with citations in Org just a bit easier.

A nicer :tangle-mode syntax

The standard way of setting a :tangle-mode has typically been by providing a closure that makes use of Elisp’s octal syntax, such as (identity #o755). This is unnecessarily verbose, and certainly doesn’t feel natural.

With the addition of a small mode-interpreting function (org-babel-interpret-file-mode) It is now possible to specify :tangle-mode using three different forms of shorthand

octal
o755 is equivalent to (identity #o755)
chmod
chmod-style inputs like u+x are now parsed to a file mode[1] with the the base/default mode set by org-babel-tangle-default-file-mode.
ls -l
strings of the form given by ls -l like rwxr-xr-x are also accepted

This means the following forms are now all equivalent:

#
:tangle-mode (identity #o755)
:tangle-mode o755
:tangle-mode a=rx,u+w
:tangle-mode rwxr-xr-x

It has also been noted on the mailing list that the :tangle-mode (identity #o755) form works by being transformed to :tangle-mode 493 during parsing. Similarly :tangle-mode 755 is equivalent to :tangle-mode (identity #o1363). For some values the decimal and octal interpretation are both valid file modes. Due to the clear potential for confusion, and since file permissions are an important security consideration, it has been suggested on the mailing list that these forms should be depreciated with a warning in future. No decision has been made yet though.

Org element parser cache

Ihor Radchenko has done some fantastic work over the past few months by overhauling parts of org-element.el to introduce extensive caching. org-element is the Org markup parser inside org-mode. This allows for a huge jump in speed, and also provides a few functions which fetch information without updating the cache — allowing for particularly speedy lookups with a small sacrifice to correctness guarantees on one or two properties in particular cases.

Several org-mode APIs now make use of the cache to dramatically improve speed. Aside from improvements to typically slow operations, this is ideal for situations involving frequent buffer edits. It’s no understatement to say that this work is transformative.

One potential beneficiary from this work is actually fontification. It has become increasingly apparent that the current regex-based method for buffer fontification is imperfect, and can actually differ from the true structure of the document as parsed (authoritatively) by org-element. This has lead to the well-received suggestion on the mailing list to rewrite the fontification code to be built on org-element instead.

Inline source block fontification

I think inline source code blocks are an underappreciated feature of Org. I don’t think it’s helped that they have not been visually treated at all differently from plain text. Now though, they have a new dedicated face (org-inline-src-block) and in the same manner as source blocks, based on org-src-fontify-natively can be fontified using the language’s major mode.

inline-src-block-fontified-vs-code.png
Figure 1: Side-by-side comparison of a identical paragraphs using code (~) markup and inline source blocks (src_).

If you aren’t familiar with inline source blocks, you’re missing out. They are very much the inline cousin of source blocks, and so support all your favourite Babel features like code execution and header arguments. This provides a fantastic capacity to inline dynamically computed expressions, and optionally show the code that produces them.

inline-src-block-julia-demo.png
Figure 2: A paragraph making use of evaluated inline source blocks. Note that the ⟨11⟩ is a prettified results macro (using a potential future org-mode patch).

Functions as default heading arguments

Matt Huszagh has contributed a patch that allows functions to be used as values for default header arguments. This is great for arguments where a sensible default can be provided by evaluating a function on-the-fly.

Consider for example the arguments required to produce a simple image using R with Babel:

Org mode
#
#+begin_src R :results graphics file :file myimage.svg
library(ggplot2)
ggplot(mpg, aes(displ, hwy, colour = class)) + geom_point()
#+end_src

In a Jupyter-style (.ipynb) or throwaway document, we likely don’t care about the file name at all. With these new capabilities, we can provide a file name dynamically as a default argument!

First we must write a function that when run at the source block will give us a suitable file name, like so

Emacs Lisp
#
(defun my/org-src-sha-to-image ()
  (concat "generated-"
          (substring
           (sha1 (org-element-property :value (org-element-at-point)))
           0 8)
          ".svg"))

Let’s also write a function to guess whether the source block produces a plot by checking if there’s a plot command on the last line.

Emacs Lisp
#
(defun my/org-src-guess-results-type ()
  (if (string-match-p "^ *\\(?:plot\\|ggplot\\)([^\n]+\n?\\'"
                      (org-element-property :value (org-element-at-point)))
      "graphics file" "replace"))

Then we can just use these function in place of a static value in the default header arguments variable — that’s all it takes.

Emacs Lisp
#
(setq org-babel-default-header-args:R
      '((:results . my/org-src-guess-results-type)
        (:file . my/org-src-sha-to-image)))

This means for most cases we can now get away without any header arguments at all.

Org mode
#
#+begin_src R
library(ggplot2)
ggplot(mpg, aes(displ, hwy, colour = class)) + geom_point()
#+end_src

It’s always lovely to see more ways of reducing boilerplate.

Proportional image widths

Previously, as long as org-image-actual-width was nil or a list of the form (default-value), org-mode would display images according to a :width attribute (e.g. #+attr_html: :width 400px) by simply looking for the first #+attr_ affiliated keyword and reading the numeric component of the :width as the number of pixels wide the image should be.

This has now become somewhat fancier. The image-width determining logic has been extracted to a new function (org-display-inline-image--width) which will now extract floating-point values like 0.7 and interpret them as that portion of the accessible text width in the buffer.

proportional-image-width.png
Figure 3: A containing with an image set to half of the accesible text width

This means that a width parameter like #+attr_latex: :width 0.7\linewidth the image will displayed as 70% of the buffer text width. This also supports percentage value, like #+attr_html: :width 80% by dividing the number before the % by 100 as a floating-point value. As always, if you don’t like the way display width is inferred here you can override it by putting a #+attr_org: :width X statement first.

Support for proportional image widths extends to the (default-value) form of org-image-actual-width, as now if you set it to say (0.9) which will cause images without any width specification to be displayed at 90% of the buffer text width.

If you want to have some images displayed as their actual width you can use the new special width parameter t to set this on a per-image basis with #+attr_org: :width t. Now all you need to do is remember to put this first. Based on current discussions on the mailing list though, soon #+attr_org will be prioritised when determining display image width, no matter which order you put the attributes in. I do like having one less thing to remember 🙂.

Other improvements

  • Allow citations immediately following an item bullet TEC
  • Allow citations immediately following a footnote definition Nicolas Goaziou
  • Update some obsolete function references Marco Wahl
  • ob-gnuplot is now maintained by Ihor Radchenko
  • Improve makescript support for ORGVERSION in tag-less mirrors Nicholas Vollmer
  • New ob-julia, now maintained by Pedro Bruel
  • Allow for no indentation, but preserving current indentation by setting org-indent-indentation-per-level to 0 David Lukes
  • Eliminate some byte-compile warnings Nicholas Vollmer Bastien
  • Support Greek smart quotes Juan Manuel Macías
  • org-mouse support for intermediate-state checkboxes Jim Porter
  • Allow nested parenthesis in org-compile-prefix-format %(sexp) expressions Ihor Radchenko
  • oc-csl / citeproc improvements András Simonyi
  • Move more unmaintained/overly niche ob-* files to the contrib repo, reducing the maintainer burden Bastien
  • Allow use of a function for org-agenda-overriding-header for dynamic headers Christopher League
  • Improve org-protocol URI decoding Max Nikulin
  • Remove some obsolete LaTeX packages from the default packages list TEC
  • Add support for text and year citation styles to oc-csl András Simonyi
  • Produce lower-case keywords in ox-org TEC
  • Improve ob-gnuplot argument processing Ihor Radchenko
  • A collection of oc-* improvements Nicholas Goaziou
  • Support bare author citations in oc-csl TEC
  • Add :options LaTeX attribute to tables Juan Manuel Macías
  • Fix display error with ob-plantuml and html export Su Lin
  • More tests! Ihor Radchenko
  • Documentation improvements! Marco Wahl Stefan Kangas Daniel Fleischer Wiliam Denton Thomas Dye Bastien Bruce D’Arcus Kyle Meyer Nicolas Goaziou

Bugfixes

  • Fix heading insertion in a case where point is before any heading Marco Wahl
  • Prevent stringp error when tangling Org from an org-src edit buffer Mark Dawson
  • Prevent indent-tabs-mode from messing with justification in ASCII exports Morgan Willcock
  • Fix form of default Babel haskell header args Ihor Radchenko
  • No more duplicated logbook entries for repeated tasks Ihor Radchenko
  • A headline fontification edge case Sébastien Miquel
  • Refactor code that needed Emacs 28 Kyle Meyer
  • Make sure a terminating emphasis marker can’t be used as a beginning emphasis marker in fontification Ihor Radchenko
  • Allow footnotes at footnote definition start Nicholas Goaziou

October 2021 2021-10-31

Turns out that life became busy enough that instead of delaying, last month’s post had to be cancelled. We’re now back to business as usual though 🙂.

There have been some cool recent developments in Org over the past two months, but you’ll have to learn about those is next month’s edition as Org 9.5 has been released 🎉. So, let’s go over some of the changes I’m most excited about, in no particular order. To get a more complete picture of the latest changes, see ORG-NEWS.

The new citation engine

August 2021 2021-08-38

Dear readers, unfortunately I’ve been unusually busy as of late, and will continue to be for a bit over a month. As such, it’s entirely likely that the next post or two may be late make use of creative dates too.

In terms of Org development, we’ve had a fairly slow month — mostly tweaks and fixes, concentrated on the new org-cite functionality. However, there have been big developments with Roam as of late.

July 2021 2021-07-31

Last month I not-at-all-subtly hinted that a certain long-awaited feature was arriving imminently. At this point, I think it’s a good idea to set the tone for the rest of this post.

celebrate-citations.svg

Citations

June 2021 2021-06-34

The previous two months have been pretty good for Org development — with many bug fixes and feature improvements. This month has been substantially slower than the last[1], but that’s not to say not much is happening: in fact, there are some rather nifty contributions lined up for the not-too-distant future and a certain long-awaited feature branch[2] is getting very close to merging 😀. You’ll just have to stick around to hear about those in a future edition of TMIO 😉.

dilbert-zenos-paradox.jpg
Figure 4: It’s right around the corner, I swear!

Customise the reference command used in LaTeX

May 2021 2021-05-31

Async Babel sessions have landed

Jack Kamm has made a fantastic contribution to Org: ob-comint.el now supports asynchronous output from source blocks with full support for sessions 🎉.

If you haven’t used sessions before, you’re in for a treat! By simply assigning a named session to a source code block, e.g. :session foo, the same process will be reused for any other code blocks that specify the foo session. To do this for a certain language across the entire document, you can set the header argument property, i.e.

Org mode
#
#+property: header-args:lang :session foo

Along with the asynchronous process communication in ob-comint.el, we have an implementation for Python, and we should see support for R and Ruby in the not-too-distant future 😍.

Read more…

April 2021 2021-04-30

A discussion on contributor support

Concerns were raised[1] about some contributors’ patches languishing, and it not being made clear how long it might take to get a response from someone.

In response to this, a the new role of Contributor Steward has been created to: help ensure contributors get a timely response, help out with preliminary patch feedback, and keep updates.orgmode.org up to date.

Org now has three Contributor Stewards to ease the process for patch submitters and core maintainers:

  • Timothy / TEC
  • Tim Cross
  • John Corless

Read more…

Welcome 2021-04-26

Introduction

Org is an absolutely marvellous project. However, a quick glance at https://orgmode.org can lead one to think “so… it’s an Emacs version of Markdown? What’s the big deal?”. While it’s easy to understand how someone might think that at first, that impression misses two crucial points:

  • While for simple constructs (bold, italic, headlines, etc.) the syntax is very much analogous[1], Org scales to much more powerful forms that are a headache to replicate in Markdown
  • Org mode was developed in Emacs, for Emacs. The integrations for Org put every other plaintext markup editing experience to shame. This is a bold statement, and I stand by it.

Read more…

CC0 To the extent possible under law, TEC has waived all copyright and related or neighboring rights to This Month in Org.