A Mailman's Digitized Grammar of Action

A chapter from a recent handbook on privacy studies by Jo Pierson and Ine Van Zeeland pointed me towards an older article by Philip E. Agre, called Surveillance and capture: Two models of privacy. Agre surely took his own work on privacy seriously, because it seems that he disappeared in 2008 and lives completely off-grid since then. Pierson and Van Zeeland particularly highlight Agre’s “capture model” that expresses how various human activities are reorganized so that they can be “captured” and tracked in ICT systems.

By analyzing the elements of patterns of human activity you can rearrange them into what Agre calls “grammars of action.” That is something we already know from industrialization and automation processes. What Agre adds is that the imposition of these grammars of actions on human activities also allows systems to better track these activities. Hence the relevance for privacy studies. It may very well be that humans end up fighting a losing battle against the informational needs of ICT systems. Since reading about Agre’s “grammars of action” I start seeing more subtle effects of digital technology in my own life. I want to share a recent example.

I currently live together with eight other people, so it happens quite often that we accept packages for housemates when they are not home themselves. Let’s call two of my roommates Alice and Bob. The other day the doorbell of my housemate Alice rang, but Alice turned out to not be home. After hearing the doorbell ring for a second time in the distance, Bob opens the door to find the mailman holding a package for Alice. Of course, Bob offered to accept the package, but to his surprise the mailman refused to hand over the package because he already registered it in the digital Track & Trace system as being not delivered. Instead, he said, Alice had to pick up the package later at the address of some store.

We first of all see that the actual job of a mailman nowadays consists for a great part of entering information about his physical activities in a digital system, specifically designed to track the whereabouts of each parcel at any time. That this ipso facto also tracks the mailman is part of the job nowadays. The presence of ICT systems thus clearly reorganizes the labour process of mailmen. Secondly, digital systems intervene in direct, physical interactions between people in quite subtle ways. It was very weird for Bob to open the door and find a mailman unwilling to hand over a package that was physically present at the intended address.

These kind of social effects of digital technology are very hard to predict from a design perspective. Nevertheless, I suspect that as digital technologies are becoming ubiquitous in our daily lives, the future years will see a rise in research precisely on their opaque rol in shaping our lifeworld.

Custom Note Tagging System with Ctags and Vim

In the previous post I expressed my requirements for an ideal note taking system and took the first steps in designing it for my preferred editor, which is Vim. My overall desire is to create an ecosystem of interconnected notes in such a way that this system does not only become an extension or recording of my thoughts, but also a quasi-independent dialogue partner in the creative process of writing. The idea is that when you are going to write something, you start by opening a note on the topic of choice, and that from there on you can effortlessly follow links to other related notes to discover new lines of thought. To this end, I wanted to implement a tagging system that is tailored to the way I make notes, in addition to the search functions for file names and their contents that are discussed in the previous post.

For example, I was reading an article from Stiegler and encountered an interesting thought on capitalism and the Anthropocene, so I added the tags “@capitalism” and “@anthropocene”. At that point that specific place in the text is connected to all my other notes on capitalism or the Anthropocene and included in many possible trajectories through the note system.

Initially I was working on my own implementation by playing around with Python and Vimscript, but I have settled on a solution that is fast, cross platform, with minimal dependencies and perfectly integrated within Vim.

The credo of most of my posts on Vim so far has been that even though Vim is known as the programmer’s editor, non-technical writers should also leverage its power and endless options for customization. This post is written in the same vein because it repurposes a technical tool called ctags. The original purpose of ctags is to go over a code project and make an index of all function names and provide a link to the place where they are declared (a feature nowadays integrated in full blown IDEs). This allows the programmer to easily navigate through a complex coding project. Over time this program has been extended to be used with many other programming languages as well, even though it has retained the reference to the C programming language in its name. And here is the crux: later variaties of the ctags program, called Exuberant Ctags and Universal Ctags allow you to define extensions towards other languages!

What this means is that I can define my own syntax for the tags I’m going to use and then let ctags create an index of those tags with links to their corresponding files. As an icing on the cake: due to its strong roots in programming culture, Vim has native support for navigating with those tags! N.B. to be clear: even though Vim has native support for ctags, it is not a plugin but an external program that does not automatically ship with Vim.

In this post I’ll walk you through the process to set this up and explain the rationale of each step along the way. Note that even though I design this for Vim, this system works well for any editor that is smart about ctags or something equivalent. If you are not interested in the technical details at all, you can skip the next section. If you have no idea what I’m talking about and just want to see some pictures, start with the last section.

Defining and parsing the syntax of your tags ¶

Because I write in Markdown the hashtag is not a candidate for our tag syntax because it already indicates a header. I chose to define tags instead as such: “@tag”. You can define your tags in a different way of course, but I suggest you keep it simple because you will have to write the rule that correctly parses your tags.

Different ctags versions offer different options, so it is good to provide a quick overview. Historically ctags has first evolved into Exuberant Ctags and recently into Universal ctags. Exuberant Ctags introduces support for other languages than C and first introduced the possibility to support other languages in two ways. The first and simplest option is to provide a regular expression, which is basically a rule to find your tag by matching a particular pattern in a string. The second option is to define your own full blown parser. We are not creating a whole new language, but only need to find simple tags, so we are obviously going for the first option.

Because non-technical writers very likely do not know how to write regular expressions (regex for short), I’ll walk you through the process of writing one. Be aware that there are many dialects for writing regex, but that all versions of ctags use an incredibly old version called Extended Regular Expressions (ERE) which really limits what we can do. Another reason to keep things simple.

This is how such a reasoning process could look like. Let’s say we write two tags on one line for example as such:

@meme-machine @vimlife

Our rule should recognize two single tags. Intuitively, the rule should be something like: find an “@” and then match all word characters until you encounter a character that clearly does not belong to the word.

This simple regex would be expressed as @(\w+):

@   find a literal "@"
(   start a "capture group", i.e. the part of the expresion that we are interested in
\w  the "@" should be followed by a "word character" (alphabetic letters and numbers)
+   there should be at least one character after the "@" but there can be infinitely more
)   close the capture group. The part within brackets is the tag. 

So when we write @vimlife in our note, the regex will find vimlife as the tag. However, this is a bad regex. The first problem is that it will not match @meme-machine correctly. Because the - is not a word character, this regex will incorrectly return meme as a tag instead of meme-machine. We could improve on this regex by refining our rule: find an “@” and then match any character until you find a space or a newline.

This regex could be expressed as @(\w.*)\s:

@   find a literal "@"
(   start a "capture group", i.e. the part of the expresion that we are interested in
\w  the "@" should be followed by a "word character" (alphabetic letters and numbers)
.   a wildcard that matches any character whatsoever, including characters such as "-"
*   there can be 0 or more of those wildcard characters
)   close the capture group. The part within brackets is the tag
\s  when we find a space we are at the end of the tag. 

This avoids the previous problem, but introduces a new one. One feature that is common nowadays but absent in the regex for ctags is a thing called lazy evaluation. If the regex would be lazy then the rule would stop matching at the first space, which separates the two tags. But unfortunately our regex is greedy, meaning he will make the match as long as possible. The combination .*\s matches everything until a space character is found, but the end of the line is also a space character type! As a result, @meme-machine @vimlife is considered to be a single tag, which is obviously not what we want.

In modern regex dialects you could explicitly make the star match lazily by appending a question mark. Then the regex would look as such: @(\w.*?)\s. But this is not possible in the ERE dialect of ctags. In other words, time to take a step back and re-evaluate how to solve this problem without lazy evaluation, which is better in any case because lazy evaluation is computationally expensive. Click away if you want to think about it yourself.

If not, my simple solution is @(\w\S*):

@   find a literal "@"
(   start a "capture group", i.e. the part of the expresion that we are interested in
\w  the "@" should be followed by a "word character" (alphabetic letters and numbers)
\S  any *non*-whitespace character (inverse of \s)
*   0 or more non-whitespace characters
)   close the capture group. The part within brackets is the tag

This is a more efficient approach with the same effect as using lazy evaluation. Because a tag now does not contain any whitespace characters by definition, the first tag is matched separately. I still enforce that the first character after the “@” has to be a word character, otherwise “@" or for example (and amusingly) the regex pattern itself would be a tag.

UPDATE: 15-1-2020

Of course, URLs that contain “@” will also be matched with the current regular expression. We can exclude these matches by requiring that “@” either occurs at the beginning of the line or is preceded by a “space” character (i.e. “@” occurs at the beginning of a new word somewhere in a sentence ). In other regex dialects you have the special \b sign to indicate word boundaries, but not in the ERE POSIX dialect. We can however write (^|[[:space:]])@(\w\S*):

( open a group
^ match the beginning of the line
| or instead match
[[:space:]] any whitespace character
) close the group
@   find a literal "@"
(   start a "capture group"; this the part of the expresion that we are interested in
\w  the "@" should be followed by a "word character" (alphabetic letters and numbers)
\S  any *non*-whitespace character (inverse of \s)
*   0 or more non-whitespace characters
)   close the capture group. The part within brackets is the tag

I adjusted the code below to this new regex. Note especially that we now have two groups, and that we are interested in the second one only, so our back reference changes from \1 to \2.

Installing and configuring ctags ¶

There is still another problem left. Modern implementations of regex engines in programming languages offer the option to find all regex matches of a given line. However, when we use regex only our pattern only matches the first tag. This means that in @meme-machine @vimlife the second tag will never be registered.

I thought about this for a bit, but long story short, this problem cannot in principle be solved with Exuberant Ctags when we take the regex route. So if you for some reason insist on using Exuberant Ctags rather than Universal Ctags the tagging system strictly requires you to only put one tag on each line. If that’s the way you want to go, then create a configuration file called .ctags in your home directory and write the following specification of our markdown tagging language.

--langdef=markdowntags
--langmap=markdowntags:.md
--regex-markdowntags=/(^|[[:space:]])@(\w\S*)/\2/t,tag,tags/

The first line defines the name of our language, the second line associates our new language with a file extension (I use .md for Markdown) and the third line specifies our regex pattern, a backreference to our capture group (\2) and lastly a specification of the type of tag this is. I just called it tag, t for short. As you might see, these options are flags that will be given to the ctags command. You can download exuberant tags here or simply with your package manager of choice.

Despite they limitations of using regex only, the successor of Exuberant Ctags called Universal Ctags does have a way to return multiple tags per line through the use of an experimental feature. Using Universal Ctags has other benefits as well. The benefits as I perceive them are:

• Support for even more languages, including Markdown!
• Does not necessarily use a system wide configuration, so you can define your needs on a per project basis
  • The config file can thus be included in your GitHub repository and you’ll be set up immediately after cloning your repository on any computer.
• Multiline support (this is actually what we abuse to find multiple tags on one line)

You can download the latest build of Universal Ctags for Windows on the project’s GitHub page. If you are using Windows, make sure you place the executable in a folder that is contained in the PATH variable, so that you can run ctags from the command line. On Linux just download the package with your package manager of choice. If you use the Arch User Repository (AUR) look for this package.

To avoid conflicts with Exuberant Ctags the configuration files are now located in a special directory. So after installing create the directory .ctags.d/ and create the file md.ctags within that directory. The configuration syntax has slightly changed. The main change is that we will use a multiline regex now. Because programming languages that rely on brackets to indicate scopes can spread structures of interest over multiple lines, the usefulness of pure regex is limited. This feature can however also be used to find multiple matches within a single line. Have a look here for documentation, if you are interested. Otherwise, copy the following configuration to your configuration file in ./.ctags.d/md.ctags, relative to your project folder.

--langdef=markdowntags
--languages=markdowntags
--langmap=markdowntags:.md
--kinddef-markdowntags=t,tag,tags
--mline-regex-markdowntags=/(^|[[:space:]])@(\w\S*)/\2/t/{mgroup=1}

Note that you can’t call your custom language just “markdown” because that language definition already exists (unlike in Exuberant Ctags). By default Markdown headers etc. will be produced as tags, but I actually do not care about that and added the second line to explicitly indicate I want to use my own language definition and not the default language also mapped to the .md extension. Almost good to go!

Creating tags ¶

Tags can now be created easily from the command line by changing your directory to your project folder (here, our notes repository), and then running ctags recursively on the current folder (recursively indicating that all subfolders will be taken into account as well):

ctags -R .

This will create a file names tags in your project folder. You can open it to inspect if everything worked out correctly. As you will see, the generation of tags is very fast as this tool is designed to still work for very large and complex code projects, where each file has many tags. We’ll have less files and significantly less tags per file.

So far this post has been completely editor agnostic. But the beauty of using ctags for our note taking tags is that Vim handles them exceptionally well.

The power of the whole command line is at your fingertips, because Vim can run external commands from within the editor. So you do not have to leave Vim to generate the tags. You can simply type :!ctags -R . , where the dot refers to the current directory.

This does however assume that Vim’s current directory is your project root folder. Verify this with the command :pwd. Alternatively, you could replace the dot with the path towards your notes directory. But the better option is to use Vim’s native cd (change directory) command and change the working directory to your notes folder. For example, type :cd ~/Documents/Notes. This also allows you to more efficiently search files by only considering your notes.

To make this whole process smooth we can easily make some mappings so we don’t have to bother typing commands anymore. Remember that <leader> is by default the backslash.

" Generate ctags
nnoremap <leader>tt :!ctags -R . <CR>

Alternatively, if you do not want to see the command output you can generate the tags silently, but a quirk with this is that you have to force a redraw of your screen afterwards. Try it out without in terminal Vim, and you’ll see what I mean.

" Generate ctags silently
nnoremap <leader>tt :silent !ctags -R . <CR>:redraw!<CR>

As shown in the previous post on note taking in Vim, I have a mapping that immediately brings me to the index of my notes and also automatically changes my directory to the project root. I strongly recommend this. If you have an idea you quickly want to write down you can jump to your notes folder within a second and start writing.

" Go to index of notes and set working directory to my notes
nnoremap <leader>ni :e $NOTES_DIR/index.md<CR>:cd $NOTES_DIR<CR>

Alternatively, you can define a function to change the directory to the root of the file you are currently editing (e.g. the index of your notes):

" Change directory to directory of current file
nnoremap <leader>cd :cd %:h<CR>

UPDATE 14/4/2020: I’ve received replies and emails specifically from MacOS users that my ctags extension does not work. I do not have access to a machine with MacOS and cannot reproduce the issue. I suspect that the universal-ctags build for MacOS uses a slightly different regex engine. Luckily, a helpful comment from Fernando offers a fix. I’ve had confirmation from at least one other MacOS user that this fixed his issue as well.

Navigating tags from within Vim ¶

As said before, Vim has great support for handling ctags. Vim knows about the location of your tags file. If Vim doesn’t find your tags, check that you are in the right directory and also make sure that the tags variable makes sense with :set tags? Alternatively, set tags explicitly in your .vimrc or ._vimrc (Windows) configuration file for example as such:

set tags+=./tags;,tags

The semicolon allows Vim to recursively move up a file tree to look for a tags file in case it doesn’t find one as explained here. You can now search tags with autocompletion with the tselect command, or ts for short.

I for example have a tag @workflow, so I would type in :ts work <TAB>, which auto completes ts workflow. This will open a menu with a numbered list of all files with the tag workflow. You can quickly jump to a file by entering its number.

Pro tip: make your search case insensitive! This makes autocompletion ignore the case, so that :ts Work<TAB> still autocompletes to :ts workflow. To achieve this, set this in your .vimrc:

" Ignore case in searches
set ignorecase

Another really nice feature is that you can search on the tag that is currently under your cursor (or one place to the right). You do this with the <Ctrl>-] command.

This will jump to the first encountered tag. What’s also really nice is that it jumps to the exact line where the tag is used, so you do not have to search further manually.

One interesting note here is that the way we use tags is really quite different than its regular use in programming. The base case in programming is that you define a function once and that it is called in many places. The desired default behavior is that from all those places where it is called, you can quickly jump to the place where that function is defined. It can however occur that you override a function definition, so that in fact you end up with an ambiguous tag where the same tag links to two different locations.

We however desire and exploit the ambiguity of tags.

The whole principle of rhizomatic navigation that I desire is exactly that tags are defined in multiple places. The tselect command already gives you all options for navigation. But if we want to find all files for the tag under the cursor rather than only the first one, we do not use <Ctrl>-] but g ] instead. This shows all ambiguous tags, i.e. all the files in which it is “defined.”

It gets even better. Because tags are so well integrated in Vim, your fuzzy finder plugin will almost certainly also be able to search the tags file. I use CtrlP because it works well both on Linux and Windows. My previous post mentions my setup for CtrlP using ripgrep. When searching using <Ctrl>-P you can toggle whether you are searching files, buffers or tags with and (backward and forward, see :help ctrlp-mappings). Alternatively, you can directly invoke the :CtrlPTag command. Various autocompletion plugins will also be able to suggest and complete tags.

UPDATE 15/4/2020: You probably want to define a quick mapping for this, for example:

" Binding for searching tags ("search tag")
nnoremap <leader>st :CtrlPTag<CR>

One last trick before I’ll share screenshots of an example workflow. If you follow a tag to another file, look around for a bit, and then want to go back to where you where before going down the rabbit hole, you can type <Ctrl>-t to go back to through what is called the tag stack. The tag stack basically tracks the trajectory you’ve taken by following tags through your notes. A beacon of light in the mess of the creative mind.

Screenshots of example workflow ¶

After opening gVim (the screenshots are from my Windows machine), I press \ni (Notes Index) to change the working directory to my notes and to open the index page.

Starting from my index page, I can’t quite remember the name of a tag, so I’ll decide to use fuzzy finding.

The detail shows the fuzzy nature of the tag search. I typed AC (randomly), but as you see also results like “Jacobs” and “aircraft” are displayed.

From the list of suggestions I chose “Jacobs”, which is the name of a university professor. This could be some author you are writing a paper about. As a result I’m now viewing lecture notes of a security course I followed, which discusses a range of topics. We hold our cursor on the tag “security”.

The command g] opens a list of all ambiguous tags. We see that another file is also about security. So let’s expand our horizon and enter its number to visit that file.

We have now reached another file with course notes on a highly related topic. It discusses security, but clearly from a more societal and philosophical perspective, i.e. the human side of computer security.

And so on. I might by now have a more specific idea to write about. If it’s a single concept I’ll make a small note in my “Zettelkasten” directory (for which I have another easy binding), where I’ll might decide to explicitly link to all the files I’ve explored. If I add the security tag there as well together with a new tag, I’ve opened up new lines of thought!

Conclusion and summary of used Vim mappings ¶

Like with my previous post on this topic, I’m writing about this while exploring ideas so everything is WIP. It is possible to define multiple regex rules for our custom language, so it’s easy to add more features to this tagging system. I might for example explore the usefulness of tracking explicit markdown links to other files with this system.

Let me know if you have suggestions! Feedback is welcomed.

If at some point I haven’t changed my system in a long time I’ll likely bundle together a .vimrc with everything you need. The system so far actually heavily depends on native Vim mappings, so you do not need much at all (Keep It Simple Stupid)! With the code below you can install CtrlP using vim-plug. There are two external dependencies, Universal Ctags and ripgrep which however are both cross-platform, minimalistic and do not require configuration outside of what is provided below. Plug and play. For now, I’ll provide a quick summary of mentioned Vim bindings and settings (and some not mentioned) as requested here:

" Specify a directory for plugins
" - Avoid using standard Vim directory names like 'plugin'
call plug#begin('~/.vim/plugged')

" Fuzzy file finding
Plug 'kien/ctrlp.vim'

" Initialize plugin system
call plug#end()

" Ignore case in searches
set ignorecase

" Generate ctags for current working directory
nnoremap <leader>tt :silent !ctags -R . <CR>:redraw!<CR>

" Change directory to directory of current file
nnoremap <leader>cd :cd %:h<CR>

" Quickly create a new entry into the "Zettelkasten" 
nnoremap <leader>z :e $NOTES_DIR/Zettelkasten/

" Go to index of notes and set working directory to my notes
nnoremap <leader>ni :e $NOTES_DIR/index.md<CR>:cd $NOTES_DIR<CR>

" 'Notes Grep' with ripgrep (see grepprg)
" -i case insensitive
" -g glob pattern
" ! to not immediately open first search result
command! -nargs=1 Ngrep :silent grep! "<args>" -i -g '*.md' $NOTES_DIR | execute ':redraw!'
nnoremap <leader>nn :Ngrep 

" Open quickfix list in a right vertical split (good for Ngrep results)
command! Vlist botright vertical copen | vertical resize 50
nnoremap <leader>v : Vlist<CR>

" Make CtrlP and grep use ripgrep
if executable('rg')
    set grepprg=rg\ --color=never\ --vimgrep
    set grepformat=%f:%l:%c:%m
    let g:ctrlp_user_command = 'rg %s --files --color=never --glob ""'
    let g:ctrlp_user_caching = 0
endif

" Binding for searching tags ("search tag")
nnoremap <leader>st :CtrlPTag<CR>

" What to ignore while searching files, speeds up CtrlP
set wildignore+=*/.git/*,*/tmp/*,*.swp

" This step is probably not necessary for you
" but I'll add it here for completeness
set tags+=./tags;,tags

Building a Note-taking System with Vanilla Vim

Vim is my preferred tool for making notes and organizing my thoughts. This posts explains how we can use Vim to build a personalized note taking ecosystem of interconnected notes, only using plain text. I have some requirements for such a system. I want:

• everything in plain text: future proof, portable, searchable.
• version control on my notes
• to be able to smoothly find notes and navigate between them
• a quick way of searching the contents of my notes
• notes to behave like hypertext: they can directly link to other notes
• to do everything within a single consistent context, i.e. never leave Vim to do basic actions.
• my note taking workflow to be cross-platform (for me: work both on Linux and Windows)
• to Keep It Simple Stupid (KISS)

This is my guide for achieving the above. This guide assumes you make notes in Markdown, but you can adapt everything to your needs.

Motivation ¶

The overall goal is to start building a personal plain text ecosystem where notes are linked together in such a way that they do not only help me remember things, but also positively stimulate the process of thinking and writing. If knowledge is your business (if you are a writer, academic, or something of the like) spending some time to optimize your note taking is worth it.

Ideas about how note taking structures your thinking are worked out by the Zettelkasten (indexing cards) system that I found while writing this article. See for example a quote from this post on the benefits of extending your mind and memory through note taking:

Some note archives never move beyond a storage of thoughts and stuff you need for reference. If you connect your notes heavily by creating links between them, though, eventually your Zettelkasten will do more than fill memory gaps. Instead, it will improve the depth of your understanding.

Note taking improves the depth of your understanding for various reasons, but the least superficial is that writing down thoughts forces you to think more coherently by requiring you to make thoughts explicit.

Similar to how your ability to teach is a good indication of whether you understand the essence of something, just so is note taking a way to see if you really get the main point of for example a paper that you are reading. There is a hermeneutical principle at work here. The philosopher Gadamer for example describes how understanding requires you to go into dialogue with whatever you are trying to understand. This implies that you make your pre-understanding or prejudice explicit so that it can can be challenged in dialogue with something other, whether that is another person with a different background in teaching or a book you are reading.

So ideally notes are really an ecosystem rather than just a storage container. Just collecting stuff is not the same as understanding it. I really like the idea of “communicating” with your notes in the creative process, also worked out on the Zettelkasten blog. As you write down more notes and keep connecting them, your notes will gain complexity up to the point where, let’s say a year later, you will find surprising connections between new thoughts and old notes. In this sense your notes can really become this other party that you enter into dialogue with and a stimulant for creative thinking.

N.B. this post thus does not focus on more dynamic plain text notes, such as todo lists or agenda’s. Vim doesn’t try to emulate the functionalities of other types of software, but tries to do one thing well and then integrate with other tools dedicated to other purposes. Having said that, doing everything in plain text is cool, but you should probably have a look at Emacs Evil Mode in combination with Org mode.

Step 1: Create a single repository for your notes ¶

Because I want version control on my notes, it makes an awful lot of sense to organize all my notes in a single directory so that I can straightforwardly use Git. You could distribute your notes over your system and still have version control using a tool like GNU Stow, but this effort is not worth it for me. Keep It Simple.

Additionally, this way you can easily organize your notes with a meaningful folder structure that is easy to remember and efficient to navigate, assuming that you will have a folder structure at all.

Step 2: Think about a folder structure for your workflow ¶

After setting up your note directory, it makes sense to give some thought to how you are going to organize your notes within that directory.

There are many systems for organizing your life that rely on various forms of note taking. This is not the main focus of this post, but here are some pointers to relevant methods:

• Getting Things Done (GTD)
• The Organized Mind
• Building A Second Brain (BASB)
• Zettelkasten

These methods generally begin with decluttering your mind by writing things down. There are plenty of online tutorials on how to implement these methods with digital note taking systems and tools, e.g. with Emacs. We will do something similar in Vim.

Personally I have not (yet?) settled for a single note taking system. In the end it’s all about finding out what works for your specific interests and use cases.

The Zettelkasten system for example urges you to only express a single idea in each note (atomicity) and rely heavily on tagging these ideas and linking them together with other notes. This implies that it is futile to try to organize everything in folders. You instead rely on searching tags and following hyperlinks between notes. Below I’ll explain how I use Vim to follow those links.

I really do try to minimize sorting files, but in my case some folder structure is meaningful to me. Having done university courses of two curricula, philosophy and artificial intelligence, it makes sense for me to at least organize long form course notes per year, e.g. under /AI/Y3/ .

If you do project-based work, where projects last for example at least half a year, it would also make sense to me to organize those notes in a folder. I have a separate directory for more free form notes.

Let’s start with the implementation in Vim.

Step 3: Create an index page ¶

Because we have all our notes organized in a single directory, we can make an index as an entry point into our note taking system.

In the root directory of your notes, make a index.md file. We are going to use this index file to make links to other files and directories that we can follow using Vim’s awesome gf command.

The gf (go to file) command opens the filename, directory or url that is currently under the cursor. This means that we can populate our index files with relative paths to files that we often use.

Only use relative paths, because this makes our note system independent of any specific system.

For example, my index file contains a “quick start” menu for useful folders:

Quick start:

- Philosophy/Thesis

All directories:

- AI
- Philosophy
- Miscellaneous
- Workflow\ &\ Skills
- Epub\ Annotations

To to one of these directories all I have to do is place my cursor somewhere on the folder name and type gf to open the file in a new window (vertical split), or gF to open it in the same window, or <C-w>f to open the file in a horizontal split. By the way, run :help key-codes for an explanation of Vim’s notation for key sequences.

Paths with spaces in them can complicate matters. If you escape the spaces the gf command works just fine, the other commands not so much (in terminal Vim at least). What works in all cases is visually selecting the path before running gf.

Next to a quick start menu, I thought it was cool to include a full tree of all notes. By default the tree command only shows the file names at nodes, but for gf to work we need valid paths. Because all notes share the same root directory, we can save space by using valid relative paths. On a Linux system, you can achieve this as such:

tree -f -I "images*|*.pdf|*.py|*.html"

The -f option displays the full paths relative to the root of the notes directory, which is necessary for gf. The -I option excludes everything matching the given pattern. I have some stray pdf, python and html files in my notes directory, so I make sure to exclude them. I also have one directory purely for images I reference in my notes. All notes have the markdown extension (and an occasional Emacs .org file).

To insert the output of the command directly in our text buffer, prepend the command with the read command or its shorthand r, or even shorter (but this overrides the current line), with . (a dot). So you run:

:.!tree -f -I "images*|*.pdf|*.py|*.html"

The resulting tree looks something like this:

.
├── ./AI
│   ├── ./AI/Master
│   │   ├── ./AI/Master/AAPS-brainstorm.md
│   │   ├── ./AI/Master/AAPS-Chemero_Radical_Embodied_Cog_Science.md
│   │   ├── ./AI/Master/AAPS-Kwisthout_Frugal_Explanations.md
│   │   ├── ./AI/Master/AAPS-Lecture_Notes.md
│   │   ├── ./AI/Master/AAPS-Olivia_Scene_perception.md
│   │   └── ./AI/Master/AAPS-presentation_skills.md
│   ├── ./AI/Other
│   │   ├── ./AI/Other/ImageRecognitionWorkshop.md
│   │   ├── ./AI/Other/Jan_Broersen-ACAIS_Responsible_AI.md
│   │   └── ./AI/Other/lamachine.md
│   ├── ./AI/TA
│   │   ├── ./AI/TA/De_Graaf-How_People_Explain_Action.md
│   │   ├── ./AI/TA/Levy-Computers_and_Populism.md
│   │   ├── ./AI/TA/Meynen-Ethics_brain_reading.md
│   │   ├── ./AI/TA/Mittelstadt-Ethics_of_algorithms.md
    
. . . . . . . . . . .. . . . .  etc.

│       ├── ./AI/Y3/RI-Logic_Block.md
│       ├── ./AI/Y3/RI-Multi_Agent_Systems.md
│       └── ./AI/Y3/SEC-Lecture_Notes.md
│
└── ./Zettelkasten
    ├── ./Zettelkasten/BASB.org
    ├── ./Zettelkasten/environment.md
    ├── ./Zettelkasten/memory_techniques.md
    └── ./Zettelkasten/writing_principles.md

You can use this full tree for navigating to any note using Vim’s gf command. To quickly jump to a file name in the tree, just use Vim’s search / in the index file.

UPDATE 14/4/2020: I almost immediately moved away from using the tree. It just looked cool, but it quickly gets unmanageable. Relying on directory structure forces you to categorize notes rather than relying on search tools and direct interlinking. I’ve switched to a “Zettelkasten” approach. However, the idea of an index page is alive and kicking! You can always insert links to heavily used project notes. I currently use a page with all my tags as my index page (for tags, see the next post).

The last thing we need to do is make a fast way to access the index file from anywhere. We can make a mapping to 1) open the index file and 2) change the Vim working directory to our notes directory, so that we can use our relative paths.

One way to do that is to add the following mapping to your .vimrc:

" Go to index of notes
nnoremap <leader>ww :e $NOTES_DIR/index.md<CR>cd $NOTES_DIR

UPDATE 19/12/2019: I now let all note related mappings start with <leader>n. This command is now mapped as <leader>ni for “note index.” This command also has a small mistake. This is the corrected version:

" Go to index of notes and set working directory to my notes
nnoremap <leader>ni :e $NOTES_DIR/index.md<CR>:cd $NOTES_DIR<CR>

N.B. the default leader key is the backslash \ .

<CR> is a carriage return, or an “Enter” in common tongue. We need <CR> to also execute the command, but it is also a cheeky way to chain two commands together on a single line.

$NOTES_DIR is a bash variable that I set in my ~/.bashrc:

export NOTES_DIR=/home/edwin/Documents/Notes

You can of course hardcore that path into your .vimrc, but I preferred to use the bash variable because then I don’t have to change my .vimrc in case I migrate my notes directory for some reason. On Windows I do hardcore the path though. Vim allows you to ignore the Windows convention of using backslashes in paths, so you can use a backslash to escape whitespaces as usual. For example, my remap on Windows looks like this:

nnoremap <Leader>ww :e C:/Users/Edwin\ Wenink/Documents/Notes/index.md<cr>

Whatever we are doing in Vim, our notes are now always reachable within a second.

Step 4: Methods for finding notes ¶

We can now quickly find notes by jumping to our index file, using Vim’s search operator / and then running gf on the search result.

A more rhizomatic way of navigation is to create links within notes to other related notes and follow them with gf. This is our method for building up an ecosystem of interconnected notes.

Another option is to use the default find function, which offers you autocomplete on paths. This is a feasible approach because all our notes are organized in a single directory with a meaningful structure.

Alternatively, just use a fuzzy file finder to quickly find files based on partial matches. A well-known option is fzf. I opted for the Ctrl-P fuzzy finder because making that run on Windows as well was easy.

You can make Ctrl-P faster by letting it use the blazingly fast tool ripgrep, by adding the following to your .vimrc (after installing ripgrep of course):

" Make Ctrlp use ripgrep
if executable('rg')
    let g:ctrlp_user_command = 'rg %s --files --color=never --glob ""'
    let g:ctrlp_user_caching = 0
endif

Another good argument for using ripgrep is that it runs both on Linux and Windows and thus fits my cross-platform requirement. Do note however that ripgrep is an external command line tool and does not ship with Vim, so you need to install it yourself. Have a look at the project page for clear installation instructions.

Something I have not done yet, is design an effective plain-text tagging system UPDATE 19/12/2019: see this post. This blog adds tags behind all files in the index, as a quick hack. This doesn’t fit with the above mentioned approach of automatically generating the navigation tree, as that would replace the tags. I also think a plain-text tagging system is ideally decentralized, i.e. tags are stored in the notes themselves. This is more robust but also tricker to implement efficiently.

Step 5: Navigating between opened files in your buffer ¶

We covered how to find a file you want to read or edit. But how do you efficiently jump back and forth between files that you opened?

Using search functions to find a file each time I need it is superfluous. When I’m writing a paper I want to have various related notes open at the same time and ready for inspection. But you shouldn’t have multiple windows of tabs open for more than two files, first of all because they fill up your precious screen space and secondly because cycling through them is slow.

You should use buffers and the jumplist markers Vim sets for you.

Everytime you open a new file, Vim will open that file in a buffer. It is important to realize that a buffer does not need to have a window, i.e. it is not necessarily visible.

You can see the complete list of open buffers using the ls command, which is familiar to all Unix users. Alternative you can type out buffers. You will see that each buffer is numbered. You can open a buffer quickly either by referring to its number, like so to open the second buffer :b2.

A very nice feature is that the buffer command can expand partial matches with filenames opened in a buffer. If you for example have your .vimrc opened in a buffer, you can type b rc<TAB> to expand rc to the full path to your .vimrc.

You can also cycle through buffers as such (with alternative mappings from vimunimpaired):

• :bprevious or [b
• :bnext or ]b

And you can delete buffers from the buffer list with :bd The command without arguments deletes the current buffer.

I have to admit that I only realized I wasn’t using Vim’s buffers properly after making a field trip to Emacs. Sometimes it is good to explore a different workflow to re-evaluate your habits, and integrate improvements. What was also more prominent in Emacs were functions to quickly jump back and forth between the buffers you visited lasts. I wish I knew that Vim could do this as well from the moment I started using it.

Vim uses movements to navigate through a text. Before each movement Vim sets a marker of your current location and stores it in the jumplist. These markers also work across files.

• With C-o you jump to the previously visited location.
• With C-i you jump one place to the front of the jumplist.

This means that if I open a new file in a buffer for example after using gf, then I can quickly jump back to the previous file with C-o.

If you perform movements in the meanwhile, you’ll have to cycle back through those movements as well. You can also directly alternate between the current file and the last opened file with C-6. Somehow introductions to Vim rarely mention these commands, but they really were eye-openers for me. I do not leave Vim anymore for file navigation, and especially on Windows this makes the whole experience much closer to using the command line.

Proper buffer navigation satisfies my wish to do everything from within Vim. Without buffers, you’ll find yourself opening new Vim instances all the time, which requires your window manager to switch between those instances.

Step 6: Search contents of your notes ¶

Additionally, in the case tag based search would not be enough (true in my case, because I did not implement it), we fall back on searching the contents of our notes.

The most straightforward tool to do this is grep, available on every Unix-like system. However, we wanted our solution to be cross-platform and Windows does not have grep.

A first good approach is to use Vim’s native grep called… vimgrep. Because this doesn’t require an external tool, any system using Vim can use this approach. There is an issue though. vimgrep and lvimgrep (a vimgrep that populates the locallist instead of the cross-file quicklist) are slooooooow.

Luckily, Vim also has a regular grep command that you can configure to use any searching tool. We just need that tool to be cross-platform. We already used ripgrep for Ctrl-P, so let’s use it for Vim’s grep command as well.

" Make :grep use ripgrep
if executable('rg')
    set grepprg=rg\ --color=never\ --vimgrep
endif

Merge the new line with the code block for the Ctrl-P fuzzy finder we saw above. The --vimgrep option emulates vimgrep behavior, i.e. it now returns each match, irregardless whether it’s in the same file or sentence as another hit, as a single result.

The next step is to write a command with a handy shortcut to search our notes and only our notes.

I was inspired by a note searching function from this video of a doctor using Vim to make notes. He uses the slow vimgrep. This is his function:

command! -nargs=1 Ngrep vimgrep "<args>" $NOTES_DIR/**/*.txt
nnoremap <leader>[ :Ngrep

I introduced four changes. We do not use vimgrep anymore, but Vim’s grep command which we redirected to rg. But the syntax of ripgrep is different. It does not take Unix style wildcards, but requires you to explicitly define a glob for the search pattern using -g. I also changed the mapping slightly, using the “n” for “notes”. Instead of .txt files, we only search through .md files. Adjust the glob pattern if you want to include more. Remember that the <leader> is a backslash by default.

" My own version, only searches markdown as well using ripgrep
" Thus depends on grepprg being set to rg
command! -nargs=1 Ngrep grep "<args>" -g "*.md" $NOTES_DIR
nnoremap <leader>nn :Ngrep 

So if we press \nn in Vim, we can immediately type our search term and get all matches in our note directory.

The results of the search populate what is called the quickfix list in Vim. This is a list you can access from any context within Vim (as opposed to the locallist which is bound to the context of the current file).

You can use the following commands to navigate the items in the quickfix list:

• :copen and :cclose for opening the list
• :cnext and :cprev for jumping to next/previous list item
• :cc {nr}: to jump to item number and echo it
• :colder and :cnewer to also navigate older quickfix lists.

The vim-unimpaired plugin provides default mappings for the quickfix list:

• ]q for :cnext
• [q for :cprev
• [Q for :cfirst
• ]Q for :clast

For who is into more advanced stuff, you can run commands on each item in the quicklist with cdo {cmd}, which runs cmd on every list item.

I was also inspired by the Vimming doctor linked above in the creation of a navigation pane for browsing the search results in the quickfix list. The following function and mapping allows you to open a sidebar with the results from our custom “note grep” with \v:

command! Vlist botright vertical copen | vertical resize 50
nnoremap <leader>v :Vlist<CR>

Clicking on a option moves to exact line in file of the search hit.

Sometimes I leave “TODO” notes in notes. This is how the custom sidebar looks after searching my notes for “TODO”:

The sidebar shows the file a match was found in and a preview of the match itself. You can adjust the size of the bar with :vert resize {size}.

Conclusion and ideas for the future ¶

The system described above only uses vanilla Vim functionality without the need for plugins.

I also deliberately do not use additional plugins for previewing my notes, because I write in Markdown and Markdown is designed to be readable in plain text. Instead I advice to explore the various Markdown plugins out there, for syntax highlighting and folding. I can really recommend vim-pandoc and the corresponding syntax plugin.

I’m interested in getting tips from you, especially about a potential implementation for “tags”.

This is my current TODO list:

• Design a tagging system and related search functions
  • UPDATE 19/12/2019: see this post
• Think more about the best file naming convention
  • e.g. use timestaming and write a function to search timeframes

• Write a function for quickly adding a new note

• Think more about my folder organization
  • UPDATE 15/4/2020: I opted for a stricter Zettelkasten approach and now use a single directory. This forces you to rely on proper searching tools and manual interlinking, following the Zettelkasten principles. Additionally, this really simplifies writing your own tools, for example for managing backlinks, because you do not have to worry about paths but only about filenames. Additionally, not relying on paths in your links makes the link robuster, e.g. they do not break when you rename some directory. I do have to mention that this was never an issue with Vim alone, because Vim’s gF can move to parent en sibling directories as well. I would like my markdown links to be strictly correct though, so that they would also automatically work when I publish my notes as a website or preview them in Github (where I store them).

If I work any of these out you can expect a new blogpost about that.

Links for inspiration ¶

• Mark Koester - Plain text life
• Mark Koester - Smart Notes
• Curated overview of Zettelkasten
• Joe Reynolds - You (probably) don’t need Vimwiki
• Conner McDaniel - Vim notetaking tips

On function creep, privacy and encryption

Some technologies have the interesting property that they take on roles that were never intended or foreseen by their designers.

Johannes Gutenberg introduced the printing press in Europe and used his technique to efficiently produce bibles, spread the word of God, and support the authority of the Catholic Church. Instead, the introduction of efficient printing techniques in Europe “introduced the era of mass communication which permanently altered the structure of society. The relatively unrestricted circulation of information—including revolutionary ideas—transcended borders, captured the masses in the Reformation and threatened the power of political and religious authorities; the sharp increase in literacy broke the monopoly of the literate elite on education and learning and bolstered the emerging middle class” ( Wikipedia).

The technology of mechanical printing changed European civilization in spite of and beyond the intentions of its designer. And in this case, we can applaud this as something beautiful: technology has the power to redefine the boundaries of thought and action and transform our social realities for the better. This transformative power is in fact something that is sought after nowadays. Take for example the Internet of Things movement that entertains the ideal of connected technologies embedded in our homes and ultimately our cities, to the point where they are ubiquitous and shape our life world. Despite this being a dream for some, others see in this the voluntary and naive submission to a surveillance society.

Function creep and security ¶

In other cases, the application of a technology outside of its intended context is simply inappropriate. When it comes to security, Bruce Schneier writes that “far too often we build security for one purpose, only to find it being used for another purpose – one it wasn’t suited for in the first place.” The correct functioning of a technology is not independent of the values of the social context in which they are embedded. Schneier mentions how US driver’s licences increasingly transformed from being a simple credential for showing that you know how to drive a car, to something you could use to prove you are old enough for buying liquor. With this changing social context, it suddenly became valuable to create fake driver’s licenses, and this bumped up its security needs. Another one: “Security systems that are good enough to protect cheap commodities from being stolen are suddenly ineffective once the price of those commodities rises high enough.” Security measures are always taken relative to how much time and money you expect an attacker to put in. When the value of the thing you secure goes up, your security measure goes from being appropriate to being a laughing stock.

The use of a technological design beyond its intended purpose is called function creep. This is not only a potential issue for security, but also for privacy if the function creep involves personal data. The linking of personal data from various sources can lead to a quite accurate personal profile that may lead to some convenient applications, such as a personalized search experience, but also to questionable practices such as political microtargeting, which may disrupt the democratic political process. The issue here is not just the particular usage of personal data that may be desirable or not, but that often the used data is originally collected for other purposes. The public outrage about the Cambridge Analytica-scandal is understandable because personal data collected in the context of a social network was used in another context for the Trump campaign. Those people may have consented to sharing their data with Facebook, but would likely not have consented to their data being used for a political campaign.

Function creep and contextual integrity ¶

Why function creep is such a threat to privacy is best explained by Helen Nissenbaum’s concept of contextual integrity. When we visit a doctor we consent to disclosing very personal information so that the doctor can adequately help us (we are obliged to give the doctor complete and accurate information, actually). If you agree that this is not a privacy breach, then you should also agree that privacy is more than straight up secrecy or confidentiality. But consent in the context of the examination room does not mean that the doctor can share this information outside of this well-defined context of medical practice. You do however likely accept that the doctor may need to consult with a specialist, and consequently you give away a bit of control over your data. Privacy is thus more than confidentiality or being in full control of your data at all times. The relative importance of all these aspects of privacy are heavily context dependent, and can therefore not simply be translated from one context into another. So neither can consent.

For the course where I work as a teaching assistant, students were asked to give an example of function creep, some positive and negative implications and how to resolve the negative implications. Reading and evaluating the resulting small essays were proof of the importance of teaching, as they forced my to formulate my own thoughts on the subject. I noticed one trend in particular, namely that many students proposed encryption as a solution to the privacy issues concerning function creep.

Is encryption a solution? ¶

Some students found a blog of Michael Zimmer where he gives an example of privacy invasion due to function creep. He mentions how a dean of a Washington high school thought he saw two girls kissing. A quick check with the footage of the surveillance camera confirmed this, and the dean consequently informed the parents of one of the girls about this. The girl got pulled out of school. The presence of a surveillance camera is acceptable for ensuring public safety, but was instead used to enforce social norms: textbook function creep (and perhaps the dean was an actual creep, who knows).

In this scenario the inappropriate function creep is caused by human interference. One proposed solution was thus the design of an autonomous AI system that encodes the used data and only extracted features relevant for its goal: e.g. ensuring public safety. Another proposal was to encrypt all collected data, so that explicit permission to use the data is always needed. Let’s leave the technical feasibility of these proposals aside for now.

These answers make sense given that they seem to assume that function creep is due to explicit human involvement (and specifically, with less than pure intentions). Encrypting data is indeed very important for protecting sensitive data against people with bad intentions that try to steal and misuse that data. But I would argue that function creep does not depend on this explicit human interference.

Good encryption ensures data protection, which is preceded by the moment of data collection. Compare how the GDPR battles function creep: before collecting data you have to specify exactly for what you intend to use it, why your need for that data is legitimate, and how long you need it. Function creep is thus already relevant at the stage of collecting data and specifying the purpose of data collection. This process of data collection can be fully automated without humans necessarily trying to make inferences on particular individuals. But the critical point is that no data is used that was legitimately recorded for another purpose, for which encryption is maybe a necessary condition but not a sufficient one. Imagine an automated AI system recording and linking data haphazardly without much oversight on the purpose of data collection. Following the first proposed solution, the AI would encrypt the collected data, protecting it against humans with malicious intent. But I would argue that in this scenario things did not get better but perhaps rather worse. The system extracts features based on data that may be inappropriately gathered and linked (function creep), and in this case encryption would even further limit the explainability of decisions made or supported by such a system.

Encryption also raises the question: Encrypted by who? Who is in control of the data? Let’s say a law obliges me to encrypt the data I gathered. Since I have the key to decrypt the data, I can still use the data when I need it, potentially for another purpose than the one I stated when I collected the data. In other words, misuse by the data controller is not necessarily counteracted by the technical measure of encryption alone. Encryption protects against misuse by other actors.

I would thus say that even though encryption is extremely important for data protection, it does not in itself prevent function creep. However, that does not mean that encryption cannot help battling function creep. The app IRMA, developed by the Privacy by Design foundation offers attribute-based authentication with zero-knowledge proofs in a way that is sensitive to contextual integrity. (They have very strong ties to Radboud University, but I am in no way affiliated.) I can for example load the attribute that I’m an inhabitant of Nijmegen from the local municipality into IRMA. If I need to authenticate to an application that needs to know where I live, I can prove that I indeed own this attribute without disclosing any other attributes, such as my age. Likewise, I can prove that I’m older than 18 to another application, without having to disclose my exact age (zero-knowledge proof) or disclose where I live (irrelevant attribute). The context sensitivity thus lies therein that the user remains in control of which attributes to disclose in each particular context and only in that context, without having to give away more information.

Such use of encryption addresses the two main points I thought the above answers on encryption missed. Firstly, IRMA is sensitive to contextual integrity because it uses cryptography to ensure only relevant attributes are used, for which the purpose of original collection matches the legitimate use case of the app. Secondly, this use of encryption is effective against function creep only by giving the user control over identity management and the authentication process in a decentralized manner.

This does not solve issues with surveillance of course, but was the first thing that came to my mind when thinking about a way in which encryption might help to battle privacy-invading function creep. Privacy by design is a good example of how encryption can be used effectively against function creep by embedding the relevant ethical and societal values in a responsible design practice.

Pick a random background on each boot (Linux)

LightDM is a display manager that requires a “greeter” that prompts a user for credentials and lets a user log in to a session. On my Arch system I currently use the LightDM GTK greeter. You can specify a background of your choice for the greeter if you think a black screen is a bit too boring. Some people also think that just having one background is too boring. This quick write-up shows how you can pick a different background each time you boot up your system. If you do not use LightDM, you can easily adapt the instructions in this post to your needs. I will discuss two different scenarios.

Scenario 1: Static Greeter background, random thereafter ¶

It could be the case that you want your LightDM greeter background to be static because you have a background that really suits the greeter, while at the same time you would like a random background after logging in. To achieve this, firstly specify the static background in the configuration file of your greeter at /etc/lightdm/lightdm-gtk-greeter.conf.

My configuration file looks like this:

[greeter]
theme-name = Adwaita-dark
icon-theme-name = HighContrast
position = 16%,center 40%,center
screensaver-timeout = 50
background = /usr/share/pixmaps/background.jpg
default-user-image = /usr/share/pixmaps/xterm-color_48x48.xpm
user-background = true

As you can see, you can set the background for the greeter here. There is a small catch: it is easiest if you place your background in /usr/share/pixmaps, because that location is readable by LightDM. This way you do not have to worry about permissions.

Secondly, you can use feh to generate a random background from a folder of backgrounds. In this scenario, the order of things is important. We use the LightDM greeter settings to set the background of the greeter, but then we want to change the background before starting up our window manager of choice (i3 or dwm in my case). We can achieve this by calling feh from ~/.xprofile, because LightDM sources ~/.xprofile before starting the window manager. Calling feh from ~/.xinitrc did not work for me in combination with LightDM, because then LightDM overwrites the background you set at the beginning of the X session.

So simply make ~/.xprofile if you do not have it yet, and enter the command feh --randomize --bg-fill ~/Pictures/backgrounds/*. This command assumes your backgrounds are stored in ~/Pictures/backgrounds/, so adjust this as needed.

Notice that that command creates a file called ~/.fehbg with your background settings.

If you do not use LightDM at all, you can use this method all the same to set a random background for your window manager, skipping the first step.

Scenario 2: random LightDM background. ¶

If you want the background of both the LightDM greeter and your window manager to be chosen randomly at startup, we can write a script to replace background.jpg in /usr/share/pixmaps/ with a random background. Alternatively, you could write a script to directly write to the greeter configuration file, but my approach is easier and has low risk, because you do not adjust any configuration file.

If you run ls -l /usr/share/pixmaps/ | grep background.jpg, you’ll see that the background file is owned by the root user by default. If you want to run a script to change the background file as a regular user, you need to make sure to give write access to the background file. One way to do that is by making your user the owner of the background file as such:

sudo chown -R edwin:users /usr/share/pixmaps/background.jpg

Where edwin is my username (run whoami to retrieve it), and users is the group name of regular users.

After doing this, running the ls command from above gives the following output:

-rw-r--r-- 1 edwin users 819492 Apr 3 2018 background.jpg

Now we can write a script to pick a random background picture from the directory where we keep our backgrounds, and replace background.jpg with that new background. I wrote a simple bash script for this:

#!/bin/bash

cd /usr/share/pixmaps

background_home=/home/edwin/Pictures/backgrounds

# Shuffle backgrounds and pick one
background=$(ls $background_home | shuf -n 1)

# Replace current LightDM greeter background
cp $background_home/$background background.jpg

Don’t forget to run chmod +x on your script to make it executable. Now all we need to do is call this script at the right place and time.

We simply call the script from ~/.xprofile like we did before. Replace the line we added for scenario 1 with a simple call to your script.

Voila! Each time you boot up your system you’ll be greeted by a new background. The background sticks around when you start up your window manager, unless you explicitly override it, like we did in scenario 1 with feh.

Scenario 3: Change background at interval ¶

If you want to keep changing your background picture with some time interval after you have logged in, you could adjust the bash script above to include a loop that updates the background for example each hour using feh.

The adjusted bash script could look like this:

#!/bin/bash

cd /usr/share/pixmaps

background_home=/home/edwin/Pictures/backgrounds

# Shuffle backgrounds and pick one
background=$(ls $background_home | shuf -n 1)

# Replace current LightDM greeter background
cp $background_home/$background background.jpg

sleep 1h
# Use feh to ad hoc set a new random background at a given time interval
while :
do
	feh --randomize --bg-fill /home/edwin/Pictures/backgrounds/*
	sleep 1h
done

Because we now introduced an infinite loop, we should really make sure that the bash process can run in the background, otherwise your system will hang and never boot into your window manager! This is not hard though. In your ~/.xprofile, just make sure to disown the bash call, for example as such:

random_background.sh & disown

Now you will have a fresh background from your collection each hour! If you manage to hang your system anyways, just open another TTY and log in there to fix the problem. Because you do not automatically start the X server at other TTYs, you can adjust your script and halt the blocking process (e.g. using htop).