Rooijakkers Software

Quote of the week #3

Sun, 09 Jun 2024 00:00:00 +0200

The Buddha in Dhammapada on debugging.

Patient endurance is the supreme practice for freeing the heart from unwholesome states.
– The Buddha

Een kijkje in de keuken van team Clojure

Mon, 06 Nov 2023 00:00:00 +0100

Bij Mediquest zijn er twee DevOps-teams: team DBA, vernoemd naar het administreren van databases; en team Clojure, vernoemd naar het programmeren met Clojure. In deze blogpost belichten we het programmeren met Clojure.

Mediquest is één van de ongeveer twintig bedrijven in Nederland die werken met deze programmeertaal. Waarom Clojure? Wat is het? Hoe is het om ermee te werken? Onze collega Erwin is softwareontwikkelaar in team Clojure en neemt ons mee.

Waarom kiest Mediquest voor Clojure?

Software begint bij code. In het algemeen geldt: hoe minder code hoe minder fouten. Clojure is een taal die ontwikkelaars in staat stelt om met relatief weinig code veel uit te drukken. Clojure-code beschrijft de essentiële complexiteit, inherent aan het op te lossen probleem, terwijl de onnodige complexiteit (de technologische overhead) wordt beperkt tot een minimum.

Met Clojure kan je op veel plaatsen minder code schrijven: in browsers met ClojureScript, op servers via Clojure en ClojureCLR tot scripting op besturingssystemen via de Clojure-variant Babashka. Zo schrijf je van voor- tot achterkant in dezelfde taal.

Deze uitdrukkingskracht en dit bereik stelt Clojure-teams in staat om met een relatief klein ontwikkelteam veel software te beheren.

Wat ontwikkelt Mediquest zoal in Clojure?

Clojure is tien jaar geleden in Mediquest geïntroduceerd door oud-collega Cees van Kemenade die een low-code platform maakte genaamd de ‘Business Process Engine’. Met dit platform kunnen procesflows (e.g. als deze data wordt aangeleverd, doe dan deze controles, verstuur deze e-mails, sla het hier versleuteld op) visueel worden gedefinieerd. Die plaatjes leiden tot uitvoerbare code. Zo kunnen mensen met weinig programmeerervaring processen programmeren. Sommige van die flows draaien nog steeds, maar vanaf vijf jaar geleden kwamen er in Mediquest meer Clojure-ontwikkelaars, ontstond team Clojure en verschoof de aandacht naar ontwikkeling van het portaal.

Dankzij Clojure biedt Mediquest nu een portaal vol applicaties met componenten (gecertificeerd op gebied van kwaliteit en privacy) voor dashboards, voor het maken en distribueren van vragenlijsten en om bestanden te uploaden en downloaden. Elk van deze bouwstenen kan worden geconfigureerd om aan specifieke klantwensen te voldoen. Zo wordt het portaal ingezet voor onder meer Mijn Positieve Gezondheid, mijnPerined en PREM.

‘Volgens mij worden sinds Clojure dingen sneller ontwikkeld.’ – Corine Horsch, Data Scientist (team Datalab)

Wat is het verhaal achter de haakjes?

Een uniek kenmerk van Clojure (of beter gezegd van de programmeertaal Lisp, waar Clojure een variant van is) zijn de ‘verkeerd geplaatste’ haakjes. Waar andere programmeertalen vaak lezen als doSomething(1, 2) lees je in Clojure (do-something 1 2). Ja, zeker voor programmeurs, is dat even schrikken. Maar gelukkig, deze haakjes tezamen met de syntax voor lijsten ‘(), [] voor reeksen, #{} voor sets en {} voor dictionaries vormen tachtig procent van de Clojure-regels.

Met die haakjes is Clojure-code zelf ook een datastructuur in Clojure, namelijk een lijst. Daarom noemen Clojure-programmeurs code ook wel data. Dit maakt het mogelijk voor programmeurs om code te schrijven die code aanpast. En zo als het ware een eigen programmeertaal binnen Clojure te ontwikkelen, die specifiek aansluit op het domein. (Mocht je als programmeur deze zogenaamde macro’s beter willen begrijpen, de blogpost The Nature of Lisp legt het uitgebreid uit.) Zo is een groot deel van de expressiviteit en kracht van Clojure dankzij deze onconventionele haakjesplaatsing.

_Bron: xkcd_

Dus jullie hebben nooit bugs?

Uiteraard kun je ook in Clojure complexe en moeilijk te onderhouden code schrijven. Echter, de taal is zo ontworpen dat dit wordt ontmoedigd. De meeste programmeerproblemen kunnen uiteindelijk worden uitgedrukt met die paar eerder genoemde basisdatastructuren zoals lijsten, sets en dictionaries, en Clojure blinkt uit in het hanteren van deze datastructuren.

‘Het is beter om honderd functies te hebben die opereren op één data-abstractie, dan tien functies op tien datastructuren.’ – Rich Hickey, maker van Clojure (in navolging van computerwetenschapper Alan Perlis)

De datastructuren zijn namelijk onveranderlijk (dit maakt het makkelijk erover te redeneren), mocht je parallel code willen aanspreken zijn daar faciliteiten voor en er zijn allerlei functies die opereren op al deze lijsten, sets en dictionaries (en ook de datastructuren van het hostplatform), omdat die alle de zogenaamde sequence-interface implementeren. Dit voorkomt een hoop bugs veroorzaakt door een taal zelf.

Bugs die optreden komen meestal voort uit fouten in het ontwerp zelf (de essentiële complexiteit), en niet dankzij programmeerfouten door de taal (onnodige complexiteit).

En verder?

Een groot deel van wat Clojure zo prettig maakt om mee te werken komt door de haakjes aan de buitenkant: je hebt als ontwikkelaar toegang tot draaiende code via een REPL; je kan vragen stellen aan draaiende code en deze wijzigen, en je kan Clojure op eenvoudige wijze uitbreiden met nieuwe concepten.

De haakjes wennen overigens wel, voor nieuwe Clojure-programmeurs is vaak het werken met onveranderlijke datastructuren (het zogenoemde functioneel programmeren) het lastigst om te leren. Hoewel er meer programmeertalen de functionele kant op zijn gegaan de afgelopen jaren.

Kortom?

Goed, ik denk dat het Clojure-evangelie nu genoeg is verkondigd. Al met al is het als ontwikkelaar prettig werken met Clojure vanwege de robuuste datastructuren, dezelfde code kunnen schrijven op voor- en achterkant, het gemak van redeneren en de krachtige abstractiemogelijkheden. Clojure speelt een essentiële rol bij het tijdig opleveren van kwalitatief hoogstaande software bij Mediquest.

Erwin – softwareontwikkelaar team Clojure

Deze blog verscheen eerst op de Mediquest-website. Zie https://home.mediquest.nl/een-kijkje-in-de-keuken-van-team-clojure/.

GDPR-compliant smart meter data on the IOTA Tangle — four lessons learned while putting the consumer in control

Sat, 08 Sep 2018 00:00:00 +0200

This blog post originally appeared on Medium.

TL;DR: We created a proof of concept where smart meter data is obtained via a Raspberry Pi and published via IOTA Masked Authenticated Messaging. A consumer can proof he owns the Raspberry Pi, and the consumer can give and revoke consent to service providers to access the smart meter measurements for a specific goal. All on the IOTA Tangle.

Alliander is a Dutch energy grid operator that ensures 6 million citizens have access to energy. At Alliander R&D we research technologies that will improve our infrastructure and lead to a more sustainable future.

Sharing smart meter data with service providers can help the energy transition. For example, in the UK energy usage was reduced after providing citizens an app with usage insights connected their smart meter. If anyone can access energy usage data it can help balance the grid and more.

At the same time smart meter data is privacy-sensitive. Power consumption can be associated with appliances used and with measurements every 0.5 seconds it is possible to see what television channel is being watched!

_{Power consumption peaks can be associated with individual appliances. (Source: Privacy Issues Related to Smart Grid Technologies, Megan J. Hertzler)}

You can ponder if we should care about privacy in the face of the slow-moving climate change apocalypse. But it's the law. Energy usage data cannot be legally shared without adhering to it.

The General Data Protection Regulation (GDPR) came into effect May 25. It provides data and privacy protection for European citizens. When dealing with personal identifiable information (like the measurements of a smart meter) a service provider has to:

State the goal for data usage clearly (and only use the data for that goal)
Have consent of the consumer to access the data
Stop collecting data when consent is revoked

(Ignoring other parts of the GDPR that cannot be encoded in software, but that need to be enforced juridically, like right of erasure).

So, for open usage of smart meter measurements a solution needs to be provided that adheres to the requirements of the GDPR. Two goals of this solution are:

Energy data goes directly from the smart meter to the service provider.
Consent (authorizations to access data) are stored decentrally.

When data or consent is stored centrally, for example in a cloud, there are privacy concerns. A cloud is a single point of failure (what if consent is irrevocable because a server is down?), that is susceptible to misuse and hacks. Central storage can also lead to vendor lock in, where the owner of the smart meter data reader determines what service providers can use the data, instead of creating a level playing field where you can authorize anyone to access the data. Distributed ledger technology could address these issues.

Dutch Smart meters have a P1 port which gives consumers access to real time measurements with a high resolution. We investigated how distributed ledger technology can be used to provide open access to P1 energy measurements where the consumer is in control. Because it scales and has no transaction fees (who wants to pay to give consent?) we use the IOTA Tangle.

_{A Raspberry Pi running IOTA interfacing software connected via a P1 cable (USB to RJ11) to a Dutch smart meter}

IOTA aims to be the backbone for the Internet of Things whereby devices can communicate and transact with each other. E.g., your washing machine buys energy from the solar panels of the neighbours while your self-driving car is instructed to pick up your kids. To read more about IOTA Tangle see The Tangle: an Illustrated Introduction.

Every data or value transaction on the IOTA Tangle must validate two earlier transactions by solving a cryptographical puzzle that takes computing power (proof of work). This way all nodes agree with each other on the state of the Tangle. For more information on how IOTA works see The Tangle: an Illustrated Introduction. (Image source) –>

Now follow four lessons we learned implementing GDPR-compliant energy data streams on the IOTA Tangle

The difficulty in providing access is handing out the keys.
Encryption is needed when sharing keys.
MAM channels can be used as decentralized storage of consent with an encrypted immutable event log, but... you now have an encrypted immutable event log.
Masked Authenticated Messaging on standard hardware is unsuited for smart meter measurements.

_{Component diagram}

The difficulty in providing access is handing out the keys

IOTA Masked Authenticated Messaging (MAM) is a feature of the IOTA Tangle that allows to publish and fetch encrypted messages. In restricted mode both the address of a data stream as well as a side key (with which the data is encrypted) needs to be known to read and decrypt the data. MAM can be used for data integrity and access control.

When I first heard about MAM I thought that it would solve all GDPR issues. Then we stumbled upon the problem of exchanging keys...

An authorized service provider needs to receive the root and side key of restricted MAM channel (key exchange)

After you have paired with the Raspberry Pi (see Scenario: Pairing with a Device for a way how ownership can be proved via IOTA using a secret on the Pi itself), when you give a service provider consent to access your data (you do this in a type of IOTA wallet that manages your paired devices and consent policies), the service provider has to receive the restricted MAM channel's root and side key. How can this be done?

In our solution, the service provider can request access to data by sending its address and the goal for usage (for what reason energy data access is required as per the GDPR) to the consent wallet (called My Home). Then when a Raspberry Pi owner gives consent to access the data, the Raspberry Pi sends the access keys to the service provider's IOTA address.

_{From the console output of the Raspberry Pi we see the creation of a zero value IOTA transaction containing the Pi's MAM root and side key to an authorized service provider's IOTA address}

In the IOTA transaction above the MAM root and side key in the IOTA transaction are encrypted. Why?

Encryption is needed!

Because on the IOTA Tangle all communication is public, the above key exchange needs to be encrypted. Otherwise eavesdroppers can see the IOTA transaction and obtain the root and side key as well. When an attacker has the keys, he has access to your privacy-sensitive energy usage data.

As stated, when using restricted MAM between two parties a key exchange needs to take place: the root and side key of the restricted MAM channel need to be communicated to the authorized service provider. IOTA has no asymmetric encryption. IOTA only has signing. So you cannot encrypt data with, for example, an IOTA address, and only decrypt it if you have the seed. So for asymmetric encryption we need something outside IOTA.

Since one of the benefits of IOTA is that its secure against quantum computers (IOTA's signing and proof of work are quantum proof), we don't want to lose this benefit by using classical asymmetric encryption algorithms like RSA). These are broken once quantum computers arrive.

The author of the CHIOTA health care chat application helped us. CHIOTA came up with a quantum proof key exchange for IOTA. A method to deterministically generate a key pair using an IOTA seed and address. We use a similar approach, but in JavaScript instead of C#. Using this method and an IOTA seed you can generate a public and private key.

I think it would be helpful if quantum-proof asymmetric encryption is integrated in IOTA as an IXI module. Since you cannot encrypt with someone's IOTA address itself, a directory of public keys is needed somewhere. CHIOTA stores the public key on a user's first IOTA address. Perhaps we will see asymmetric encryption integrated in the IOTA Tangle or in MAM in the future.

Encrypting the MAM root and side key

By using the quantum proof encryption described, the MAM root and side key can be encrypted with the public key of the service provider. This way, although communication is public, only the service provider can decrypt it.

In our solution a tryte-encoded public key is sent along with the data access request, but a possible solution could also be that the public key is available in a central place. Something like an energy data usage app store where consumers can vouch for the trustworthiness of service providers.

_{The service provider wattt.nl receives the MAM data (root and side key) from the device and decrypts it}

Revoking access

Thanks to Harm van den Brink for sharing this idea.

Asymmetric encryption is also used in the case of access revocation. Multiple service providers can read the same MAM data stream with P1 energy data. When the software on the Raspberry Pi is instructed that a service provider's access is revoked (see next issue), it changes its side key and sends out a key rotation message. This message is published on the same MAM stream as the energy data, so service provider's only need to listen to that. The published message contains the new side key, encrypted with the public keys of the remaining authorized service providers (click here for the code). This way all service providers that are listening that are still authorized can decrypt the message and update the side key, but the service provider of which consent is revoked cannot.

_{The service provider shows plots earlier received data, but is not included in the KEY_ROTATION message, so is no longer authorized and is unable to fetch new data}

MAM channels can be used as decentralized storage of consent in an encrypted immutable event log, but... you now have an encrypted immutable event log

Once the Raspberry Pi (connected to the smart meter) is paired, it starts listening to the consent management front-end's MAM channel (click here for the code). Every time an event of type AUTHORIZED is published on this channel by the front end, the Raspberry Pi will communicate its current root and side key (encrypted with the service provider's public key) to the service provider in the message (via an IOTA transaction). And when an event of type AUTHORIZATION_REVOKED is published on the MAM channel the Raspberry Pi will publish a message of type KEY_ROTATION on the MAM channel. So the Raspberry Pi can handle authorizations via this MAM channel.

The front-end uses the same channel to obtain its state, but it creates another view (click here for a code example) from the events (for example show a consent policy when an event of type AUTHORIZED has taken place).

ydD0JHK6kU9qQ.png" alt="Give Service Provider consent" width="100%">

_{In My Home's consent management wallet a paired device can be selected that the service provider can then access. The device is obtained from an event on the MAM channel (type of DEVICE_ADDED). And when access is given an event of type AUTHORIZED is published to which the paired Raspberry Pi will respond.}

Using an immutable log of events as a backend is called event sourcing. Event sourcing works as a backend for both the consent management front-end and the Raspberry Pi.

The MAM channel root is communicated to an authorized service provider. This way all parties involved (Raspberry Pi, the consumer, and the service provider, see component diagram above) have access to the immutable audit trail and can show whom had consent and when to use the data.

_{Events that are published on the consent and devices wallet (My Home)'s MAM channel. A user has added a device, provided consent, revoked consent and deleted the device.}

Very nice. But now we have an immutable audit log only accessible by parties involved. On permanodes, this will never be erased. The same goes for the published energy measurement data by the way, it is just like the policies, forever stored on permanodes in the IOTA Tangle.

Is this a problem? Although it is encrypted now, data might later be decryptable with new technologies or exploits. Other than that I don't see much difference between publicly storing encrypted data and handing out keys, or sending data in another way. For a malicious service provider it might be a bit more convenient to give someone else access when using keys. Instead of sending the raw received data they only have to send the keys. But raw data can also be easily send. It might be a problem in the future if the encryption is hacked, the rest of the argument against public storing of encrypted data seems based on emotion (although this needs to be addressed in the case of widespread adoption).

In relation to the GDPR's right to erasure, is it enough for a service provider to delete the root and side key? The GDPR does not address what erasure precisely means. I personally think it is fine to have publicly available encrypted data with removed keys in a cloud or on a distributed ledger. It's similar to the practice of crypto-shredding. But in a larger scale roll out of distributed ledger software for GDPR, will citizens feel the same? We will see what interpretation of the legislation will dominate.

What is a problem with MAM is that it is unsuited for high frequency smart meter measurements.

MAM proof of work is too slow for smart meter measurements (on standard hardware)

MAM's proof of work is a problem for publishing high frequency smart meter data on MAM. MAM is, with the current speed of proof of work on nodes, unsuited for P1 energy data. Even when offloading the proof of work from the Raspberry Pi to a node, it is too slow. On the main net one P1 energy data MAM message can be published roughly every 40 seconds. On the testnet it does work for smart meters that send data once every 10 seconds (click here for the P1 data publishing code). Newer Dutch smart meter models, however, send data every second!

In its current state IOTA is not suited for the IoT industry (see this IOTA Report Overview). At the moment, the energy required for the proof of work for data publishing, is higher than the estimated energy savings from usage of the data (back-of-the-envelope calculation: it takes 0.00005 kWh) per IOTA transaction, three transactions per MAM message is 0.00015 kWh per message, one message every 10 seconds for older smart meters is 473 kWh proof of work (PoW) per year per household. On average 3500 kWh and 16000 kWh gas energy usage per household, insight into energy usage leads to 2% energy savings by a conservative estimate, and 2% of the energy usage is 0.02 * (3500 + 16000) is roughly 390 kWh, so with current IOTA transaction speed for an average household more energy is used for the proof of work (473 kWh) than the expected savings (390 kWh)).

It is possible to move to an off-Tangle protocol like MQTT, but in that case the service provider is required to host an MQTT broker or similar. You lose the ease of access of an MAM.fetch.

The PoW problem, however, can be solved. There are advancements where dedicated hardware can run proof of work much quicker. For example, an FPGA for the Raspberry Pi can run proof of work for one transaction in 18 milliseconds! This uses much less energy. This does mean that people wanting to share their energy data for the energy transition must buy such dedicated hardware, or it must be provided in smart meters.

Proof of work speed needs to be improved for IOTA to be suitable for the IoT industry. This will happen once more devices start using the IOTA Tangle and the IoT industry starts using hardware that is optimized for IOTA's proof of work.

In practice

Examples of the software connected to the IOTA test net are deployed. Below we go through the flow on a laptop.

Energy data on IOTA with consent
Check out this video on Streamable using your phone, tablet or desktop.

_{Here we see the pairing of a Raspberry Pi, requesting consent from a service provider to access its data via the consent wallet My IOTA Home, consent (key exchange) for two service providers (wattt.nl from localhost and wattt.nl from domain), viewing the data once consent is given, and revoking the consent for one of the two service providers (key rotation). Note that this video is sped up.}

Remarks

In the future IOTA's Qubic protocol can possibly be integrated. Its oracles might be useful for storing consent. Also the next version of MAM called MAM+ will probably contain a public key infrastructure with possibly a key exchange mechanism. And in the future identity management system like Sovrin or IRMA can be integrated to proof that you own a smart meter without revealing anything else about yourself, and for signing the consent.

The code is open source and available on GitHub. It's a proof of concept and it's frail, but the happy flow (publishing energy data, pairing a Raspberry Pi, authorizing a service provider, viewing the data, revoking consent) has been tested and works.

If you have a Dutch smart meter, a Raspberry Pi and a RJ11 P1 cable you can download and install the software and connect yourself! Those who want to create their own service provider can do so and interface with the My Home consent management wallet as well.

For more technical information check the documentation. The ideas are possibly applicable to other fields where personally identifiable information is used. For example in health care when a patient wants to share his medical records.

Conclusion

We hope this will be a step towards a privacy-preserving infrastructure that opens smart meter data to the world. Where energy data can be used by everyone who you trust. This way smart meter owners can be in control of their data, and everyone can create services that use it.

The infrastructure for sharing energy data while adhering to the GDPR, can be realized using a decentralized consent protocol on the IOTA Tangle. Although IOTA has some open issues like:

Improving the speed of proof of work for publishing MAM data messages (for example with dedicated hardware).
Legally clarifying if removal of keys of encrypted data on a distributed ledger adheres to the GDPR's right to erasure.

Thanks to my colleagues at Alliander and IOTA users for their help. And thank you for reading. Let me know what you think.

This blogpost was originally published at https://medium.com/@erwinrooijakkers/gdpr-compliant-smart-meter-data-on-the-iota-tangle-four-lessons-learned-while-putting-the-dea852a5b2aa

Simple but effective

Fri, 31 Mar 2017 00:00:00 +0200

(defn debug-panel
  "Debug panel, pretty prints any data you pass in on the screen."
  [d]
  [:pre
   (with-out-str (cljs.pprint/pprint d))])

[debug-panel <data structure>].

Lovely.

Four reasons why HelloData is written in Clojure

Sat, 26 Nov 2016 00:00:00 +0100

The HelloData platform aims to connect smart meters, apps and consumers with one another, while still ensuring consumers retain ownership and control of their data. Technically speaking, this creates challenges when it comes to data streams (scalability), security (access rights; protection against malicious users), and the interface between the source (e.g. the smart meter) and services (the apps that use the data).

Because the first version of HelloData, written in Ruby on Rails, had performance issues, we decided to (re)build the platform in Clojure.

But why Clojure? In this blog post I explain four reasons behind this decision:

REASON 1. Clojure is a dialect of Lisp

I first came across Lisp at the outset of my career as a software developer in the article Beating the Averages by Paul Graham (2001). In it, Graham describes how his startup (Viaweb, later Yahoo Stores) became successful thanks to a "secret weapon".

This secret weapon was Lisp: a family of programming languages that has its origins back in 1958 and is characterized by extensive use of open and close parentheses.

Graham’s article convinced me that Lisp is the most powerful model of computation in existence. And that by seeing things through my Java specs (Java is the world’s most widely-used programming language, and the one with which I, too, started out) I had been suffering from the ‘Blub Paradox’: as a programmer I always thought in the programming language Blub, and so could only understand languages that were less powerful than Blub. And my Blub-induced mindset had therefore led me to dismiss any languages more powerful than Blub, such as Lisp, as gibberish.

And so began my quest to learn Lisp

I filled my evenings and weekends with mind-blowing exercises that rejigged my mind. Initially, I had to get used to the parentheses. But then... I got it!^¹

The power of Lisp lies in parentheses

The fundamental data structure in Lisp (short for: LISt Processor) is the list. Not only does the data that the Lisp programmer uses consist of lists enclosed by parentheses; so, too, does the code itself. This means that code can be treated as data, and data as code. This is called homoiconicity.

Homoiconicity makes it possible to create transformations out of source code, to write programs that create programs (in Lisp these programs are known as macros), and to introduce new syntax. This is what makes Lisp so powerful, and why Paul Graham described it as the "secret weapon".

In the context of HelloData, we make use of Lisp’s power in two ways:

Within Clojure, we’re trying to write our ‘own language’ that describes our domain. The flexibility of the abstractions means that we are scarcely, if at all, hampered by the limitations of a programming language.
This same flexibility means it’s possible to integrate features of other languages into Lisp, so they are available as libraries (code shared by others, so that a developer doesn’t having to write them themselves). An example being Go routines from the Go programming language, which can be introduced into Clojure via the core.async library. The difference with Go itself is that, in Clojure, such language features are not a fundamental property of the language.

So is everything with Lisp always just beer and skittles?

No.

Precisely because Lisp is so powerful and flexible, we also have the phenomenon sometimes refered to as The Curse of Lisp (for more on this, see Rudolf Winestock’s essay The Lisp Curse): Lisp’s being so easy to extend means there are also many language extensions. Resulting in many different dialects and libraries that are only partially and poorly documented, and really only suitable for the specific domain of their creator.

Clojure is a Lisp dialect that is less affected by the Curse of Lisp, as in Clojure the consistency of the core libraries and the standard means of doing things are guarded by its author, Rich Hickey, and his company Cognitect.

What’s more, Clojure has access to the Java Virtual Machine (JVM) and, in the form of ClojureScript, to JavaScript, on which many standard libraries are available. Which neatly brings us to the second reason for opting for Clojure.

REASON 2. Clojure runs on the JVM and JavaScript runtime

The JVM is a platform-independent environment (the JVM runs on Linux, Windows, macOS, etc.) for executing Java bytecode. The JVM has been around since the 90s, since when it has been optimized by many different people.

On the JVM you have access to a vast number of libraries. From web servers to user interfaces — you name it and there’s a library for it. Clojure is converted into Java bytecode, and thus benefits from the stability, popularity, speed and safety of the JVM.

All libraries on the JVM are simple to use from Clojure. For example, in our registration flow we wanted to use Macaroons: exchange keys described in this paper by Google. We don’t have to implement Macaroons ourselves, because a Java library already existed which —aside from one spelling mistake — we could easily use.

Clojurescript does the same with Javascript that Clojure does on the JVM. JavaScript is the world’s most widely-used programming language and runs on pretty well all browsers.

What’s in a name?

Though their names may be very similar, Java and JavaScript have about as much in common as a car and a carp. Clojure and ClojureScript, on the other hand, are the same language. This means that, as long as you don’t use libraries specifically intended for the host, you can reuse Clojure code from the backend on the frontend. Letting you write an entire application in Clojure and, for example, use the validation code in both the frontend and backend.

Best thing about ClojureScript? The unique development experience.

Normally speaking, the development of websites involves the following two steps:

Adjust code.
Refresh the browser to see the changes.

Step 1 is unavoidable, but Step 2 is tedious and slows things down. The developer has to perform an extra act and the website loses its state. If, for example, there’s been a few clicks and fields have been filled in on the website for testing, following a browser refresh the developer is back to square one.

The Figwheel library offers a solution that makes this continual refreshing unnecessary, as it ensures that the code that has been modified is sent directly to the browser. If you have the browser displayed on the left of your screen and the code you are adapting on the right, when you make a code adjustment on the right, you immediately see on the left its effect on the website. In this video, Figwheel’s creator gives a demo:

This perfectly describes our development experience at HelloData and is one of the reasons why the development team always looks so cheerful.

Although Clojure runs on the JVM, the code written does not look like Java - the most popular programming language on the JVM - at all. This is partly due to the parentheses, but mainly because of Clojure’s data orientation. What this is you’ll read in the third reason.

REASON 3. Clojure is data oriented

In Clojure much less code is needed to say the same things as in Java. Code that is written is only about the problem that it’s solving. Syntax, design patterns, and other ceremonies that are not about the problem, disappear.

This is because Clojure treats data as data.

Treating data as data – what does that mean? Programmers work with data. In many other programming languages, like for example Java, data is not treated as data. Before it is used the data is first transformed to a, by the programmer created specific data structure (for example a class), only used for that specific use case, that is difficult to extend, and makes it difficult to create another implementation.

In Clojure the programmer often just uses Clojure datastructures (lists; vectors; hash-maps; sets).

In practice this means that the programmers makes transformations on Clojure datastructures with all the free functions that are available.

It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures. –- Alan Perlis (Epigrams in Programming)

To drive the point home I can advise you to watch a few minutes of the Clojure made Simple video below, from 51 minutes and 15 seconds onwards. This video contains a rant of Rich Hickey, the Clojure creator, where he compares a specific Java-datastructure with the same representation in Clojure.

Clojure’s data-orientation (treatment of data as data) is – besides the level of abstraction that can be reached – the reason that Clojure code bases are relatively small. Smaller programs have fewer bugs. And almost all code is about the problem that is solved. Not about something else. This makes Clojure code simple and robust.

So, its flexibility, the host platforms and the treatment of data as data, are three reasons to use Clojure. Another important point is that Clojure is made for parallel and concurrent programming.

REASON 4. Clojure embraces concurrent programming

According to Moore’s Law, the performance of processors doubles every 18 months. So taking account of physical limitations, Moore’s Law becomes this:

Source: IEEE Computer Society

In the graph above, we see how the limit on processor performance is reached. Right now, technologies are being developed to make computers even faster, namely:

1. More processors on a chip (multi-core processors, see bottom line of graph); 2. Utilizing multiple computers, for example in the cloud (distributed programming).

The future of programming is thus multi-core and distributed².

To make the most of multi-core processors and distributed machines, software developers should use concurrent programming (writing code that does things in parallel). After all, when source code is sequential, you can’t run it on multiple computers or processors simultaneously.

The problem with concurrent programming...?

Quite simply, it’s difficult to do in most programming languages.

Concurrent programming is tricky due to so-called ‘race conditions’. Race conditions occur when two processes simultaneously want to read or write shared data and the sequence or timing of this is uncontrollable. They are the cause of many bugs and often difficult to trace.

Race conditions manifest themselves as occasionally an erroneous situation.

One way to avoid race conditions is to identify critical sections at points in the source code that may only be used by one process at a time. But programming these critical sections itself has its problems. Take, for example, deadlocks, where two processes both wait for each other and the software freezes. Or starvation, where a process never gets its turn and is simply kept waiting. As well as these problems, code with critical sections cannot be executed in parallel.

The creator of Clojure says in his presentation, Clojure Concurrency, that we can learn two things from a famous Java book with techniques for concurrent programming:

1. Concurrent Programming in Java is incredibly difficult. 2. The immutability (unchangability) of values is repeatedly cited as the solution.

Race conditions are caused by changes: one process wants to write a value at precisely the moment that another process is reading it; or two processes want to write a value at the same time. But what if assigned values were immutable? Then there’d be no more racing.

Immutability of values makes critical sections redundant and race conditions impossible. So-called ‘functional programming languages’ are based on this principle. Just as with mathematical functions, the results of calculations in a functional programming language are always the same. In non-functional programming languages, you can’t be sure about outcomes and different processes or side effects can occur that change a value in the background. In functional programming languages, you can be certain (this is known as ‘referential transparency’); so it doesn’t matter in what order you execute code, which makes concurrency easy.

Clojure supplies tools to program functionally and immutable values are fundamentally built into the language. Once assigned, a value in Clojure is ‘persistent’. If you want to change something, you need to make copies of previous data structures with the changes added. In order to avoid all these data structures occupying too much space in the computer’s memory, Clojure uses a method called ‘structural sharing’. This ensures efficient, persistent data structures, as shown in this diagram:

Source: Functional Advantage with Clojure

In Clojure, it seems like the programmer is working with copies of data, but under the hood Clojure reuses everything that remains unchanged. So despite all the copies, Clojure still only uses minimal memory.

Sometimes you do need to change a value. After all, in the real world things change and in a pragmatic programming language you have to be able to work with variable values.

But happily, changes to variables are possible in Clojure, and so the language is not purely functional. Clojure provides constructs for change. There are mechanisms built into Clojure to change values which ensure that the system is always consistent. All without the need for the developer having to update the values manually, using error-prone critical sections.

Through this focus on functional programming, persistent data structures and language constructs, we can see how Clojure truly embraces concurrent programming.

AND THERE’S LOADS MORE REASONS TO LOVE CLOJURE

The origins in Lisp, the support of host platforms and the focus on concurrent programming are three of the reasons why HelloData is built in Clojure. And I’ve not even mentioned any of these others:

Clojure’s REPL, which allows the developer to communicate with their code
the many supporting programming paradigms - such as logic programming, where you can say: "this is happening, give me a solution".
practical polymorphism
integrated editors
flexibility for change
clojure.spec
the GPU (Graphical Processing Unit) tools
Adoption within the industry
the build tool
and the friendly Clojure community.

The future is parallel – and Clojure is ready for it!

There’s just one problem with Clojure. And only you can solve it.

Besides problems programmers with a different background have adapting to Clojure, there is the additional problem of actually finding good, experienced Clojure developers.

Are you an experienced Clojure developer? Do you want to contribute to a sustainable future? If so, we’d be delighted to welcome you to HelloData.

Simply get in touch with me, at erwin.rooijakkers@mpare.net, and I'll be happy to tell you more about HelloData.

This blog post originally appeared on the MPARE website. See https://www.mpare.net/en/blog-drie-redenen-waarom-hellodata-wordt-gebouwd-in-clojure/.

This moment of enlightenment is wonderfully illustrated in a lecture by Gerald Jay Sussman at the point when he finishes writing out Lisp code (fitted onto a single school blackboard), in which any programming language can be expressed. See https://youtu.be/0m6hoOelZH8?t=34m34s.↩
Extensive research is being done into quantum computers, with promising results. But to keep things to a reasonable length and not dive into areas where I have limited expertise,I’ve decided not to cover the quantum computer, and programming it, in this blog post.↩

Quote of the week #2

Mon, 03 Oct 2016 00:00:00 +0200

Steve McConnel in Code Complete on the personality of expert software engineers.

If you haven't spent at least a month working on the same program – working 16 hours a day, dreaming about it during the remaining 8 hours of restless sleep, working several nights straight through trying to eliminate that "one last bug" from the program – then you haven't really written a complicated computer program. And you may not have the sense that there is something exhilarating about programming.
– Edward Yourdon
This lusty tribute to programming machismo is pure B.S. and an almost certain recipe for failure. Those all-night programming stints make you feel like the greatest programmer in the world, but then you have to spend several weeks correcting the defects you installed during your blaze of glory. By all means, get excited about programming. But excitement is no substitute for competency. Remember which is more important.

Quote of the week #1

Thu, 22 Sep 2016 00:00:00 +0200

Alan Perlis on the first page of Structure and Interpretation of Computer Programs.

I think that it’s extraordinarily important that we in computer science keep fun in computing. When it started out, it was an awful lot of fun. Of course, the paying customers got shafted every now and then, and after a while we began to take their complaints seriously. We began to feel as if we really were responsible for the successful, error-free perfect use of these machines. I don’t think we are. I think we’re responsible for stretching them, setting them off in new directions, and keeping fun in the house. I hope the field of computer science never loses its sense of fun. Above all, I hope we don’t become missionaries. Don’t feel as if you’re Bible salesmen. The world has too many of those already. What you know about computing other people will learn. Don’t feel as if the key to successful computing is only in your hands. What's in your hands, I think and hope, is intelligence: the ability to see the machine as more than when you were first led up to it, that you can make it more.
– Alan J. Perlis (April 1, 1922-February 7, 1990)

PAIP in Clojure Chapter 4

Sat, 17 Sep 2016 00:00:00 +0200

The fourth chapter of Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp on the General Problem Solver.

SX359BO1,204,203,200_.jpg">

Summary

The chapter starts with a quote by Herbert Simon, a Nobel Prize winner and one of the inventors of the General Problem Solver (GPS):

There are now in the world machines that think.

This GPS machine that thought was the first useful Artificial Intelligence program. It was written to solve every problem using the same algorithm (reasoning). GPS was written in (1959) in a computer language created by the authors called Information Processing Language (IPL): an assembly language for manipulating lists. As can be imagined by looking at IPL's Wikipedia page the code was complex. Nowadays using a language at a higher level of abstraction a GPS can be written in a few lines of code and we can have a clearer grasp of the concepts at work. It turns out the program is not so general as it name suggests. According to Norvig the exaggerated claims probably had to do with the complex code: if the code is complex the program must be doing something interesting.

GPS is a searching algorithm that given various means with their pre- and postconditions finds means to a specific end. Simon and his colleagues realized humans often use means-end analysis to solve a problem. Here's an example:

I want to take my son to nursery school. What’s the difference between what I have and what I want? One of distance. What changes distance? My automobile. My automobile won’t work. What is needed to make it work? A new battery. What has new batteries? An auto repair shop. I want the repair shop to put in a new battery; but the shop doesn’t know I need one. What is the difficulty? One of communication. What allows communication? A telephone… and so on. — Newell and Simon’s example from their 1972 book Human Problem Solving

We can pose this problem when we structure it of consisting of various goals with starting conditions and operators.

In Clojure we can introduce a record called Op:

(defrecord Op [action preconds add-list del-list])

Op consist of an action (goal), and sets of preconditions, additions once the action is executed and a list of things that are no longer the case (del-list).

Now we can structure the taking our son to school problem posed above as consisting of the following operators:

(def school-ops
  #{(Op. :drive-son-to-school
         #{:son-at-home :car-works}
         #{:son-at-school}
         #{:son-at-home})
    (Op. :shop-installs-battery
         #{:car-needs-battery :shop-knows-problem :shop-has-money}
         #{:car-works}
         nil)
    (Op. :tell-shop-problem
         #{:in-communication-with-shop}
         #{:shop-knows-problem}
         nil)
    (Op. :telephone-shop
         #{:know-phone-number}
         #{:in-communication-with-shop}
         nil)
    (Op. :look-up-number
         #{:have-phone-book}
         #{:know-phone-number}
         nil)
    (Op. :give-shop-money
         #{:have-money}
         #{:shop-has-money}
         #{:have-money})})

GPS can solve any problem that is posed this way when it is given an initial state, a goal and available operations:

(GPS #{:son-at-home :car-needs-battery :have-money :have-phone-book} ; initial state
     #{:son-at-school} ; goal
     school-ops) ; available operations

Some problems, however, take too much time to solve in such a way. Computing the solution to these problems takes exponentially more time as the size of the problem grows. It is generally assumed [citation needed] these problems cannot be solved by an algorithm: there is simply not enough time to find a solution. Problems like that are called NP-hard. So GPS is not general since it fails to solve these problems in a reasonable amount of time.

Intermezzo The physicist Arkardy Bolotin states in Computational Solution to Quantum Foundational Problems that NP-hard problems can manifest itself in our physical reality (read in this blog post). Quantum physics breaks down at a macroscopic level (i.e., we do not see people in superposition or tunneling through walls, but electrons do), but there is no reason why it should break down. Bolotin argues that finding a solution to the quantum equations takes too much time (is NP-hard) and that therefore macroscopic objects in our world do not show quantum behaviour.

/edit: a computer scientist I respect called Scott Aaronson states the above is complete garbage and ignores basic physics and computer science:

It gives me no pleasure to respond to this sort of thing—it would be far better to let papers this gobsmackingly uninformed about the relevant issues fade away in quiet obscurity—but since that no longer seems to be possible in the age of social media Response: https://www.scottaaronson.com/blog/?p=1767#comment-103591

End intermezzo

Here's, anyhow, the first implementation of the GPS algorithm in the book translated to Clojure:

(def state
  "The current state: a list of conditions."
  (atom #{}))

(def ops
  "A list of available operators."
  (atom #{}))

(defrecord Op [action preconds add-list del-list])

(defn appropriate?
  "An op is appropriate to a goal if it is in its add list."
  [goal op]
  (some #(= % goal) (:add-list op)))

(declare apply-op)
(defn achieve
  "A goal is achieved if it already holds,
  or if there is an appropriate op for it that is applicable."
  [goal]
  (or (some #(= % goal) @state)
      (some apply-op
            (filter #(appropriate? goal %) @ops))))

(defn apply-op
  "Print a message and update *state* if op is applicable."
  [op]
  (when (every? achieve (:preconds op))
    (println "Executing" (:action op))
    (reset! state (clojure.set/difference @state (:del-list op)))
    (reset! state (clojure.set/union @state (:add-list op)))))

(defn GPS
  "General Problem Solver: achieve all goals using *ops*."
  [state' goals ops']
  (reset! state state')
  (reset! ops ops')
  (if (every? achieve goals) :solved))

When we execute GPS for the son to school problem

(GPS #{:son-at-home :car-needs-battery :have-money :have-phone-book}
     #{:son-at-school}
     school-ops)

this is the result:

Executing :look-up-number
Executing :telephone-shop
Executing :tell-shop-problem
Executing :give-shop-money
Executing :shop-installs-battery
Executing :drive-son-to-school

GPS finds the means to an end.

NP-hardness is not the only problem of the current GPS algorithm. Norvig explains numerous other problems with it in PAIP. In the exercises he wants us to solve some of them. I'll do my best, but I am not sure I have enough time.

Exercises

Exercise 4.2 [m]

Write a function that generates all permutations of its input.

(This is probably useful for improving GPS later.)

Seems like a good idea to use a recursive strategy. The permutations of zero elements is only the element itself. Permutations of a list are the permutations of the rest plus the first element added at every position in its permutations.

(defn add-all-positions
  "Creates collection where element elem is added at every position of coll."
  [elem coll]
  (let [size (count coll)]
    (loop [n 0 acc []]
      (if (> n size) acc
          (let [[before after] (split-at n coll)
                permutation (concat before [elem] after)]
            (recur (inc n) (conj acc permutation)))))))

(defn permutations
  "Generates permutations of input bag."
  [bag]
  (if (empty? bag) [[]]
      (conj (mapcat (partial add-all-positions (first bag))
                    (permutations (rest bag))))))

Testing:

(count (permutations [1 2 3 4]))
;; => 24

This algorithm is O(N²) since it loops over all previous permutations. According to Norvig finding permutations can be improved to O(N), but I do not see how.

Exercise 4.3 [h]

GPS does not recognize the situation where a goal is accidentally solved as part of achieving another goal. Consider the goal of eating dessert. Assume that there are two operators available: eating ice cream (which requires having the ice cream) and eating cake (which requires having the cake). Assume that we can buy a cake, and that the bakery has a deal where it gives out free ice cream to each customer who purchases and eats a cake.

(1) Design a list of operators to represent this situation.

(def dessert-ops
  #{(Op. :eating-ice-cream
         #{:having-ice-cream}
         #{:eating-dessert}
         #{:having-ice-cream})
    (Op. :eating-cake
         #{:having-cake}
         #{:eating-dessert :having-ice-cream}
         #{:having-cake})
    (Op. :buying-cake
         #{:able-to-buy-cake}
         #{:having-cake}
         nil)})

(2) give GPS the goal of eating dessert. Show that with the right list of operators, GPS will device to eat ice cream, then decide to buy and eat the cake in order to get the free ice cream, and then go ahead and eat the ice cream, even though the goal of eating dessert has already been achieved by eating the cake.

(GPS #{:able-to-buy-cake}
     #{:eating-dessert}
     dessert-ops)
;; => Executing :buying-cake
;;    Executing :eating-cake
;;    Executing :eating-ice-cream

(3) Fix GPS so that it does not manifest this problem.

This can be solved by adding a check to see if operations are already achieved:

(defn already-achieved?
  "Checks if new operations are not already achieved"
  [ops]
  (clojure.set/subset? ops @state))

And adding this check to apply-op (and ensure to return true since the truthy return value is used in GPS):

(defn apply-op
  "Print a message and update *state* if op is applicable."
  [op]
  (when (every? achieve (:preconds op))
    (when-not (already-achieved? (:add-list op))
      (reset! state (clojure.set/difference (set @state) (:del-list op)))
      (reset! state (clojure.set/union (set @state) (:add-list op)))
      (println "Executing" (:action op)))
    true))

This leads to:

;; => Executing :buying-cake
;;    Executing :eating-cake

Now, there are various other problems that can be solved. Norvig wants me to introduce a GPS that does: pattern matching; solves the problem of "leaping before you look" (e.g., when a subgoal is :jump-of-cliff GPS has already jumped and cannot go back if it turns out to be a bad idea); can reorder goals; identifies problems that cannot be solved with reordering goals ("Sussman anomaly"), and solves mazes. I decided to only look at some solutions, and not implement them myself.

If you want to know more you can view Norvig's source code for GPS that solves some of these problems at Norvig's website. More about the history and problems of the GPS can be found at General Problem Solver (GPS)

PAIP in Clojure Chapter 3

Mon, 22 Aug 2016 00:00:00 +0200

This is the chapter of Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp (PAIP) that dives into Common Lisp syntax.

SX359BO1,204,203,200_.jpg">

Summary for Chapter 3: Overview of Lisp

The chapter starts with a quote by Guy L. Steele, Jr.:

No doubt about it. Common Lisp is a big language.

There is a lot to Common Lisp. In this chapter Norvig shows us the subset of Lisp he uses in his book.

There are six maxims every programmer should know about:

Be specific. (E.g., use when instead of if if there is only one clause.)
Use abstractions. (E.g., provide a first-name method when dealing with a list of names.)
Be concise.
Use the provided tools.
Don't be obscure.
Be consistent. (Sometimes principles are in conflict with each other, then choose one and stick with it.)

Exercise 3.1 [m]

(let [x 6
      y (* x x)]
  (+ x y)) ; => 42

Show a lambda expression that is equivalent to the above let* expression. You may need more than one lambda.

((fn [x]
   ((fn [y] (+ x y))
    (* x x)))
 6) ; => 42

Exercise 3.2 [s]

The function cons can be seen as a special case of one of the other functions listed previously. Which one?

list*. See:

(= (list* 1 '(2 3 4)) 
   (cons 1 '(2 3 4))) ; => true

Exercise 3.3 [m]

Write a function that will print an expression in dotted pair notation. Use the built-in function princ to print each component of the expression.

Note: princ prints suitable output without escape characters and binds some values to special Common Lisp parameters. I use Clojure's print.

Dotted pair notation is used in Lisps to show we are dealing with an improper list: a linked list where the second element of a cell is not a list. This is notated with a dotted pair. E.g., (cons 1 2) ; => '(1 . 2).

Clojure has no dotted pair notation since it avoids the linked list data structure and uses the abstraction of the sequence. See sequences:

A seq is a logical list, and unlike most Lisps where the list is represented by a concrete, 2-slot structure, Clojure uses the ISeq interface to allow many data structures to provide access to their elements as sequences.

For a list (special type of sequence) we can introduce dotted pair notation. To do so we need a method that can check for elements that have a single value (e.g., 1, :a). Elements which are therefore not a sequence.

I used the following method from an answer on Stack Overflow to check if a method is single-valued (or an atom in other Lisps than Clojure).

(defn single-valued?
  "Checks if value is single-valued."
  [x]
  (not (or (nil? x)
           (.. x getClass isArray)
           (some #(instance? % x) [clojure.lang.Counted
                                   clojure.lang.IPersistentCollection
                                   java.util.Collection
                                   java.util.Map]))))

Next we can create a case analysis and recursively generate a dotted-pair representation of a sequence.

(declare create-dotted-pair)
(defn create-dotted-pair-rest
  "Creates dotted pair string representation for the rest of a list."
  [lst]
  (str " . " (create-dotted-pair lst)))

(defn create-dotted-pair
  "Creates dotted-pair string representation of list."
  [lst]
  (cond (single-valued? lst) (str lst)
        (empty? lst) (str "'()")
        (seq? lst) (str "("
                        (create-dotted-pair (first lst))
                        (create-dotted-pair-rest (rest lst))
                        ")")
        :else (str lst)))

(defn print-dotted-pair
  "Prints dotted-pair string representation of list."
  [lst]
  (print (create-dotted-pair lst)))

This works as follows for a list created by a chain of cons cells, the shorter notation for a list, and for a nested list:

(print-dotted-pair (cons 1 (cons 2 (cons 3 (cons 4 '())))))
; => (1 . (2 . (3 . (4 . '()))))
(print-dotted-pair '(1 2 3 4))
; => (1 . (2 . (3 . (4 . '()))))
(print-dotted-pair '(1 2 (3 4)))
; => (1 . (2 . ((3 . (4 . '())) . '())))

Exercise 3.4 [m]

Write a function that, like the regular print function, will print an expression in dotted pair notation when necessary but will use normal list notation when possible.

Dotted pair notation is only necessary when the rest of a list is not a list. To implement this we have to make changes to the create-dotted-pair-rest method. Unfortunately it is impossible to create an improper list with native Clojure data structures.

(cons 2 3) ; => IllegalArgumentException (because 3 is not a sequence)

This makes it impossible to write a test without implementing a linked list data structure in Clojure (like Max Countryman did). Therefore I'll skip this exercise.

Now the first hard exercise:

Exercise 3.5 [h]

(Exercise in altering structure.) Write a program that will play the role of guesser in the game Twenty Questions. The user of the program will have in mind any type of thing. The program will ask questions of the user, which must be answered yes or no, or "it" when the program has guessed it. If the program runs out of guesses, it gives up and asks the user what "it" was. At first the program will not play well, but each time it plays, it will remember the user's replies and use them for subsequent guesses.

The idea seems to be that the program has a tree of questions and remembers the answer for each node.

To solve it we need state. For mutable state we create a map which we hold in an atom.

I use the following map to build a tree of questions and answers:

(def questions-and-answers
  "Questions and answers database atom.
  Keywords are used to traverse the tree.
  An answer needs to be non-existent (nil) or in the form of a string."
  (atom
   {:is_it_a_programming_language?
    {:yes
     {:is_it_statically_typed?
      {:yes
       nil
       :no
       nil}}
     :no
     {:ca_va?
      {:yes
       nil
       :no
       nil}}}}))

The program could be made more interesting if it was possible to add new questions and answers. This way the database of questions and answers can keep growing. For now, we just ask the questions and update if the user has an answer.

(defn get-remaining
  "Gets the remaining questions and answers from the questions-and-answers database.
  previous should be a vector of the previous questions and answers."
  [previous]
  (if (empty? previous) @questions-and-answers
      (get-in @questions-and-answers previous)))

(defn get-next-question
  "Gets the next question from questions-and-answers
  previous should be a vector of the previous questions and answers."
  [previous]
  (-> previous get-remaining ffirst))

(defn further-questions?
  "True if there is an answer for a question. False if not."
  [previous]
  (if (empty? previous) true
      (not (nil? (get-remaining previous)))))

(defn found-answer?
  "True if we have found the answer."
  [previous]
  (string? (get-remaining previous)))

(defn set-answer!
  "Sets the answer to the value provided.
  previous should be a vector of the previous questions and answers."
  [previous answer]
  (swap! questions-and-answers assoc-in previous answer))

(defn key->string
  "Create a string of the answer of question key."
  [key]
  (-> (name key) (clojure.string/replace "_" " ") clojure.string/capitalize))

(defn string->key
  "Create a key of the answer string."
  [answer]
  (-> answer (clojure.string/replace " " "_") clojure.string/lower-case keyword))

(defn driver-loop
  "Asks the questions, updates the answers.
  previous is vector containing the previous questions."
  ([] (driver-loop [])) ; Start with no previous provided answers.
  ([previous]
   (let [question (get-next-question previous)
         ask-it (println (key->string question)) ; ask-it is dummy variable to not create nested lets
         answer (read)
         print-answer (println answer) ; dummy, idem
         previous (conj previous question (string->key answer))]
     (cond (found-answer? previous)
           (let [found-answer (get-remaining previous)]
             (println (str "The answer is: " found-answer)))
           (not (further-questions? previous))
           (do (println "What was it?")
               (let [given-answer (str (read))]
                 (println given-answer)
                 (set-answer! previous given-answer)
                 (println "Thank you.")))
           :else (recur previous)))))

; (driver-loop)

Usage of the program looks like this:

The node in the tree is updated after an answer is given. The next time the same sequence of questions are answered the program responds with the answer the user added earlier. The representation is shown by dereferencing the atom.

Exercise 3.6 [h]

(def a 'global-a)
(def ^:dynamic *b* 'global-b)
(def f (fn [] *b*))

(let [a   'local-a
      *b* 'local-b]
  (list a *b* (f) (var-get #'a) (var-get #'*b*)))

Given the initialization for the lexical variable a and the special variable *b* above, what will be the value of the 1st form?

This returns (local-a local-b global-b global-a global-b).

I translated the Common Lisp example from the book to Clojure. That example had different behavior. I hoped to see the dynamic scoping at work in (var-get #'*b*) so that the shadowed *b* was returned in the second position. I will probably figure out later how to do this.

Exercise 3.7 [s]

Why do you think the leftmost of two keys is the one that counts, rather than the rightmost?

Quicker search to argument list and easier to cons up new value to the beginning of a list (in Common Lisp). The latter is not the case in Clojure with its vector argument lists.

Exercise 3.8 [m]

Some versions of Kyoto Common Lisp (KCL) have a bug wherein they use the rightmost value when more than one keyword/value pair is specified for the same keyword. Change the definition of find-all so that it works in KCL.

(defn llast
  "Gets second to last value."
  [coll]
  (-> coll butlast last))

(defn get-value
  "Finds the value for a key from right to left. Returns value or nil if key not found."
  [key coll]
  (cond (< (count coll) 2) nil
        (= key (llast coll)) (last coll)
        :else (recur key (drop-last 2 coll))))

(defn find-all
  "Find all those elements of sequence that match item, according to the keywords.
  Doesn't alter sequence."
  [item coll & options]
  (let [test-fn (or (get-value :test-fn options) '=)
        key-fn (or (get-value :key-fn options) 'identity)
        result (remove #(not (test-fn % item)) coll)
        final-result (map key-fn result)]
    final-result))

(find-all 1 '(1 1 2 3 4 5 6) :test-fn = :key-fn inc :test-fn >) returns '(3 4 5 6 7). The rightmost key-fn: is used (>).

Exercise 3.9 [m]

Write a version of 1ength using the function reduce.

(defn length
  "Finds the length of coll using the function reduce."
  [coll]
  (reduce (fn [acc e] (inc acc)) coll))

Exercise 3.10 [m]

Use a reference manual or describe to figure out what the functions lcm and nreconc do. lcm finds the lowest common denominator and nreconc does some sort of reversing. The first is available in Clojure contrib library, the second is not available in Clojure.

Exercise 3.11 [m]

There is a built-in Common Lisp function that, given a key, a value, and an association list, returns a new association list that is extended to include the key/value pair. What is the name of this function?

Clojure's assoc does this:

(assoc {:other-key 'other-value} :key 'value) ; => {:other-key other-value, :key value}

Exercise 3.12 [m]

Write a single expression using format that will take a list of words and print them as a sentence, with the first word capitalized and a period after the last word. You will have to consult a reference to learn new format directives.

I think this cannot be done by format. It can for example be done via clojure.string methods:

(defn as-sentence
  "Takes a list of words and creates a sentence with first word capitalized and period at end."
  [word-list]
  (str (clojure.string/capitalize (clojure.string/join " " word-list)) "."))

That's it. In Chapter 4 we will built a General Problem Solver.

PAIP in Clojure Chapter 2

Thu, 11 Aug 2016 00:00:00 +0200

After some vacation and working on other things I finally got round to Chapter 2 of Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp (PAIP).

SX359BO1,204,203,200_.jpg">

Summary for Chapter 2: A Simple Lisp Program

The chapter starts with a quote by the Italian royal historiographer Giovanni Battista Vico (1668-1744):

Certum quod factum.

For those that do not speak Latin, this is the rough translation: "One is certain only of what one builds."

In this chapter we build a more elaborate Lisp program and therefore live a more certain life. The program we will build generates random English sentences. In fancy terms we call the program generative syntax for a context-free phrase-structure grammar.

The original Common Lisp code from the book can be viewed here. First I translated this code to Clojure below.

(ns paip.simple
  (:gen-class))

; Clojure translation of the Common Lisp code in Chapter 2.
; Original: https://norvig.com/paip/simple.lisp

;;; Code translated from Paradigms of Artificial Intelligence Programming
;;; Copyright (c) 1991 Peter Norvig

(defn one-of
  "Pick one element of set, and make a list of it."
  [set]
  (-> set rand-nth list))

(def simple-grammar
  "A grammar for a trivial subset of English"
  {:sentence    [[:noun-phrase :verb-phrase]]
   :noun-phrase [[:article :noun]]
   :verb-phrase [[:verb :noun-phrase]]
   :article     #{"the" "a"}
   :noun        #{"man" "ball" "woman" "table"}
   :verb        #{"hit" "took" "saw" "liked"}})

(def bigger-grammar
  "A somewhat bigger grammar for a trivial subset of English"
  {:sentence    [[:noun-phrase :verb-phrase]]
   :noun-phrase [[:article :adj* :noun :pp*] [:name] [:pronoun]]
   :verb-phrase [[:verb :noun-phrase :pp*]]
   :pp*         [[] [:pp :pp*]]
   :adj*        [[] [:adj :adj*]]
   :pp          [[:prep :noun-phrase]]
   :prep        #{"to" "in" "by" "with" "on"}
   :adj         #{"big" "little" "blue" "green" "adiabatic"}
   :article     #{"the" "a"}
   :name        #{"Pat" "Kim" "Lee" "Terry" "Robin"}
   :noun        #{"man" "ball" "woman" "table"}
   :verb        #{"hit" "took" "saw" "liked"}
   :pronoun     #{"he" "she" "it" "these" "those" "that"}})

;;; ==============================

(def grammar
  "The grammar used by generate.  Initially, this is
  simple-grammar, but we can switch to other grammars."
  simple-grammar)

;;; ==============================

(defn rule-lhs
  "The left hand side of a rule."
  [rule]
  (->> rule keys (apply concat)))

(defn rule-rhs
  "The right hand side of a rule."
  [rule]
  (->> rule vals (apply concat)))

(defn rewrites
  "Return a list of the possible rewrites for this category."
  [category]
  (rule-rhs (select-keys grammar [category])))

;;; ==============================

(defn generate
  "Generate a random sentence or phrase"
  [phrase]
  (cond (vector? phrase)
        (mapcat generate phrase)
        (seq (rewrites phrase))
        (generate (rand-nth (rewrites phrase)))
        :else
        (list phrase)))

(generate :sentence)
; => (One example)
;    ("Kim" "saw" "Pat")

;;; ==============================

(defn generate-tree
  "Generate a random sentence or phrase,
  with a complete parse tree."
  [phrase]
  (cond (vector? phrase)
        (map generate-tree phrase)
        (seq (rewrites phrase))
        (cons phrase
              (generate-tree (rand-nth (rewrites phrase))))
        :else (list phrase)))

(generate-tree :sentence)
; => (one example)
;    (:sentence (:noun-phrase (:Article "the")
;                             (:Adj*)
;                             (:Noun "ball")
;                             (:PP*))
;               (:verb-phrase (:Verb "liked")
;                             (:noun-phrase
;                              (:Name "Robin"))
;                             (:PP*)))

;;;; ==============================

(defn combine-all
  "Return a list of lists formed by appending a y to an x.
  E.g., (combine-all '((a) (b)) '((1) (2)))
  -> ((A 1) (B 1) (A 2) (B 2))."
  [xlist ylist]
  (mapcat (fn [y]
            (map (fn [x] (concat x y)) xlist))
          ylist))

;; TODO: There is a bug in generate-all. For sentences that are constructed out of noun-phrase
;;       or verb-phrases the resulting sentences are displayed in characters.
(defn generate-all
  "Generate a list of all possible expansions of this phrase.
   Only works for non-recursive grammars."
  [phrase]
  (cond (vector? phrase)
        (combine-all (generate-all (first phrase))
                     (mapcat generate-all (rest phrase)))
        (seq (rewrites phrase))
        (mapcat generate-all (rewrites phrase))
        :else
        (list phrase)))

The translation between the different built-in methods and difference in behavior between Clojure and Common Lisp cost me quite some time, but it was worth it: I know more about Clojure than I did before, and I have a working sentence generator.

The lessons from the chapter are:

data-driven programming (where data drives what a program does next) makes it easy to extend functionality of a program just by adding new input data.In the parser above simple-grammar and bigger-grammar both work without having the program to change).
Code is also data, in what Norvig calls the one-data/multiple-program approach we can create slightly different versions of the same program which accomplish different tasks using the same data. generate-tree and generate-all programs are example of one-data/multiple-program: slight changes lead to different related functionality.

generate-all is not fully functional since somehow the words are cut down into characters when they are created by a verb-phrase or noun-phrase. Perhaps that problem will somehow be solved in the exercises.

Exercises

The source code with unit tests is available at https://www.github.com/transducer/paip.

Exercise 2.1 [m]

Write a version of generate that uses cond but avoids calling rewrites twice.

Instead of checking it the rewrites are non-empty, in our way of defining the grammar (using the native Clojure map) we can perform the same check by checking if the phrase is a keyword:

(defn generate
  "Generate a random sentence or phrase"
  [phrase]
  (cond (vector? phrase) (mapcat generate phrase)
        (keyword? phrase) (generate (rand-nth (rewrites phrase)))
        :else (list phrase)))

Exercise 2.2 [m]

Write a version of generate that explicitly differentiates between terminal symbols (those with no rewrite rules) and non-terminal symbols.

Just extract a method and provide it with a meaningful name.

(defn non-terminal?
  "Determines if a phrase is non terminal."
  [phrase]
  (keyword? phrase))

(defn generate
  "Generate a random sentence or phrase"
  [phrase]
  (cond (vector? phrase) (mapcat generate phrase)
        (non-terminal? phrase) (generate (rand-nth (rewrites phrase)))
        :else (list phrase)))

Exercise 2.3 [m]

Write a trivial grammer for some other language. This can be a natural language other than English, or perhaps a subset of a computer language.

A subset of "Clojure":

(def simple-clj-grammar
  "A grammar for a trivial subset of Clojure"
  {:code                [[:definition-phrase :function-definition]]
   :definition-phrase   [[:definition :names :docstring]]
   :function-definition [[:library-definitions :parameter-types]]
   :definition          #{"defn"}
   :docstring           #{"wtf?" "TODO"}
   :names               #{"my-method" "awesome?" "change-it!"}
   :library-definitions #{"map" "mapcat" "inc" "dec"}
   :parameter-types     #{"string" "int" "long" "bool"}})

Exercise 2.4 [m]

One way of describing combine-all is that it calculates the cross-product of the function append on the argument lists. Write the higher-order function cross-product, and define combine-all in terms of it. The moral is to make your code as general as possible, because you never know what you may want to do with it next.

It is possible to write cross-product using a for-comprehension:

(defn cross-product
  "Calculates the cross product."
  [f coll1 coll2]
  (if (empty? coll1)
    '(())
    (for [x coll1
          y coll2]
      (f x y))))

(defn combine-all
  [xlist ylist]
  (cross-product concat xlist ylist))

The cross-product function in use:

(cross-product + '(1 2 3) '(10 20 30))
; => '(11 21 31
;      12 22 32
;      13 23 33)))

(cross-product list '(a b) '(1 2))
; => '((a 1) (a 2) (b 1) (b 2))))))

That was a full day of work. I will start with Chapter 3 tomorrow.

PAIP in Clojure Chapter 1

Wed, 13 Jul 2016 00:00:00 +0200

From my parents I received the book Paradigms of Artificial Intelligence Programming by Peter Norvig for my 28th birthday.

SX359BO1,204,203,200_.jpg">

PAIP is a classic with high standard Common Lisp code to solve problems in artificial intelligence. The coming year I will study this book.

There are 25 chapters and there are 52 weeks till 13 July 2017. This means I have to finish a chapter roughly every two weeks.

Summary for Chapter 1: Introduction to Lisp

The chapter starts with a quote by Alan Perlis:

You think you know when you learn, are more sure
when you can write, even more when you can teach,
but certain when you can program.

The rest of the chapter explain the fundamentals of programming in Lisp and why it is awesome.

Exercises

I will do a selection of the exercises in this book. I assume that you as a reader have not read the book, so I'll explain the exercises if necessary. The source code with unit tests is available at https://www.github.com/transducer/paip.

Exercise 1.1 [m]

Define a version of last-name that handles "Rex Morgan Md," "Morton Downey, Jr.," and whatever other cases you can think of.

In the book we had a function first-name to retrieve the first names from a list of names, where a name is a list of strings like ("Robert" "Downey" "Jr."). The first-name method first checked if the first element of the name existed in the parameter list *titles*. If it did first-name invoked itself recursively with the rest of the name.

We have to do the same, but instead of looking for the first element, we have to look for the last. And instead of validating the element does not exist in *titles*, we look in a parameter called suffixes.

(def suffixes '("Jr." "Md"))

(defn last-name
  "Select the last name from a name represented as a list."
  [name]
  (if (some #(= (last name) %) suffixes)
    (last-name (drop-last name))
    (last name)))

The library functions drop-last and last are convenient.

Exercise 1.2 [m]

Write a function to exponentiate, or raise a number to an integer power. For example: (power 3 2) = 3² = 9.

With the use of an iterative loop:

(defn power
  "Exponentiate a base to a power"
  [base power]
  (loop [b base
         n power
         result 1]
    (if (= 0 n)
      result
      (recur b (dec n) (* b result)))))

Exercise 1.3 [m]

Write a function that counts the number of atoms in an expression. For example: (count-atoms '(a (b) c)) = 3. Notice that there is something of an ambiguity in this: should (a nil c) count as three atoms, or as two, because it is equivalent to (a () c)?

(defn count-atoms
  "Counts the number of elements in an expression. nil is considered an atom.
   Clojure does not have true atoms, so better to call this method count-elements"
  [expression]
  (->> expression
       flatten
       count))

Exercise 1.4 [m]

Write a function that counts the number of times an expression occurs anywhere within another expression. Example: (count-anywhere 'a '(a ((a) b) a)) ; => 3.

(defn count-anywhere
  "Counts the number of times an expression occurs anywhere within another
   expression"
  [expression-to-search expression]
  (->> expression
       flatten
       (filter #{expression-to-search})
       count))

Exercise 1.5 [m]

Write a function to compute the dot product of two sequences of numbers, represented as lists. The dot product is computed by multiplying corresponding elements and then adding up the resulting products. Example:

(dot-product '(10 20) '(3 4)) = 10 x 3 + 20 x 4 = 110

(defn dot-product
  "Calculates the dot product. Assuming vectors are of even length and consist of numbers."
  [coll1 coll2]
  (if (empty? coll1) 0
    (+ (* (first coll1) (first coll2))
       (dot-product (rest coll1) (rest coll2)))))

How a Petri net can help you simplify your business logic

Thu, 02 Jun 2016 00:00:00 +0200

In enterprise organizations it is bound to occur that different teams are building similar software independently of each other. For example, multiple teams write code which interacts with various services and performs some business logic in a similar manner. The complexity of creating software is usually not in creating components which interacts with external software itself (e.g., connecting with a RESTful API via a service client class). The complexity resides in the means of combination of these various components. I.e. the modeling of the business process (the flow of work through an organization).

A Petri net is a mathematical modeling language which is particularly suited to model business processes. In the foreword of Modeling Business Processes: A Petri Net-Oriented Approach by Aalst & Stahl (2011) it is stated that:

Petri nets provide the foundation of the graphical notation and the basic primitives for modeling concurrency, communication, and synchronization. The extension of these basic primitives with data (color) ... makes it possible to model and analyze complex artifacts. (p. iv).

To facilitate flexibility in the face of changing requirements, to encapsulate the business rules in one central place, and to possibly tackle the problem of re-usability described above, a Petri net can be used to model a business process. Below I will show an example of using a Petri net using a domain specific language (DSL) which we have started creating at work. From this DSL which describes a business process an event-driven Petri net representation can be generated. All that is left of complex business logic in an application is the need to send the right events (with data) to the Petri net representation.

In this article I will elaborate on Petri nets, the DSL created and an event-driven Petri net representation.

Petri nets

In 1939 Carl Adam Petri was only 13 years old when he invented the Petri net with the goal of describing a chemical process. A Petri net is a directed bipartite graph consisting of places and transitions, where every place only connects with transitions and every transition only connects with places.

Below a visualization of a Petri net is provided. Transitions are visualized as rectangles and places as circles.

(Source: Wikipedia)

In the places tokens can exist (the dots that appear in the image above). Once all in-adjacent places towards a transition are filled, the transition can fire and consume the tokens, and thereby produce new tokens in the out-adjacent places out of the transition. Parallelism is supported (e.g., multiple places next to each other coming out of a transition or multiple places leading to one transition).

There are various extensions to the Petri net. A colored Petri net is a special type of Petri net where the tokens can also have data. A colored Petri net is particularly suited for modeling business processes.

Especially in Europe much research has been done on Petri nets and certain properties can be proven about them. It is possible to mathematically reason about reachability ("can we reach a certain state?"), liveness ("can a transition fire?") and boundedness ("how many tokens can there be?"). Reasoning about such properties is useful for example for:

Showing how much memory or disk space a process will consume.
Checking the validity of a business process.
Analyze bottle necks in the flow.

By using Petri Net Markup Language (PNML) to describe the Petri net analysis can be done via existing tools.

DSL

When a chef cooks a recipe she combines various ingredients. When certain events happen, or when ingredients are ready, a chef knows what action(s) to execute next. E.g., when a chef receives an event WaterBoiling she knows that she can start cooking the pasta. And once the chef has a knife and tomatoes she will start cutting them. If the chef is skilled, this results in pasta sauce when finished.

In the DSL below a similar recipe can be provided to a chef – only the chef is an execution engine written in Scala. Ingredients do not consist of edible substances but of software components. Actions to be executed are not certain kitchen skills, but instead methods on these software components. The preconditions for these actions are data or events that are provided.

The chef is all that is needed to handle flows through our business process.

Event-driven Petri net representation

In the current setup the recipe is represented as a Java map where the keys are the interfaces of the components and the values a representation of the actions. The representation of the action has properties to specify which methods are needed and which preconditions they have. These preconditions are specified as events or data.

Once the chef received the recipe it can start cooking a Petri net topology. The chef "cooks" by using reflection to get all the information out of the recipe and the ingredients. This topology is shared with various states of the Petri net (a state describes in what place which data/tokens are).

The chef can listen for events that are happening. Once it processes an event it knows to which Petri net state it belongs by looking at the process identifier, and it will perform the action specified in the recipe and update the state accordingly with the results.

The Petri net is modeled in-memory via the Kagera library which is maintained by one of my team members at work (who happens to be really enthusiastic about Petri nets). In the future it will be possible to store the state on disk using event sourcing – with the added benefit that all historical flows can be replayed and analyzed.

The chef has the following abilities:

It can be provided with real or stub implementations of the ingredients in its constructor.
It must be told what events are used.
It is possible to retrieve all state which belongs to a certain process.
It provides some more methods to get a GraphViz or PNML representation of the recipe for visualization and analysis purposes.

Below we see the interface of the chef:

class Chef(recipe: Map[Class[_], Seq[Action]],
           ingredientImpls: Map[Class[_], AnyRef],
           events: Set[Class[_]]) {

  /* Cook the recipe for a specific process id */
  def cook(processId: UUID): Unit = /* (...) */ 

  /* Get all accumulated state from the Petri net marking */
  def getAccumulatedState(processId: UUID): Map[String, Optional[Any]] = /* (...) */ 

  /* Tell the chef an event has happened for a specific process id */
  def tellEventHappened(processId: UUID, event: AnyRef): Unit = /* (...) */ 

  /* Get the GraphViz (graph visualization software) implementation of the recipe */
  def getVisualRecipe: String = /* (...) */ 

  /* Get the PNML (Petri Net Modelling Language) representation of the recipe */
  def getPnmlRecipe: String = /* (...) */ 
}

When getVisualRecipe is called the following GraphViz visualization of the recipe describing a simple cooking process is returned:

Here events are modeled as blue diamonds, red circles represent data (places in Petri net terminology), and grey rectangles are the actions on the ingredients (transitions in Petri net terminology).

In the code of your application only the user-facing parts need to be created. Here the chef can be notified by events via the tellEventHappened method.

For example in a Java API, an instance of the Chef is provided in the classes providing API endpoints. This can look roughly as follows:

@Autowired
private JChef chef;

@POST
@Path("pastaSauce")
public Response waterBoiling(UUID processId, Sauce sauce) {
    chef.tellEventHappened(processId, new PastaSauceReady(sauce));

    return Response.ok().build();
}

Once the API is called on the pastaSauce resource there is no complex if-then-else business logic. The chef is just told whom it is that is calling our API, that it is the PastaSauceReady event that has happened and what sauce data was provided. All logic on what to do with this information is provided by the recipe and encapsulated in the chef.

Closing remarks

By using the modeling language of Petri nets and by creating a DSL it is possible to facilitate reuse, flexibility and encapsulation of complexity in the modeling of a business process. Changing the business logic has become as easy as changing the recipe. Business analysts can read or view the recipe and see if it adheres to their ideas. The chef should even be able to tell if it can cook a specific recipe. In theory, it is possible to share the ingredients and the chef with other teams and let them create new recipes reusing the components. And where in classically build APIs it can be difficult to trace where things went wrong, because of the properties of Petri nets it is possible to thoroughly analyze flows taken through the application.

If you want to read more:

Fowler, M. (2010). Domain-Specific Languages. Addison-Wesley Professional.
Van der Aalst, W.M.P. & Stahl, C (2011). Modeling Business Processes: A Petri Net-Oriented Approach. The MIT Press.

Programming and meditation

Tue, 17 May 2016 00:00:00 +0200

Our minds jump from past memory to future fantasy and are extraordinarily out of control. Buddhism provides a technique which helps us to tame our wild minds: meditation.

Meditation is sitting down quietly and being aware of all that goes on, without comment. In meditation (Vipassana) the idea is to stay with reality as it is experienced in this very moment. It brings about purification and insight.

I learned the technique of meditation at a 10-day retreat of dhamma.org. Last year I went to Thailand and I have lived 10 days in a similar manner as Buddhist monks and nuns at Wat Ram Poeng.

Blending in at a meditation retreat at Wat Ram Poeng

This has led to me appreciate this wonderful technique for feeling at ease and dealing with the vicissitudes of life.

Where meditation helps me to stay grounded, programming, on the other hand, can be an activity which moves me away from everyday reality. As a programmer I work with abstractions.

As Fred Brooks writes in The Mythical Man-Month:

The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds castles in the air, from air, creating by exertion of the imagination.

Meditation can not only provide a welcome counterweight to this work with abstractions, it also cultives 10 qualities of character (Pali: paramis) that are useful during the practice of programming.

Below I will list these 10 qualities and explain how they might benefit the programmer.

Dana – Generosity

Meditation teaches you to give freely without expecting anything in return. Won't be surprised when you start coding for others for the sheer joy of giving.

Sila – Morality

Meditation helps you to speak properly, perform good actions and have a right means of livelihood. Software engineering involves communication with others and is a way to sustain yourself. Cultivating the urgency to do or say the right thing will help you with this.

Nekkhamma – Renunciation

By renouncing the thoughts of for example sensual pleasures you become more adequate at dealing with that what is actually going on. When you code you code. Nothing else.

Panna – Wisdom/understanding

Meditation makes you realize on a fundamental level that everything is impermanent (Anicca) and that because we as human beings cling to and identify with (Anatta) this changing reality we suffer (Dukkha). One thing to learn from this is that you are not your code. This helps when receiving feedback.

Viriya - Effort

Mindfulness trains the mind to not procrastinate. It leads to prolonged periods of focus. Focus comes in handy during programming.

Khanti – Patience / tolerance

Meditation gives you the patience and tolerance to just fill in that Excel sheet for the next hour.

Sacca – Truthfulness

You will tell your boss the true state of affairs: the software will not be finished before the deadline.

Adhitthana – Strong determination

Just like the Buddha did not leave the tree till he was enlightened, you will not stand up from your chair till you have finished that piece of code.

Metta – Loving-kindness

Meditation teaches you to love what you do and the people you do it with. Enjoying what you do and cultivating good relationships with colleagues or friends helps with programming just as much as everything else.

Upekkha – Equanimity

Sitting through (enjoying even) hours of pain in your legs without responding to it has taught you to stay calm in the face of adversity. These 20 more use cases that you had initially overseen that will need to be implemented? No problem.

As a bonus, what I love of meditation is how aware you become of bodily sensations. For example: you notice earlier that your body is aching and that it is time to walk and stretch. It also helps with creativity. It's impressive how ideas start to flow when you watch your belly go up and down 30 minutes in the morning.

Interested? A good way to learn the technique of Vipassana meditation and experience its beneficial results is at a 10-day retreat at dhamma.org. Alternatively you can go to a country like Thailand and learn the technique at one of the meditation centers there.

Shameless plug:

Vim and Emacs

Mon, 02 May 2016 00:00:00 +0200

I'll show you why I love Vim and why I switched to Emacs and why you might too.

Vim

Vim is really really awesome. Inspired by a lot of great hackers who swear by Vim (like Bram Moolenaar and Vim Diesel) and by the greatest ode to a text editor ever ("Vim Creep"), eight months ago I decided to take the plunge.

First I did vimtutor at home, then I installed VimEmu in Visual Studio (I programmed C# at the time), I printed the Vim cheat sheet, and broke up with my mouse.

Against the advice in the excellent video Learning Vim in a week (see below) I did start at work.

Work is the place where I edit the most text and I needed to do some boring work at the time. I have no regrets learning Vim there. Within a week my speed was back to my average before that. I became slower in certain parts but faster in others.

More importantly, boring, repetitive tasks were no longer boring and repetitive. In Vim, writing boring code is seriously fun because you stretch your mind on creating efficient text manipulations. I stayed interested doing the work, and I learned Vim. In that period both my employer and myself have benefited from me learning Vim at work.

Eight months later I use Vim everywhere. Nowadays even a mundane task as updating my TODO list is exhilarating.

The following example of the awesomeness of Vim is based on a real life situation where I wanted to unroll a T-SQL loop. Suppose you want to create four separate update statements from the four table names (First, Second, Third, Fourth) and columns above the EXECUTE block below:

INSERT INTO @TABLES_COLUMNS (TABLE_NAME, COLUMN_NAME)
    VALUES
    ('[dbo].[First]', '[Yadadada]'),
    ('[dbo].[Second]', '[Blabla]'),
    ('[dbo].[Third]', '[Abacadabra]'),
    ('[dbo].[Fourth]', '[Yeehah]'),
    (...)

    WHILE 1=1
    /* Set the variables of table name and column name (substitute with VALUES above) */
    EXECUTE
    (
        'UPDATE ' + @TABLE_NAME + ' ' +
        'SET ' + @COLUMN_NAME + ' = temp.OLD
        'FROM ' + @TABLE_NAME + 'AS t ' +
        'INNER JOIN #temp AS temp ' +
        'ON t.' + @COLUMN_NAME + ' = temp.NEW'
    )

(Note that this code is a bit simplified: the setting of the loop variables and extra values are skipped.)

In the real life example this was based on there were about 15 of those substitutions instead of four. This is what I would consider a boring task. Since the task is short enough that writing a script to automate it is slower than actually doing it, in normal (non-Vim) life I would to do this by hand.

What needs to be done above is:

Cleanup the "+" and quotes from the EXECUTE statement.
Copy the cleaned up EXECUTE statement.
Copy the first variable value and substitute it for @TABLE_NAME.
Copy the second variable value and substitute it for @COLUMN_NAME.
Repeat step 2 till 4 for the next entries in @TABLES_COLUMNS.

I would have to do this about 15 times. I think this would take me about 10 minutes.

With Vim it is different.

I recorded the operations on the text once, and then I executed the operation I just did on the next two variable values. Then on the next, then on the next. This is programming your text. And it is so easy. You should probably see the recording and execution of the macro to understand:

_{Note: after recording I see that showing these keypresses doesn't really help since a lot of them are missed and you cannot easily see the difference between lower and upper case. :) Apologies.}

We see lots of missed efficiencies and some wrong keypresses. It is a dirty recording. But life in Vim is messy. Editing code with a mouse doesn't go right the first time either.

What we see in the gif above is that first I remove the no longer necessary EXECUTE statement after easily having jumped to the end of the file (G). (Note that navigating in Vim goes with h, j, k and l – you never have to leave your home row.) I visually select the statement with something like V5k (visually select five lines [V] upwards [k]).

In Vim it is as if you speak to your computer: you say a verb, you say how many times and you say what.

Then I substitute the "'" and "+" from the selected region with a command (:) with nothing globally:

:s/[+']//g

Next I delete the selection and I yank the UPDATE statement plus one blank line in register u ("ud [delete into register u]). Now I navigate to the top of the page (gg) and start a recording in register a (qa) (I pick register a for no other reason than that it is close to q). I navigate down two lines (2j), find the first quote (f') of the first variable and yank (copy) everything inside the quotes (') into register f ("fyi'). Next I go to the end of the line ($) and copy the second variable into register s. (I use register f for first and s for second.) So now we have the table name we want to substitute stored in register f and the column name in register s.

Then I go to the end of the file (G), paste everything inside register u ("up). This way the UPDATE-statement comes at the bottom of the page. i Next I visually select the word of the table name variable and paste the f register over it ("fp) (so: vw"fp). I also substitute the column name for the new value inside register s.

After having substituted all the column names and table names variables for the new values, I jump to the top of the file and go two lines down (gg2k). I remove the line (dd), jump back to the top (gg) and stop recording (q).

Next I execute that what I have just recorded three times (3@a)! Since the recording was on a meta-level of navigation, selection and substitution this works perfectly fine for the other variable names. Each time copying the statement, removing and substituting the values of the table and column name and ending on the top of the file. The whole file is done. This would have worked just as fast for 100 rows by the way.

I find these possibilities so cool.

/edit As Veedrac showed in a comment on reddit in Sublime Text this refactoring is also possible and arguably more intuitive using multiple cursors. Vedrac made a video where multiple cursors are used to do the same refactoring. This video can be watched here.

Note that the jumping around in Vim and selecting text is usually the biggest time saver. As JangoSteve writes in the comments on Hacker News in response to the article Vim Creep:

vim doesn't save you five minutes, twenty times a week; it saves you 0.75 seconds, eight thousand times a week. Same time savings, but much harder notice the pain before you've solved it.

And indeed the most time savings are just all the basic movements where you would have normally grabbed your mouse repeatedly. But Vim is also absolutely wonderful for these large refactorings as above.

To be honest, when I recorded the macro for the first time (in the real life thing where the example was based on) it only really worked the tenth time I tried. (In my defense: it was during a late night cowboy deploy that went horribly wrong because of a query that performed a tad bit worse on production than acceptance and there were people watching my actions via the remote which gave me performance anxiety.) But finally it did succeed and I learned from it and this time while recreating it I got it in one go. In the excellent book Learning the vi and Vim Editors something like this is said about forcing yourself to learn efficiency gains: first some action or learning a regex will take you some time, but this pays back many times over the next time when hyper-efficient text-editing has become embedded in your muscle memory.

Note that one of the lessons of the Learning Vim in a Week video above is that "There is always a better way." Typing this I see I could have better substituted the variable names in a similar manner as how I substituted the quotes and plus from the full statement. Also I used B or W instead of b and w to jump Backwards or forwards a big Word (including the @). So there was indeed a better way with less keypresses. There surely is even a (much) better way.

Furthermore, the fact that I am now recreating this situation shows how much fun it is. I really don't mind.

I got a bit carried away already, but Vim is awesome. Some of the things we have seen is easily navigating with the keyboard, executing commands over lines of text, storing stuff in multiple registers, editing selecting or deleting specific blocks of text, and recording a macro to re-execute a sequence of keypresses.

This is made possible by different modes that are available in Vim: normal mode for navigation, insert mode for inserting text, command mode for executing a command and visual mode for selection.

Some more awesomeness that is not shown in the example above are for example:

The .-command to repeat your last action.
Editing the .vimrc file to easily bind new keys and add plugins. (Well this is shown, you see for example syntax highlighting and relative line numbers).
The awesome Vim plugin surround.vim to easily change surrounding quotes or braces into something else or add or delete them.
Marking places in the text and jumping back (ma – mark this position with a and 'a to jump to position a).
Reselecting the region you just had selected with gv.
Doing a block selection with CTRL-V and changing the selection size on the other side with o.
Navigating undo trees (e.g., go to the state of your file five minutes back in time with :earlier 5m).
Pair programming by working in the same file after sharing your buffer over SSH using tmux.
Everything else other text-editors can do.

If you want to jump into Vim I can advice you to watch the video on Learning Vim in a Week above and do the vimtutor.

Stick with it. You will not regret it.

Emacs

I love Lisp and it turns out that for Lisp editing it is somewhat more logical to use Emacs (although for example the expert Lisper Doug Hoyte swears by vi – the non-improved version of Vim). The reason for that is it has tight integration with the language and you can use a version of Lisp called ELisp to configure and modify the editor. (Note that some people get really carried away with editing their editor instead of editing arguably more useful code).

But even before I used Emacs to learn the Lisp-dialect Clojure I had already switched to it. The reason for that was the video "Evil Mode: Or, How I Learned to Stop Worrying and Love Emacs":

(Warning: it is a bit long.)

Evil Mode is an Extensible vi layer for Emacs. This means you can have Vim keybindings while having the Emacs environment.

Why would you want that? Well, Emacs is a much better environment than Vim. Some people call it an operating system instead of a text-editor. And it is true. Where Vim will sometimes hang when performing large operations, Emacs just runs on easily.

In the video above examples are shown how things that would block normal Vim can sometimes go on in Emacs on a background thread. And for me Emacs with Evil mode just feels lighter than Vim. There are some other attempts to improve the performance of Vim (notably NeoVim), but this would still not have the Lisp integration Emacs has.

Furthermore, things like searching and replacing are better. You visually see in the text what will change while a substitution command is typed. And search matches light up before you finished typing your search. Examples of search and replace are shown in the video at the end. And when you start typing in the command area it automatically starts showing suggestions. There are a lot of these minor things that make Emacs with Evil mode very pleasurable.

Because of the Lisp integration (firing up a REPL and never having to leave Emacs), nice quirks as command completion, and the better performance I use Emacs.

To configure Emacs you can edit a .emacs file or the ~/.emacs.d/init.el file. Evil mode provides a lot of ways to hook into the execution flow and plugins. For editing Clojure I just use the Emacs setup as described in Clojure for the Brave and True extended with Evil mode.

Some of the non-Evil features I use for editing Clojure are mentioned there. Things like CTRL-c CTRL-k to compile code and CTRL-c CTRL-d CTRL-d to view the library definition (if I am not mistaken, fact of the matter is the keybindings for Emacs to me are much more unlogical than those of Vim).

Now I will visually show some of the other awesomeness, but related to Evil mode. Just to give an impression.

Redoing things with the .-command work just as in Vim:

Above we see the . command used to change text inside the parentheses " (ci").

Next I have bound CTRL-b to display all available buffers and easily switch between them. This is done in such a way you can use the familiar hjkl to navigate.

And we can display multiple windows vertically (CTRL-W v) or split them horizontally (CTRl-W s):

(Since I did not want to not use these buffers at all, note that we made a selection upper case with U and it changed in all views of the buffer simultaneously.)

Another really awesome thing which I believe comes out of the box is undolist, with it you can visually navigate the undo history. Yes, you can view the branches you decided to delete. Undo history can easily be viewed and navigated by typing the command:

:undolist

How cool is that?

And we can add a very rich set of plugins to Emacs. Of course we have all the plugins added for Clojure by the author of Clojure for the Brave and True. Which provide syntax highlighting and tight integration with documentation, the REPL and executing code.

I'll now show two other plugins I find very interesting.

We can display the file system on the left if we want with F8 by using NeoTree:

Above we see me pressing F8 and navigating to an uninteresting folder on my file system.

Furthermore, look at this "slurping" of some elements from a string with > (SHIFT-.):

(Note that the five lines are combine with 5J [J is for joining two lines] – another feature I wanted to force into this).

Slurping on one end or the other or barfing things out on one side of the other is a feature of ParEdit. ParEdit is helpful for manipulating S-expressions (the parentheses in Lisp) but can also be used for other structures like the quotes above.

Is everything rose shine and moon light? No. I have identified the following shortcomings so far:

Sometimes I want to quit an Emacs buffer (q), but then I cannot do so because I first have to enter insert mode. See my question about it on the Emacs Stack Exchange here.
The window jumps halfways unwantedly when scrolling up and down (CTRL-E CTRL-Y btw). This can probably be fixed.
Paredit can be annoying. You have to press CTRL-Q before inserting a parenthesis. But you can of course disable it.

That's all I can come up with and I have thought really hard.

Closing remarks

I think Emacs Evil mode is a very nice environment to edit text. I hope I was able to show some of that by the examples. If you want some more of Evil mode I can recommend this great demonstration:

You can download the Clojure for the Brave and True emac setup which I extended with Evil mode as descibed above by clicking here. If you install Emacs and place the repository in the ~/.emacs.d directory everything will install automatically. Note that there are probably better dotfiles out there. The way to install plain Vim can be found here.

One final note: if you start using Vim do not forget to remap your Capslock to Escape. You will be pressing Escape a lot and on todays keyboards it is too faraway from the homerow.

Thank you for reading. Comments are welcome.

The meta-circular evaluator

Mon, 25 Apr 2016 00:00:00 +0200

Meta-circular evaluators are awe-inspiring.

This will be a walkthrough of the meta-circular evaluator demonstrated in Chapter 4 and Lecture 7A of The Structure and Interpretation of Computer Programs (SICP). The chapter and video are on the subject of "meta-linguistic abstraction" – the establishing of new languages.

To evaluate a computer language you need an evaluator (also called interpreter) for that language. Evaluating an expression in a programming language means that the evaluator performs or executes the instructions described in the expression. An evaluator is called meta-circular if it evaluates the same language as it is written in using the same language constructs as the language itself (circular definitions).

Why would you want such a thing? One of the reasons is that having a meta-circular evaluator makes it very practical to implement new languages on top of the implementation language. Using a meta-circular evaluator you can, for example, create a language that is particularly suited for a problem at hand (a Domain Specific Language). Another reason is that it is insightful for educational and experimental purposes. The basic eval-apply structure below can be used to write interpreters for all kinds of languages. It is the kernel for every computer language.

What does a meta-circular evaluator look like? In SICP it is stated that:

It is no exaggeration to regard this as the most fundamental idea in programming:

The evaluator, which determines the meaning of expressions in a programming language, is just another program.

So it is just another program. In this post I have translated the code from meta-circular evaluator program from SICP (you can find the original code here) to Clojure.

Clojure is interesting because:

It runs on the Java Virtual Machine (therefore having access to all Java libraries and being able to interoperate with existing Java code bases).
It has a functional core (therefore being particularly suited for concurrent programming, necessary to utilize multiple CPUs).
And it is a dialect of Lisp (therefore having all the interesting properties described below).

Now, before I continue, let me be clear. I am not an expert on the subject matter. It is impossible for me to improve on the explanation in SICP. What I have attempted to do here is translate the Scheme code of SICP to Clojure and thereby deeper understand what is going on in the meta-circular evaluator and also learn something about Clojure. I have tried to put into my own words what is described in SICP. If you have a background in computer science this post is probably not much new. After all, SICP is an introductory level computer science book. I don't have such a background so for me this was all pretty mind blowing. I learned quite a lot from this post myself. Before I started to write I thought I understood, but when converting the SICP code to Clojure I realized I did not understand it as thoroughly as I thought. I needed quite a lot of help from the web and the end result is basically different sources copy-pasted together till I had a reasonably working evaluator.

It is very likely that the text in this post contains mistakes and I would be happy if they will be pointed out to me as well.

Now, let's get started.

Background

The idea of a meta-circular evaluator was first described by John McCarthy in his paper Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I.

(There has by the way never been a Part 2, although some people have some ideas what it would look like).

This is the paper where McCarthy invented Lisp (acronym for LIst Processing). It contained the definition for eval – the universal function. eval is a universal machine: if you feed it a Lisp program, eval starts behaving as that program. It required inventing a notation where you could represent programs as data.

For McCarthy, writing this function was just a theoretical exercise. It was never his idea that the language or the function was actually implemented. What then happened was that Steve Russell, one of McCarthy's graduate students (who also invented one of the first computer games) transformed his theory into an actual programming language. As McCarthy stated in an interview:

Steve Russell said, look, why don't I program this eval..., and I said to him, ho, ho, you're confusing theory with practice, this eval is intended for reading, not for computing. But he went ahead and did it. That is, he compiled the eval in my paper into IBM 704 machine code, fixing a bug, and then advertised this as a Lisp interpreter, which it certainly was. So at that point Lisp had essentially the form that it has today..." (Source)

Alan Kay (famous computer scientist and creator of the language Smalltalk) describes his excitement when he first saw this eval and understood that it was Lisp written in itself.

These were "Maxwells Equations of Software!" This is the whole world of programming in a few lines that I can put my hand over.

^{Note: the Maxwell equations are four extremely elegant equations from which everything there is to know about the electromagnetic field can be derived.}

Lisp is special because it does not enforce syntactic and semantic rules: it can be programmed at a higher level of abstraction than other languages. Lisp can be transformed in other languages. Some of that we will see below.

So what does Lisp look like?

Lisp, Lisp, Lisp

The nice thing about Lisp is that is has almost no syntax. The rules are simple (like Go), but the possibilities within these rules are virtually endless (like Go). Lisp is infamous for its parentheses.

(Source: https://xkcd.com/297/)

Combined with prefix notation these parentheses makes it so that Lisp data structures have the same makeup as Lisp code. The fundamental data structure in Lisp is a list (in Clojure we have some more data structures, we will not worry about that now). The first part of a list is interpreted as the operator, and the rest as the operands.

So for example in:

(+ 1 2 (* 3 4))

The operator is + and the operands are 1, 2, and (* 3 4). Lisp applies the operator to the operands and this leads to the result of 12 in the case of (* 3 4) and 15 in the case of (+ 1 2 12).

The abstract syntax tree (AST) looks like this:

(Source: https://github.com/jiacai2050/JCScheme)

As can be seen the AST of Lisp is the same as Lisp itself. A Lisp is a language that is written in an AST, when the language is interpreted or compiled these code trees can be manipulated.

Now, let's look at the seven or so special forms of the language. The rest of Lisp can be implemented in these special forms. (Note: in reality Lisp is not implemented this way, there are some more special forms used in most Lisps for for example performance and usability reasons).

My description of these special forms (from which eval can be created) is roughly based on the article Roots of Lisp of Paul Graham where he explains what McCarthy has discovered. The original definitions of McCarthy can be found here.

In Lisp there are about seven fundamental forms:

`atom`

Everything in Lisp is either a list or an atom. (In Clojure we also have some other types like vectors, but we do not worry about that.) Atoms are for example numbers (1), strings ("hello") or symbols (add1). Lists consist of parentheses surrounding atoms separated by whitespace or other lists or both.

`quote`

Quoting an expression means that the expression does not need to be evaluated. This is similar to our use of quotes in natural language, where we use quotes to refer to the symbol itself instead to the meaning. To "say your name" I reply with "Erwin", to "say 'your name'" I reply with "your name". The quote operator can be used to blur the distinction between code and data. By quoting code you can pass it as data to another method which can then inspect and evaluate the expression. To quote you can use the operator quote or put a ' in front of the expression.

Example: '(+ 1 2 3). Which is equal to (quote (+ 1 2 3)). When this expression is evaluated the result is (+ 1 2 3) (instead of 6, the result of evaluating the unquoted (+ 1 2 3)).

`eq? a b`

(eq? a b) returns true if the values of a and b are the same atom or both the empty list (defined as '()), and false otherwise.

`cons a l`

cons stands for construct and is used to add one element a to a list l. Lisp lists are chains of cons cells, where the second element points to the first element of the next cons cell (a linked list). The last element of a list is the empty list '() (or nil).

Note that Clojure does not use the cons cell as described above to create a list. Clojure has sequences. Again, we will not worry about that now since the operations supported are similar, only the underlying implementation is different.

`car l`

(car l) returns the first item in a list. In Clojure this form is called first.

`cdr l`

(cdr l) returns everything but the first item in a list. In Clojure this form is called rest.

car comes from "Contents of the Address part of Register number" and cdr from "Contents of the Decrement part of Register number". They refer to the way cons cells were stored in early computers. This is no longer the way it is, but Lispers have decided to stay with it because you can easily combine them. E.g., the car of the cdr (the second element of a list) is the cadr. Clojure breaks this connection with these roots, but has introduced other forms to make it easier to chain methods together (e.g., ->) and has special utility methods like nth to obtain the nth element from a list.

`cond <p1> <e1> ... <pn> <en>`

A conditional (conditionals were among other things first introduced in the paper of McCarthy). If p_n is true e_n is evaluated. Also an :else clause can be introduced if no predicates are true.

So in (cond (p1 n1) (p2 n2) :else "Hi") the result is n₁ if p₁ is true, n₂ if p₁ is false, and p₂ is true, and "Hi" if both p₁ and p₂ are false.

Besides these fundamental forms you can create functions with fn. In a function you have a list of arguments and a list of expressions. The value of the arguments are assigned to the values that are passed during the function call (this becomes relevant later). A lambda expression (anonymous function) is created like this:

(fn (arg) (+ arg 1))

And you call the lambda expression as follows:

((fn (arg) (+ arg 1)) 1) ;; => 2

This can be read as "call the lambda expression with argument arg and body arg plus 1 with the argument 1 substituted for arg, resulting in (+ 1 1) which evaluates to 2".

Expressions can be bound to a symbol in Clojure with defn as follows:

(defn add1 [arg]
 (+ arg 1)))

Here add1 with argument arg is bound to (+ arg 1).

As said Clojure has more collection types than the list. Clojure also contains maps, vectors and sets.

It is interesting to note that these other data structures are implemented in Clojure in such a way that they are immutable (unchangeable). And because they share underlying structures when being copied they have very good performance characteristics. To learn some more about this you can view a video where the creator of Clojure explains this how these data structures are implemented.

One more thing you will see in the implementation below: some data structures are demarcated by different types of symbols. The vector uses the [ and ]. In Clojure the argument list to a function or definition is a vector (not a list) and looks like this:

(fn [arg] (+ arg 1) 1) ;; => 2

Since it supports the same basic operations as a list (like first, empty, or rest`) it is fine to think about it as a list.

Example of use: implementing `map`

To define functions and to evaluate them often recursion is used. Below we will define a procedure that is called map. map applies a given function f to every argument in a list (coll) and returns a list with the results. So when we do (map add1 '(1 2 3)) (using add1 defined above) the result will be (2 3 4).

(defn map [f coll]
 (if (empty? coll) '()
  (cons (f (first coll)) (map f (rest coll)))))

This is done by cdring ("resting" in the case of Clojure) down a list and consing up the result. It applies f to the first element of the list, and combines (cons) the result with f mapped to the rest of the list. The recursion breaks if the collection is empty. The evaluator has a lot of these type of recursive definitions as well so it is crucial to understand.

The seven forms above (map is not a primitive) are all the primitives you need to create a meta-circular evaluator.

Now, in an actual Lisp implementation there are a lot more primitives. The code belows also uses more forms. Most of them can be implemented in the code below, but others can't. The reason for that is that sometimes the evaluation order matters.

Another form we will see is let, which is of the form

(let [<var1> <val1> <varn> <valn>] <body>)

let can be used to bind variables (var) to value (val) (to declare variables basically). These are in scope in the body. This let can roughly be written as:

((fn [<var1> <varn>] <body>) <val1> <valn>)

Where you create a lambda where the variables are substituted for the values.

This is impossible to implement in a normal function, since the variables are always evaluated. When evaluation order needs to be controlled, a so-called macro can be created.

A macro hooks provides a convenient way to hook directly into the abstract syntax tree and change it at compile time (before the code runs). We will not be using or supporting macro's in this meta-circular evaluator, but it is definitely possible. If you want to learn more about macros you can read Paul Graham's On Lisp or Doug Hoyte's Let Over Lambda.

Now we have defined all the basic operators. Let's look at the meta-circular evaluator.

The implementation of the meta-circular evaluator

The meta-circular evaluator consists of two methods: eval and apply.

These methods call each other recursively as follows. This can be displayed as follows:

(Source: https://github.com/jiacai2050/JCScheme)

If no descriptive names are used for definitions inside eval and apply, the whole evaluator fits on one page. As shown in the Lisp manual (p. 13) it fits on one page, and as demonstrated in SICP lecture 7A: on a blackboard. In the current implementation a bit more descriptive names are used since it helps for clarity. And also a somewhat sophisticated environment structure is used to store procedures. Therefore this code does not fit on one page, but it is definitely still a short program.

Unfortunately the implementation described below is possibly not truly a meta-circular evaluator. A defining characteristic of a meta-circular evaluator is that it is written in the language it evaluates. This evaluator is written in Clojure, but it cannot evaluate everything of Clojure. What it can evaluate looks a lot like Scheme translated to Clojure, because in fact it is. But... Since Clojure is a Lisp and Scheme is a Lisp, I think the name meta-circular evaluator is still warranted.

Now we will implement eval and apply.

Eval

eval is a procedure that takes two arguments: an expression and an environment. And like every interesting procedure, it is a case analysis:

(defn eval [exp env]
      (cond (self-evaluating? exp) exp
            (variable? exp) (lookup-variable-value exp env)
            (quoted? exp) (text-of-quotation exp)
            (assignment? exp) (eval-assignment exp env)
            (definition? exp) (eval-definition exp env)
            (if? exp) (eval-if exp env)
            (lambda? exp) (make-procedure (lambda-parameters exp)
                                          (lambda-body exp)
                                          env)
            (begin? exp) (eval-sequence (begin-actions exp) env)
            (cond? exp) (eval (cond->if exp) env)
            (application? exp) (apply (eval (operator exp) env)
                                      (list-of-values (operands exp) env))
            :else (error "Unknown expression type -- EVAL" exp)))

When we call eval with a quoted expression to evaluate (e.g., (eval 'definition 'env) in an environment, it will look at what type of expression we are dealing with, and then dispatch to another procedure to handle it.

I'll first go over it just describing eval in words, and next dive deeper into the routines. When an expression exp is self-evaluating (like a string ["hello"], a number [1] or a boolean [true]) eval just gets the expression back. If exp is a variable (a), we look up the value of the variable in the environment (we will discuss how this works later). If exp is a definition we want to define the value in the environment. (If we define (defn a "hello") we want "a" stored in the environment to be "hello", and when we look up the variable a later we want to have "hello" returned.) If exp is an assignment (set! a "hi"), we will look up a in the environment, and set it to a new value. If exp is a lambda expression (anonymous function) we create a procedure of this lambda expression (we create a list tagged with the flag 'procedure), where we store the parameters and the body of the procedure alongside with the current environment captured in it (see make-procedure below). Next, we have an if-statement and conditional: a conditional is transformed to a list of if-statements via a routine named cond->if. In an if-statement: if the predicate is true the first clause is evaluated, and if false the second clause is evaluated. For example, (if p 'n1 'n2) leads to n1 if p is true and to n2 otherwise.

The above is all reasonably straightforward. It is a way to implement all the reserved words. As an aside, Gerald Jay Sussman states:

The number of reserved words that should exist in a language should be no more than a person can remember on its fingers and toes. I get very upset with languages which have hundreds or reserved words. (Source)

If the expression does not start with a reserved word, the expression is an application. This means we apply the result of evaluating the operator of the expression in the environment, to the list of values that follow, each evaluated in the environment.

First we look at list-of-values, which is implemented as follows:

(defn list-of-values [exps env]
  (if (no-operands? exps)
    '()
    (cons (eval (first-operand exps) env)
          (list-of-values (rest-operands exps) env))))

This is a recursive definition (like map above) where we construct a new list with as first argument the first operand evaluated in the environment, and as the rest, the rest of the operands evaluated in the environment. When we come to the empty list '() (the second element of the last cons cell in a linked list) the recursion breaks and we return the empty list '().

We will discuss the implementation of apply after we have looked at the inner definitions of eval a little bit closer. These consist of routines to detect expression, routines to get information out of expressions, and routines to transform expressions.

Routines to detect expressions.

First a little heads up: all definitions used above to get information out of expressions (like first-operand and rest-operands) are all implemented with cars and cdrs (first and rest in Clojure). E.g.,

(defn first-operand [args] (first args))
(defn rest-operands [args] (rest args))
(defn no-operands? [args] (empty? args))

Detecting what type of an expression we are dealing with, is done by comparing the first element of the list via tagged-list:

(defn tagged-list? [exp tag]
  (and (list? exp) (= (first exp) tag)))

This checks if the expression is a list, and (using the code inspecting facilities of Lisp) if the first expression is equal to the tag.

For example, to see if a variable is a definition or quoted we check if the first element is respectively the symbol 'quote or 'defn:

(defn quoted? [exp] (tagged-list? exp 'quote))
(defn definition? [exp] (tagged-list? exp 'defn))

And in a similar manner:

(defn lambda? [exp] (tagged-list? exp 'fn))
(defn assignment? [exp] (tagged-list? exp 'set!))
(defn begin? [exp] (tagged-list? exp 'do))
(defn true? [x] (not (= x false)))
(defn false? [x] (= x false))
(defn cond? [exp] (tagged-list? exp 'cond))
(defn compound-procedure? [p] (tagged-list? p 'procedure))
(...)

Note that if a definition contains a boolean result (true or false), by convention we end the definition with a question mark ?.

This is how routines to detect expressions are implemented.

Routines that get information out of expressions

Beside routines to detect expressions we have some procedures to get information out of expressions. We know the syntax of our language and we know at which position certain elements should be located. First we define some helper methods to get certain elements out of a list besides the rest and the first, so for example the second is:

(defn second [exp] (first (rest exp))) ; cadr
(defn rest-after-second [exp] (rest (rest exp))) ; cddr
(...)

These routines are implemented in Clojure as follows:

(defn operator [exp] (first exp))
(defn operands [exp] (rest exp))
(defn text-of-quotation [exp] (second exp))
(defn assignment-variable [exp] (second exp))
(defn lambda-parameters [exp] (second exp))
(defn lambda-body [exp] (rest-after-second exp))
(...)

Here we see that for example lamba-body is the rest-after-second. Which is indeed the case. If we look at:

(fn [a b c d] (+ a b (* c d)))

We see that everything after the second element (a b c d) is the body.

All the other implementations of routines that get information out of expressions are defined in the same manner.

Routines to manipulate expressions

As seen in the definition of eval above, when an expression is a lambda expression (as identified by lambda?), we make a procedure. This is implemented as follows:

(defn make-procedure [parameters body env]
  (list 'procedure parameters body env))

So we create a new list via the procedure list (which is a routine to create a linked list via a chain of cons'es). Note that the procedure creates a new list where the environment is stored in. This is a way to create something called a closure: the environment at the moment the procedure was called is captured. Next when we look at apply we will see how this environment is retrieved. Furthermore, we will discuss the difference between two types of scope (dynamic and lexical) and their implementation later.

In a similar manner as make-procedure we can create a new type of expression from an if-expresion in cond->if.

Note: using -> is a convention to signify the routine is a conversion.

Via cond->if a cond expressions (conditional) can be converted using an internal procedure named expand-clauses. expand-clauses recursively expands every clause of a cond expression to nested if-statements.

So eval evaluated an expression by detecting the type of expression, getting the information out of the expression and possibly converting it to something else, If it does not start with one of the reserved words it is treated as an application. This will be handled with apply in the next segment.

Apply

We have arrived at apply. apply is called in eval if the expression is not one of the reserved words (like if, cond, fn, defn and set!). An expression is an application if it is not one of the special forms, but still a list (e.g., in the case of (+ 1 2), + is not in the list of the reserved words). Then the expression will be handled by this part of eval (see above):

(...)
(apply (eval (operator exp) env)
       (list-of-values (operands exp) env)
(...)

We already looked at list-of-values that evaluated all the operands in an environment. What remains is applying the operator to the arguments. apply then feeds back the expression and the environment to eval, et cetera.

apply is a procedure that takes as input a procedure and its arguments, and produces an expression and an environment:

(defn apply [procedure arguments]
  (cond (primitive-procedure? procedure)
        (apply-primitive-procedure procedure arguments)
        (compound-procedure? procedure)
        (eval-sequence
         (procedure-body procedure)
         (extend-environment (procedure-parameters procedure)    ; variables
                             arguments                           ; values
                             (procedure-environment procedure))) ; base environment
        :else (error "Unknown procedure type -- APPLY" procedure)))

Again it is a case analysis. Let's go through it. We first check if a procedure is a primitive procedure with primitive-procedure?. A procedure is a primitive procedure if it is looked up in the environment and it was in the list of primitive procedures defined in the environment. The primitive procedures are defined as follows, and are later added to the environment (how the environment is constructed will be discussed later).

(def primitive-procedures
  (list (list 'first first)
        (list 'rest rest)
        (list 'cons cons)
        (list 'nil? nil?)
        (list 'empty empty)
        (list 'list list)
        (list '+ +)
        (list '- -)
        (list '* *)
        (list '/ /)
        ;;      more primitives
        ))

If the procedure is a primitive procedure we apply the primitive procedure with the underlying apply of Clojure. At the beginning of the evaluator we store the underlying apply to not confuse it with the newly defined apply:

(defn apply-from-underlying-lisp [procedure arguments] (apply procedure arguments))

How this apply that is built into Clojure works, is outside of the scope of this post (to be honest I don't know). Interesting to note is that these types of primitives are not essential for computation. You can do everything with lambda and something called a Y Combinator. But using that is not as convenient.

The application of a primitive procedure is defined as follows:

(defn apply-primitive-procedure [proc args]
  (apply-from-underlying-lisp
   (primitive-implementation proc) args))

Where primitive-implementation is a method to retrieve the procedure:

(defn primitive-implementation [proc] (second proc))

As can be seen above where the primitives-procedures are defined the primitive-implementation is a symbol like first or +. Again, these are evaluated by the underlying Clojure implementation.

If we are dealing with a compound procedure (a list tagged with procedure), we evaluate the sequence of elements with the routine eval-sequence:

(defn eval-sequence [exps env]
  (if (last-exp? exps)
    (eval (first-exp exps) env)
    (do
      (eval (first-exp exps) env)
      (eval-sequence (rest-exps exps) env))))

Note that do is used to evaluate multiple expressions (in eval it is detected by begin?, a similar construct in Scheme).

eval-sequence is a recursive definition where we, if we have only one expression, only evaluate this one expression in the environment, and if we have more than one expression, we evaluate the first expression and eval-sequence the rest of the expressions in the environment.

Here last-exp, first-exp and rest-exp are defined as follows:

(defn last-exp? [seq] (empty? (rest seq)))
(defn first-exp [seq] (first seq))
(defn rest-exps [seq] (rest seq))

Naturally the first expression is the first element of the sequence (of expressions). The rest of the expressions is the rest of the sequence. And we are dealing with the last expression if the rest of the sequence after it is empty.

Important to note is that when we use eval-sequence in the definition of apply we extend the environment of the evaluation with the procedure parameters, arguments, and the procedure environment itself. This way the environment of the procedure (which was stored when it was created) are available during evaluation.

So what exactly is this extending of an environment? And in what way are symbols added to it?

The environment

We now go step by step through the environment.

An environment consists of frames stacked together in a list. A frame is empty or consists of two lists: one lists with variables, and one list with values. When a definition is looked up we look at the first frame to see if the variable is defined there. If we find it we return the value. If it is not, we look one frame further to see if it is defined there. Until we finally get to the global environment. If it is not found there we have an error ("Unbound variable"). So the goal of an environment is to store variables and the values associated with them.

A simple environment structure looks like this:

Source: https://mitpress.mit.edu/sicp/full-text/book/book-Z-H-21.html#%sec3.2.

Here we see multiple frames (I, II and III) that are connected. A will find that the value of the variable x is 7, but C will find 3.

The environment is crucial to the evaluation process, because it determines the context in which an expression will be evaluated. Each time a procedure is applied, a new frame is created.

It turns out that my assumed immutability (I thought nothing could be changed, which was not completely true from a user's perspective) of Clojure made the implementation quite difficult (for me). Clojure has, unlike Scheme (the language where this evaluator was translated from), no mutable (changeable) cons cells (no actual cons cell at all as described earlier). Where in Scheme you can change the first element or last element of a list respectively with the routines set-car! and set-cdr!, this is not possible in Clojure.

We can however accomplish mutation by defining a so-called Atom (different from the basic building element atom defined above). An Atom in Clojure is a mutable value, that can be mutated in an atomic way, free of concurrency problems (see this chapter on Metaphysics in Clojure for the Brave and True for an explanation).

The algorithm used in the Scheme version checks the first frame, loops over it (with a method called scan) and if it finds the variable, it returns the value associated with it. If it does not find anything it will loop over the enclosing environment and scans there.

When redefining a variable in the environment, a similar process is used. When the variable is found however, set-car! is used to overwrite the variable value in that frame.

After using some environment structures with bugs in them (for example being unable to define recursive definitions) I found a better environment definition in the SICP answers of Jake McCrary. The implementation has the same behavior as that of the original of the SICP.

We now go step by step through this environment structure by Jake McCrary.

The empty environment is an empty list.

(def the-empty-environment '())

(Note that def is similar to defn without arguments.)

To get the enclosing environment (frame) of the current environment (frame), we look at the rest from the frame we are in now.

(defn enclosing-environment [env] (rest env))
(defn first-frame [env] (first env))

To make a frame, we use a routine zipmap to add two lists of variables together which "returns a map with the keys mapped to the corresponding vals" (ClojureDocs). A map is a list of key-value pairs.

(defn make-frame [variables values]
  (atom (zipmap variables values)))
(defn frame-variables [frame] (keys @frame))
(defn frame-values [frame] (vals @frame))

Let's illustrate the behavior of this creating of frames with zipmap.

We have a list of variables to be added to the environment:

(def variables (list '+
                     'rest
                     'add1))

A list of corresponding values (the primitive + and rest operator, and a lambda expression and a created procedure to be added to the environment:

(def values (list +
                  rest
                  '(fn (x) (+ x 1))
                  '(procedure (x) (fn (x) (+ x 1)) {:+ + :x 2 (...)})))

When we zipmap them:

(zipmap variables values)

This leads to the following result:

{+ #object[clojure.core$PLUS (...)],
rest #object[clojure.core$rest_ (...))],
add1 (procedure (x) (fn (x) (+ x 1)) {:+ +, :x 2, (...)})}

So a frame is a map (key value pairs, differentiated with { and } in Clojure) with variables (keys of the map) bound to values (values of the map) (in this case the primitive objects from the underlying Clojure).

Next we get methods to add bindings to frames:

(defn add-binding-to-frame! [var val frame]
  (swap! frame assoc var val))

Note that by convention routines that mutate state are ended with an exclamation mark !.

add-binding-to-frame! uses swap! in combination with assoc to associate a variable and new value to a frame. This is one of the ways to have (safe) mutable state.

Next we need a way to extend our environment (this is the method used in eval and eval-sequence to add variables and values to a frame, for example when storing the variables of a procedure in the environment where it will be evaluated in, or when new definitions are added):

(defn extend-environment [vars vals base-env]
  (if (= (count vars) (count vals))
    (cons (make-frame vars vals) base-env)
    (if (< (count vars) (count vals))
      (error "Too many arguments supplied " vars)
      (error "Too few arguments supplied " vars))))

extend-environment takes variables and values and a base environment. It then makes a frame of the variables and values and conses that up to the base environment. The rest is error handling for when the amount of variables and values applied are not the same (e.g., when evaluating ((fn (x) (+ x 1)) 2 3) it will returns "Too many arguments supplied").

Next, we have a method to lookup a variable in an environment. This one looks a bit more complicated because of the inner methods.

Know that letfn is just like let, only now we define functions and bind them to a name only available in the scope. Furthermore, contains? checks if a variable is available in a map and returns true if it does, false it it does not.

(defn lookup-variable-value [variable env]
  (letfn [(env-loop [env]
            (letfn [(scan [frame]
                      (if (contains? frame variable)
                        (let [value (get frame variable)]
                          value)
                        (env-loop (enclosing-environment env))))]
              (if (= env the-empty-environment)
                (error "Unbound variable -- LOOKUP" variable)
                (let [frame (first-frame env)]
                  (scan @frame)))))]
    (env-loop env)))

What happens at the bottom of lookup-variable-value is that we start an env-loop with the env. env-loop defines a method scan to look for a variable in a frame.

If the frame contains the variable we get (get is a way to retrieve the value belonging to a key) the value belonging to the variable from the frame. Otherwise, we start a new env-loop over the enclosing environment (the next frame or frames).

When the env-loop is started we first check if we are dealing with the empty environment. If that is the case we break the recursion: we have scanned all frames in scope and we are therefore dealing with an unbound variable. If we are not dealing with the empty environment, we are getting the first frame from the (remaining) environment (remember that the environment is a list of frames), get the value from the atom by using @, and scan that.

So now we can understand how to lookup a variable, we also need a way to define a variable value in the environment. To accomplish this we have a similar procedure, which uses a similar scanning method as above.

(defn set-variable-value! [variable value env]
  (letfn [(env-loop [env]
            (letfn [(scan [frame]
                      (if (contains? @frame variable)
                        (swap! frame assoc variable value)
                        (env-loop (enclosing-environment env))))]
              (if (= env the-empty-environment)
                (error "Unbound variable -- SET!" variable)
                (scan (first-frame env)))))]
    (env-loop env)))

The difference with lookup-variable-value is that in set-variable-value! we are trying to find the variable, and in the first frame we find which contains the variable we change this variable's value for the current value. If the frame doesn't contain it we start the env-loop over the enclosing environment. The recursion breaks when we end up with the empty environment (unbound variable) or when we find a variable and have set it.

Furthermore,to define a variable we get the first frame of the environment and add the variable as a key with the value as a value. (So we add it to the top-level frame, the global environment.)

(defn define-variable! [variable value env]
  (swap! (first-frame env) assoc variable value))

Finally, at the beginning we setup the environment:

(defn setup-environment []
  (let [initial-env
        (extend-environment primitive-procedure-names
                            primitive-procedure-objects
                            the-empty-environment)]
    initial-env))

We load the primitive procedures in the environment and we set the-empty-environment as our base environment.

Now you've seen everything but a way to interact with this evaluator.

Interacting with the evaluator

Last but not least we have a way to interact with the evaluator. When we run the code a driver-loop asks for input, and when it has received input it will evaluate the input in our environment:

(def input-prompt ";;; Eval input:")
(def output-prompt ";;; Eval value:")

(defn driver-loop []
  (prompt-for-input input-prompt)
  (let [input (read)]
    (user-print input)
    (let [output (eval input the-global-environment)]
      (announce-output output-prompt)
      (user-print output)))
  (driver-loop))

(defn prompt-for-input [string]
  (newline) (println string) (newline))

(defn announce-output [string]
  (newline) (println string) (newline))

(defn user-print [object]
  (if (compound-procedure? object)
    (println (list 'compound-procedure
                   (procedure-parameters object)
                   (procedure-body object)
                   '<procedure-env>))
    (println object)))

'METACIRCULAR-EVALUATOR-LOADED

(def the-global-environment (setup-environment))

(driver-loop)

That's it. Works like a charm.

Example interaction

The goal of this thing was to be able to feed the evaluator program a Lisp expression, and have it return the result. So when we feed it (eval '(defn add1 (fn (x) (+ x 1))) <e>) it should know what this means and add the symbol add1 to the environment.

Here we see some interactions with the evaluator:

So "Hello" evaluates to itself since it is self-evaluating. + is found in the environment and termined to be a primitive operation from the underlying Clojure. And we can also define functions like map and add1 and apply these to arguments.

If you would like to see a full evaluation of an expression by hand, I can recommend watching the second part of Lecture 7A of SICP. You can also run the code of the evaluator and put some break points or printlns at strategic points.

Dynamically scoped environment

In my journey to create a working version I also stumbled upon this implementation of Mathieu Gauthron.

A dynamically scoped environment is a bit simpler to implement, what is defined latest is found first (instead of passing an environment along in a certain procedure scope, we just update the environment). It is a bit harder to reason about though, because in a dynamically scoped environment it is only at runtime defined what is in scope.

After we define what lexical and dynamic scope is, below will follow an experiment with the dynamic evaluator.

Lexical scoping versus dynamic scoping

Lexical scope means that the structure of the program determines which variables are in scope (for example in C-like languages the scope is defined by curly braces { and }. In dynamic scope the scope is defined at runtime, that what was defined latest when the program is evaluated at runtime is found first. Currently there are almost no languages with dynamic scope. Lexical scope is more logical to reason about. Nevertheless there are valid use cases for dynamic scope. A nice example is exception handling (throwing an exception and catching it):

Exception handling in most languages utilizes dynamic scoping; when an exception occurs control will be transferred back to the closest handler on the (dynamic) activation stack.

One way to test if a language is dynamic or static is by the following idea:

First you define a method that uses a variable scope in its body:

(defn test (fn [] scope))

When you call this method:

(test)

this returns

Is not a valid symbol – LOOKUP-VARIABLE-VALUE scope

This is because scope is not available in the environment. It is not defined.

Next when you introduce a new binding with the same name (scope) wherein you call test:

((fn [scope] (test)) (quote dynamic))

A lexically scoped language will return "Is not a valid symbol – LOOKUP-VARIABLE-VALUE scope" as above, because the environment when the definition of test was created does not contain this binding. A dynamically scoped language will return 'dynamic. It does not care about what the value of scope was when test was defined and just looks up the nearest definition. In other words: the lexical environment was not captured.

(The idea for the test was obtained from this Stack Overflow answer, but slightly modified since our language does not know how to evaluate let.

As expected from the above description, the evaluator with the dynamic environment has the following behavior:

Closing remarks

What we have seen above is some nice copy, paste and explaining. For this I was inspired by SICP, and the evaluators of Greg Sexton, Jake McCrary and Mathieu Gauthron.

What we have seen above is an elegant model of computation. Using about seven primitives we can define eval in which we can implement our language.

The idea that an evaluator is just another program (and such a short one) is remarkable. Once you understand how the meta-circular evaluator works, it becomes possible to change the implementation and create interpreters for other languages.

Indeed, new languages are often invented by first writing an evaluator that embeds the new language within an existing high-level language.

(Source)

The fact that Lisp code trees can be changed at runtime or compile time makes it so that it is possible to deeply influence the language. Where in languages like Java you have to wait for a committee to put a new feature into a language, in Lisp you don't have to wait and can implement these features yourself.

One quote (incorrectly stated according to the supposed author but nevertheless instructive) describes the difference between languages without the ability to modify its structure (like Java or APL) and Lisp as follows:

APL is like a beautiful diamond - flawless, beautifully symmetrical. But you can't add anything to it. If you try to glue on another diamond, you don't get a bigger diamond. Lisp is like a ball of mud. Add more and it's still a ball of mud - it still looks like Lisp.

One nice example of the ball of mud idea is the go-routine that is added to Clojure for asynchronous programming (communicating sequential processes). Go also has them, but for Go is it a fundamental language feature. Because the language can be fundamentally and syntactically changed, it could be introduced as a library in Clojure. (See here for more information.)

In SICP among other things the evaluator is extended and modified to become an evaluator for:

And the same idea is used to create an evaluator for machine language and to create a compiler that converts Scheme to a machine language. In SICP it is also explained how the mechanisms works by which procedures call other procedures and return values to their callers (which the meta-circular evaluator borrows from the underlying Lisp).

One of the other things that is missing in the above implementation is a way to define macros, which are used for defining syntax language extensions. I am not so well-versed in them yet. I might play with them in a later post.

I did learn a great deal from the above implementation. I hope it is useful for others as well.

The source code of the evaluators can be found at: