The LilyPond Report #27

Greetings everybody, and welcome to this twenty-seventh issue of the LilyPond Report!

This issue raises some important questions as well as completely irrelevant shallow factoids. That’s how we roll, just bear with us. On the plus side, you will learn quite a lot about Scheme integration in LilyPond and some brand-new niceties that David Kastrup has recently added to our feature set. Also featured in this installment: how to take black-and-white pictures without intending to, what is the silliest possible name for a software project, and what future is there for fancy night clubs. (Yes, we know — but don’t say we did’nt warn you.)
As always, you can post your comments at the bottom of the page, or send your contributions to the LilyPond Report’s next issues.

 Editorial

by Valentin Villenave.

Here’s a piece of news for you: LilyPond is no more. It has filed for bankruptcy and disappeared from the map, it’s been shut down, it has ceased to be, it’s expired and gone to meet its maker!

Now, I’m evidently not talking about the GNU LilyPond free software we use and love. What I’m referring to here, is the LilyPond night club in East Hampton, one of the fanciest and most expensive places in one of the fanciest and most expensive locations in the world: Lily Pond Lane. I’ve been trying for the past several years to make a shallow and not-funny joke about it, vainly searching for some link between an outrageously indecent "1% amongst the 1%" place, and our own small community of dedicated volunteers... Only to learn that it had gone bankrupt and been closed — or rather, merely changed its name. (That sarcastic tone? That’s just me refraining from weeping any tear.)

Anyhow. Are we in any danger of closing down and filing for bankruptcy? Sometimes I do wonder. On one hand, some things are looking pretty good this year: we’ve been awarded with a gizmo and we’ve been having our very first Google Summer of Code sponsorship (go Janek, we’re counting on you!). On the other hand, David Kastrup’s request for funding, although moderately successful at first, isn’t doing so well these days. You’ll find his May-June report below, and the next one is not on its way to look any less bleak. Now, there’s probably a lesson to be inferred here, but I’ll let you fill in the blanks!

 Release news

by Valentin Villenave.

As this Report went online, a new development version (2.15.42) has just been released. You may want to give it a go for testing purposes; please note, however, that due to some critical bugs remaining, this is not the next "Release Candidate" for our upcoming next stable version. For normal users, we still officially recommend the current stable version 2.14. (Although, if you ask me, the development branch brings just too much awesomeness to be overlooked. More below.)

 Investors’ report

by David Kastrup.

First I have to apologize for delaying the report for May until it made no more sense to release it on its own. So here are the numbers for May and June, in sequence. The original request for funding made in February in the LilyPond Report 24 still contains all the relevant details regarding my financial situation and the need for funding while I am working exclusively on LilyPond.

I have not separated one-time payments and regular payments this time: they average out in the whole, and where regular payments come into disagreement with the intended plan, keeping track becomes really complex, and running a reminder system would be even more complex, and of course somewhat pointless when we are talking about a voluntary commitment.

As a rough estimate, probably about €400 per month can by now be attributed to regular or semi-regular payments.

Regarding the report for April, there are €100 missing that I had counted as half of a two-month installment when it really has been a one-month payment. Special thanks for that! So April, with a total of €821 also managed to surpass the prospected bare minimum of €800 per month required to keep my financial situation in limbo.

I count payments in US$ as €0,80 here, and other currencies as the result in Euro after Paypal’s conversion and fee deduction.

May

Fixed payments (€)	Variable plans (€)
250 200 160 100 72 37 6×25 24 2×20 15 3×10	25 + 50 (target €1200)
Total: 1078	Total: 75
Totals: €1153

Until May 20th, there were about €200 in donations. I used the opportunity when posting a link to the video of a LilyPond-related talk I gave, to point out that things were not really looking well, and in hardly more than a week, the totals surpassed that of any month so far.

June

Fixed payments (€)	Variable plans (€)
160 100 50 2×30 5×25 20 3×10	25 + 50 (target €1200)
Total: 545	Total: 75
Totals: €620

In contrast, June has dropped to a record low, most likely due to the delayed report for May. Averaging both months still put me in the position to pay rent, health insurance and food till the end of June. June has really shown itself a month of the faithful (to the degree where some payments likely intended for July already arrived). As one positive result, about two thirds of the payment arrived via SEPA bank transfers, quite the most cost-effective option within the Euro zone (banks should not bill more than for in-country transfers).

Reasons for delayed report

What is the reason that this report has been delayed so much? Mostly, I am not happy with the current development situation and how I am contributing to it, to the degree where I am not sure I am giving a good return of the considerable investment LilyPond users take in me. That’s one reason I felt not really justified asking for more. And while the large rally at the month of May has on the one hand been elating and inspiring, it also meant responsibility to live up to the expectations.

Recently, Graham made an anonymous survey of formerly more active LilyPond developers, asking for the reasons of their declined participation. One answer in particular was worrying to me.

I am citing one paragraph:

The last "teamwork" problem is the amount of "monetization" going on. I didn’t join lilypond to help other people make money off it. I don’t have any good suggestions for this, but the question asked if I was contributing less and this is one reason. Having some people making money and others not makes it feel like less of a "team".

Now it is not that much that I am ``making money off LilyPond’’: at the end of the month everything is gone without reserve so far. It is more like I am making time for LilyPond development from the monetary support of LilyPond users and developers: I can afford the luxury to work on a project I like. Being independently wealthy would be even nicer, but I have to work with what I’ve got.

I am quite grateful that users of LilyPond have been willing to pitch in and let me continue. But what this buys them is just a single active developer, and if that makes other developers stop contributing, it is not helping.

Another thing in this feedback about reduced involvement points out a deterioration in the style of discussions on the LilyPond mailing lists. And it is not hard to make a connection to my lack of decent communication skills.

LilyPond developer and user meeting in August

So I am trying to address this with a meeting of LilyPond developers and users at August 24th to 28th (check out the detailed announcement here) at my place. I have made the experience that it is easier to behave civilized to people one has actually met already instead of knowing them just as letters on the screen. It also makes it easier to understand what people are trying to say, even when they are not having their best day. With a `critical mass’ of core people reasonably understanding one another, maybe some more or less frequently occuring problems might become insignificant.

Of course, another purpose of the meeting is better distributing specialized knowledge, and make more people able to deal with the programming languages and subsystems of LilyPond, and also figure out where the stumbling blocks are.

So maybe this meeting, where a number of core developers will be turning up, will be able to improve the atmosphere, productivity, and feeling of belonging to a really worthwhile community again.

Lilypond and Scheme

Some of the recent Email discussion (hopefully to be considered civilized) were centered around the use of Scheme as LilyPond’s extension language. My personal take on this is that embedding Scheme at the syntactical LilyPond layer (namely in *.ly files) is working out well enough. LilyPond is more a language of declarations and expressions rather than actions, and a functional language like Scheme fits nicely with that. The current interface using #, $, #{ … #} as a way of switching the language between expressions is actually rather straightforward. Interfacing an imperative language, where the basic unit of execution is not an expression, would likely be harder. The end of June saw the additions of #@ and $@ operators (see below).

As a general purpose programming language (namely what we see in *.scm files), the dearth of visually recognizable subunits is the most common complaint about Scheme. As long as good user interfaces manage to keep the complexity of the required user-level programming at bay, I don’t consider this a show-stopper. LilyPond has quite a lot of potential and need for further improvement here, even though version 2.16 will offer a lot over 2.14 in that department already.

With version 2.15.41, the eighth release candidate for 2.16, we are again on the finishing line for stable releases, and hopefully it will not be yanked from our feet by newly discovered regressions again. There are changes in the release policies under way, so perhaps 2.18 will not take as long as 2.16 did. While 2.14 took even longer, there was less of a feeling of urgency about it.

What happened in the code?

Apart from talk, what kind of work on LilyPond did I do in May? I’ve made some progress with trying to reinvent some central data structures in the framework of GOOPS, GUILE’s Object Oriented Programming System, but this project is moving forward slowly. I am still getting the hang of how to create a design that falls into place naturally and is easy to use. Basically, this is still issue 2507.

Apart from several bugfixes and some slight syntax changes, most noteworthy are probably the work on tweaks allowing for something like

<\tweak Accidental #'color #red   cis4
\tweak Accidental #'color #green es
                                 g>

(previously, only the notehead would have been susceptible to a tweak), completing the work on event classes meaning that string indications like

c\5

don’t need to be part of a chord to be displayed, and a redesign of the \footnote interface. The latter took several iterations before it actually became reasonably boring (that is, consistent and versatile), and an overhaul of its documentation has not yet been completed since I ran out of steam and decided doing something else before getting deadlocked. There would be more potential for improvement of the \footnote interface, but it would require rather tricky parser work concerning optional arguments. With all the iterations on the footnote interface and the documentation, this project took up more time and energy than planned for, and unfortunately not just from me. Indeed, there are still more fixes to the footnote code that have spilled into even July.

In June, the focus has been mostly on bug fixes (some critical), but one highlight in my opinion was the addition of `splicing operators’, mainly $@. Basically, this means that

$@(list a b c)

is the same as

$a $b $c

If you try

git grep ",@" scm

in the LilyPond code repository, you’ll see that this kind of thing is pretty useful in the world of Scheme quoting, so it’s not amiss to have it as a tool in Scheme LilyPonding.

It allows to write

{ $@xxx }

instead of

$(make-sequential-music xxx)

so again, it is a measure to get work done when one is more comfortable using LilyPond constructs rather than their Scheme counterparts. (See below for a detailed article on this matter.)

Here is a somewhat extravagant example what one can do with the splicing operators, inserting pitch lists into a chord, and putting durations and articulations (in this case, slurs) after it, and splicing the whole list of chords into the surrounding music.

The output also demonstrates that our algorithms for typesetting ties have potential for improvement.

{
 $@(map!
     (lambda (p d) #{ < $@p >$@d #})
; this $@ produces elements for a sequential music list via map!.  Each
; element is constructed from p, a list of pitches making up a chord,
; and from d, which is a list first containing a duration followed by
; _optional_ articulations, so $@d actually can return several tokens of
; _different_ type.
;
; The following form constructs the list of pitch lists for use in p
     (map!
      (lambda (i)
        (map!
         (lambda (i)
           (ly:make-pitch 2 (+ 1 (* 9 (quotient i 5)) (* i -2)) 0))
         (iota 5 i)))
      (append! (iota 34) (list 32)))
; The following form constructs an (end-less) list of lists containing
; a duration and maybe articulations.
     (apply
      circular-list
      (map!
        (lambda (m) (cons (ly:music-property m 'duration)
                          (ly:music-property m 'articulations)))
        (extract-typed-music
           #{ s4.~ s~ s4~ s2~ s2 #}
           'skip-event))))
 <c' e' g' a' c''>1 \bar "|."
}

 Many meetings

by Valentin Villenave.

If you haven’t heard, there’s this big unprecedented International LilyPond meeting taking place somewhere in Germany later this month, thanks to David. Lots of exciting talks and interesting people — which, unfortunately, doesn’t include your truly (because of an administrative as well as financial hiccup). However, living in Paris still allows me to meet with a lot of LilyPond contributors: Rune Zedeler in 2007, Paco Vila in 2009, John Mandereau whenever he’s not in Italy, Graham "Grumpy" Percival every now and then, Mike Solomon every other day...

This should be in color

if I knew how to use a camera.

And, last but not least, Joe Neeman just this afternoon. For those of us who don’t know him yet, Joe is one of our most active long-time developer; although he doesn’t spend much time goofing around with us mere mortals in silly jokes on the lists, his contributions are always of a very high quality.

Meeting with him was just plain cool (he’s actually much younger than I expected, yet another of these clues that tell me how my own life is a failure); Joe is a student in statistics and plays the violin, not at all the hardcore programmer I’d have expected judging from his code. He doesn’t seem bothered at all with LilyPond business, and even mentioned he had "some projects" in regard to future development.

What does he do for fun? Well, he develops a logic-based video game with his girlfriend, written in Python and remotely inspired by a HoM&M mini-game. Isn’t life awesome when you’re a geek?

 A Kind of Magic

by David Kastrup.

Exposition
In a recent mailing list discussion, I came up with the following piece of code working with recent development versions of LilyPond (2.15.41 or later):

Now apart from having a reasonably nice effect (pattern takes two lists of symbols, the first specifying the input order of its arguments, and the second specifying the desired output order), the code is both short as well as sophisticated. We will now first analyze the logic and elements of the code, and then take a look at how LilyPond manages to make it work.

The storyline

• Scheme macros

are a variation of Scheme functions that changes execution order. When a Scheme function is called, first its arguments get evaluated, and then the function is executed with the evaluated arguments, and its return value is the result of the function call.

A macro changes the order: the macro is called first on the unevaluated arguments (in the case of the call of pattern, the first argument is a list with the symbols A, B, C and D). The result of the macro call is then evaluated in the Scheme evaluator, and the result of this evaluation after executing the macro becomes the result of the macro call.

Since unevaluated Scheme code usually consists of nested lists, it is often convenient to specify the macro body in the form of mostly quoted lists (quotes keep the list from premature evaluation), but with a few variable elements spliced in.

• Quasiquotes

are the tool of choice for working with mostly constant lists with a few variable elements, and are seen in connection with most macros. The whole quasiquote is introduced with the backward quote character "` instead of the normal quote character "'". They are `quasiquotes’ because inside of such a quoted expression, you can `unquote’ parts by preceding them with a comma ",", and you can `unquote-splice’ a list expression into a surrounding list by preceding them with comma-at ",@".

So, for example,

`((+ 1 2) ,(+ 1 2) ,@'(+ 1 2))

evaluates to the list

((+ 1 2) 3 + 1 2)

Namely the starting list (+ 1 2) containing the symbol "+ and the numbers 1 and 2 is retained unchanged. The same construct with an unquote before it is evaluated to 3 before being put in the list, and the quoted list '(+ 1 2) evaluates by removing the quote, to (+ 1 2) which is then spliced into the surrounding list, effectively removing one level of parentheses.

• The call to pattern

Now the call to pattern looks like

(pattern (A B C D) (A B D A C D))

and, considering that macro arguments are not evaluated, the result of substituting the arguments is simply

`(define-music-function (parser location ,@'(A B C D))
  ,(make-list (length '(A B C D)) 'ly:music?)
  #{ $@(list ,@'(A B D A C D)) #})

(we have inserted quotes appropriately to indicate that the lists are not to be executed, so that one can feed this into a Scheme sandbox inside of LilyPond) and if we now execute the quasiquote, we are left with the following expression when finishing executing the macro:

(define-music-function (parser location A B C D)
  (ly:music? ly:music? ly:music? ly:music?)
  #{ $@(list A B D A C D) #})

This is the definition of a music function doing the required transformation of four music arguments to the requested order.

• LilyPond’s "$@" operator

is a variation on the Scheme splicing operator and has been introduced in LilyPond 2.15.41: it can be used for splicing a list of expressions into the surrounding #{ … #} construct, and so

#{ $@(list A B D A C D) #}

is essentially the same as

#{ $A $B $D $A $C $D #}

To achieve the same effect in older LilyPond versions, we would have had to write

(make-sequential-music (map ly:music-deep-copy (list ,@result)))

instead of

#{ $@(list ,@result) #}

since one effect of "$" is to create a copy. Whenever a music expression may be used more than once, we need to copy it since many functions processing music change their input while processing it. So the new splicing operator saves us from knowing make-sequential-music and some other things.

Behind the scenes

• Why does this even work?

This question is indeed puzzling since macro expansion is a complex beast, and #{ … #}, embedded LilyPond, is actually a piece of LilyPond code that is stored away in a string and later executed in a copy of the LilyPond parser. The LilyPond manual states that Scheme expressions inside of embedded LilyPond are executed in "lexical closure". However, macro expansion happens at an earlier point of time. So how can this work out?

• At the first layer,

let us ask the Scheme sandbox (slightly reformatted):

lilypond scheme-sandbox
GNU LilyPond 2.15.42
Processing `[...]/scheme-sandbox.ly'
Parsing...
guile> (define-macro (pattern args result)
 `(define-music-function (parser location ,@args)
    ,(make-list (length args) 'ly:music?)
    #{ $@(list ,@result) #}))
guile> (macroexpand-1 '(pattern (A B C D) (A B D A C D)))
(define-music-function (parser location A B C D)
                      (ly:music? ly:music? ly:music? ly:music?)
 (#<procedure embedded-lilypond
    (parser lily-string filename line closures)>
   parser
   " $@(list ,@result) "
   #f 3
   (list (cons 2 (lambda () (list A B D A C D))))))
guile>

After expanding the macro, we arrive at a call to the internal function embedded-lilypond which gets, as expected, the contents of #{ … #} as a string. But it also gets "closures" a list containing pairs, with the first element being an offset into the string, and the second being an anonymous "lambda" function for evaluating the embedded Scheme expression at that point in the string.

LilyPond’s parser uses this anonymous function that has been created in the lexical environment (namely its placement in Scheme source code) of the #{ … #} expression instead of actually interpreting the Scheme expression from the text at the time it passes the whole embedded LilyPond string to the parser. And it turns out that this lambda function magically already is the right expression! How did it arrive there?

The answer, of course, is somehow buried in the innards of scm/parser-ly-from-scheme.scm in the LilyPond source, the code that is responsible for turning #{ … #} into Scheme.

• Let us dig deeper.

Let us take a look at the embedded Scheme expression itself, outside of the macro and quasiquote and unevaluated:

guile> '#{ $@(list ,@result) #}
(#<procedure embedded-lilypond
 (parser lily-string filename line closures)>
 parser
 " $@(list ,@result) "
 #f 6
 (list (cons 2 (lambda () (list (unquote-splicing result))))))

Note that "(unquote-splicing result)" is just a different representation for ",@result", like we are now seeing a different representation for the whole #{ … #} construct. If this expression is now part of a larger quasiquote construct, the "unquote-splicing" operator will kick in and substitute the value of "result" into the list, and that is just what we are seeing here.

• A bit of history:

The current form of the closures argument as an association list of string offsets to anonymous functions was introduced as an emergency measure because Guile version 2.0 was lacking local-eval, a feature of Guile used previously for the purpose of evaluating Scheme inside of embedded LilyPond in lexical closure (the syntactical environment containing local variables) of the outward Scheme layer. Version 2.0.4 of Guile (released in January 2012) saw a return of "local-eval". Would this kind of construct work using local-eval in either Guile 1.8 or Guile 2.0?

Frankly, I don’t know. Capturing a lexical environment in the middle of expanding a quasiquote seems like an audacious enterprise. In afterthought, expecting this to work was somewhat optimistic. But LilyPond delivered.

Résumé
As straightforward as this little gem may have looked at first sight: if we take a thorough look behind the scenes to see just what makes it work, the degree to which the intertwining of LilyPond and Guile manages to rise to the level of `naïve’ expectations is actually rather impressive.

The $@ and #@ operators are a little addition in the toolbox for crossing between the lands of LilyPond and Scheme and should come in handy for some things that might otherwise require staying in Scheme for things one would rather tackle in LilyPond syntax.

 LilyPond’s (lesser-known) companions

by Valentin Villenave

Since 2008, the LilyPond Report tries to make a complete list of little-known projects pertaining to LilyPond. To be blunt: this task has proved to be impossible. LilyPond’s children are all over the place; everybody and their dog seems to be coding some interface, some plugin, some markup gizmo whatchamacallit. Which is nice — don’t take it otherwise. But every other week I seem to stumbling upon things I had no idea existed...

For example, let’s talk about converters. Some of us unlucky former Sibelius users may have heard about Kirill Sidorov’s sib2ly tool, that has been developed for at least (from what I gather) seven years... But did you know about Alberto Vignani’s ove2ly converter? (Hell, I’m not even sure what Overture is...) And there’s recently been some interest in converting Encore scores: Felipe Castro has introduced his own enc2ly converter, and LilyPond’s founder himself, Han-Wen Nienhuys, appears to be working on something similar.

Moving on to interfaces and editors, you may already know about the editing environment we all love (LilyPondTool, from our hungarian contributor Bertalan Fodor) written in the toolkit we all love to hate (Java, and the jEdit editor). But have you had a look at its counterpart Elysium (formerly LilyPondBeans), written by our other hungarian contributor Dénes Harmath for another Java editor (Eclipse)? Do check it out — or at least its website: it is so ostensibly relaxing I find it stressful

If you’re looking for something lighter, did you know there once was an FLTK-based editor named Lied? There’s also this new thing called Lyg (apparently still in its infancy), whatever that means. And dozens of others.

On a different note, you may have seen that (former?) Denemo developer Nils Gey, has now launched his own LilyPond interface called Laborejo. What does it bring? Check it out and let us know!

Finally, I have to announce (proudly) that the record for the Silliest Software Name, Ever, is currently held by a French music program whose name roughly translates as... "crepe batter". Think about that, next time someone tells you "LilyPond" is a stupid name.

That concludes our twenty-seventh issue of the LilyPond Report.

Cheers,
Valentin Villenave & David Kastrup.