LilyPond news

The LilyPond Report #28

David Kastrup, Janek Warchoł, Valentin Villenave — 2012-10-07T18:22:02Z

This short, informal opinion column is about the GNU LilyPond project: its team, its world, its community. It is not, however, an official documentation resource. Reader comments welcome; reader contributions even more appreciated!

Table of contents

Greetings everybody, and welcome to this twenty-eighth issue of the LilyPond Report!

Perhaps unsurprisingly, this installment gives a large place to the first LilyPond international meeting, with no less than three special correspondents reporting on that special event! As for the new LilyPond stable release, we have covered it enough in previous issues so that we can afford to skip ahead today, and give you a glance at our future projects, to begin with: the ongoing Grand LilyPond Syntax Stabilization work. All these, and more, are to be found in this month's issue.
As always, you can post your comments at the bottom of the page, or send your contributions to the LilyPond Report's next issues.

Editorial

by Valentin Villenave.

Well well, hasn't it been a busy couple of months. First things first: the brand-new/shiny/futuristic/pure-awesomeness new LilyPond 2.16 we've been talking about for the past year-and-a-half, is here! But more on that below. There are even more exciting news with our very first international LilyPond meeting having taken place in Germany in late August, which you'll read all about from several key members of our community. Several key issues where discussed, and some wilder long-term ideas were raised (among which, a better integration with the Mutopia project, or a LilyPond-dedicated Kickstarter fundraising dedicated to publish authoritative editions of an ancient music score). One minor disappointment this summer resulted from our involvement in the Google Summer of Code (see LilyPond Reports #25, #26 and #27), which (partly due to unforeseen, sad circumstances) did not fulfill everything we had hoped for. But if anything, this only gives us something else to look forward to next year!

With so much excitement lately, the question remains: what to do now? Well, there are a number of ambitious plans ahead, some of which may have already started on our development mailing list. Now is the time where people are gathering ideas, comments and testimonies, to determine what would best fit LilyPond's future and our (potential) users' priorities. The Grand LilyPond Syntax Stabilization project (GLISS), first imagined by Graham Percival no less than four years ago, is now in action (in all its glorious messiness, so far — more on that below) and one can only dream what it will bring us!

Well, perhaps not only dream: as a matter of fact, you can help too. We're always actively trying to make it easier for new contributors to join us; there is a position for everyone depending on your time and skills (and most importantly, your adventurousness); even if you feel like you're missing either or both, our senior developer David Kastrup is less picky and simply relies on... your checkbook. Isn't life sometimes brillant in its simpleness?

Release news

by Valentin Villenave.

You wanted it? There you have it. LilyPond's stable version is now 2.16.0 and we really, really do encourage you to go ahead and take the leap. Of the hundreds of changes and bugfixes, most will probably appeal more to the advanced user or programmer (or, you know, your average free-software geek); there are, however, quite a number of changes that will make you realize what you'd be missing out by overlooking our newest, greatest release yet.

And that's not all! Development has evidently not halted, and as soon as 2.16.0 was released, the new 2.17 development cycle issued its first unstable release — we're now at version 2.17.4 on the development branch (and counting). This branch is, obviously, intended for testing purposes and not production use.

Investors' report (1)

by David Kastrup.

[Editors' note: With some delay, here are the results of senior developer David Kastrup's continued call for funding, and the development activity your donations allow him to pursue. You will find the (more recent) August report further down below, and September results will be included in our next installment.]

July

Let's look at the numbers for July. Again, the report could have come earlier, but there has been a lot of ongoing work that seemed rather important to get done. I'll get back to that after the numbers. Again, I have not separated one-time payments and regular payments (excepting the variable payment plans) in order to get a picture fast, and all payments are attributed to the month they arrived in.

Fixed payments (€)	Variable plans (€)
180 175 2×100 50 30 3×25 2×20 5×10	25 + 50 (target €1200)
Total: 800	Total: 75
Totals: €875

Now the good news is that with the May&June report late and no additional announcements, my needs have been met, partly due to a one-time payment of €175, but mostly due to the faithful keeping up donations.

The bad news is that medical insurance will likely put me down as "self-employed" rather than "subsisting" soonish, gobbling up another €150 per month since they are then basing their payment rates on an assumed `minimum' income of about €1300 per month.

I'll try finding some more people willing to sponsor the work I do on LilyPond, but it makes sense if you help spreading the story.

Results
Now what has actually happened in July? A lot, actually. There has been a flurry of bug fixes and improvements across the board regarding programming interfaces, some performance fixes, and several cleanups of LilyPond syntax. Several programming interfaces dated from the time where Guile did not yet support rational numbers as a native number type, and I've made progress in simplifying them in connection with musical moments.

Quite a bit of effort was spent on strategic discussions in the mailing list again: some concerned the way we want to see LilyPond syntax heading, with a view of providing long-term stability.

But the most pressing topic of discussion was likely centered around our stable release criteria. As the main result, I have let myself get appointed dictator for the release of the overdue stable version 2.16. Just as of this writing we have seen the release of 2.15.95, the last unfrozen development release in the 2.15 series. The stable release branch has been split off from development, and the next release created from the development branch will be 2.17.0. Any further 2.15 releases will occur from the stable release branch, and only to cure absolutely release-critical conditions before releasing 2.16.0. The release 2.16.1 will likely happen about a month afterwards, to integrate fixes that have seen exposure in 2.17.0 and found to work well.

A plausible release date for 2.16.0 might be during the developer and user meeting for LilyPond in Waltrop at my place, happening from August 24th to 28th. LilyPond build script magician John will be present as well as project leader Graham, so releasing 2.16.0 as part of the meeting course work would seem quite feasible if nothing really forbidding intervenes.

Outlook
It is not clear to me yet how things will continue. I am glad for the time I have so far been able to spend on LilyPond thanks to the commitment of a large number of people sponsoring my work, and I think that LilyPond is seeing reasonable benefit from its results. After the efforts for releasing 2.16 and having the meeting at my place will peter out, I expect to pick up work on LilyPond syntax again and be able to finish most of the music function work in one or two months of time.

I also would like to continue with the work reorganizing context, context definitions and other LilyPond internals: this can't really be done in small parts. I tend to work on that in bursts of one or two weeks at a time, so it will likely still take several months before something ready to test will arrive from that.

And of course, there are lots of smaller things always cropping up and getting solved. If you take a look at the bug tracker, you'll see that there is no shortage of issue reports, and I do a fair amount of solving problems reported by others as well as those `invented' by myself. And that further slows down some of the `big' developments.

The month-to-month uncertainty of making ends meet even while several people are taking large personal investments (and not only monetary ones) remains troublesome, and I don't really feel like reaching the point where I would say ``now everything I wanted to see done is finished'' anytime soon.

Just recently news made the headlines that Avid, the owner of the proprietary notation software Sibelius, closed its Sibelius development headquarters in the UK, discharging the original developers. An offer by them to buy Sibelius back from Avid was refused, so they are not allowed to continue development independently. If dispensing of the expertise from the previous core programming team including its creators is supposed to make economic sense, it seems unlikely that serious further development is planned.

Now while active development of LilyPond certainly has moved through a number of hands as well, nobody is getting locked out from it. And even non-programming users have the guarantee that the code remains available for bug fixing and further developments, and that they can hire programmers should business and/or their private score collection depend on it.

Compare that to the consequences of proprietary business models: the right to make such an offer has been bought off the creators of Sibelius.

A Postcard From Waltrop

by Graham Percival.

The Waltrop LilyPond meeting was fantastic. I've met other LilyPond developers before (Bordeaux in 2010, Paris in 2011, Linz in 2012), but this time it was on a completely different level.

From left to right: John Mandereau, Graham Percival, Jan Nieuwenhuizen

I think there were two reasons why it felt so different. First, it was a multi-day affair with computers, rather than meeting people for an afternoon or two while sight-seeing. There's nothing wrong with purely social affairs, but sitting side-by-side while working on LilyPond gave me much more of a feeling of teamwork. Second, there were enough people there such that I could watch the interactions of other developers. It was fascinating to watch small groups working, such as Jan and John fighting with GUB, or Mike and Janek passing patches and regtests between each other to find and fix bugs in the vertical skyline. There was a mixture of technical LilyPond discussion, tips on tool use (git, emacs, scripts), and jokes about pineapples.

I was also blown away by the speed Mike and Jan's workflow. I've always considered myself to be a fast typist and fast at switching between windows and desktops, but they completely outclass me. I was particularly stunned by Mike, since he was working inside a virtual machine! The best analogy I can think of is an organist. I've seen skilled pianists all my life, but the first time I saw a skilled organist up close — playing on 2 or 3 keyboards, twiddling the stops, playing pedals... all with a delay of a second or two — my jaw almost dropped. I'm not trying to start a fight about which instrument requires the most skill (how can we really judge the "skill" of manipulating keyboards vs. manipulating vocal cords, tongue, lips, etc?), so let's just say that the organ has one of the most *visible* displays of skill.

I didn't get a lot of work done on the weekend. I have trouble concentrating when there's people talking around me unless I completely drown them out with really loud music in headphones. I considered moving to a different room to work, but that would have been really antisocial and I wasn't really planning on doing much work anyway. I viewed it as a holiday from my PhD, so I was happy to "hang out" in the main room and do incidental tasks. I'm glad that I stayed in the main room, though — I feel more enthusiastic about LilyPond than I have for the past year or more. I've only been "going through the motions" for the past few months, but that's changing due to Waltrop.

I definitely hope it happens again. The energy was incredible, and it made me wonder how much LilyPond could improve if we had regular face-to-face meetings. It's not possible given our world-wide nature of our developers and contributors, but it's an interesting thought experiment. I'm a bit sad that I didn't try harder to organize a British LilyPond meeting, but it's too late now as I'm leaving the country in a few days.

Graham Percival (looking surprisingly confident.)

From Waltrop with love

by Janek Warchoł

Breakfast time.

I am very happy that i participated in Waltrop meeting. It was the first time that i've seen any of the developers in person - in most cases i was quite surprised! Granted, i've talked and collaborated with you for about two years, but in text-only communication people appear differently than in real-life (now, that's not an impressive discovery :P). I think that Graham was the one who surprised me the most - turns out that he's not that grumpy after all! Did you know that he can actually be enthusiastic about some LilyPond proposals? (if you, dear reader, are now thinking that i must have had hallucinations, i assure you that i'm 100% sane!)

Also, i think that i can understand other developers significantly better - i just imagine them speaking whenever i'm reading an email from them, and bang! 50% better understanding Yes, i think that may be one of the most important results of the meeting - more effective communication (as English isn't my native language, i'm never sure if my messages are clear enough - it was great to have real-time, visual feedback from other people on what i say).

There are some measurable effects, too - i got interested in GUB, and when i'll receive my shiny new laptop [it's actually a ThinkPad, so it won't be shiny in any literal sense ] i'm going to try building GUB there. Never expected that — wasn't GUB always the most scary thing in Lily development?

All in all, i believe that together we can craft the best piece of music software in the world!

Janek "the Pineapple Man"

Impressions from the Waltrop meeting

by David Kastrup.

In a nutshell, the meeting quite surpassed my wildest expectations. Partly due to my lack of insistence, a few weeks before the meeting it looked like Graham, Marc, Harm and myself would spend some leisure time with each other, and that would be about it. But then with Mike, John, and Jan a critical mass of heavy-weights enrolled, and furious activity ensued.

Friday
On Friday, basically between Graham and John the release of 2.16.0 was getting under way, completing more or less by noon, starting the whole meeting off with a large boost of excitement, motivation, and a sense of entitlement. Janek started to install and compile GUB, LilyPond's Grand Unified Builder on his machine, a process continuing in the next days and spreading out to more machines, including one dedicated for future release work from my place. With Mike, and later Jan arriving on Friday, the plan was to release 2.17.0 running with accumulated updates to the build system in the evening.


Leaving the evil bright yellow thing outside

It turned out that this was quite a bit too optimistic and that my resistance against admitting any non-trivial work into the 2.16.0 release after 2.15.95 had been prudent, even though the freezes I put down were rather abrupt and provided little opportunity for firming up 2.16.0 better. Soon after splitting the stable release branch off early in August it became clear that several small regressions would require non-trivial fixes, and that exposing them in an unstable release before reintegrating them into the stable branch would be desirable. So there was (and is) definitely a release 2.16.1 going to happen in due time, and clamping down hard on 2.16.0 was opportune. The alternative of softening up the freeze until the last fixes were in place was something we tried for about a year again and again: without an unstable release series serving as a testbed, it did not work.

In this manner, we were able to ride on the euphoria of having released 2.16.0 early in the meeting. Nothing seemed impossible, and the programmers tended to stay busy until late at night, or early morning, with vigor working on 2.17.0, making GUB run on several systems, and in the last night, integrating Mike's large skyline work. Graham had set up his bed in the meeting room and alternated between sleeping, working, and overseeing things.

Weekend
Saturday saw the arrival of Karin Höthker and Dominik Hörnel from the Scorio project (see below). A lot of discussions ensued, a tablet computer with a score application was handed around, and the LCD projector rented for the meeting was put to use. In other departments, Rodolfo assumed the reins in the kitchen, resulting in a marked increase in food quality at lunch (pasta and salad) and dinner (barbecue).

Increasingly the importance of having LilyPond support MusicXML for output became obvious. Patrick Schmidt from Philomelos, arriving on Sunday, did nothing to detract from this impression, and neither did Nils Gey, the Laborejo head honcho (see The LilyPond Report #27).

Another recurring theme was the stability/dependability of LilyPond syntax. Neither of those projects ended up using LilyPond as their storage format. Philomelos, when importing LilyPond, uses \displayMusic to print it, then parses the resulting Scheme code using a parser written in another programming language (could have been PHP but I don't quite remember). The Philomelos GitHub account sports a fork of the MusicXML exporter from LilyPond, so it would not seem like this is the only strategy, or maybe I just confused things. Scorio uses LilyPond just for export and typesetting, and looking at their web page listing the used "Open Technologies", there is a notable absence of LilyPond. It is mentioned on the Platform page, though, which makes it likely that this somewhat disconcerting omission is not due to hiding credit where credit would be due, but rather due to the lack even of self-inflicted standards LilyPond obeys.

Harm had arrived on Saturday, but although he indulged in some bar line customization work with Marc, the main focus of the conference activities turned out to be not in the Scheme/Guile department. An embarassingly large part of the conference appeal might be described as "Wifi and AC power", and the developers taking things from there. The flurry of release and build system activity culminating in the release of 2.17.0 on Sunday and the followup integration of Mike's skyline patch on Monday and the release of 2.17.1 in the following night concluded the storm of development activity.

Apart from the various coding parties, another important aspect were the frequent discussions about where corporate users of LilyPond were seeing problems affecting their workflow, and lots of discussions about where we see LilyPond heading.

A recurrent theme were the question what to do about knitting LilyPond into the music fabric of the Internet, and what to do about Mutopia ailing rather than profiting from the increasing growth of user-provided content management and/or crowd sourcing.

We had Han-Wen on phone conference from Brazil on Saturday night, but connection quality and lack of physical presence (creating a video connection was not reasonably successful) turned this more into a radio-amateur-like experience rather than an interchange with a high level of interactivity. While the phone conference petered out, I brought Karin and Dominik to their hotel in Datteln, having to drag Conny from the car in the process where she had sought shelter from the thundering voice of Han-Wen permeating the house.

Monday
On Monday, Patrick brought in Julien Lerouge, the programmer responsible for Philomelos' import of LilyPond (if I understand correctly). I did not actually follow most of the discussions that happened on Monday since most happened in small circles. Also, I was somewhat exhausted, and intermittently trying to get GUB to work on a donated computer.

Tuesday
People left by and by. Hopefully not too many were affected by my mischaracterization of the local transport ticket vending machines (which did require stamping even single-person tickets before use, contrary to more central ticket vending machines). I don't think that any party dropped off at the subway station had more than 1 minute to spare except for John.


Coding party.

Summary

The good
The meeting was quite productive regarding work on LilyPond. It was particularly helpful that Jan, as the author of GUB, LilyPond's Grand Unified Builder and one of the LilyPond founders, was working together with John to bring some of the requirements forward to newer systems.
It was great meeting all the people one knew only from mail communications and associate a face and personality with them.
Indeed, not all that many of the participants had previously met with Jan. It was particularly nice to see him enthusiastic about the meeting itself, the people he met, and some newer developments and possibilities, like when I showed the amount of introspection possible using the Scheme sandbox and the much energized #{...#} to repeated interjections of "It's beautiful!".
While organization was rather on the thin side, the meeting attracted American, Canadian, Dutch, French, German, Italian, and Polish participants from France, Franconia, Germany, the Netherlands, Poland, Scotland, and Switzerland.
The bad
There were quite a few lost opportunities.
The picturesque surroundings were not really keeping anyone distracted for any significant amount of time. Most participants probably did not likely traverse more than 20m² of ground during their stay.
Apart from Marc's wife Tine, nobody actually got to see the town of Waltrop exhibiting its main yearly attraction in the form of the Waltroper Parkfest.
The lack of a prepared talk schedule resulted in a dearth of "entertainment" for participants not currently coding. Having talks and discussions in parallel would not likely have distracted the focus of the coders significantly, so this was a bit of lost opportunity as well.
There were too few interactive sessions focusing on the exchange of programming, music entry and debugging techniques and establishing standards for coding and review.
Apart from my own Thursday evening icebreaking attempts, no music was actually performed during the conference, probably in order not to prompt repeat performances.
My lack of planning (and sleep) did not always lead to the most convincing results in the food department. Part of that was softened by our favorite Indian fast food joint which even provided "typically German" fast food in the form of "Zigeunerschnitzel" and "Hawaiischnitzel" to those participants who would have been loath to leave without a taste of German fastfood classics.
The lack of organization ahead of time lead to a somewhat random collection of guests and interests and probably made it hard to get affordable transportation. On the other hand, finding sleeping places for a significantly larger assembly would likely have been difficult.
Conclusion
I think that the meeting was an important step for refocusing on our common goal. Most of the participants had fun: basically all of them had to be dragged from their computers and to the subway in the last possible minute. I hope that this fun will stay with people and carry them to our next meeting. Even though the participants did not get to leave the house much, the meeting place was adequate for our purposes.
I consider doing this again next year. Maybe I should offload part of the organizational burden next time in order to increase the meeting's effectiveness.

Investors' report (2)

by David Kastrup.

August

So here, somewhat late are the numbers for August. Let me first state that I am carrying over an impressive €420 payment "to be distributed as I see fit" into September (see below) as it appears more necessary there. So this is not included in the August numbers. Apart from this rather large sum, the Waltrop meeting more or less carried itself. I set a "conference fee" target of €10 per day and person. This might have been overoptimistic on the days we ordered food, but attendees tended to round up, and there were some donations dedicated to the meeting (including covering the LCD projector). For mixed donations and meeting fees, I have deducted the nominal daily fee before listing the donation. So the meeting is accounted for.

Fixed payments (€)	Variable plans (€)
175 160 152 119 100 3×80 75 70 60 40 39 32 30 5×25 20 15 2×10	25 + 50 (target €1200)
Total: 1472	Total: 75
Totals: €1547

Now that's rather amazing, and of course the single variable plan is down to €25 for September. There are a few other amazing things: most of the larger contributions have been from previous contributors, making their total contribution quite larger than what appears in the monthly statements.

One of the larger single amounts was from somebody who is taking his vow of poverty and considered it appropriate, as he had been making use of LilyPond for creating readable score sheets for brethren with worse eyesight, to take LilyPond's development into consideration when divesting himself of his last possessions.

Again: I continue being amazed at your support, and it is rather uplifting to see that continued amount of support and dedication. The question is how to stabilize it: if everybody now thinks everything is in safe waters, it won't be. So again I want to remind people of the possibility of pledging a variable payment plan, where the payment depends on the situation in the past month.

I have taken the opportunity of actually taking a look at my personal financial situation. My original statement of a minimum of €800 for continuing on LilyPond was somewhat optimistic: with rent (about €400 including energy, phone, and internet) and insurance and other fixed costs, about €600 were already gone (and insurance, as told last month, is going up), and that leaves about €7 per day for food and similar niceties. It turns out that my bank account from the middle of September, as compared to March, has dropped by a mere €200, while the remaining unconverted PayPal balance in $US is a bit above €1000. However, pretty much at the start of this period, our "shopping accounting" for close to a year was done, and I got about €600 out of that as I am mostly the one of us two buying food.

So it means that even with the spectacular August balance (including the carry-over amount), I have been scraping even. In that time, I took one conference trip to Chemnitz (for a LilyPond conference), and a 10-day climbing vacation with friends in Italy at a rather affordable apartment. Not all that much of luxury. And there will still be taxes to pay, as your support puts me above the poverty line regarding the German taxation system.

The €800 projected sum (which was pretty much what the monthly average of the previous months had been) was just that: an amount allowing me to break even, and with the bit of cheating in August, it did do so before taxes. But if you consider contributing to a variable payment plan, in the interest of long-term sustainability I would suggest specifying the target sum after which your payments are allowed to decrease somewhat higher as I don't really see all that much wiggling room for further cutting my expenses.

While it is too early for a detailed report on September's results, latest numbers show a sharp decline. However predictable that decline was, it confirms how fragile this system remains, and how your constant involvement and benevolence is needed.

Results of August
The main focus in August has been the developer meeting at my home in Waltrop (see above), and its results. LilyPond 2.16.0 was released on that meeting as well as 2.17.0, and 2.17.1. The preparation of the 2.16.0 release had been left to my discretion after extensive discussion, and while there are still some cleanups requiring a 2.16.1 release, its overall shape appears fine to me, and being able to start the meeting off with releasing it was quite important for setting the mood. So no regrets here, and it is definitely quite a step forward for our users.

While I managed to get off the wrong foot for much of the after-meeting discussions and work again, I still think that having met personally will help everybody manage to get a broader perspective concerning the shortcomings of our communication.

What's happening now
So this was rather uplifting, and the mood of the Waltrop meeting also helped. Back into development, the atmosphere on the mailing list, after initial revitalization, has become strained again. To a large degree, this is attributable to my manner of handling proposals I do not consider fit for inclusion into LilyPond.

Discussions about syntax changes have been opened, and since my own priorities are set on consolidating the current state and possibilities while many of the suggestions don't really fit well with current and planned developments or overall consistency, I have spent an extraordinary amount of time and energy mostly discouraging proposals or expounding on the problems they would cause both in their own implementation as well as when considered as part of a whole.

Let's just say that I have not really managed to do this in a manner that has been well-received.

Ongoing tasks
It turns out that syntax changes are continuing, with much of the patches being somewhat obtuse in the description. I expect to spend at least the next month on continued cleanup and reorganization, with the goal of making user-definable functions a much more versatile and ubiquitous tool than they are now.

There are some discussions about various syntax changes, and some of them are more implementable or useful than others. Those that are both of more invasive nature as well as worthwhile will likely have to wait until I finished with my ongoing reorganization work, as they would get in the way before this is reasonably finished.

After the syntax has been done, the property system is still in for an overhaul in order to make it work more closely to expectations. There are currently surveys planned in the national user lists and forums for LilyPond regarding the problems people have using LilyPond, but some of the preceding discussions indicated that this is an area of interest.

And the Guile v.2 migration is also going slowly on intermittedly. In all of those areas, I consider it important to be involved, and your support allows me to do that.

Online Typesetting with LilyPond

by Karin Höthker and Dominik Hörnel.

At the Lilypond developer meeting in Waltrop we presented our online music editor (see The LilyPond Report #24). We talked about how we attempted to tackle the performance problem one inevitably gets when developing a WYSIWYG music editor with Lilypond. Typesetting remotely on a server adds to the problem. In the meantime we have done some measurements that we will present in the following.

When starting with scorio, we quickly agreed on LilyPond as our music typesetting engine because of three major reasons:

its beautiful music engraving
the ability to produce scores from semantical music information without the need to specify a lot of graphical details thus avoiding extensive re-work with adjusting text, slurs and other notation elements afterwards
because it can be extended and integrated into a GUI-based note editor.

At first, it was not obvious how to achieve a round trip in an acceptable time involving

typing a note into an editor,
sending a request to a server,
running Lilypond on the server,
sending the result back to the client and
displaying the result in the browser.

Although there are excellent editors supporting input of Lilypond syntax and Lilypond compiling (like Frescobaldi), it is hard to find WYSIWIG editors for Lilypond (Denemo is an example). One reason may be that LilyPond offers an input format that is excellent for writing music in a text editor, but challenging as far as automatic score manipulation is concerned.

In the following section, we briefly explain the round-trip in scorio and how we attempted to master the performance issue.

Score manipulation
As mentioned above we decided against doing edit operations on LilyPond source files. So, we had to look for an alternative representation. For a short moment, we thought about developing our own internal music format, but soon rejected the idea: Why re-invent the wheel? Furthermore, we would have locked users in into scorio, since import from and export to other music formats had yet to be written. We were rather aiming at setting up an open music notation platform, where users could import existing scores, edit and share them online, and later export them again to their favourite notation program.

So, we ended up using a slightly enhanced MusicXML format as our internal representation. Although not perfect, MusicXML makes score manipulation, import and export easier as it has become a de facto standard in music notation industry supported by all major notation programs. The graphical representation in scorio is pure HTML and therefore makes it possible to use scorio in all major browsers on every platform (PC, Mac, and even iPad). To produce HTML for a LilyPond score we wrote our own HTML backend as an addition to the existing Postscript and SVG backend in LilyPond.


Figure 1: Score manipulation round-trip

So the round-trip basically works as follows (see figure 1): When the user clicks into the score, an edit command is created, e.g. “create new note before existing one”. The edit command inserts the note into the MusicXML tree. Then the MusicXML tree is converted into a Lilypond file and processed by LilyPond. The HTML backend produces an HTML page and finally sends it back to the editor. The rendering of the score is done by the browser.

The HTML representation of the music piece mainly consists of images. However, the client needs more information in order to decide what action the user wants to perform next. For this purpose, the HTML contains additional attributes attached to each graphical object (or grob) of the score. For example, its part and staff are attached to each note. This is achieved by adding “annotations” when generating the LilyPond score. Annotations are created using a custom \annotate command that adds them to the Scheme music tree. In this way they will be attached to the grobs they belong to, and can be added to the HTML in the HTML backend.

The following excerpt from a generated Lilypond file shows an example of annotated Lilypond:

\score {
 <<
 \new Staff="'((sc-part . P1) (sc-staff . 1))" <<
 \new Voice {
 \numericTimeSignature \annotate #'(
 (sc-id . 1298541892071_0.6492787506599623)
 (sc-beats . 4)
 (sc-beat-type . 4))
 \time 4/4
 }
 \new Voice {
 \annotate #'(
 (sc-id . 1270804615382_0.44181689312608075)
 (sc-part . P1)
 (sc-staff . 1)
 (sc-voice . 1)
 (sc-fifths . 0)
 (sc-mode . major))
 \key c \major
 \annotate #'(
 (sc-id . 1270804615379_0.08310412084356789)
 (sc-part . P1)
 (sc-staff . 1)
 (sc-voice . 1)
 (sc-sign . G)
 (sc-line . 2))
 \clef violin
 \annotate #'(
 (sc-id . 1270804615382_0.06348578421654139)
 (sc-part . P1)
 (sc-staff . 1)
 (sc-voice . 1)
 (sc-step . C)
 (sc-octave . 5)
 (sc-note-type . quarter)
 (sc-duration . 480))
 c''4
 }
 >>
 >>
 }

So far, we have seen the editing round trip. The performance problem however is still open.

Improving performance
The round-trip works well for smaller scores. But as the score is growing larger and larger the performance significantly drops and makes editing practically unusable when rendering the entire score after each edit command. We also noticed that the Lilypond processing time does not increase linearly but superlinearly with the size of the score due to the calculation of optimal line breaking in LilyPond. To overcome this performance issue we use the \skipTypesetting command. It allows producing graphical output for an excerpt of the score thus almost reducing the time consumed for typesetting to this excerpt. Consider a user working on a system in the score. As scorio shows one page at a time we may skip typesetting for all other pages. Even more, if the user is editing notes on a specific system it is not necessary to update the entire page each time but rather the system on which the user is working. We therefore do typesetting for this system only. After the user has finished working on the system s/he may refresh the page and later the entire score. We call this concept partial update: When an edit command is performed on a system, typesetting is restricted to that system (system update). In case the edit command produces more than one system due to a system break, a page update is automatically performed. Only in case an edit command influences the entire score such as a score transposition, or if the user manually refreshes the score, a score update is performed.

The following figures show some measurements of the LilyPond performance with score, page, and system update. The sample score simply repeated the notes Bb-A-C-B up to a maximum of 20 pages.


Figure 2: Runtime performance with LilyPond.

As can be seen in figure 2, the runtime increases superlinearly (quadratically?) with the number of pages.


Figure 3: Page update with LilyPond.

In figure 3 however, creating one page took about 3 seconds. When increasing the number of score pages, but typesetting one page only the runtime slowly increases linearly. Rendering one page on a 20 page score took about 4.4 seconds.


Figure 4: System update with LilyPond.

Similarly, in figure 4 creating one page took about 1.5 seconds. When increasing the number of score pages, but typesetting one system only the runtime slowly increases linearly. Rendering one system on a 20 page score took about 2.4 seconds.

Conclusion
We have described how the online music notation site scorio.com typesets music with Lilypond running on a server. Performance measurements show that the approach works well for smaller scores. Since scorio's user interface is designed to be as simple as possible, it targets users who want to typeset music with only little training and typically would like to typeset short musical pieces for daily use rather than extensive compositions. At the same time, we feel that there is still room for improving the performance of the most widely used operation in scorio: editing a single note in a system. 200ms – the reaction time commonly perceived as fast in user interfaces – may be out of reach. But a turn-around duration well below 1 second would improve the user experience.

LilyPond Syntax oddities

by Valentin Villenave.

Let's be blunt: for the past couple of years, I haven't been as much involved with the LilyPond project as I'd love to (or at least, as I used to love to). I never stopped using LilyPond obviously, but the discussions, the developments, so much intelligence and brightness flowing up everyday on our mailing lists, that was just more than I could handle.

Recently, I happily accepted a commission from our contributor James Lowe to create an orchestral score for him. Since he told me the violins in his orchestra were mostly beginners, I thought I'd use quite a lot of open strings; so I thought, hey, there must be some sort of a shorthand for that. Since d-> produces an accented note and d-. produces a staccato, I had reasons to hope that, say, d-° would print an open string.

And it did.

Now, James being a LilyPond connoisseur of sorts, I thought I'd send him the .ly code instead of a PDF. And since I was using a fairly recent development build, there was hardly going to be any problem for him to compile my code, right? Right. Except... he couldn't.

Because what I had been using was actually not a known shorthand, but an almost-undocumented feature of LilyPond that allowed people to print markups without using double quotes. And the ° character I mistook for an "open string" sign, actually was just that: a single-character markup. And thanks to David Kastrup clarifying the lexer and getting rid of this undocumented feature (that looked, actually, like a bug or an inconsistency of sorts), the newest version of LilyPond — the one James was using — would have thrown an error, thereby forcing me to clean my code, and use the actual, proper \open symbol instead:

What I mean is: syntax consistency does matter. To all of us, even to experienced users like myself or James. Granted, all of us do not have much time to spare for such discussions. But there _should_ be a way for as many people as possible to gain awareness and take part in this collective thought process. This cute silly story of mine actually predates the Grand LilyPond Syntax Stabilization project that is now in action (for better or worse, see below), but LilyPond is full of such undocumented half-features/half-bugs.

To wit: we recently mentioned the empty chord construct, which may be one of the nicest of those; however, look at another undocumented oddity David has just exposed:

xxx #'j #'k #'l = 7 #(format #t "xxx = ~a\n" xxx) xxx = #'() xxx #'j #'k #'l = 7 #(format #t "xxx = ~a\n" xxx)

Not for the faint of heart, eh?

LilyPond Syntax discussions

by Valentin Villenave.

So, the GLISS project is finally here and people are chiming in with their suggestions. Whilst it had been suggested that these discussions ought to take place on a mailing list of their own (a list I opened four years ago but that hasn't seen much activity), it has been decided that our regular lilypond-devel mailing list would do just fine, provided that mail subjects are neatly prefixed with [GLISS] for general-purpose gliss-oriented conversations, [talk] for informal talks, [proposal] for more carefully thought-out ideas, etc.

It does make for very technical, high-density, high-volume conversations; some generous souls have offered to keep regular users posted (particularly users who don't speak English on foreign mailing lists) but that doesn't change the fact that our -devel list, which was already quite crowded, is becoming somewhat of a mess — where some discussions tend to become heated (as David mentioned above).

Here at the Report, we've been receiving a few letters from people involved in these discussions and wondering if there could be a better way to handle these discussions. Here's what Ian Hulin (who has been working on several proposals for a new tuplet syntax) had to say on the matter:

I reckon what has happened has provided us with some useful insights.

Unless someone takes ownership of a discussion and is prepared to spend time moderating it, it will tend to peter out and go nowhere.
Being a moderator can be like being a eunuch in Imperial China: they knew an awful lot about sex in great detail but couldn't do it themselves. Moderating a discussion means guiding developers to a consensus and then finding one of them comes up with one which does 90% of the job (cf. Keith O'Hara's latest post).
Moderating means you've got to be prepared to repeat yourself an awful lot.
Moderating means you've got to be patient, and you've got to be infinitely polite.
Moderating also means you've also sometimes also have to be deliberately stupid so you can get a particularly enthusiastic developer to dump a bit more out of their head into writing so the rest of can buy in to what he's thinking.
Moderating means keeping track of the discussions and producing a and constantly refining a proposal until it becomes a design doc for the feature. We could also use it as the basis for entries in NEWS if the feature needs one.
Using [talk] only really works if we use it as a sub-list of -devel.
Moderating a proposal means spending time on it, which you may feel could be spent on implementing it. It's actually quite difficult to do (hence the eunuch reference above).
Using this process maybe allows us to capture the WHY of a development feature rather than answers to WHAT and HOW which is often what is documented, if it is documented at all.

This somehow meets another proposal by Janek:

i suggest to visibly separate discussing problems from discussing solutions. Currently we are mostly discussing ideas for syntax, i.e. "let's have syntax X doing Y". I suggest that for several weeks we shall focus on "i find syntax W confusing" and "i find notation Z difficult/inconvenient to express in current syntax" instead. We would add syntax problems that we identify as issues to the tracker. [Editor's note: David isn't sure the tracker would be an appropriate place for that, and I tend to agree.] After we've finished gathering them, we'll sort the issues and *then* we would discuss how to solve them.

Why do it this way?

we'll see the big picture better
we'll be able to schedule the discussions about solutions, so it'll be easier to participate in them

And perhaps most importantly: when someone posts a syntax *idea*, there's a chance that syntax experts will reply "omg wtf?! this won't work". This leads to frustration. On the other hand, if we discuss our *problems*, syntax experts can just answer "it would be reasonable to solve it this or that way" - and voila! less frustration.

So, where do _you_, dear reader, stand on these issues? Is the current way appropriate to you, or do you feel somehow left out? Is there a place that would seem more likeable, more convenient, for these discussions, that you can think of? Or is it a purely human problem (assuming, of course, there is a problem at all)? Like many times in the past, the LilyPond Report intends to help however it can, by gathering opinions, synthesizing current debates and providing its readers with both the latest topics at hand, as well as a place to comment on them while taking a step back.

That concludes our twenty-eighth issue of the LilyPond Report.

Cheers,
Valentin Villenave, Janek Warchoł, Graham Percival & David Kastrup.

The LilyPond Report #27

David Kastrup, Valentin Villenave — 2012-07-31T20:17:00Z

Table of contents

Greetings everybody, and welcome to this twenty-seventh issue of the LilyPond Report!

This issue raises some important questions as well as completely irrelevant shallow factoids. That's how we roll, just bear with us. On the plus side, you will learn quite a lot about Scheme integration in LilyPond and some brand-new niceties that David Kastrup has recently added to our feature set. Also featured in this installment: how to take black-and-white pictures without intending to, what is the silliest possible name for a software project, and what future is there for fancy night clubs. (Yes, we know — but don't say we did'nt warn you.)
As always, you can post your comments at the bottom of the page, or send your contributions to the LilyPond Report's next issues.

Editorial

by Valentin Villenave.

Here's a piece of news for you: LilyPond is no more. It has filed for bankruptcy and disappeared from the map, it's been shut down, it has ceased to be, it's expired and gone to meet its maker!

Now, I'm evidently not talking about the GNU LilyPond free software we use and love. What I'm referring to here, is the LilyPond night club in East Hampton, one of the fanciest and most expensive places in one of the fanciest and most expensive locations in the world: Lily Pond Lane. I've been trying for the past several years to make a shallow and not-funny joke about it, vainly searching for some link between an outrageously indecent "1% amongst the 1%" place, and our own small community of dedicated volunteers... Only to learn that it had gone bankrupt and been closed — or rather, merely changed its name. (That sarcastic tone? That's just me refraining from weeping any tear.)

Anyhow. Are we in any danger of closing down and filing for bankruptcy? Sometimes I do wonder. On one hand, some things are looking pretty good this year: we've been awarded with a gizmo and we've been having our very first Google Summer of Code sponsorship (go Janek, we're counting on you!). On the other hand, David Kastrup's request for funding, although moderately successful at first, isn't doing so well these days. You'll find his May-June report below, and the next one is not on its way to look any less bleak. Now, there's probably a lesson to be inferred here, but I'll let you fill in the blanks!

Release news

by Valentin Villenave.

As this Report went online, a new development version (2.15.42) has just been released. You may want to give it a go for testing purposes; please note, however, that due to some critical bugs remaining, this is not the next "Release Candidate" for our upcoming next stable version. For normal users, we still officially recommend the current stable version 2.14. (Although, if you ask me, the development branch brings just too much awesomeness to be overlooked. More below.)

Investors' report

by David Kastrup.

First I have to apologize for delaying the report for May until it made no more sense to release it on its own. So here are the numbers for May and June, in sequence. The original request for funding made in February in the LilyPond Report 24 still contains all the relevant details regarding my financial situation and the need for funding while I am working exclusively on LilyPond.

I have not separated one-time payments and regular payments this time: they average out in the whole, and where regular payments come into disagreement with the intended plan, keeping track becomes really complex, and running a reminder system would be even more complex, and of course somewhat pointless when we are talking about a voluntary commitment.

As a rough estimate, probably about €400 per month can by now be attributed to regular or semi-regular payments.

Regarding the report for April, there are €100 missing that I had counted as half of a two-month installment when it really has been a one-month payment. Special thanks for that! So April, with a total of €821 also managed to surpass the prospected bare minimum of €800 per month required to keep my financial situation in limbo.

I count payments in US$ as €0,80 here, and other currencies as the result in Euro after Paypal's conversion and fee deduction.

May

Fixed payments (€)	Variable plans (€)
250 200 160 100 72 37 6×25 24 2×20 15 3×10	25 + 50 (target €1200)
Total: 1078	Total: 75
Totals: €1153

Until May 20th, there were about €200 in donations. I used the opportunity when posting a link to the video of a LilyPond-related talk I gave, to point out that things were not really looking well, and in hardly more than a week, the totals surpassed that of any month so far.

June

Fixed payments (€)	Variable plans (€)
160 100 50 2×30 5×25 20 3×10	25 + 50 (target €1200)
Total: 545	Total: 75
Totals: €620

In contrast, June has dropped to a record low, most likely due to the delayed report for May. Averaging both months still put me in the position to pay rent, health insurance and food till the end of June. June has really shown itself a month of the faithful (to the degree where some payments likely intended for July already arrived). As one positive result, about two thirds of the payment arrived via SEPA bank transfers, quite the most cost-effective option within the Euro zone (banks should not bill more than for in-country transfers).

Reasons for delayed report

What is the reason that this report has been delayed so much? Mostly, I am not happy with the current development situation and how I am contributing to it, to the degree where I am not sure I am giving a good return of the considerable investment LilyPond users take in me. That's one reason I felt not really justified asking for more. And while the large rally at the month of May has on the one hand been elating and inspiring, it also meant responsibility to live up to the expectations.

Recently, Graham made an anonymous survey of formerly more active LilyPond developers, asking for the reasons of their declined participation. One answer in particular was worrying to me.

I am citing one paragraph:

The last "teamwork" problem is the amount of "monetization" going on. I didn't join lilypond to help other people make money off it. I don't have any good suggestions for this, but the question asked if I was contributing less and this is one reason. Having some people making money and others not makes it feel like less of a "team".

Now it is not that much that I am ``making money off LilyPond'': at the end of the month everything is gone without reserve so far. It is more like I am making time for LilyPond development from the monetary support of LilyPond users and developers: I can afford the luxury to work on a project I like. Being independently wealthy would be even nicer, but I have to work with what I've got.

I am quite grateful that users of LilyPond have been willing to pitch in and let me continue. But what this buys them is just a single active developer, and if that makes other developers stop contributing, it is not helping.

Another thing in this feedback about reduced involvement points out a deterioration in the style of discussions on the LilyPond mailing lists. And it is not hard to make a connection to my lack of decent communication skills.

LilyPond developer and user meeting in August

So I am trying to address this with a meeting of LilyPond developers and users at August 24th to 28th (check out the detailed announcement here) at my place. I have made the experience that it is easier to behave civilized to people one has actually met already instead of knowing them just as letters on the screen. It also makes it easier to understand what people are trying to say, even when they are not having their best day. With a `critical mass' of core people reasonably understanding one another, maybe some more or less frequently occuring problems might become insignificant.

Of course, another purpose of the meeting is better distributing specialized knowledge, and make more people able to deal with the programming languages and subsystems of LilyPond, and also figure out where the stumbling blocks are.

So maybe this meeting, where a number of core developers will be turning up, will be able to improve the atmosphere, productivity, and feeling of belonging to a really worthwhile community again.

Lilypond and Scheme

Some of the recent Email discussion (hopefully to be considered civilized) were centered around the use of Scheme as LilyPond's extension language. My personal take on this is that embedding Scheme at the syntactical LilyPond layer (namely in *.ly files) is working out well enough. LilyPond is more a language of declarations and expressions rather than actions, and a functional language like Scheme fits nicely with that. The current interface using #, $, #{ … #} as a way of switching the language between expressions is actually rather straightforward. Interfacing an imperative language, where the basic unit of execution is not an expression, would likely be harder. The end of June saw the additions of #@ and $@ operators (see below).

As a general purpose programming language (namely what we see in *.scm files), the dearth of visually recognizable subunits is the most common complaint about Scheme. As long as good user interfaces manage to keep the complexity of the required user-level programming at bay, I don't consider this a show-stopper. LilyPond has quite a lot of potential and need for further improvement here, even though version 2.16 will offer a lot over 2.14 in that department already.

With version 2.15.41, the eighth release candidate for 2.16, we are again on the finishing line for stable releases, and hopefully it will not be yanked from our feet by newly discovered regressions again. There are changes in the release policies under way, so perhaps 2.18 will not take as long as 2.16 did. While 2.14 took even longer, there was less of a feeling of urgency about it.

What happened in the code?

Apart from talk, what kind of work on LilyPond did I do in May? I've made some progress with trying to reinvent some central data structures in the framework of GOOPS, GUILE's Object Oriented Programming System, but this project is moving forward slowly. I am still getting the hang of how to create a design that falls into place naturally and is easy to use. Basically, this is still issue 2507.

Apart from several bugfixes and some slight syntax changes, most noteworthy are probably the work on tweaks allowing for something like

<\tweak Accidental #'color #red cis4 \tweak Accidental #'color #green es g>

(previously, only the notehead would have been susceptible to a tweak), completing the work on event classes meaning that string indications like

c\5

don't need to be part of a chord to be displayed, and a redesign of the \footnote interface. The latter took several iterations before it actually became reasonably boring (that is, consistent and versatile), and an overhaul of its documentation has not yet been completed since I ran out of steam and decided doing something else before getting deadlocked. There would be more potential for improvement of the \footnote interface, but it would require rather tricky parser work concerning optional arguments. With all the iterations on the footnote interface and the documentation, this project took up more time and energy than planned for, and unfortunately not just from me. Indeed, there are still more fixes to the footnote code that have spilled into even July.

In June, the focus has been mostly on bug fixes (some critical), but one highlight in my opinion was the addition of `splicing operators', mainly $@. Basically, this means that

$@(list a b c)

is the same as

$a $b $c

If you try

git grep ",@" scm

in the LilyPond code repository, you'll see that this kind of thing is pretty useful in the world of Scheme quoting, so it's not amiss to have it as a tool in Scheme LilyPonding.

It allows to write

{ $@xxx }

instead of

$(make-sequential-music xxx)

so again, it is a measure to get work done when one is more comfortable using LilyPond constructs rather than their Scheme counterparts. (See below for a detailed article on this matter.)

Here is a somewhat extravagant example what one can do with the splicing operators, inserting pitch lists into a chord, and putting durations and articulations (in this case, slurs) after it, and splicing the whole list of chords into the surrounding music.

The output also demonstrates that our algorithms for typesetting ties have potential for improvement.

{ $@(map! (lambda (p d) #{ < $@p >$@d #}) ; this $@ produces elements for a sequential music list via map!. Each ; element is constructed from p, a list of pitches making up a chord, ; and from d, which is a list first containing a duration followed by ; _optional_ articulations, so $@d actually can return several tokens of ; _different_ type. ; ; The following form constructs the list of pitch lists for use in p (map! (lambda (i) (map! (lambda (i) (ly:make-pitch 2 (+ 1 (* 9 (quotient i 5)) (* i -2)) 0)) (iota 5 i))) (append! (iota 34) (list 32))) ; The following form constructs an (end-less) list of lists containing ; a duration and maybe articulations. (apply circular-list (map! (lambda (m) (cons (ly:music-property m 'duration) (ly:music-property m 'articulations))) (extract-typed-music #{ s4.~ s~ s4~ s2~ s2 #} 'skip-event)))) 1 \bar "|." }

Many meetings

by Valentin Villenave.

If you haven't heard, there's this big unprecedented International LilyPond meeting taking place somewhere in Germany later this month, thanks to David. Lots of exciting talks and interesting people — which, unfortunately, doesn't include your truly (because of an administrative as well as financial hiccup). However, living in Paris still allows me to meet with a lot of LilyPond contributors: Rune Zedeler in 2007, Paco Vila in 2009, John Mandereau whenever he's not in Italy, Graham "Grumpy" Percival every now and then, Mike Solomon every other day...


This should be in color: if I knew how to use a camera.

And, last but not least, Joe Neeman just this afternoon. For those of us who don't know him yet, Joe is one of our most active long-time developer; although he doesn't spend much time goofing around with us mere mortals in silly jokes on the lists, his contributions are always of a very high quality.

Meeting with him was just plain cool (he's actually much younger than I expected, yet another of these clues that tell me how my own life is a failure); Joe is a student in statistics and plays the violin, not at all the hardcore programmer I'd have expected judging from his code. He doesn't seem bothered at all with LilyPond business, and even mentioned he had "some projects" in regard to future development.

What does he do for fun? Well, he develops a logic-based video game with his girlfriend, written in Python and remotely inspired by a HoM&M mini-game. Isn't life awesome when you're a geek?

A Kind of Magic

by David Kastrup.

Exposition
In a recent mailing list discussion, I came up with the following piece of code working with recent development versions of LilyPond (2.15.41 or later):

#(define-macro (pattern args result)
 `(define-music-function (parser location ,@args)
 ,(make-list (length args) 'ly:music?)
 #{ $@(list ,@result) #}))
 
 $(pattern (A B C D) (A B D A C D))
 { a' a' a' a' }
 { b' b' b' b' }
 { c'' c'' c'' c'' }
 { d'' d'' d'' d'' }

Now apart from having a reasonably nice effect (pattern takes two lists of symbols, the first specifying the input order of its arguments, and the second specifying the desired output order), the code is both short as well as sophisticated. We will now first analyze the logic and elements of the code, and then take a look at how LilyPond manages to make it work.

The storyline

Scheme macros

are a variation of Scheme functions that changes execution order. When a Scheme function is called, first its arguments get evaluated, and then the function is executed with the evaluated arguments, and its return value is the result of the function call.

A macro changes the order: the macro is called first on the unevaluated arguments (in the case of the call of pattern, the first argument is a list with the symbols A, B, C and D). The result of the macro call is then evaluated in the Scheme evaluator, and the result of this evaluation after executing the macro becomes the result of the macro call.

Since unevaluated Scheme code usually consists of nested lists, it is often convenient to specify the macro body in the form of mostly quoted lists (quotes keep the list from premature evaluation), but with a few variable elements spliced in.

Quasiquotes

are the tool of choice for working with mostly constant lists with a few variable elements, and are seen in connection with most macros. The whole quasiquote is introduced with the backward quote character "` instead of the normal quote character "'". They are `quasiquotes' because inside of such a quoted expression, you can `unquote' parts by preceding them with a comma ",", and you can `unquote-splice' a list expression into a surrounding list by preceding them with comma-at ",@".

So, for example,

`((+ 1 2) ,(+ 1 2) ,@'(+ 1 2))

evaluates to the list

((+ 1 2) 3 + 1 2)

Namely the starting list (+ 1 2) containing the symbol "+ and the numbers 1 and 2 is retained unchanged. The same construct with an unquote before it is evaluated to 3 before being put in the list, and the quoted list '(+ 1 2) evaluates by removing the quote, to (+ 1 2) which is then spliced into the surrounding list, effectively removing one level of parentheses.

The call to pattern

Now the call to pattern looks like

(pattern (A B C D) (A B D A C D))

and, considering that macro arguments are not evaluated, the result of substituting the arguments is simply

`(define-music-function (parser location ,@'(A B C D)) ,(make-list (length '(A B C D)) 'ly:music?) #{ $@(list ,@'(A B D A C D)) #})

(we have inserted quotes appropriately to indicate that the lists are not to be executed, so that one can feed this into a Scheme sandbox inside of LilyPond) and if we now execute the quasiquote, we are left with the following expression when finishing executing the macro:

(define-music-function (parser location A B C D) (ly:music? ly:music? ly:music? ly:music?) #{ $@(list A B D A C D) #})

This is the definition of a music function doing the required transformation of four music arguments to the requested order.

LilyPond's "$@" operator

is a variation on the Scheme splicing operator and has been introduced in LilyPond 2.15.41: it can be used for splicing a list of expressions into the surrounding #{ … #} construct, and so

#{ $@(list A B D A C D) #}

is essentially the same as

#{ $A $B $D $A $C $D #}

To achieve the same effect in older LilyPond versions, we would have had to write

(make-sequential-music (map ly:music-deep-copy (list ,@result)))

instead of

#{ $@(list ,@result) #}

since one effect of "$" is to create a copy. Whenever a music expression may be used more than once, we need to copy it since many functions processing music change their input while processing it. So the new splicing operator saves us from knowing make-sequential-music and some other things.

Behind the scenes

Why does this even work?

This question is indeed puzzling since macro expansion is a complex beast, and #{ … #}, embedded LilyPond, is actually a piece of LilyPond code that is stored away in a string and later executed in a copy of the LilyPond parser. The LilyPond manual states that Scheme expressions inside of embedded LilyPond are executed in "lexical closure". However, macro expansion happens at an earlier point of time. So how can this work out?

At the first layer,

let us ask the Scheme sandbox (slightly reformatted):

lilypond scheme-sandbox GNU LilyPond 2.15.42 Processing `[...]/scheme-sandbox.ly' Parsing... guile> (define-macro (pattern args result) `(define-music-function (parser location ,@args) ,(make-list (length args) 'ly:music?) #{ $@(list ,@result) #})) guile> (macroexpand-1 '(pattern (A B C D) (A B D A C D))) (define-music-function (parser location A B C D) (ly:music? ly:music? ly:music? ly:music?) (# (parser lily-string filename line closures)> parser " $@(list ,@result) " #f 3 (list (cons 2 (lambda () (list A B D A C D)))))) guile>

After expanding the macro, we arrive at a call to the internal function embedded-lilypond which gets, as expected, the contents of #{ … #} as a string. But it also gets "closures" a list containing pairs, with the first element being an offset into the string, and the second being an anonymous "lambda" function for evaluating the embedded Scheme expression at that point in the string.

LilyPond's parser uses this anonymous function that has been created in the lexical environment (namely its placement in Scheme source code) of the #{ … #} expression instead of actually interpreting the Scheme expression from the text at the time it passes the whole embedded LilyPond string to the parser. And it turns out that this lambda function magically already is the right expression! How did it arrive there?

The answer, of course, is somehow buried in the innards of scm/parser-ly-from-scheme.scm in the LilyPond source, the code that is responsible for turning #{ … #} into Scheme.

Let us dig deeper.

Let us take a look at the embedded Scheme expression itself, outside of the macro and quasiquote and unevaluated:

guile> '#{ $@(list ,@result) #} (# (parser lily-string filename line closures)> parser " $@(list ,@result) " #f 6 (list (cons 2 (lambda () (list (unquote-splicing result))))))

Note that "(unquote-splicing result)" is just a different representation for ",@result", like we are now seeing a different representation for the whole #{ … #} construct. If this expression is now part of a larger quasiquote construct, the "unquote-splicing" operator will kick in and substitute the value of "result" into the list, and that is just what we are seeing here.

A bit of history:

The current form of the closures argument as an association list of string offsets to anonymous functions was introduced as an emergency measure because Guile version 2.0 was lacking local-eval, a feature of Guile used previously for the purpose of evaluating Scheme inside of embedded LilyPond in lexical closure (the syntactical environment containing local variables) of the outward Scheme layer. Version 2.0.4 of Guile (released in January 2012) saw a return of "local-eval". Would this kind of construct work using local-eval in either Guile 1.8 or Guile 2.0?

Frankly, I don't know. Capturing a lexical environment in the middle of expanding a quasiquote seems like an audacious enterprise. In afterthought, expecting this to work was somewhat optimistic. But LilyPond delivered.

Résumé
As straightforward as this little gem may have looked at first sight: if we take a thorough look behind the scenes to see just what makes it work, the degree to which the intertwining of LilyPond and Guile manages to rise to the level of `naïve' expectations is actually rather impressive.

The $@ and #@ operators are a little addition in the toolbox for crossing between the lands of LilyPond and Scheme and should come in handy for some things that might otherwise require staying in Scheme for things one would rather tackle in LilyPond syntax.

LilyPond's (lesser-known) companions

by Valentin Villenave

Since 2008, the LilyPond Report tries to make a complete list of little-known projects pertaining to LilyPond. To be blunt: this task has proved to be impossible. LilyPond's children are all over the place; everybody and their dog seems to be coding some interface, some plugin, some markup gizmo whatchamacallit. Which is nice — don't take it otherwise. But every other week I seem to stumbling upon things I had no idea existed...

For example, let's talk about converters. Some of us unlucky former Sibelius users may have heard about Kirill Sidorov's sib2ly tool, that has been developed for at least (from what I gather) seven years... But did you know about Alberto Vignani's ove2ly converter? (Hell, I'm not even sure what Overture is...) And there's recently been some interest in converting Encore scores: Felipe Castro has introduced his own enc2ly converter, and LilyPond's founder himself, Han-Wen Nienhuys, appears to be working on something similar.

Moving on to interfaces and editors, you may already know about the editing environment we all love (LilyPondTool, from our hungarian contributor Bertalan Fodor) written in the toolkit we all love to hate (Java, and the jEdit editor). But have you had a look at its counterpart Elysium (formerly LilyPondBeans), written by our other hungarian contributor Dénes Harmath for another Java editor (Eclipse)? Do check it out — or at least its website: it is so ostensibly relaxing I find it stressful

If you're looking for something lighter, did you know there once was an FLTK-based editor named Lied? There's also this new thing called Lyg (apparently still in its infancy), whatever that means. And dozens of others.

On a different note, you may have seen that (former?) Denemo developer Nils Gey, has now launched his own LilyPond interface called Laborejo. What does it bring? Check it out and let us know!

Finally, I have to announce (proudly) that the record for the Silliest Software Name, Ever, is currently held by a French music program whose name roughly translates as... "crepe batter". Think about that, next time someone tells you "LilyPond" is a stupid name.

That concludes our twenty-seventh issue of the LilyPond Report.

Cheers,
Valentin Villenave & David Kastrup.

The LilyPond Report #26

David Kastrup, Janek Warchoł, Valentin Villenave — 2012-05-18T16:49:53Z

Table of contents

Welcome to this twenty-sixth issue of the LilyPond Report!

No funny pictures or cooking recipes this month, the Report is back with a very serious issue full of long, technical articles. Whilst we're getting ready for the new LilyPond stable release, David Kastrup takes a look at our current development model and what needs to be ironed out. Janek has a pretty detailed analysis of LilyPond's current output and how it measures to our beloved hand-engraved references; at last, the Report as found a new contributor, namely kernel developer Pavel Roskin, who'll tell us about how he saved the world (well at least, the GCC part of it) thanks to LilyPond. All that, and many other things are discussed in this month's issue: read on to find out!
As always, do feel free to post your comments at the bottom of this page, or send your contributions to the LilyPond Report's future issues.

Editorial

(by Valentin Villenave)

How do you ensure that a Free Software project remains viable on the long run? As Graham Percival (our current LilyPond project manager) noted a couple of years ago, developers are but a non-renewable energy source. The question then becomes: what do you do with however few resources and time you have at hand? Quoting Graham again in a recent message: "I personally am interested in helping people to do whatever they want with lilypond. [But:] Simply creating an atmosphere where this is easy takes more resources than we have available." Thus, perhaps the crucial part of our discussions these days is about organisation : which tasks should take priority, which ones could be automatized, how to make sure nobody's time will be spent in vain (for example by banishing any duplicative work), etc.

Such will be the subject of upcoming discussions, with a revival of Graham's "Grand Organisation Project" (GOP2.0, nope, not this one), to be (hopefully) followed by the long-awaited "Grand LilyPond Syntax Stabilisation" project (a.k.a. GLISS). Where and how will these discussions take place? Via a live intercontinental remote-conference system? Or even in real life (David welcomes you this summer in Germany)? It's a fairly safe bet that the main place will remain our -devel mailing list, as with all serious discussions. (Well, not too serious.)

Then again, another way of tackling the problem would be to actually increase the resources (read: developer time) we have available: in other words, getting one or more paid developers to work on LilyPond.

A way to do that is to apply for grants and hope : as you may know, our Polish contributor Janek Warchoł was recently accepted as a "Google Summer of Code" student, but there's more: thanks to Mike Solomon, LilyPond recently won the first prize in a prestigious music software competition!

Another (perhaps more reliable) way, is simply to ask _you_, dear readers, for money. We're all grateful for the Free Software movement that allows us to do our computing, express ourselves, communicate and create every single day; sometimes our gratitude can be expressed by contributing: reporting bugs, helping others, promoting our favourite programs... and yet, at times the best way to contribute is the most common: throwing in some dough. That's the way the ReactOS project has recently been choosing, and it has been (rightly so, if you ask me) commended for that decision as paving the way for Free Software at large.

The LilyPond world is no stranger to paid development: a special sponsoring webpage on our website allows you to pay people for their work on LilyPond, and you may have read from David Kastrup, the only developer currently listed on this page, as he recently asked you to join him towards a stable development funding model. How's it working for him so far? Read below to find out, and do not hesitate to join the plan!

Finally, another way of addressing the issue is, rather than asking for direct payment, to help our top contributors lead a comfortable enough career in music editing, writing or performing; a career that, in turn, will allow them to work on LilyPond as much as they'd like to. Mike Solomon, for example, is currently trying to raise money for his vocal ensemble on KickStarter; he has been trying to get this announced on our website but there is room for debate as to whether or not that kind of announcements should be featured prominently. In any case, there are a few days left for his fund-raiser, so feel free to chip in!

The road to 2.16

(by David Kastrup)

Development statistics

As the release of LilyPond 2.16 is looming closer and closer and closer, let us take a look at how the contributions to it are spread out. Running the following Gnuplot program

set terminal pdf font ",5"
 set output "contrib.pdf"
 set logscale y 2
 plot "< git shortlog -n -s release/2.14.2-1..origin" using :1:2 \
 with labels rotate by 45, "" using :1 with lines

leaves us with the following graphics right now:

We have 50 contributors and a rather straightforward logarithmic distribution of commits each, resulting in 2062 commits. If we cut the graph off after 37 contributors because of rounding issues (along with their commits), we arrive at the law

Solving for

as the expected number of commits from the most prolific contributor we arrive at

which numerically leaves us with . This is somewhat below the 336 we find in reality, but then the cutoff had been somewhat arbitrary. All in all, logarithmic law seems to match the reality quite well. If we take a look at the number of commits between 2.12.3-1 and 2.14.0-1, it turns out that we had 5242 commits there, so we are still quite below the last number of changes between stable releases for 2.16.

Critical issues

If we take a look at the stream of critical issues that are keeping us from releasing LilyPond 2.16, we see a mixture of several causes.

Detailed table (unfold)

regressions
ID	Opened	Owner	Summary
2532	May/13/2012		regression - hairpins begin/end in wrong places (incorrect spanner bounds)
2523	May/08/2012	dak_at_gnu.org	Patch: Display postevents on drum notes, rests and spacer rests
2505	Apr/28/2012	pkx166h	Patch: Doc: NR clarified \footnote command as a TextScript
2496	Apr/25/2012	mtsolo	Add announcements to the upper right corner of the website
2486	Apr/21/2012		beamed rests overestimate system-height
2484	Apr/17/2012	dak_at_gnu.org	Patch: Let pitched trills in articulations be transposed as well
2468	Apr/07/2012	k-ohara5a5a_at_oco.net	augmentation dot of a moved rest is placed on a line
2447	Mar/28/2012	mtsolo	Stems under beams no longer adjust for merged note heads
2446	Mar/28/2012	dak_at_gnu.org	iffy git commands in release checklist
2441	Mar/27/2012	mtsolo	Stems no longer adjust for merged note heads
2409	Mar/15/2012		Full measure rest staff position ignores \override for subsequent rests
2408	Mar/15/2012	Carl.D.Sorensen	Inconsistent automatic beaming when using multiple tuplets next to each other
2403	Mar/15/2012	dak_at_gnu.org	MultiMeasureRest's 'staff-position in RhythmicStaff is wrong per default
2399	Mar/14/2012	julien.rioux	website links to 2.13 docs for 'help us' points to a non-existent page
2397	Mar/14/2012	mtsolo	Tuplet number incorrectly placed when using additonal articulations and/or abreak
2386	Mar/10/2012	Carl.D.Sorensen	Inconsistent Beam Subdivision
2380	Mar/05/2012	dak_at_gnu.org	formatting of lyrics
2356	Feb/26/2012		Lilypond segfaults
2338	Feb/20/2012		OS X LilyPad not working
2337	Feb/20/2012	benko.pal	Endless Loop
2304	Feb/08/2012	dak_at_gnu.org	Patch: Fix barre snippet.
2301	Feb/07/2012	mtsolo	Patch: Fixes cross stem glissandi
2289	Feb/06/2012	dak_at_gnu.org	repeat percent with skips can crash
2287	Feb/05/2012		lyrics spaced too far from staff
2283	Feb/04/2012		Patch: Build: website should rely on $LILYPOND_GIT
2276	Feb/03/2012		multiple stanzas chord symbols collide with staff
2272	Feb/02/2012	mtsolo	Long monosyllabic words collide with barlines
2271	Feb/02/2012		2.15.27 won't open on Mac 10.4 PPC
2268	Feb/01/2012		Patch: Gets left Beam bounds correct with tempo indications
2267	Feb/01/2012		Patch: Gets left Hairpin bounds correct with tempo indications
2260	Jan/28/2012		line-broken spanners start too far to the right, with \tempo and \time
2257	Jan/27/2012		reg-test beam-shortened-lengths.ly fails
2256	Jan/27/2012	Carl.D.Sorensen	empty strings: tabStaff insists on stringnumbers and looses notes
2251	Jan/25/2012		website still doesn't auto-update
2244	Jan/22/2012		Beamlet orientation is wrong in compound meters
2243	Jan/22/2012	Carl.D.Sorensen	Incorrect subdivision of beams
2242	Jan/22/2012		Incorrect tabstaff fingering
2240	Jan/21/2012	dak_at_gnu.org	Patch: Don't wrap EventChord around rhythmic events by default.
2231	Jan/18/2012	mtsolo	Tuplet bracket makes space for dynamic but dynamic is placed underneath bracket
2228	Jan/16/2012	Carl.D.Sorensen	Some users prefer to minimize partial beams rather than beam by the beat
2224	Jan/15/2012	julien.rioux	LM: clickable examples in the docs have stopped working.
2222	Jan/15/2012	julien.rioux	lilypond-book fails with html input
2215	Jan/12/2012	Carl.D.Sorensen	lost commits jan 7
2205	Jan/09/2012	mtsolo	Breathing sign is positioned incorrectly after changing Staff or TabStaff size
2193	Jan/07/2012	mtsolo	collision between beams and flags
2180	Jan/05/2012	mtsolo	beams are detached from stems
2160	Dec/31/2011		document LILYPOND_WEB_MEDIA_GIT
2156	Dec/31/2011	PhilEHolmes	lost commits
2100	Dec/12/2011	Carl.D.Sorensen	Patch: CG: explanation of branches for the impatient
2094	Dec/10/2011	Carl.D.Sorensen	spacing-measure-length.ly regtest has time sig overlaying rest
2061	Nov/29/2011	mtsolo	Lyrics collide with bar lines
2031	Nov/08/2011	Carl.D.Sorensen	no lilydev available
1998	Oct/30/2011		GUB problem with curl, SSLv2_client_method
1997	Oct/30/2011	dak_at_gnu.org	segfault in tablature-negative-fret.ly
1975	Oct/18/2011	mtsolo	VerticalAxisGroup staff-staff-spacing gets insane spring distances
1971	Oct/16/2011		Slashed flags no longer work
1968	Oct/15/2011		test full GUB rebuild
1967	Oct/15/2011	dak_at_gnu.org	Concurrent slurs that start and end at the same moment should not produce a warning
1966	Oct/15/2011		win7 environment variable penetration
1948	Sep/30/2011		Windows install clobbered system PATH
1943	Sep/28/2011	mtsolo	lilypond after 2.15.8 fails on x86 Macs
1933	Sep/25/2011		Lilypond-book subprocess doesn't work on mingw
1929	Sep/25/2011	mtsolo	stem-tremolo-position.ly broken
1926	Sep/24/2011		2.15.12 processing speed problems
1919	Sep/23/2011		lilypond 2.15.12 fails on darwin-ppc
1903	Sep/18/2011		Page breaking does not work between different scores
1896	Sep/17/2011		lost instrument names
1892	Sep/16/2011		lilypond.org can't build website
1881	Sep/11/2011		Multiple cyclic dependency errors for Beam/Stem
1858	Aug/31/2011		unexplained increase in profile reports
1856	Aug/31/2011	k-ohara5a5a_at_oco.net	lots of extra space at beginning of bar
1832	Aug/23/2011	reinhold.kainhofer	make doc broken
1831	Aug/23/2011	v.villenave	no server for lilydev
1827	Aug/20/2011		GUB failure for darwin-x86::cross/gcc target
1821	Aug/15/2011	reinhold.kainhofer	GUB argument list too long
1491	Jan/25/2011		git-cl hangs when uploading patches due to bad mime-type

While there _are_ some changes wich did not make sense to postpone (2240 was one of those, and it has been responsible for several followup regressions), most critical issues nowadays are due to discovered regressions. Our release criteria require a two-week dormant phase in which no critical regressions are being discovered, where a _critical_ regression is defined as something that has worked already in either 2.14 or 2.12.

Other holdups have been the lack of support for MacOSX (that has actually caused quite a bit of delay until we found people willing to invest the work required for calling MacOSX a supported platform). A number of the regressions have been caused by developments of invasive nature: while the number of commits during the 2.14 development phase is not yet half that of those while 2.12 has been current, there have been lots of larger-scale projects.

2.14.0 has been tagged for release on June 5th 2011, about 11 months ago. 2.12.0, in contrast, was created on December 22nd 2008, 2.5 years earlier. So while the density of commits has slightly picked up in 2.14, we also have the situation that so many important changes have been created in that time that a new stable release is rather desirable.

So even while several developers have, for health or other reasons, mostly withdrawn from active development, development itself is still going forward. In fact, there were times when the limited human resources available for working on our development infrastructures were affecting the development to a degree where work morale and atmosphere were suffering.

But even while the quality assurance department is still not exactly overstaffed (volunteers welcome), we are currently in reasonably good shape.

However, both the depth as well as the breadth of the development that is happening make it less likely that the two-week window for not discovering new regressions will pass without occurrences. Discussions about changes to the release processes are slated to reopen in summer. Hopefully by that time we will not be under the pressure of still having to release 2.16.

Regressions, once discovered, tend to be fixed rather speedily. The above table does not show this, but the average life time of a newly discovered bug, in particular of critical nature, tends to be remarkably short.

Investor's report

(by David Kastrup)

So here are the operating results for my work on LilyPond in April, the second month where my work on LilyPond has been enabled by LilyPond users pitching in after my request for funding in February::

Onetime payments (€)	Payment plans (€)
2×100 80 50 2×25 22 19 10	100 25+50 3×25 4×10 4
Total:431	Total: 290
Totals: €721

The projections for next month are currently €290. There are some ameliorating circumstances for this month: I have been offered €200 if I write a proper bill in Germany for that, but I am currently checking my financial status with financial administration to make sure that I can make use of a regulation foregoing sales tax (this makes no net difference for corporate customers, but private `customers' can't reclaim sales tax, so it would be a nuisance if I had to actually cash and divert sales tax: it is already noticeable that about 5% evaporates through PayPal, and sales tax is about 4 times that). There has also been a payment of €200 from a supporter I had figured in for a monthly payment of €100, and I have counted this for next month until I hear otherwise. So one way or another, April has good chances to meet the minimal target of €800 yet that I need for treading water financially.

While one-time payments have declined somewhat, some more people pitched in with monthly payments for several months (3 to 12 months). A surprisingly large ratio of one or more-time contributors have not committed to regular plans because they don't feel that their own financial/job situation allows them to plan ahead that far. I was reminded of Steinbeck's ``Grapes of Wrath'' where Ma Joad, after getting credit in a store from a clerk rather than the store, says "`I'm learnin' one thing good,' she said. `Learnin' it all a time, ever' day. If you're in trouble or hurt or need— go to poor people. They're the only ones that'll help— the only ones.'" Of course, the analogy is not all that fitting since I am appealing to those who have a fortune, namely that of being able to feel excited about a project like LilyPond.

Now what has been accomplished? It turns out that April has been slow not just for me as

git shortlog -n --since "1 month ago"

shows. There is probably some exhaustion involved as we are still gearing up for the release of 2.16 which has seen a stream of last-minute regressions being discovered on the brink of release. But while in line with a general decrease of activities, the list of changes checked in and projects tackled by me still appears embarrassingly small compared to that of last month.

Now like last month, I have been alone in running the regression testing most of the month despite of my comparatively slow setup (James has just now taken over, a development I am not altogether happy with since he already does a lot of grunt work). But while taking a week off for my yearly Easter climbing trip did not help, the trouble actually started before that and is still on:

I was trying to tackle a task involving adding new event types to LilyPond, and I figured out that the current sketchy support for this in LilyPond was broken by concept. So I had to redesign the way event classes are supported in order to stop the existing code example and interface from doing the equivalent of modifying constants. I had to move things like the event class hierarchy into dynamic, per-context data structures, and while it was workable, it was not really a pretty fit. And when things are not pretty and I don't like what I have to program, I get slow and ineffective because I spend more time brooding than coding.

You can find most of the accomplished work in the bug tracker connected with issue 2449 what I arrived here fits with LilyPond's architecture logically (and doing so already ironed out a number of user interface aggravations in connection with fingering indications and string numbers and closed a few long-standing bugs) but while what I have been doing does no longer clash with the current architecture of LilyPond, fitting it as a sort of in-between between LilyPond's grob properties and context properties did not feel good, and an incommensurate amount of brooding and scribbling and bothered sleep was spent on what amounts to a comparatively small amount of actual code.

As a result, I am now working on issue 2507 (and am in the design phase, so again there is little visible output yet) in order to tie context and and user-settable grob properties (which are implemented as an incompatible special case of context properties) into a coherent and user-extensible logical whole, and then fit user-extensible events and grobs and similar entities into that whole.

I can't really think of immediately visible end user visible benefits. But things should get quite more logical for people willing to program extensions for LilyPond, and it will be good to be able to simplify and streamline the documentation sections concerning grob and context properties when I am done, and get rid of a lot of handwaving in the process.

One consequence should be that

\set SomeProperty = value \unset SomeProperty

becomes equivalent to

\override SomeProperty #'() = value \revert SomeProperty #'()

namely let the grob property changing commands (overrides) become equivalent to alist changes of a context property. While the implementation of grob and general context properties at the current point of time is somewhat shared, it is in such a half-baked manner that crashes will ensue if you try interchanging them.

Having a single coherent model for structured per-context data will make it much easier to design and work with extensions, and a redesign of this area will address several long-standing bugs in connection with nested grob properties.

The actual task I set out to do originally (implementing a way to typeset multiple marks at the same place, still pending) is still not finished because of my dislike of having to pick an ugly way for implementing it, and because I am now redoing the landscape in order to make for a nicer path.

And while I feel bad about the sluggish progress I am making right now, this is actually about redesigning fundamental parts of LilyPond that have been taken for granted (or beyond improvement?) for very long and that nobody else feels like touching. And it will not just benefit the particular extension I had set out to do but make a number of things, not just for me, more straightforward.

This sort of ``too much, too late'' effect where I take far too long for doing far more than planned is why task-independent funding makes good use of my skills and `programmer personality': while it is certainly nice when I manage to tackle a lot of small and medium-size things (like in March) that would take quite longer without my involvement, tackling a few big roadblocks that would not be moved at all otherwise is also important.

What are the outlooks for May? Partly due to the recent slow rate of change, I consider it likely that we'll manage to release version 2.16 after all. It is high time for getting a lot of improvements into everybody's hands in the form of a stable release. Adapting LilyPond to using Guile 2.0 has not moved forward in April, and this issue of compatibility and forward-going is likely the next topic of general interest after LilyPond 2.16 has been released.

With regard to contributors, actually not just for my own finances but for the various workloads of the whole project: I'd like to see the load distributed over more shoulders. Maybe it is to be expected that most of the large contributions come from people who can be seen to have a large piece of their heart in the project. But I am sure that there are many users for which LilyPond is useful and for which some progress is a little important to them, and if those would decide to contribute regularly a little, both in the project itself as well as supporting my ability to continue focusing on it fully, I think this could make quite a difference.

My thanks to all those who contributed to LilyPond in April in either manner, and those who continue doing so.
David Kastrup.

Compiler adventures, episode 1

(by David Kastrup)

With the release of Ubuntu 11.10, the regtest suite stopped working when compiling an optimized version of LilyPond. The initial report was in October. The test in question, tablature-negative-fret.ly, displayed a warning "Requested string for pitch requires negative fret: string 1 pitch #" (quite as intended) and then crashed with a segmentation fault. So naturally the first suspect was that it did not properly deal with the condition that the warning was about.


segfault backtrace

The backtrace started with

#0 Grob::internal_set_property (this=0x1a, sym=0xb607cb90, v=0x1a)
 at /home/gperciva/src/lilypond/lily/grob-property.cc:112
 #1 0x082c0c0b in Engraver_dispatch_list::apply (this=0x85d08e8, gi=...)
 at /home/gperciva/src/lilypond/lily/translator-dispatch-list.cc:35

Which was kind of a bummer: while it was obvious that Grob::internal_set_property was being called with an invalid value for "this", the displayed caller, Engraver_dispatch_list::apply, did not actually contain a call to this property. The function in its entirety looks like

void
 Engraver_dispatch_list::apply (Grob_info gi)
 {
 Translator *origin = gi.origin_translator ();
 for (vsize i = 0; i < dispatch_entries_.size (); i++)
 {
 Engraver_dispatch_entry const &e (dispatch_entries_[i]);
 if (e.engraver_ == origin)
 continue;
 
 (*e.function_) (e.engraver_, gi);
 }
 }

So the problem apparently was with the function called via the function pointer "e.function_". Unfortunately, both e and gi had been optimized out and were not available for debugging at the point where the problem occured. To add insult to injury, "dispatch_entries_" is a C++ vector, and asking the debugger to display any member of it resulted in the debugger complaining that "operator []" was not available for calling: all of its occurences in the code had been compiled right into its surroundings, and the compiler had not kept a separate copy around for the sake of calling from the debugger.

Now this is not the end of the world: the compiler we use, GCC, has a gazillion options for about every conceivable purpose, and indeed there is one for keeping a copy of inlined functions for the sake of debugging or whatever else:

-fkeep-inline-functions
In C, emit `static' functions that are declared `inline' into the
object file, even if the function has been inlined into all of its
callers. This switch does not affect functions using the `extern
inline' extension in GNU C90. In C++, emit any and all inline
functions into the object file.

To my dismay, the crash disappeared (to date, I don't see a good reason for this). So while debugging became somewhat more feasible, there was no bug left to analyze!

After running the problem through Valgrind, a memory analyzer, we had a lot of inconclusive data. The memory analysis was reasonable: most bugs of that kind tend to be caused by Guile's garbage collector reusing memory locations that were still being used for a different purpose, with the most likely reason being a programmer forgetting to properly protect it while it was still needed.

About a month later, I boiled this down to a minimal example still crashing. It still looked like an unitialized variable handling error by a programmer at first.

I finally got a handle on the problem while trying various things with the debugger gdb: I noticed that it did not just have a command for executing a program step by step, but that there was a way to step _backwards_. One needed to use a few special commands (basically, something like "set target record"), and then gdb would, while eating lots of time and memory, record a history of states. Stepping backwards from the crash, I finally discovered that the actual function that had crashed was not even visible in the backtrace. Now being able to step backwards, I finally got to the location where the trouble occured, and the trouble did not make sense.

It crashed inside of a function that, judging from the function arguments, was not really supposed to crash. Since I could not pinpoint why and where stuff started going wrong, I suspected the debugger of not telling the whole truth and switched the single-step display to (disassembled) machine language rather than C++, to find out the exact instruction where things went wrong.

And it was then that I discovered that the machine language did not quite correspond to the C++ source: the compiler had miscompiled the C++ program.

You can see the further progress of work on the bug by continuing to read the bug report I reported the bug to GCC development and followed up with a boiled down example. The problem was picked up and handled rather fast: apparently Jakub Jelinek had worked on the particular part of the compiler anyway, and thus was able to recognize and test the bug rather fast, creating a relevant minimal test case (which demonstrated the problem in C, meaning that it affected all programming languages) on his own.

The bug has been fixed as of GCC-4.6.3 or 4.7.0. In contrast to this bug which occurred under rather limited circumstances and mostly on the 32bit Intel platforms, the bug Pavel uncovered recently in GCC-4.7.0 (more on that below) occurs on all platforms, and under quite more common circumstances in C++. I consider it way more scary.

Having gotten used to the idea that sometimes the computer can be wrong, I was able to provide feedback, guesses and suggestions when Pavel was seeing a similar kind of problem recently.

With experience and teamwork, the bug was handled much more thoroughly and lots faster than the one I described here, and the GCC developers were provided with decidedly better input. So while we were not lucky to have a fix in the queue already (that just needed to get accepted more speedily and backported to 4.6) this time, the bug turnaround time was still much faster and led to a fix within days. Frankly, I hope that the fix will get picked into the upcoming Fedora's compiler: I am scared by the recent one more than by the one I tracked down previously.

Compiler adventures, episode 2

How I made the world a better place
(by Pavel Roskin)

I installed Fedora 17 beta on one of my systems. I tried the version of Lilypond that was included with the distro. It was version 2.15.29 at the time when version 2.15.37 was current.

Actually, I would be fine with Fedora 17 shipping Lilypond 2.14.2 until 2.16 is released. But it turned out that the maintainer had a minor compile problem with 2.14.2 and switched to the development branch, but kept updating only versions for RawHide (the rolling unstable Fedora) leaving Fedora 17 with an old development version.

I tried the stock Lilypond with my Lilypond files and found that it would give bogus barcheck errors.

So I checked out the latest Lilypond code from git repository. Imagine my surprise when I got the same barcheck errors! I tried Lilypond 2.14.2 fixing the compile issue and sure enough, the barcheck errors were still there.

It was clear that something was wrong with the system. I reduced the piece to a short example. Barchecks were gone, but there was another error message about "moving backwards in time". The PDF output was obviously wrong with an extra measure appearing at the end.

Since Fedora 17 came with a brand new version of gcc (4.7.0), the compiler was an obvious suspect. I tried compiling with the optimisation flag "-O1" instead of "-O2", and sure enough, it fixed Lilypond.

I posted my finding to the Lilypond mailing list and got some valuable suggestions. But most importantly, I found that the issue is of great interest to the developers, which was very encouraging.

Everything from this point was done in cooperation with Lilypond and gcc developers and could be seen in the mailing list archives and bug tracking systems.

I started looking for the miscompiled file. My initial approach was to write a script that would compile N files with -O2 and then switch to -O1. That approach turned out the be ineffective and slow, as I would have to recompile everything every time and I would not be able to use parallel build.

The approach that worked was to remove some object files, recompile them with different flags and see if that would change the behavior. That's how I found the problematic file, simultaneous-music-iterator.cc.

Then I looked for the function suffering from the gcc bug. I copied the contents of simultaneous-music-iterator.cc into another file. I would comment some functions in the original file. The remaining functions were commented in the other file. That way, I found the problematic function, Simultaneous_music_iterator::pending_moment().

The next step was to find which optimization caused the problem. The "-fverbose-asm" option to gcc reported all optimization flags enabled for compilation. I quickly found that "-fno-tree-vrp" was fixing the problem. At this point, I had a working patch, so I submitted it as an issue.

As a result of the discussion in the Lilypond bug tracker, I actually had to write a script and test all other optimization options that would fix Lilypond. "-fno-inline" was the only other option, but it was obviously much more intrusive than "-fno-tree-vrp".

David Kastrup asked me to post the assembly code so that he would look for problems in the generated code. I posted the x86_64 code. David asked me to post i386 code. It turned out that some i386 libraries needed by Lilypond are not available in x86_64 Fedora. So I installed Fedora 17 for i386 on a spare SSD.

But at that point David has already found the problem. The compiler output in the broken case wasn't interested in the result of comparison used in the min() function.

I wrote a wrapper around min() that would fix LilyPond. But it wasn't an acceptable fix. What if there other calls to min() that gcc miscompiles? So the solution was to give the "-fno-tree-vrp" flag to gcc 4.7.0 and report the bug to the GCC developers so that the next version of GCC would not have to be worked around.

I opened an issue in the GCC bug tracker. It had the preprocessed version of simultaneous-music-iterator.cc with everything but the problematic function commented out. I mentioned the findings of David Kastrup about the assemble output.

GCC developers reacted quickly and confirmed that it was a bug. However, they wanted a self-contained test. LilyPond was too big to be such a test.

I wrote a new main() function that would call Simultaneous_music_iterator::pending_moment() with the same arguments as it would happen in LilyPond processing the problematic LilyPond file. I replaced a linked list with an array for simplicity. The result of the function would be printed on stdout.

It's a little embarrassing to admit that the program was printing "denominator/numerator" rather that the other way around. I can only tell in my defense that I attended school not in an English-speaking country. Anyway, what mattered was that the output would depend on the compile flags.

I wrote a bash script that would compile and link the files the way how Lilypond build system would do it. I found a small set of object files that could be linked into a working executable after their sources were stripped of unneeded code.

I was happy to see that the program output was different with -O1 and -O2 optimization. The next step was to copy the needed sources to a separate directory together with the compile script.

I put the new directory under git control and would commit every change that would show the "interesting" behavior. I didn't waste any time on descriptions, I would just do

git commit -a -m "simplify"

I made a few wrong commits, but I was able to revert them quickly.

I cleaned everything I could, removing files, merging them, expanding macros and just trying to remove almost every line. Special thanks go to astyle, a program I used to re-indent C++ code after every round of simplification. That's how I came to a short example that I posted to the GCC bug tracker.

The GCC developers made a fix quickly. It is slated to appear in gcc 4.7.1. I wrote about it in the Lilypond bug tracker, and my fix was committed shortly.

It took me a lot of time, but I'm glad that Fedora 17 would have a working version of Lilypond and GCC would be fixed too. GCC bugs are scary. So much code is compiled with it, including the Linux kernel. That's how I made the world a less scary place

LilyPond output analysis

(by Janek Warchoł)

Some of the readers may be familiar with my articles in the previous LilyPond Report, as well as my multi-issue Lyrics Bug Report. Here I would like to present a detailed examination of an exemplary LilyPond output — the viola part from Mozart's Requiem in d-minor (first two movements — Introit and Kyrie).

The International Music Score Library Project hosts a scan of a hand-engraved edition published by Breitkopf & Hartel at the end of 19th century, that we'll use as a reference throughout this article. Will the LilyPond-made score stand up to this print made by a well-known publisher?


K626, viola part - Breitkopf and Hartel: (click to view in full-size)

You may notice that this score is set very, very tightly: it was common practice in 19th century to use space available on page to the maximum, aiming for scores set on as few pages as possible. In fact, judging by today's standards it's engraved way too tight; no modern publisher sets music so densely — but nevertheless Breitkopf's score remains legible.

Let's see what will happen when we use LilyPond to engrave this piece. We will begin our analysis with 100% default settings — just what you'd get after compiling with LilyPond version 2.15.36 (latests as of this writing) a lilypond file containing just the music that Mozart wrote, without any layout tweaks.

If you have a decent printer at hand i recommend printing both scores and comparing them on paper — resolution of computer screens is limited and therefore some inconsistencies in display appear (for example lines that have the same thickness appear to be different).

Here's what LilyPond gives us without any manual intervention:


K626 viola - default LilyPond output: (click to view in full-size)

[click here to see it in an embedded viewer]

click here to see the source code

\version "2.15.33"
 \pointAndClickOff
 
 \header {
 title = "Requiem d-minor K626"
 subtitle = "I. Introit et Kyrie"
 composer = "W. A. Mozart"
 instrument = "Viola"
 }
 
 \score {
 \relative c {
 \key d \minor
 \tempo Adagio
 \time 4/4
 \clef C
 
 r8 f-.\p r a-. r g-. r bes-.
 r8 a-. r a-. r a-. r b-.
 r8 c r d r c r d
 r8 d r a r a r b
 r8 f' r f r e r a
 r8 f r f r e r d
 r8 c r g'\f r f r e
 \mark \default
 r8 d r d r d r cis
 d8 r e r f r e r
 d8 r e r f e d g
 c,4 d g, c
 f4 bes8 a g f16 e a8 g
 f8 e d cis d4 c
 bes4 bes' a8 r g r
 \mark \default
 f4 r r16 f( a) f-. c-. f-. a,-. c-.
 f,4 r r16 f'( a) f-. c-. f-. a,-. c-.
 es8 r d r r c'\p r bes
 r8 g, r g r bes r a
 bes8 bes r bes r c r c
 bes4 a bes8( g es f)
 bes4 r r8 d~ ( d16 es d c)
 bes16( a g f) es4 f r8 f'~(
 f16 g f es) d( es d c) bes ( c' bes a g a g f)
 e!4( f) r2
 r8 d~ ( d16 es d c bes a bes c) d4~
 \mark \default
 d4 r16. bes32 \f g'16. es32 f4 r16. a32 c16. a32
 f4 r16. d32 f16. d32 a'4 r16. fis32 a16. fis32
 bes4 r16. bes,32 d16. bes32 es8 f4 es8
 d4 r16. d32 f16. d32 bes4 r16. bes32 d16. bes32
 g4 r16. g'32 bes16. g32 g,4 r16. es'32 g,16. g'32
 fis8[ r16. fis32] e!8[ r16. e32] d8 d d d
 g,8 g\p r bes r a r a
 g4( fis) g8( es c d)
 \mark \default
 g8 r bes\f r a r a r
 d8 r e! r f! r d r
 e,8 r e r a4 b
 c4 d~d8 d4 d8~
 d8 d~( d16 es d c ) bes ( d c bes) a( c bes a)
 g8.( a16) bes( a) g( bes) a( g) f( g) a( g) a( f)
 c'8 c, c'2 e4
 f4 a,8 a a4 cis
 d4 f,8 f f4 a
 \mark \default
 bes4 r r16 bes'( d) bes-. f-. d-. bes-. d-.
 a4 r r16 d'( f) d-. a-. f-. d-. f-.
 gis,8 b' b b b b b b
 a8 a, r cis \p d4 e
 d4 a f( e8 d)
 a'2 a\fermata
 \bar "||" \tempo Allegro \time 4/4 
 R1*4
 r2 r8 b\f b b
 c16 d c b c d b c d e d c d e c d
 e8 f16 e d c b a gis8 b e d
 \mark \default
 c8 cis d e a, d, g! f
 e8 g bes4 a r
 R1
 r2 d4. d8
 c4 f gis,4. gis8
 \mark \default
 a4 r8 a b4. a16 b
 c8 b16 c d8 c16 d e8 d c b
 a16( b c4) cis8 d d, e fis
 g16( a bes4) b8 c! bes a g
 a16 f g a bes g a bes c8 g c4~
 c8 bes16 a bes c a bes g8 a16 bes c4
 R1
 \mark \default
 d4. d8 bes4 es
 fis,4. fis8 g4 r8 g
 a4. g16 a bes8 a16 bes c8 bes16 c
 d8 c bes a g f' es b
 c8 g' f16 es d c d c b a g as f g
 \mark \default
 es8 c as'4. f8 bes4~
 bes4 as g8 a16 b c8 d
 es16 ( d c bes a8) bes4 a8( bes c)
 d4 r r8 f, f f
 g16 a g f g a f g a bes a g a bes g a
 bes4 r r8 c c c
 \mark \default
 d16 es d c d es c d es f es d es f d es
 f8 d e! f g4 f8 e
 f8 f, bes g c4 r
 r2 r8 g g g
 as16 bes as g a bes g a bes c bes a b c a b
 \mark \default
 c8 g c( bes!) a d, d'( c)
 bes4 r r8 g g a
 a4 a8. a16 b4 b8. b16
 cis8 a( b cis) d4. d8
 \mark \default
 e4 a,4. f8 bes4~
 bes8 g e' e d cis16 d e f g f
 e8 d cis8. d16 e4 a,8 cis
 d8( f e d) e a,( b cis)
 d8 d,( g e) a( bes! a g)
 \mark \default
 f8 d g( f) e c a' a
 f4 d' g, r
 r8 a d( c) b b e( d)
 c4 d8. d16 d4 e8. e16
 e4 d8( f) e4. e8
 f2 r4\fermata \tempo Adagio f8. f16
 e4 f2 e4
 d1 \bar "|."
 }
 \layout { }
 }

We instantly recognize that LilyPond decided to set the music much more loosely: the original fits one page (15 systems), while Lily's version is 1,5 pages long (21 systems). I personally find it a bit too "airy". While you don't have to agree with my opinion on this matter, my goal in engraving this piece was specifically to demonstrate that LilyPond can produce music set as tightly as the Breitkopf edition (one of the violists in our orchestra said that she much prefers tightly set scores). Also, sometimes there is no other choice than to set music tightly - for example when optimal page breaking requires putting a lot of music on one page, or when measures are so long and full of notes that they simply need serious squeezing to fit on the page at all.

Thus, let's change the overall horizontal spacing to see how Lily will handle it. Some of the problems in engravings below will appear exactly because of this decision, but generally condensing the music doesn't change much in program's behaviour — most things you can also find in the uncondensed version.

To change horizontal spacing, I use the method mentioned in the manuals:
\override Score.SpacingSpanner #'common-shortest-duration = #(ly:make-moment 1 4 )
Here are the results:

[embedded viewer]

That's definitely tighter than the default output — I quite like it! However, have a look at letter E, measures 49-52 (after the tempo change): these are four measures, each filled with rests. What for? That's a waste of space; they should be typeset as a MultiMeasure rest. Of course, it's easy to add a \compressFullBarRests command, but I don't quite get the point why this isn't the default behavior. I don't recall any score in which the rests weren't compressed...

Going back to the horizontal spacing: the result is now nicer, but still not as tight as the Breitkopf engraving. Let's try more:
\override Score.SpacingSpanner #'common-shortest-duration = #(ly:make-moment 1 1 )

[embedded viewer]

What's that? Nothing changed... or did it?

Yes, there are changes, but not for the best at all — the spacing went all weird, see the markings here:

[embedded viewer]

(I suggest you to do an Alt-Tab comparison, i.e. open both files in full size and switch between the windows (or browser tabs) using Alt-Tab or whatever shortcut your operating system uses)

I find this behavior very strange, not only because it results in a clearly ugly spacing, but also because obviously there is still a lot of whitespace left between the notes, so it *should be possible* to actually do some shrinking. For example, 16ths are separated with about 0.5 staffspace, while in hand-engraved scores it is not uncommon for short notes to almost touch each other. Consider the fourth system: there are twenty-four 16th notes in it. Reducing the space between them to 0.1 staffspace — that's close, but still legible — would be enough to add another measure to that system (without changing anything about longer notes' spacing)! The situation is very similar in systems 10 and 14.

Ok, so how to achieve spacing as tight as in Breitkopf edition? I've tried forcing the number of systems, but this resulted in music overflowing the page in the last system. Finally i managed to do this by reducing font size to 18 and forcing page-count to 1, but it's not exactly what i was looking for.

[embedded viewer]

As you can see, this setting revealed some problems that were not instantly recognizable in the previous versions — for example the piano in measure 46 and the F rehearsal mark seemed previously okay, but here it is obvious that they stick out too much. If you look closely at the hand-engraved edition, you'll see that the F rehearsal mark is not perfectly centered on the barline, but shifted a bit to the right, to allow placing it closer to the staffline. Dynamic stickout is greatly reduced in Breitkopf's edition by moving the dynamic horizontally — see measure 26 for an example. This isn't hard to fix in LilyPond — but it seems possible to automate such decisions.

Nevertheless, as I've said; the Breitkopf edition is set too tight, so let's go back to the Lily version created by overriding the common shortest duration to 1/4 — it is definitely the best. Time to examine it closely.

[embedded viewer]

The first thing that drew my attention was bes32\f g'16. es32 figure in measure 26: Lily output is very "airy" compared to the engraved version:

That's because the distances between notes, augmentation dots and accidentals are much smaller in Breitkopf score. LilyPond's distances are quite good for "normal" scores, but they stay the same even when notes are placed very close to each other, leading to poor results in tight scores - see also issue 2142 in our tracker (it is about accidental spacing, but the problem is really similar in case of any other object).

So, I've overriden the spacing properties of dots and accidentals to have these objects closer to notes:

[embedded viewer]

Unfortunately, this solution isn't perfect — while flats and sharps are positioned quite nicely, the naturals are definitely too close when the notes are down-stemmed:

To achieve a pleasing result one would have to write a function that would apply a different right-padding depending on the type of accidental and stem direction. While that's doable, I think it would make more sense to write a "true & complete" fix for issue 2142 to get rid of this problem once and for all.

Going back to note spacing, all 16th notes seem to get too much space, at least compared to longer notes: there is little difference between 8ths and 16ths in this regard, unlike in the Breitkopf score:

There are more issues present, marked in the pdf below. For greater clarity, only one instance of a problem is marked in color (additional places where an issue appears are marked in gray).

When you hover your mouse over the markings, you should see an explanation text. I know that this feature works for me when i open pdfs using Adobe Reader (for example the one bundled inside Firefox). If you don't see explanation texts, please download the pdf and try some external viewer (for example Okular).

: (click to view)

[embedded viewer]

That concludes my analysis. For an almost-default output, the results of LilyPond work are quite impressive — but they're not a publication-quality engraving yet: there are noticeable flaws. Most of them concern note spacing and beaming, some are caused by lack of intelligent and flexible placement of objects like dynamics and rehearsal marks. The most visible things are easy to fix manually - after all, no computer program will be smart enough to engrave everything perfectly. However, all the problems present seem to be fixable (in other words, it should be possible to write an algorithm that would solve them).

There surely is lots of room for our Development Team to improve things (and we encourage you to help us!)

What are your opinions? I invite you to discuss this engraving - either in comments here, or on LilyPond development mailing list.

Post Scriptum: wrong positioning of the augmentation dot in measure 31 was fixed in LilyPond's version 2.15.38.

Janek Warchoł, april 2012.

The snippet of the month

(by Valentin Villenave)

Anyone who has been discovering and using LilyPond in the past decade will tell you about the wonder and bewilderment that we encounter when we compile our first scores and discover the beautiful results. And yet, anyone who has been using LilyPond a tad bit longer, will also testify to the incredibly annoying limitations we're all bound to stumble upon, sooner or later.

Of all these bugs, one that annoys me the most is the way LilyPond writes tuplets: it can't handle slur-shaped tuplet brackets, it tends to be dumb when placing its tuplet brackets, and most of all it has absolutely no idea where to place cross-staff tuplets, which may result in ugly collisions.

Which is why I was thrilled to recently (re)discover this snippet, that does make things look (a little) better.

Here's the code, I'm not sure who is to thank for this snippet but it's well written and nicely commented.

Positioning tuplet numbers close to kneed beams

The (undocumented) feature of the month

(by Valentin Villenave)

It is not uncommon for some advanced LilyPonders, to use invisible "anchors" in their music. These can be useful for adding text above full-measure rests...

\relative c'{ c1 s1*0^"hello" R1 c1 s1*0^"there" R1*2 c1 }

to end crescendos more elegantly...

\relative c'{ d2\> c\! d2\>^"better:" c s1*0\! }

or to integrate articulations and texts with music stored in a variable:

notes = \relative c'{ c4 c g' g c,1 } \score { \new Voice { s1*0-> \notes << { s1 s1*0\fermata} { \oneVoice \notes } >> } }

However, this syntax has a pitfall: using an invisible rest with a null duration, effectively changes the "last-used-duration" for LilyPond, and therefore can play tricks with your score depending on how you use it (see below).

Earlier this month, David Kastrup noticed that there actually always had been an other way to enter an invisible anchor in LilyPond: the empty chord construct, noted as <>. He went on to explain how that might prove to be less confusing for users:

Quick: tell me what you would expect without too much thinking (imagine you are a naive user) from the following:

\new Staff << \relative c'' { c4 d e f s1*0-\markup Oops c d e f g1 } \\ \relative c' { c4 d e f <>-\markup Wow c d e f g1 } >>

Whilst he definitely had a point there, and nobody is really happy with s1*0, some contributors disagreed with the idea of somehow "officially" advertising the <> syntax in our documentation, to the point where names were called (Haskell, really? That's a bit harsh.) and people appeared to be on the verge of leaving the project altogether! If we had not known this already, it again shows that LilyPond developers can be quite passionate about details.

As for me, all I can see is the awesomeness of discovering an undocumented feature that has been right there under our noses for 16 years. That alone, without any parser modification needed or whatever, is terrifically cool from a pure geek point of view. (To what David accurately told me: "Well, that was part of the problem. It was divisive between geeks and non-geeks." And LilyPond is, and must remain, meant for both types of people.)

Then the question remains: is <> elegant and "LilyPond-ish" enough to get a prominent place in our manuals? I'd say it is, but history has proven there's room for ~~flamewar~~ debate here. Whether we end up including it, implementing a new keyword (either \null or, to avoid confusion with the markup command of the same name, something else, or even a new letter such as z) which would require a parser modification, or — much more likely — nothing at all and just stick with s1*0 until someone resuscitate the issue in a dozen years, it really doesn't matter that much. Invisible anchors are really not used everyday for an average LilyPonder; and when they are, I suspect whoever uses them is already advanced enough to be able to cope with whatever is thrown at her, be it tricky null durations, new parser keywords/letters or weird Haskell-like constructs.

The closing word for now will be this comment from Janek, as I was telling him how hard reading the thread made me laugh: "I can't believe you guys went all flaming for such a silly thing!" — "Well, he replied, I couldn't believe that you quit LilyPond for a few months last year because of a silly mailing list question ."

Oh, well. Touché.

Here endeth the twenty-sixth issue of the LilyPond Report. With just a bit of luck, our next installment will be released after the upcoming LilyPond stable release, and there will be a lot to report vis-à-vis brand new exciting work being started...
In the meantime, feel free to send us your contributions !

Cheers,
David Kastrup, Janek Warchoł & Valentin Villenave

The LilyPond Report #25

David Kastrup, Janek Warchoł, Valentin Villenave — 2012-04-01T16:33:26Z

Table of contents

Greetings everybody, and welcome to this twenty-fifth issue of the LilyPond Report!

As promised, this month's issue features an interview with our fellow contributor Janek Warchoł. As well as an article by Janek Warchoł. And another article from Janek Warchoł. In other news, Janek Warchoł reports on some recent developments. Also to be found in this issue is a personal appeal from Janek Warchoł; and for a touch of fun, don't miss our special section, written this month by — wait for it — Janek Warchoł. Now, there _is_ other stuff too. But rest assured everything has been validated and approved by Janek Warchoł.
As always, you can post your comments at the bottom of the page, or send your contributions to the LilyPond Report's next issues.

Editorial

by David Kastrup.


Bing Crosby

Since my recent call for funding has not quite met the resonance I had hoped for, I went looking for alternate means of financing. I am glad to announce that I have been able to secure a combined grant from Microsoft, the RIAA, and the Hohner company for the purpose of typesetting the complete works of Bing Crosby for accordion.

To commemorate the passing away of Bing, the release is planned for October 14th, so it is not likely I'll be able to work on much more than accordion support of LilyPond in the next half year.

Since I will not be able to do all of the heavy lifting unaided, I would ask all developers concerned about LilyPond's well-being to please acquire an accordion and get acquainted with the local notation: the project calls for producing Russian, American, German, Italian, and French accordion chord notation from the same sources.


Embrace ze future of LilyPond

For later releases, a bandonion edition in the "Waschleinensystem" might get executed as well. Styrian harmonica Griffschrift will not be supported natively, unless Arnold Schwarzenegger can be interested in joining the project.

News from the 'pond

by Valentin Villenave.

Whilst our upcoming 2.16 stable release is getting nearer by the hour, its development cycle is not quite over yet. Our current development release is 2.15.35; a few Critical issues remain, particularly with note-heads, and therefore this is not our next (and maybe last?) "release candidate".

In other news, a new LilyPond-auto mailing list has been announced by Graham Percival, that is not intended for human posters, but... for robots and automated systems! If you're at all interested in what our bug-tracker or patch revision system has to say, this is the place to be. (Beware: they can get chatty at times.)

Finally, for those of us who use FreeBSD, LilyPond's development version is now available as a "port", in a more usual way than the self-contained binaries we distribute via our website. That's a nice thought!

Here's an addition by Janek Warchoł.

As some of you may know, we applied to Google Summer of code program (Google pays students stipends for working on open source programs).

Our application was rejected, probably because we aren't a big enough project (indeed we're quite small compared to some accepted organizations). However, LilyPond is a part of the GNU project - and the GNU project was accepted as a GSoC mentoring organization, so we "joined the GNU umbrella" together with a score other projects. Of course you can't expect that Google will give you everything you want — in particular, there will be a limited number of students funded by Google. No-one knows how many of these "student slots" will GNU receive - will it be more like 5 or 25? We also don't know how many of GNU's slots will LilyPond receive, but they're worth fighting for. If you are a student, here's your last chance. Go read GSoC FAQ and see the guidelines for applying to GNU projects. Student application deadline is on April 6th, 19:00 UTC. I'm going to submit two applications (i'll discuss them on -devel mailing list first, please give me your feedback!) - wish me luck!

The recipe of the month

A new section in this Report: cooking! This month's recipe was noticed by... you'll never guess.

Without further adieu: Frogs in the Lily Pond!

3 x 85g packets green jelly
12 large jelly frogs or 24 small frogs
Flower sweets

Make the jelly according to instructions on the packet and pour half of it evenly between 12 small wide glasses. Place glasses in the fridge until the jelly is almost set. Place 1 or 2 frogs on top of each jelly and top up glasses with remaining jelly. Refrigerate until set. Decorate with small flowers.

An Interview with Janek Warchoł

presented by Valentin Villenave.

The LilyPond Report — Greetings Janek, thanks for lending us a bit of your time today! Well, this will be my first question actually: is time a rare and valuable resource to you, or is time something you can spend happily doing anything you like, something you can afford giving to anyone asking for your help? How busy is your life these days?


Janek's self-portrait: No doubt: LilyPond can rely on serious people. (And good-looking, at that.)

Janek Warchoł — Hi Valentin, and greetings to everyone!
I am very busy these days. My engineering education (is that's how it's called in English? The education system in US/UK is quite different from the Polish one and I always have trouble translating these terms...) does take some amount of time, and I have a lot of other activities. The day would have to be at least 41.168 hours long to enable me to "spend my time freely"
In the case of LilyPond, I'm definitely spending more time on it than i can afford... But I hope it will pay back in the future.

TLR — "Pay back"? Do you mean something precise by that or is it just a figure of speech?

JW — I've already learned a lot about programming, learned how to use git. I also doubt I'd have many opportunities to work on international, over-a-million-lines-of-code project if it wasn't for Free Software.
It's also possible that I'll become a music engraver, who knows. I've already achieved one paid project with LilyPond (apart from preparing some scores for my choir).

TLR — I'm sure our readers would be interested in knowing where you grew up and where you live at the moment.

JW — Sure. I'm 8699 days old (as of April 1st, 2012); I was born, grew up and live in Warsaw — capital of Poland. My city isn't very interesting, as most big cities... Especially that Nazi Germans wiped out 80% of it during World War II, so we (I mean Poles) had to start from scratch — under the supervision of totalitarian USSR. Not nice.


A view from Warsaw: "not nice" ?

TLR — Indeed. So you're preparing to become an engineer? What kind of?

JW — Mechatronics engineer. It's a multidisciplinary field combining mechanics, electronics and computers; an example of a mechatronics project is a camera autofocus system. However, it is quite common for mechatronic students to work as programmers; I've just started, so i don't know yet how my career will proceed.

TLR — Does music take an important place in your life (on a symbolic level or just the amount of time)? What kind of music? (I'm guessing "written", but I could be wrong )

JW — Of course! I'm involved in four kinds of musical activity:

listening to music — communing with beauty
performing music — a way to express myself
engraving music — creating artworks
programming LilyPond — a brain-twisting exercise! (actually, it's not THAT brain-twisting — don't be afraid to try yourself!)

What do I listen to? Classical music (although I have some very strong and unconventional opinions here) and metal. The latter isn't precisely about beauty - it's rather about power. You know, that's the sort of music you play as a soundtrack to yourself when you're doing something cool.
As for performing, it's almost exclusively classical vocal music — I sing bass in "Epifania" choir.


And now, a little game: "find the geek"!

We're performing Mozart's d-minor Requiem now — it's really great, but our score is made with Finale... I had to download something else from imslp.org and print myself; it's better but not perfect, either.
Recently I've seen a very good 1952 Breitkopf edition of Requiem, and you know what? I almost thought it was LilyPond!

TLR — ... Which brings us to your involvment with LilyPond. When and how did you first hear about LilyPond? Were you already familiar with the concept of "Free Software"?

JW — No idea how I learned about LilyPond. It was in 2007, I think; I could have read about her on Wikipedia when I was searching for an alternative to Finale. I'm sure about one thing: the moment i've read the Essay on music engraving, I became 100% hooked.


A flag with Finale: One word: #fail.

I'm the kind of guy who cares about every smallest detail (did you discover that flags in Finale scores are slightly misaligned with stems?) (image) Once or twice I posted a LilyPond snippet saying "look, spacing is wrong here" and people replied "where?" - they didn't notice any problem at all :P
After some time of using Lily, I had a period of disappointment - we all know that LilyPond isn't (yet) as easy to use as we'd like it to be. I'm not talking about text input, it's the difficulties in correcting Lily mistakes — for example, a bad slur is a real pain to fix. Still, there's nothing better than Lily, so I came back to it in 2010.
As for the "Free Software" — sure, I was familiar with this concept before, although I've never used Linux as my main operating system.

TLR — I assume you mean "GNU/Linux" here... Was it natural for you to become a contributor, or would you have been fine with being "just another user"?

JW — I'm the kind of guy who's full of ideas (my personal LilyPond To-do list has about 100 entries). and I love improving things. However, when I began to contribute back in 2010, I knew that I was way too inexperienced to even try tackling any of the issues that were interesting to me (I'm still not skilled enough to handle them on my own). Fortunately there was a nice coincidence: the vertical spacing engine was being changed at the moment and testers were needed. That's how it began...

TLR — Can you sum up what you've accomplished (or help accomplish) this past year for the LilyPond project? Of all these tasks, which one felt most enjoyable/rewarding to you?

JW — When I started contributing, I attacked an issue that turned out to be too hard for me at the time - I tried to add shorter versions of flags to Feta font, to be used with shortened stems. After 2-3 months i abandoned it (time for resurrection, maybe?), but thanks to great support from our team (especially Carl and Mike, but also Graham and many other people) some of my work was actually used and I didn't get discouraged. After that, all I did was some nitpicking, mostly to gain experience, but one detail turned out to be really cool: the change of Feta G clef shape.


Comparison: click here for a GIF animation (yes, it's 1996 all over again)

It was perhaps the most rewarding thing I've done for Lily so far: the amount of feedback I received was enormous (even Han-Wen and Jan approved the new design!), and if you think about it, it's a change that's present in almost every LilyPond-made score!

One other rewarding thing was a bug report that I recently opened. It seems ordinary, but the nice thing is that it describes a general problem instead of separated symptoms (a few other issues were merged into it). And the best part — Mike Solomon had started working on it, a 1500-line patch was written and is now being refined — should be ready for inclusion quite soon! Not long ago Mike solved a similar problem with slurs; I think that the most important part of my report was realizing that the solution was already there.

TLR — You make it look as if everything is fine and dandy in the community; however some of our readers may be aware of a few controversies that are, in fact, quite common in the world of Free Software. Users complaining about developers, developers not feeling supported enough,... Have you, personally been confronted with some of these frustrating situations?

JW — Sure. First of all, I find myself with a split personality: the "user" part of me is always complaining that LilyPond isn't good enough, while the "developer" part doesn't have enough time and skill to do what he wants, and feels unappreciated sometimes. What's interesting is that my basic need is engraving music (so I'm essentially a user), but right now I'm spending more time developing LilyPond than using it.
This reminds me of David Kastrup: he wanted to write accordion music with LilyPond and ended working on very low-level stuff like parser and iterators (which is very important but not very pleasant to work with, thus no one fixed these problems before). And he's doing it full time now. We've recently seen some results of his work, they are really amazing! I wouldn't manage to write the Prelude code Nicolas Sceaux wrote, but David's version is really simple! Kudos to him!
And you know what? I guess that we forget sometimes that in Free Software people are not divided between users and developers. Almost everyone is a user, and everyone has little time to spare, too. Moreover, it's not that developers have completely different needs than users and ignore their requests — quite often they simply have "different" skills. For example, as a user i was very irritated by rectangular outllines of slurs and other objects — but I didn't have the skills to fix it.

TLR — What would your personal advice be for a better LilyPond community in general? Can you think of some governance issues, some things that could be handled better or less, er, passionately?

JW — Listen to Graham Percival when he says "we need more automation", because he is right.
And always assume good will, especially when you're absolutely sure that the other guy is the bad one.
The worst thing about Lily development is that our resources are so limited. Sure, we do add great new features and can launch a big project from time to time, but nevertheless the team is quite small compared to the project size. There are lots, lots of things that would speed up our development process itself, but we are so busy that they often wait long until someone addresses them; I guess the problem is that no-one is particularly suited to do this. For example, asking David to take care of this would be a waste of his skill. Graham is very busy and the most important task for him is keeping the project running. As for myself, I tried for a bit, but I'm not very skilled — and every now and then something else requires my attention. Let me tell you my LilyPond activities:

I've spent a lot of time making sure that LilyPond will be part of the Google Summer of Code program — I wrote a large part of our application, prepared our list of projects for students, etc.,

Writing my own GSoC application (as a student — if I succeed, I'll be able to work on Lily the whole summer!) took me about a dozen hours,

I'd like to start our own Kickstarter project, similar to Open Goldberg Variations — doing this will take a lot of time, too (I'd appreciate some help here),

I'm thinking about how our Contributors' Guide could be rearranged to make it more straightforward for newcomers,

I'm organizing a team of volunteers that is examining our regression tests — it's been too long since anyone has had a careful look at them. If you can spare 5 minutes a week (no programming skills required!), check this out.

I tried to improve some aspects of our Patchy script, but failed tremendously. I'd like to try again, but the above issues were more serious or urgent,

I'd also like to improve git-cl script to make our development process more automated, but again GSoC and other things took my available time,

(This one's quite obvious) I participated in this interview and wrote two articles for this LilyPond Report.

Besides that, I'd also like to finish several patches I've started long ago, and finish three huge reports on the quality of Lily's typesetting that I've been writing for *months*: beam, tie and lyric reports. And when I say huge, I really mean it — the biggest one already has over 500 examples. I love writing such reports, but it requires a lot of time and concentration. I wish someone would fund my work on them...
As you see, many of these are about improving our development, not directly the program. Sadly, even if I were working full-time on Lily, i wouldn't manage to do all of this...
Now imagine how many cool things would Graham do if he had the time!

TLR — Overall, what's your personal feeling regarding the LilyPond community (possibly in comparison with other online communities you may have been involved with)? Would you describe it as a "pleasant" place? Are you happy with the LilyPond project as it is now?

JW — I'm not involved with any other online community, but I'd say that LilyPond is doing well right now. Users and developers are helpful, I'd also say they are quite patient (if you know how to ask politely). I have met many great people here and I hope that some day I'll have enough money to travel to other countries to meet them in person.
And, sure, we're not perfect yet; in particular the learning curve for new contributors could be improved. However, we are moving in the right direction, I'd say.

TLR — One last question, on a more political note. You may be aware that most developed countries in the world have been told by industrials to sign an international "anti-counterfeiting" treaty (ACTA) written secretly without any democratic control whatsoever. This treaty poses serious threats to democracy and national sovereignty everywhere, including Free Software. Fortunately there's been a growing anti-ACTA movement, that somehow originated in Poland. As a Polish citizen, do you have any opinion on the matter?

JW — I am alarmed by the ACTA and I hope that the protests will take effect; protesting is the least we can do. However, I'm afraid that it will take more than just protesting to protect our freedom. The main problem I see is that changes in law often go unnoticed and through small steps; it's easy to miss them — if it were not for the protests, we wouldn't even know about ACTA!
As for the Polish politicians mentioned in the articles you linked, i'm quite skeptical. I think that prime minister Tusk refrained from supporting ACTA because of social pressure. Neither do I like Palikot's Movement; they are troublemakers and provocateurs. I find it quite sad that parties like Palikot's Movement seem to be the core political supporters of anti-ACTA movement; it makes protesters lose some credibility.

TLR — Well, thank you Janek for this interview! See you soon on the mailing lists, and good luck for your future work!

JW — Thanks, and best wishes to you all, too!

Working with LilyPond: a personal experience

by Janek Warchoł

My LilyPond experience

As you've already read in the interview, I've known LilyPond for quite a long period of time. However, I don't consider myself an advanced user: the tasks I did with Lily were quite ordinary — choral music predating the 20^th century. Since it's automatic engraving that fascinates me the most, I don't like fiddling with Lily functions and writing my own Scheme stuff.

One could say that I'm some kind of a minimalist — in my opinion, the simpler the input looks, the better (and the fewer commands are used, the more I like it) — as long as it is a robust design that could be easily modified.

So, what was my experience with LilyPond during these years? One thing I have to say is that it was fantastic to see how she changed and see that with every new version features I needed were added. It's impressive that such a small development team, consisting of volunteers only, can produce so much great code!

As for the current version, I find Lily great, but uneven. In many areas she delivers excellent typography and is a joy to use, but every now and then a small quirk appears which spoils the overall effect for me (bear in mind that I truly am a perfectionist, so things that irk me may be totally okay with you).

Since the advantages of using LilyPond are too numerous to write, I intend to explore some the problems I faced, hoping that such feedback will be useful for our Development Team.

Slurs and page layout
Perhaps the biggest and most time consuming difficulties I face are things that often need manual tweaks and recompiling to see if I got the result right. One example of such issues is slur shape.

Slurs are generally difficult to tweak; fortunately, quite a lot of them look good by default, but corrections are needed far too often to say that's an insignificant issue. There are quite a lot of methods for modifying slurs — one can set the positions of its ends, explicitly define control points of the bezier curves, or add some offsets to the default slurs using a very nice function written collaboratively by Neil Puttock, Dmytro Redchuck and David Nalesnik. I can even imagine at least two more slur-modifying functions and how they could work like — but it still won't solve the problem completely. That's because all these fixes can very easily go wrong: any new version of LilyPond may introduce small changes in spacing which would result in different line breaking — and many slurs would suddenly look completely different; I'd have to redo corrections for every slur that crosses a line break at a different place. Very painful; that's probably the worst part of the slur problem.

The other thing that takes a lot of time to get it right, is vertical spacing and page layout. Despite the fact that I began using Lily in 2007, I wasn't bothered by these issues until recently, so I cannot compare the new vertical spacing system (introduced in 2.14) to what was before.

Generally, for almost every score I need to determine spacing values from scratch, experimenting a lot; I haven't found any set of values that would work good enough as a default for all my scores, even though I want a similar look-and-feel for every piece. That's interesting: either the padding-distance-stretchability system currently used has some limitations, or I don't know all commands that might be useful. For example, I need flexible header formatting — the distance between the title (plus composer and other indications) and the first system of the music should correspond to the general spacing of the systems: when there's a lot of space on the first page, the gap should be big, but with dense spacing it should be reduced accordingly.

The problem is that spacing properties basic- and minimum-distance are measured between reference points of the objects. In this case the reference point of the first system in the middle line of the top staff (quite reasonably), but the reference point of the header markup is its top border (yes, it doesn't matter how high the markup is!). Thus, the result doesn't make any sense — a big gap for small titles and no gap for very high titles. Now, if I use padding instead, the results are more predictable, but I still cannot use one value for all scores — that's because padding is unstretchable (and I definitely want this gap to adapt to the space available).

Structural problems
...are my second headache. Generally, LilyPond is very good at encouraging logically structured scores — and it's one of Lily's greatest strengths, I love it! However, there are significant shortcomings.

The first one is in \partcombine. It is a wonderful tool, but it doesn't support well articulations, dynamics and — above all — lyrics. Therefore, when i have a vocal piece with, let's say, two basses in some places, I have to write the second bass as a temporary voice (or using chords). If I ever want to split these — for example when I want to produce educational MIDI files for my choir, with every MIDI file containing exactly one voice — I have a problem.

The second difficulty is dealing with structural elements that are common to the whole score, such as line breaks, time and key signatures, barlines, rehearsal marks and so on. Every time I'm about to typeset one of them, I spend a lot of time hesitating: is it better to write it inside the music variables (for soprano, alto, etc.), or in a separate voice? In case of breaks, the decision is obvious: they are completely unrelated to music content and change too often; they should be separate. But what about the others? Each solution has its shortcomings:

if I write these elements inside each music variable, the fact that they affect the whole score is not represented well. Every time I want to change one of them, I need to do this in every voice variable (its easy to overlook something). Also, specifying all this in every music variable means more typing; additionally, when I want to copy some fragment and use it elsewhere, I need to remove them.
if I create a separate voice for these settings, I have to type a lot of cumbersome skips (which aren't convenient to update, either) and, above all, the musical content gets separated. I would find it much more elegant if the voice variables contained all musical information - i could then compile my music variables (which are held in their own \included files) to get parts from my score. That would be a structural perfection, truly LilyPondish.

Unfortunately, I'm afraid that LilyPond herself can't do much about this problem. However, it would be a great feature for an editor like Frescobaldi — there's already a "copy/paste durations" tool, and I guess this could be solved similarly — a tool that would copy/cut selected types of items and paste them over some music or into an all-skip expression.

Scheme
Finally, I have some problems with Scheme. I don't like Scheme syntax; it looks cryptic and inelegant to me. After quite a long time of using Lily I still have problems with the special characters used: quote and pseudo-quote for example ('`), — I have little idea as to what they mean and I occasionally forget which one I should use. For example, I was very surprised that to change a grob's color to blue I have to type #blue, not #'blue.

Actually, my problems with syntax are not limited to Scheme: there was a time where I had to type Lily code without access to the Internet (thus, without access to the manuals — I use development versions, and it doesn't make sense to download a new manual each time a new version is released, so I don't download them at all). When I tried to use \tweak, I was confronted with a serious problem: how did the syntax look like? I remembered that it was somewhat different from the \override syntax, but how exactly?... It took me at least six attempts until I finally got it right. David commented later that most of the trouble was caused by me forgetting to use \tweak as a postevent (i.e. to write -\tweak). He was probably right, but the funny thing is that I had recently had a very hard time trying to write a tempo markup properly... because I tried to use -\tempo (as a postevent!), which is an incorrect syntax :P

Back to the Scheme topic, I wish that more things could be achieved in LilyPond without it. That's partly because I don't like writing Scheme, but also because I'd find it more elegant to have predefined commands for all commonly used functions (as i said, I like high-level commands and concise code).

Take bar numbers for example. The default ones are not bad (way better than in Finale), but I'm not satisfied with them: I'd like the first bar number in the system to be bigger than the others, and I want to have bar numbers centered over the bar — except for the first number in the system, which should be left-aligned (to avoid placement problems caused by StaffGroup brackets and clefs). Finally, instead of showing all bar numbers (as I usually do), I occasionally want to show every 5th number (and the number of first measure in system). Now, how does one accomplish this? I remember reading a snippet with a Scheme function that manipulated objects depending on whether they were on line break or not. It was quite simple, yet I had to spend a lot of time modifying it according to my needs. I eventually succeeded — here's my override that makes Bar numbers at the beginning of the system bigger:

\override BarNumber #'stencil =
 #(lambda (grob)
 (let ((break-dir (ly:item-break-dir grob)))
 (set! (ly:grob-property grob 'font-size)
 (if (= break-dir RIGHT)
 -1
 -3))
 (ly:text-interface::print grob)))

One could write a very similar function for bar number alignment, and I imagine that showing only desired bar numbers is also achievable using Scheme. However, I don't think I'd have any chance of succeeding if I hadn't found the original snippet that served as a base for modification.

The best solution that comes to my mind would be to create some kind of "FirstBarNumber subclass", i.e. an object type derived from BarNumber. That would enable a solution looking like this:

\override BarNumber #'self-alignment-X = CENTER
 \override FirstBarNumber #'self-alignment-X = LEFT
 \override BarNumber #'font-size = -3
 \override FirstBarNumber #'font-size = -1

In my opinion such a solution would be more elegant and user-friendly. This also reminds me of two other inconveniences: when I want to move some lyrics, I have to override properties of LyricText, LyricHyphen and LyricExtender. It would be nice if I could specify an override for all of these simultaneously, like this: \override Lyrics #'Y-offset = #2

Another useful addition would be grouping properties together: currently to create small staves or notes one has to override font-size, staff-space and thickness. Why not create a "size" property that would set all these at once?

There are more similar changes that I could suggest, but that's not exactly what this article is about; I hope that GLISS (the Grand LilyPond Syntax Stabilization project) will happen this year and sort out such issues.

Bottom line: LilyPond is a great piece of software and it can definitely can be used for pretty much every serious engraving project i can think of, but it still requires quite a lot of skill from the user to achieve more advanced tasks; there is plenty of room for smoothing rough edges to create a more intuitive and powerful user experience.

The Bug Report(s) of the month

It's been a long time since we haven't had a "bug report of the month". Therefore, we relied on Janek Warchoł (who else?) to provide us with not one but dozens of bug reports. See for example this nice sequenza:

Bug 2141: some accidentals are too far from each other
Bug 2142: accidental right-padding should depend on the tightness of music
Bug 2143: accidental arrangement should depend on tightness of music
Bug 2144: accidentals should be able to slide over/under suspended notes
Bug 2145: shorter versions of accidentals for use in tight situations

Investor's report

by David Kastrup.

So here are the operating results of people providing financial support for letting me work exclusively on LilyPond in March, the first month after making my case in the LilyPond Report #24:

Onetime payments (€)	Payment plans (€)
2×200 150 62 2×25 20	25+50 2×25 10 4
Total: 682	Total: 139
Totals: €821

Strictly speaking, one of the one-time payments was still in February, and it was actually a partial payment for contracted work on LilyPond, but since the goal is to make a living (or rather currently a survival) from LilyPond, I decided to figure it in.

What are the projections for next month?

Payment plans (€)
100 25+50 2×25 10 4
Total: 239

Now what are we to make of it? As explained earlier, I need about €800 per month to cover basic expenses of living (including health care but excluding other kinds of social security) while working on LilyPond, and about €1200 to sustain operations at a level where I can afford not looking for other options eventually. In March, I have been able to make ends meet without further depleting my leftovers from the last "proper'' job I held.

On the one hand, I am impressed and grateful for the amount of support for my work on LilyPond that this has shown. If anybody in addition to those who already contributed had chosen to pick the monthly payment plan "life saver'', his whole payment would have carried over into April.

But on the other hand, I consider it likely that in April, it would have gotten used up. Most of the payments were one-time payments triggered by the article in the last LilyPond report. So there is a lot of uncertainty about how long things are going to continue working out even barely. There was just one participant for a variable monthly payment scheme (25 fixed, and up to 50 for contributing to a target of 1200). And of course I'd like to become able to report to him that the target of €1200 could be reached without fully using up the variable part of his pledge, since obviously the variable schemes are intended to become less taxing once enough people join in.

Now what use was made of the contributions in March?

git shortlog -n --since "5 weeks ago"

tells us (try it if you have a repository of LilyPond checked out) that several commands have been added, problems in connection with Ghostscript 9 have been cured (that resulted in bad output in viewers as well as print), several critical bugs have been addressed (unfortunately, we still have not managed to release 2.16), it has become possible to place things like \accidentalStyle and \tempo commands in \layout and \midi blocks, a number of smaller bugs have been fixed, some work has been done on making the regression test process about 30% faster, I have been pretty much the only person running the review tests for all changes to LilyPond, some internals were rearranged and made more robust. In the course of development, one merge from the translators went rather bad, and in consequence the effects of dozens of commits were removed from the repository, while git was of the opinion that it had kept them and consequently refused readmitting them.

After about a day of emergency work and testing several solutions for their viability, I managed to bring the repository back into a state where there was no danger that any branch (including unknown ones on external servers that might already have merged the problematic branch) would be in danger of reintroducing the problem. The bug squad made impressively short work of the remaining cleanup: verifying manually that all of the changes that went missing (from about two to three development versions worth) were indeed properly reintroduced by my fix.

Upcoming goals (some of which I would have wished to have completed already) are managing multiple marks (and other events) at the same time, actually working ways to programmatically extend grobs and events, further unifying work on the parser regarding music functions.

Of course, the goal of releasing 2.16 is foremost in most people's mind (and I'll likely help tackling a few more problems that keep popping up whenever we think that there is nothing else missing). It is also likely that guile-2.0 compatibility will become a quite more relevant topic soon (Ubuntu 12.04 is going to provide Guile 2.0.5 if I am not mistaken), and Ian Hulin, who has been doing most of the Guile 2 related work in the past, is currently unable to contribute much. But it will become rather more important to move LilyPond to the next generation of its internal Scheme interpreter soon. And again, this is a project requiring enough heavy lifting that spreading it over a longer amount of time and/or a lot of people would cause wasteful extra work.

So there is not really much of a shortage of projects for April. While I will take the week before Easter off for a private vacation (and thus have to plead "guilty'' to not using every cent of my remaining savings exclusively for working on LilyPond), it will not be without computer and an agenda for LilyPond.

With regard to contributors: I'd like to see the load distributed over more shoulders. As you can see, rather few people committed to a regular "affordable" plan. If enough people were willing to contribute a small regular amount, the payoff for everyone would be good and not really involve much of a sacrifice, but I am afraid that I don't really know how to convince more people that joining the ranks makes sense. If you do, tell me. Or them. It is one of the "it may seem insignificant in itself, but it is very important that you do it'' kind of things.

As a rehash from the article in the last LilyPond Report (probably worth reading if you you haven't done so already), here are the payment schemes I suggested:

My mail address dak@gnu.org is registered at Paypal, and if you want to skip the middle man for regular or larger amounts of money, you can ask me for my banking details (I am living in Germany). People prefer not to think about details, so here are some payment plans. The idea is to contribute a fixed minimum, and if a specified target is not reached by all contributions, you contribute proportionally up to a cap. Of course, you are free to pick all three numbers yourself, but here are a few models:

[Regular] €25 per month fixed, no cap. This is the payment plan to pick once everything is sailing smoothly and you don't want to contribute unduly much or think about it unduly much.
[Lifesaver] Minimum €0, cap €250 per month, monthly target €800. That means that if the target (which basically allows me to postpone my decision to work elsewhere) is reached with everybody's minimum already, you are not billed. This is the option to pick if you don't want to support a single person as much as keep the LilyPond project from losing me. You do what is necessary to avoid my leaving, but nothing else. Yes, it will be annoying if it turns out you have to pay the cap more than once, but it will also be annoying for me not even to afford survival in spite of highly qualified work.
[Torchbearer] Minimum €50, cap €150 per month, monthly target €1200. This is a model aimed at being reasonably comfortable for you as well as for me if everything works out.

Since I have been surprised in the past by the percentage of people actually coming through with their expressed intentions, you would start payment with your cap, and I would inform you in time for your next payment how much of that could be carried over due to others pitching in. The more variable schemes are designed to be effective with a few highly dedicated people, but allow them to back off once more people join.

My thanks to all those who contributed financially to my work on LilyPond in March, and those who continue to do so. Most of those who supported me financially also spend a considerable amount of time working on LilyPond or helping other users, and thus are doubly dedicated to its well-being.

And now, a personal appeal from Janek Warchoł on this matter.

Dear LilyPond users around the world,
as we've read in previous LilyPond Report, David Kastrup — one of our most experienced and skilled developers, and the only one who works on LilyPond full-time — is running out of money. He asked us to sponsor his work on LilyPond by sending him some funds each month; his request was answered, but the amount of money collected is too small to sustain him.

Now, I don't know if you follow what happens in Lily development. I do, and i've seen what David did to improve LilyPond. In many cases, I wouldn't even know where to start doing things he did, and it would take me weeks to write what he wrote in few days. One time he rescued us from a really big trouble, saving 3 weeks worth of all other developers' work. His work and experience with Lily code base is really invaluable.

Think about it. If David decided to work for some software company, his salary would be tens of thousands of dollars per year - or to say it differently, if we (LilyPond community) decided to hire a programmer (to replace David when he leaves due to lack of money), we'd have to pay several thousands dollars each month. David asks us only for 1200€, because he has a Spartan lifestyle and enjoys working on Lily. Can we ever imagine getting a better offer?

There are hundreds of LilyPond users — it would be a shame if we failed to find the money needed.

Can you afford giving David 15€ per month? I think that's a fair price for using LilyPond, and without David's work Lily wouldn't be what she is now.

Ok, some of you may not afford 15€ per month. But can you afford $2, or $5? I know that it sounds like an insult to David's skills, but believe me, he will be grateful. And don't worry about the transfer fees — send money for 3 months in advance, and anyway PayPal takes less than 10% of that.

Please support David. I've already sent him money for April and May.

LilyPond's future

by Janek Warchoł.

What is the goal of LilyPond project?

Needless to say, LilyPond is a large project. With its history spanning 15 years and total size exceeding one million lines of code, it's the biggest Free Software tool for musical score preparation in the world.

We know where we come from, and we know who we are. The obvious question is: where are we going, what will our future be? Thanks to continuous efforts from many people, LilyPond keeps evolving — but can we, and should we, shape her future?

I think we should. The scope of LilyPond is really big; without a goal our development efforts will be scattered and less effective than they could be. But how to decide what's most important to LilyPond project? I suggest to use the 'catchphrases' mentioned on our website and in the essay: "music engraving for everyone" and "automatic music engraving". Combining these two into "automated music engraving for everyone" gives us (in my opinion) a very good tool for measuring issue importance. If we decide that providing automatic music engraving for everyone is our main goal, I guess that would mean two things:

making Lily easier to use,
having tasks that appear most commonly even more automated.
(notice that fixing uncommon bugs and adding special features for modern notation doesn't quite fit any of the above. While I'm not at all arguing against doing these, I suggest that we focus on something else).

The first thing is a very complicated issue, and I'll just scratch the surface here: it requires rewriting LilyPond internals (David Kastrup is doing a really amazing job here), standarizing our syntax and adding some nice yet powerful and versatile shortcuts. For example, it would be nice (in my opinion) to have a set of predefined "instruments" that would make entering music easier and the code more elegant, i.e.

\new Staff = "tenor" { \music }

would be equivalent to:

\new Staff = "tenor" \with {
 \consists Ambitus_engraver
 } { 
 \clef "G_8"
 \set Staff.instrumentName = "Tenor"
 \set Staff.shortInstrumentName = "T"
 \set Staff.midiInstrument = "Choir Aahs"
 \autoBeamOff
 \music
 }

One can also argue that it should include putting more effort into creating a fully-featured graphical interface — that's a topic for a whole other discussion.

Concerning the second area (automating common tasks), there are of course loads of things that could use more automation — we shall decide which are most important for achieving the "automated music engraving for everyone" goal. In my opinion two types of issues are crucial:

basic notation,
notation elements that heavily depend on surroundings.

An example of such an issue is tie formatting. Ties are necessary for notating rhythms, so they are a very basic element of notation and appear frequently. Since their shape changes with note placement and overall surroundings, fixing bad tie shapes is difficult. Imagine an orchestral piece containing a clarinet in B flat; I want to produce an orchestral score in concert pitch and parts with appropriate transposition. If there is a bad-looking tie somewhere, I have a problem: the tie in full score can be quite different from the tie in part (if the notes connected are between lines in score, they are placed on lines in part due to transposition, and also horizontal spacing - thus, the length of the tie itself - may be different; full score may also contain some cautionary accidentals that further complicate tie formatting problem). Fixing such a bad tie is difficult; and if you ever decide to transpose the piece you get yet another tie setup (and this time your previous overrides might actually spoil the default good output).

Now, that's just one example. What is basic notation in general, then? I suggest defining it as single-staff, single-voice music consisting of:

notes ( = noteheads, stems, flags, beams, ledger lines)
rests
accidentals
ties
augmentation dots
lyrics

These are necessary for notating essential music information - pitch, rhythm and (for vocal music) text to be performed. In my opinion we should focus on perfecting these areas. I say "perfecting", because we already support all basic notation elements — it's just a matter of having them work really smartly in any reasonable situation, out of the box. If you are wondering "what on earth needs improving? everything is good!", wait for the next LilyPond Report, where I'm going to discuss some real-life examples — or try playing the game below! Many of the problems I report feel like insignificant nitpicks, but actually the difference between acceptable and outstanding music engraving is all about details!

The game of the month

Here is a PDF file from a default LilyPond output, compiled by — you guessed it — Janek Warchoł.


Credo (Mozart).

Can you spot at least seven layout problems?

Found it? No?

Okay, here's the answer.


Credo (Mozart), with annotations.

What do these colors mean?

magenta: accidental spacing (generally some accidentals take too much space).

dark violet: lyrics horizontal spacing, and friends.

dark and medium blue: vertical positioning breaking into segments

light blue: vertical spacing should use skyline integrals

Stay tuned for more examples of default LilyPond output in the next LilyPond Report!

The joke of the month

This month's quote is from David Kastrup, but we wouldn't have dreamt of including it if it hadn't been deemed worth it by you-know-who:

— What impressed you most about the LilyPond code base?
— No comment.
— But I really want to know!
— I just told you.

That concludes our twentyfifth issue of the LilyPond Report.

Cheers,
Janek Warchoł, Janek Warchoł, Janek Warchoł and Janek Warchoł (some additional help provided by Valentin Villenave and David Kastrup, with special permission from Janek Warchoł).

(P.S. Editor's note: No Janek Warchoł was harmed in the making of this Report.)

The LilyPond Report #24

David Kastrup, Valentin Villenave — 2012-03-04T08:28:43Z

Table of contents

Greetings everybody, and welcome to this twenty-fourth issue of the LilyPond Report!

First things first: this slightly unusual issue features a rather important request from our co-editor David Kastrup, who also happens to be arguably the most involved and efficient LilyPond developer at the moment — and unarguably, our only full-time developer. Also featured in this issue is an overview of some aspects of the new, simplified syntax in LilyPond. At last, we'll be discussing a long-overdue topic, to wit: LilyPond-based web applications, with a long-overdue contribution from scorio.com founder Johannes Feulner!
As always, you can post your comments at the bottom of the page, or send your contributions to the LilyPond Report's next issues.

Editorial

by Valentin Villenave & Graham Percival.

The LilyPond Report was started exactly four years ago this month; although it has been a bumpy ride, with occasional long periods of inactivity, we hope it has proven a valuable medium for the GNU LilyPond project and the Free software world in general. It has given us a chance to meet dozens of interesting developers and contributors, to address various music-related questions and to provide a hindsight into LilyPond's development process. This month however, we are contributing to the LilyPond project in a new, more serious way, upon request from our co-editor and colleague David Kastrup.

While several projects depending on LilyPond have managed to fund full-time developers, LilyPond development itself has not really seen significant monetary support from its users so far. Every so often, however, users ask if they can contribute financially to LilyPond. Creating a single legal entity to handle such matters would raise a number of legal and administrative problems which we are not equipped to handle at the moment. However, we have a specific place where developers can indicate their interest in private arrangements:
http://lilypond.org/sponsoring.html

Currently the only developer on this page is David. As you can see from the git statistics, he has been very active over the past few months. Both his self-chosen projects as well as the expertise he has provided for work on our infrastructure and the projects of others have given LilyPond an important push forward.

As you can see in his plea below, David is in dire financial straits. Unless he receives money to support his LilyPond work, he will need to leave the project and find work elsewhere — which would be bad news for LilyPond.

Users of LilyPond, contributors and enthusiasts, you now have the chance to secure an important contributor for full-time work. If you are interested in supporting LilyPond financially, we warmly suggest that you sponsor David to fix bugs or add features that interest you.

On a personal note, both of us have unrelatedly sponsored David to add new features for us in the past, and in both cases his work was fast and high-quality.

Graham Percival, GNU LilyPond project manager;
Valentin Villenave, LilyPond newsletter editor.

An urgent request for funding

David Kastrup

Read this personal appeal from LilyPond developer David Kastrup.

Let's begin with the most important bit. At the current point of time, my financial situation is in free fall with very few slowdowns. Its terminal velocity may be restrained by my rather basic lifestyle. But if you think me having to split wood at ten degrees below zero because of the price of oil heating is helping much, think again: that shaves off something like maybe €40 from the monthly bills at best. It merely spreads my reserves a bit thinner.

Splitting wood.

"As the temperature has risen above 5 degrees, we fortunately have stopped heating" — D. Kastrup, 2012

So impact is looming close. The common answer to that is ``Get a job!''. If you take a look at the activities in the LilyPond repository in the recent months, you'll find that I already have a full-time job. I am working for you, making LilyPond a greater tool to work with for you and other musicians. And much as I like this job and enjoy working for you, the time has come to talk about a raise.

Now I am not the only one working on LilyPond, and you may question why you should consider paying me instead of someone else for that work. That is a valid question as long as you are actually paying someone else rather than using it as an excuse not to do anything at all. Check out http://www.lilypond.org/sponsoring.html for your options at any given point of time. At the time of writing this article, there are no other requests for funding. I deliver excellent value, and there is a limited window of opportunity before I have to close down operations and am no longer available. So buy yourself in while you still can.

There are others working on LilyPond in their spare time (and some of them are doing a really fabulous job). Now for one thing, the rallying point that a full-time developer provides helps even them to be more effective at what they do and can achieve, even if this full-time developer would not be first pick for the actual project lead or other tasks focused on communication skills.

But for another, it turns out that my own situation is somewhat special, and so I have an additional personal interest in you making it possible to continue my work on LilyPond, going beyond a mere job search.

Primary means of transportation (front and left).

D. Kastrup, 2012.

And so that you know what you buy into and how to make the most of it, I am telling you.

After figuring out that there were some things affecting my productivity in a number of ways that don't seem to affect other people nearly as much, I actually asked for professional opinion and got diagnosed with several medical conditions. Now "conditions'' are things to take into account, "medical'' means phenomena fitting a pattern so that one can focus on approaches that have shown to be effective in similar cases.

After lots of experimenting around, it turns out that in practice I mostly have to live with being David, and you have to deal with me being David, because that is quite solidly hard-wired. And this can actually be a rather good deal for you and others, even if some parts of it may seem unusual.

If we cut through the skinny of buzzphrases like "Asperger's'' and "ADHS'', here are key points worth knowing about me:

I can't stand things being `wrong' in a number of ways. This means that I am a good fit for working in a setting with ethical principles (call it "ideological'' if you want) like Free Software. It also means that I am uncommonly good at finding and exterminating even more obscure bugs (like tracking down a GCC compiler bug that threatened to make LilyPond unusable on newer systems).
I get irritated and distracted rather easily. That means that I can be spectacularly bad at boring but necessary tasks which is not really good when operating in normal employment situations. It actually is not really good when leading any kind of life, either, but that is a different topic.
I see people react, but am bad at predicting it. If you meet me face to face, cues that are absent in non-interactive communication do a lot for avoiding misunderstandings and annoyances. Like with many other things, this is not unusual in itself but in the degree to which it happens.

While I do not really shine in human relations when doing `batch processing', I am pretty good with computers, and indeed I started working with them at a time when the communication literally consisted of a batch of punch cards and, a few hours later, output in print.

The low frustration tolerance for `boring' tasks still shows when working with computers, but I fare rather good at `hard' and `impossible' tasks as the `mind' of computers tends to be more accessible to me than that of fellow human beings.

Music is in the Air.

D. Kastrup, 2012.

Now at the moment, LilyPond has captured my attention and provides a lot of motivation, and other developers and their needs and work put me back on track when I get distracted. That means that I am quite productive overall, though not necessarily according to any plan. My actual impetus had been extending LilyPond for typesetting music for accordion. I have not exactly been making lots of progress on that front, instead getting myself caught up in improving an increasing number of areas within LilyPond itself and the workflows focused around using and developing it, to a degree where I have, in my single-mindedness, not enough time and focus left doing another job.

As a result of my involvement, programming LilyPond has become quite more powerful and enjoyable: most of the rather extensive changes in LilyPond's input and music processing during the 2.15 development phase are due to me, and its integration with its extension language Scheme has become more logical and powerful at the same time.

LilyPond has more areas than what I have been working on: there have been other large improvements particularly in the typesetting stage of LilyPond, and those improvements did not depend on work I have done. It is my impression, however, that the motivation of users on LilyPond's user list to solve problems by writing Scheme code on their own and teaching others how to do it has picked up. I certainly hope to see this trend to accelerate after we manage to make the next stable release, 2.16, and that it will contribute to attracting new developers over time.

We don't have a general LilyPond development fund at the current point of time, and it would pose organizational and legal problems that are not worth tackling before there is positive proof that enough users are willing to invest in the continued well-being of LilyPond, and put their money rather than their foot where their mouth is.

Fortunately, my current situation is the perfect litmus test for you to demonstrate that. It is hard to imagine a more convincing incentive to contribute more than just talk. I might add that even with a general fund, you could do worse than contribute to me in person, since funds play to their strengths when paying for systematically planned tasks, and I am by far most effective when working on stuff that just happens to annoy me at a given point of time.

Patchy at work.

D. Kastrup, 2012.

There would be some things I could apply for even if everything was run through a fund: the laptop I am currently working on has been paid by another developer, one who is already putting in more time and energy than anyone could demand. It is a second-hand system from 2007 and, in spite of being quite better to work with than the system from 2000 I had to use before, is tasked a lot with test runs for LilyPond since we are just now getting them streamlined enough to be able to offload them to less experienced developers with better computing facilities. I am holding a talk about LilyPond at a major GNU/Linux fair in Chemnitz (the home of the Chemnitzer Concertina) in a few weeks, and this incurs travelling and accommodation costs that are not yet accounted for.

But most of the stuff I do is hard to plan and account for: I do it because it feels necessary and is the right thing at a given point of time. And that is enough to compel me. I hope it is for you as well: if I have to turn to a different endeavor for keeping myself afloat, I will have to cut out working on LilyPond completely since it would continue being a distraction, and distractions are something I can deal with very badly.

My mail address dak[at]gnu[dot]org is registered at Paypal, and if you want to skip the middle man for regular or larger amounts of money, you can ask me for my banking details (I am living in Germany). People prefer not to think about details, so here are some payment plans. The idea is to contribute a fixed minimum, and if a specified target is not reached by all contributions, you contribute proportionally up to a cap. Of course, you are free to pick all three numbers yourself, but here are a few models:

[Regular] €25 per month fixed, no cap. This is the payment plan to pick once everything is sailing smoothly and you don't want to contribute unduly much or think about it unduly much.
[Lifesaver] Minimum €0, cap €250 per month, monthly target €800. That means that if the target (which basically allows me to postpone my decision to work elsewhere) is reached with everybody's minimum already, you are not billed. This is the option to pick if you don't want to support a single person as much as keep the LilyPond project from losing me. You do what is necessary to avoid my leaving, but nothing else. Yes, it will be annoying if it turns out you have to pay the cap more than once, but it will also be annoying for me not even to afford survival in spite of highly qualified work.
[Torchbearer] Minimum €50, cap €150 per month, monthly target €1200. This is a model aimed at being reasonably comfortable for you as well as for me if everything works out.

Since I have been surprised in the past by the percentage of people actually coming through with their expressed intentions, you would start payment with your cap, and I would inform you in time for your next payment how much of that could be carried over due to others pitching in. The more variable schemes are designed to be effective with a few highly dedicated people, but allow them to back off once more people join.

A tool like LilyPond, providing a simple, open way of describing music and typesetting it to high standards, is important not just for personal use, but also as a base to work with music that has already passed into the public domain and that forms a large part of our cultural heritage, and that LilyPond can help to make easier to access and cast into new forms.

I do not suggest that there are no other ways to invest money or energy for making LilyPond a better tool for working with music. But don't take that as an excuse to do nothing at all. If you find that LilyPond would be better served by you working or investing on aspects not covered by me, then do it. Don't merely think or talk about it: that may feel noble, but achieves nothing. Don't postpone it. You have a chance of making the world a better place for music and, if you choose to do it through me, a limited window of opportunity. Don't blow it.

Release news

by Valentin Villenave.

Although LilyPond's next stable release, 2.16, is getting nearer every day, it will take a little more time and intermediate "development" versions. As we were hoping to release 2.16 in early March 2012, a number of new critical issues have been found, thereby delaying the release. The good news is, these issues are now solved and we're ready to proceed with the development cycle — that is, at least one more Release Candidate will be needed.

Speaking of which, it is important to point out that, as 2.16 is looming, you should test the release candidate — by "you" we mean all users, not just the most adventurous ones: the more people test it and report any problem, regressions or unwanted behavior, the more our "stable" release will be worthy of that name. So, please download and install our release candidate!

What will this new stable version bring to the table? We'll take an in-depth look later on, but here is a complete list of the most noteworthy changes, be sure to check it out and discover LilyPond's new, exciting features!

Please also note that the LilyPond team is currently in need of a new Mac OS X maintainer. This LilyPond port definitely needs some love, and it has proven hard for regular developers to keep up with Apple's latest ~~broken~~ ~~incompatible~~ ~~absurd~~ fancy new features. If you're experienced enough (both with LilyPond build process and the Mac OS X environment) and feel up to the task, please don't wait and get in touch!

LilyPond grammar update

Some LilyPond Parser Changes:
Recent Work and Plans
by David Kastrup.

LilyPond's parser is written using the parser generator "Bison''. Bison generates parsers of the LALR(1) class which basically means that it makes its parsing decisions by considering a set of rules like

property_operation: STRING '=' scalar | "\unset" simple_string | "\override" simple_string property_path '=' scalar | "\revert" simple_string embedded_scm

(the LilyPond grammar can be found formatted in that way in the Notation Reference). It gets "tokens'' (such as integers, inter-punctuation characters, complete words, note names) as input, and makes its decisions about which rule to apply next based on the combination of its current state (of which there are about 700 distinct ones), the production on the top of a push-down stack, and sometimes by looking at the next token. When it applies a rule, it may execute actions associated with that rule, and when it has reduced everything to the first rule (the start rule) it is finished.

For a compiler, the actions are commonly used for generating code, or as a first step to that purpose, a parse tree representing the input. In contrast, LilyPond works mostly as an interpreter. This means that it executes the actions associated with input as soon as it is able to exercise a rule.

Turning the LilyPond input language into a compiler would present considerable challenges because of the design of the language: some decisions, like assigning a syntactic class to an escaped \keyword or an immediate Scheme expression such as $(ly:make-pitch 0 0 0) are inherently dependent on the current state as evolved by the interpreter.

However, the purpose of the input language is almost exclusively limited to parsing and generating music lists; and actually transforming those lists into a printed representation via engravers and Midi via performers is what takes the bulk of processing time.

Parser developments

Optional music function arguments
Since the last stable versions (2.14), music functions with optional arguments have made it into LilyPond. One result has been that a number of syntactic constructs hardwired into LilyPond's grammar could be turned into music functions. For example, in LilyPond 2.14, the command \relative was parsed in the LilyPond grammar with the rule

relative_music: RELATIVE absolute_pitch music { Pitch start = *unsmob_pitch ($2); $$ = make_music_relative (start, $3, @$); } | RELATIVE composite_music { Pitch middle_c (0, 0, 0); $$ = make_music_relative (middle_c, $2, @$); } ;

Now it is parsed using the following definition:

relative = #(define-music-function (parser location pitch music) ((ly:pitch? (ly:make-pitch 0 0 0)) ly:music?) (_i "Make @var{music} relative to @var{pitch} (default @code{c'}).") (ly:make-music-relative! music pitch) (make-music 'RelativeOctaveMusic 'element music))

For one thing, having fewer hard-coded exceptions makes the syntax more regular. For another, the functionality is now documented in the notation reference together with other music functions (the given documentation string is tunneled there in the process of creating the manuals). Another advantage is that changing the functionality does not require recompilation of LilyPond.

More importantly, users can now create functionality in Scheme (and thus as part of writing a LilyPond source file) that formerly required changing and recompiling the parser.

The workhorse #{ ... #}
In 2.14, you could employ the construct #{ ... #} for writing sequential music in Scheme. It has been slightly extended to also allow writing pitches, markups, scores, books and bookparts, single music expressions, postevents, context definitions, layout definitions and some other things. Basically, it has turned into a general interface for specifying details in LilyPond syntax when writing in Scheme, and, as one example, now provides a user-friendly alternative for the markup macro. markup is a tricky construct for writing markups with a syntax that is almost, but not quite, entirely unlike LilyPond.

Previously, there was a special interface using $ for getting material from local variables into #{ ... #}. Some aspects of it have been turned into a general feature ("immediate'' Scheme expressions) available throughout LilyPond, but more importantly, all Scheme expressions inside of #{ ... #} are evaluated in `lexical closure', meaning that they can access the local variables of surrounding Scheme code. So #{ ... #} has turned into a quite more powerful feature, at the same time behaving more like `regular' LilyPond.

If we take the above example for relative music and change it such that the default will be to make the music relative to f (which means that the first note will actually be written with absolute pitch), we can call this \ablative (because it starts out as absolute, and continues relative):

ablative = #(define-music-function (parser location pitch music) ((ly:pitch? #{ f #}) ly:music?) #{ \relative $pitch #music #})

We did a few things here typical for user-defined functions: there is little point in a documentation string, and we generously employ #{ ... #}. Actually, since \relative remains available, there is little point in accepting an optional pitch argument here, but it makes for a somewhat more interesting example.

Using #{ ... #} extensively makes it possible to employ Scheme mostly as glue code between LilyPond fragments.

Future developments

Further unification of argument types
At the current point of time, there are still two argument types recognized separately in the grammar: ly:duration? and ly:pitch?. As a result, the last example had to use an immediate Scheme expression $pitch (as otherwise the argument would have been interpreted as type "Scheme''), while it was possible to write #music since arguments of type music and of type Scheme are interchangeable already, relying only on evaluation of the predicate for type checking.

Removing those special argument types provides a bit of a challenge. Pitches are challenging because a pitch is a valid start of a music expression (and that implies a bit of incremental parsing to make them fit with the general scheme of things since one does not want to recognize pitches using lookahead in optional argument settings, and pitches can contain an arbitrary number of octave marks).

Durations are challenging in a similar manner because of being able to be confused with numbers, because of optional following dots, and partly because they can become part of a preceding music expression such as when preceded by a pitch.

Removal of "closed music'' category
At the current point of time, music arguments before durations or after skipped optional arguments need to be enclosed in braces or other delimiters in order to avoid ambiguities in the interpretation of things like c 8: is the 8 part of the music argument, or is it a duration following it? And an optional music function argument has to be first created, then checked against its predicate, and if that does not match, backed up in the grammar. And in a stateful parser like that of LilyPond, backing up is only possible when the parser has not seen further tokens and/or has entered into states depending on them.

Some of the more recent changes in the LilyPond parser have been in the area of resolving such ambiguities by looking at the context and potentially backtracking and/or parsing stuff incrementally.

The principal idea is that one parses first tentatively without lookahead, and checking possible expression types that can arise against the optional argument predicates, even though the actual expression might not be complete. This means that the argument predicates must not have side-effects, but just depend on their arguments, a reasonable assumption.

However, when the expression itself results from a music or scheme function call, the amount of possible reparsing is limited since requiring music or scheme functions to be without side-effect would be excessive: for example the main point of \displayMusic is its side effect. So if \displayMusic is in itself a potentially optional argument of a music function, we either have to be able to parse its argument without lookahead, or we won't be able to back it up again in case it is not accepted by the optional argument predicate.

As one example, a \repeat construct at the current point of time is not closed music, because LilyPond needs to be able to parse an item without lookahead when it may or may not accept it as an optional argument. And a \repeat construct may be followed by \alternatives, something that is impossible to see without lookahead. The solution is to first parse just up to the conclusion of the \repeat, and then call the optional argument predicate on the resulting music expression. If it is found acceptable, LilyPond can resume parsing after having made the decision that it will not skip this optional argument, and the resumed parsing can now work using lookahead tokens freely. If it turns out that the construct with alternatives is not acceptable any more, we blame the user (meaning that we don't deal with surprising amounts of backtracking).

Removing the concept of "closed music'' (or at least reducing its impact) would reduce the number of things a user has to understand in order to create and use music functions. Here is one example of a set-back: at some point of time, \displayMusic was given an optional port argument before the music expression. Since only closed music expressions are allowed after a skipped optional argument, \displayMusic c'' ceased being valid input, a side effect discovered more than a week after making the change. The change has been reverted for now, but at one point of time, it should be possible to bring it back without such repercussions.

The idea is to have the user just see results equivalent to greedy and backtracking parsing compatible with evaluating the given music function predicates: this is conceptually easy to understand, explain, and extend, but mapping this into code that can be tackled by an LALR(1) parser generator is challenging.

LilyPond online editors

by Valentin Villenave.

Although some of our youngest readers may not remember, there actually was a time, not too many years ago, where people did rely on their computers to do their computing, where your phone's primary use was to make phone calls, where using a webmail service was considered bad taste (and JavaScript, an unnecessary ugliness), where thin clients were regarded as subpar computers to be used exclusively in a corporate environment and usually meant your boss was kinda cheap; a time where the "cloud computing" buzzword — can you imagine — didn't even exist!

One of LilyPond's major downsides when it comes to dealing with inexperienced users, and probably one of the reasons why it hasn't seen more widespread adoption amongst musicians, is its setup process that may seem peculiar to people unfamiliar with the notion of "source code". Installing LilyPond itself has become increasingly easy over the years; setting up your work environment (text editor, PDF preview, command-line invocation) is a bit more involved... assuming you can think about it: I can't count the people around me to whom it has never occurred that they could actually be using something else than the default NotePad application for Windows™, or the LilyPad text editor we provide for Mac OS X.

In this regard, being given the ability to discover the full potential of LilyPond (including syntax-colored source code and score preview) in a Web browser without having to install anything whatsoever, may be an excellent introduction. As early as 2006, a notorious visionary — namely, myself — predicted that "LilyPond's future might well be web-oriented". I wasn't, to be fair, the first one: in January 2005, Sebastiano Vigna had introduced the LilyPond Snippet Repository that provided the very first public LilyPond server. The LSR was intended for documentation purposes and required a bit of work to get around its ERW-based interface, but it is still in use today.

Setting it up, to be honest, is quite an involved task and demands more than just courage, but also a bit of masochism and recklessness. By that I am not only referring to Sebastiano's peculiar taste in terms of server-side Java, but the main point is that LilyPond, as a program and as a language, presents an extreme security risk for any server where you want to run it. Since it allows for embedded Scheme, that in turn can convey system calls, it can — and will eventually, if you leave it open to anyone on the interwebs — wipe out your system and kill your hamster! That's not the only issue: even perfectly harmless code can eat up a considerable amount of resources, there's also the problem of \include instructions that would require users to upload multiple files and thereby multiply injection risks, etc.

In short: children, do not do it at home!

But that kind of warnings has never stopped any self-respecting geek, has it?

Interestingly enough, all of the subsequent prototypes, based on Sebastiano's work, weren't conceived as standalone web-applications, but rather tried integrate LilyPond into existing Content Management Systems. In 2006 Chris Lamb came up with a Wordpress plugin; slightly later our developer and git expert Johannes Schindelin introduced a MediaWiki extension that raised a lot of eyebrows (man, LilyPond in Wikipedia! — hold your horses, more on that later). In 2007, Christophe Richard also turned it into a plugin for the SPIP cms, and it was even added as a plugin to the FCK editor. Last but not least, in 2009 Harmath Dénes integrated LilyPond into Google Wave^® the brand-new disruptively-sexy Next Big Thing that was gonna revolutionize the world as we knew it.

In May 2007, I had introduced the first prototype of an online real-time syntax-highlighting and auto-indenting LilyPond editor. It was mostly a hack around the Codepress Javascript editor, but it was fun to make and it became part of a French effort to provide LilyPond-ready web applications (around the now-defunct lilyserv.net server, modeled after Sebastiano's code). We were beginning to discuss a possible standalone web-application (a then new-and-shiny buzzword, as Gmail&Co were gaining traction). Laurent Ducos even suggested it should sport an online graphical interface such as Denemo or LilyComp, but that never went anywhere.

Then came what I'd refer to as the "second generation" of LilyPond-based online efforts: the era of standalone, user-friendly web applications. In this regard, the year 2010 certainly was some kind of a turning point. The iPhone^® (and its ~~useless~~ brother the iPad^®) was everywhere, and running LilyPond on it wouldn't have been an option; taking advantage of that, a Daniel Glover released the proprietary, LilyPond-reliant Etude App, that is said to have sold quite well. For those of us who don't have an Apple product and actually care about the open Web, the same year Mike Blackstock (after having tried to set up a LilyPond-friendly wiki — in vain ) announced the availability of a collection of online music editing tools, aptly named OMET. Then there was also LilyPondTool contributor Joshua Koo and his project (beware, that's a Facebook link) of a "Google Docs for music" that became JabTunes (the homepage is ugly and contains some Flash, but check out his labs for some pretty nifty HTML5 experiments). Finally, the new kid on the block has been announced quite recently by Trevor Dixon as Lilybin, that looks quite promising. I've also seen other people (including Dominic Neumann, Eric Knapp, Philip Peterson) announce that they were working on online web interfaces for LilyPond; there definitely is a growing trend here.

As early as May 2009, an unidentified developer appeared to be working on a project called WebLily. It wasn't made public until March 2010, as a LilyPond Report comment, although we still didn't know (other than through whois) who was behind it. Learning about it made LilyPond project manager Graham Percival unusually enthusiastic:

Interesting! Never heard of it before. [...] I have to admit that at first I was expecting tons of advertizing (I have a nasty mind), but it all looks legit. The "clickable notation reference" is a neat idea.

(Evidently, Graham's usual grumpy self took over soon thereafter, as he roasted "Mr. WebLily" for his alleged disregard of security risks. [Update: he then apologized, but his concerns were nothing if not legitimate in the first place.])

"Mr. WebLily" soon appeared to have a name, as he not only maintained his WebLily service but also launched a new, more commercial service: Scorio.com — although LilyPond isn't credited prominently on this new website. Johannes Feulner — founder of WebLily and Scorio — also happens to be one of our readers, and he generously sent us the following contribution more than a year ago.

Scorio.com — An online WYSIWYG editor for LilyPond?
by Johannes Feulner

Creative Commons 3.0 BY-SA
Contact: johannes.feulner[at]scorio[dot]com

WYSIWYG? “Impossible”, “stupid idea”, “out of scope” might well be the first reaction of a passionate LilyPond user and LilyPond lover. However at scorio.com we tried to tackle the task and you are invited to see how we did.

scorio.com is a website where you can write, edit, publish and share sheet music right from within your web browser. No software installation is needed, it's all HTML running in every browser. The music is engraved by LilyPond, giving you sheet music of its wonderful quality and beauty. Nonetheless, the editor comes with a graphical user interface completely hiding the user from the intricacies and subtle nuances of LilyPond's mighty input language (Ly code).

In this way scorio.com makes LilyPond available to musicians who do not have programming skills or lack the time to acquire them. scorio.com is well suited for beginners and learners. But being a work in progress you may expect it to cover increasingly larger subsets of LilyPond's endless possibilities to make it a useful tool for all sheet music writing musicians.

How it all began

In early 2009 I became excited about GWT, the Google Web Toolkit to write AJAX-Applications for web browsers in plain Java. The promises were many: No pains with JavaScript to produce effects like Drag&Drop and other AJAX effects, simple server side integration, Eclipse as development tool allowing to debug through client and server side in a single session, runs in any browser, multi language support and many more. Even though I had studied computer science, I've chosen the management track and had not written a line of code for many years. Thus the promise of ease of development was especially alluring to me. And it proved to be true.

Since I am also a professionally trained organist with great love for classical music (my top internet radio station is of course http://www.organlive.com) I came up with the wild idea of writing a score writer for the internet as a demo project. Wading through Google's search I got the impression, that something like this was not yet available on the net. Only later I learned about Noteflight, a Flash based score writer, which might already have been in existence at that time.

But the best thing I found during my search was LilyPond. I was impressed and excited about this marvellous piece of software and wondered why I hadn't found it before, because I really do not like the various proprietary “market leading” score writers I had been using. Their user interfaces are weird, the music layout they produce has quite some potential for improvement.

So my first idea was to write a web based graphical user interface as a frontend to LilyPond. I decided to stick to GWT (all Java based) and to run LilyPond on the server through system calls from my Java servlet.

The precursor: WebLily.net

After visiting the Google IO 2009 conference I was full of enthusiasm, filled up with positive energy and started to hack right away in evenings and on weekends. This project became WebLily.net, a website where you have an input window to type in your LilyPond code (things like { c e g }) and an output window where the graphical result is instantly shown in HTML.

This is pretty much a light version of the Frescobaldi editor. However, the advantage is that the score display is updated while you are typing your Ly code. This is extremely useful for beginners starting to learn the syntax. The type-translate-review cycle is extremely fast for small examples.

Screenshot of WebLily.net with textual input (left) and instant HTML rendering (right)

As a special feature I added the Clickable LilyPond Reference. Whenever you find an example in the reference you can simply click onto the example to start up WebLily's editor. This is very helpful, when you learn about the various features of LilyPond and want to try them out right away.

Another useful feature is that you can click onto a note head and the position of that note head is marked in the Ly code window. To produce the HTML-output I wrote an HTML backend to LilyPond. See below for technical details.

Even though it works quite well and WebLily.net has attracted some actual users, I was not really satisfied. Because what I've really wanted in the first place, is an interactive Editor where I could change the pitch of a note by drag&drop and insert new notes by clicking on them. Doing drag&drop using GWT is not that difficult to realize. However, when you stop the drag&drop operation you must determine the note affected, compute the new pitch and then update the Ly code. WebLily.net uses the point-and-click feature of LilyPond which passes for most graphical objects their line number and character position into the backend. So when I click on a note head I can actually know where this note head has been defined e.g. in line 4 starting at character position 9.

The limitation of this approach is the difficulty to insert the edited changes back into the Ly code. A note head may be a single note (c), part of a chord , durations and octaves may be involved (c'2, 4), there may be trailing parameters (c-.\ff), there may be tweaks (4) and due to the expressional power of LilyPond's syntax (including Scheme expressions) there is no easy way to handle arbitrary inputs. This approach might work for a well defined subset of LilyPond. But then you loose many of the syntactic possibilities and advantages. So I stopped at this point and started thinking of a different solution to this problem.

WebLily.net is free to use for all friends of LilyPond and I want to keep it that way as long as I can pay for the server. You may of course support the project by clicking on the wonderful ads beautifying the website.

The real thing: scorio.com

The WebLily.net project convinced me that an HTML-based WYSIWYG editor can be built but that it will take much more effort to do so. I got into contact with Karin Höthker and Dominik Hörnel, both experienced software developers with a strong background in music. We had worked together for several years at Karlsruhe University in a research project on Informational Structures in Music. We made neural networks learn how to create music in the styles of historic composers such as J.S. Bach and others. But then we lost sight of each other. Both were excited about doing a music related project and so we started building scorio.com.

Now our goal is to create a place on the web where musicians can write and edit music in an interactive score writer, publish their works if they want to and to provide a search engine to find the sheet music others have written. All you should need for this is a computer and a web browser. This portal has started in November 2010 with a first, still limited version of the score writer (note editor).

Having learnt from WebLily we decided to use LilyPond as engraver (layout engine) but not to use Ly code as internal representation for our scores. We rather selected MusicXML as internal format. Finding, inserting, updating notes is much easier with MusicXML, since Java provides powerful tools to read, write and modify XML-Documents and MusicXML comes with a (mostly) well defined semantic.

Screenshot of scorio.com's note editor

So on every click you do on scorio.com we update our MusicXML model (e.g. change a pitch), convert the score into Ly code, run LilyPond with our HTML backend on that code and then display the HTML output as update to the user. Even though this seems a lot to do on each single mouse click the performance is not all that bad (see below). Where WebLily attaches line numbers and character offsets to HTML images, scorio.com uses references into the XML-structure to backlink the HTML output to its corresponding input.

On scorio.com you can produce scores in perfect LilyPond quality. However, there is no way for direct input of Ly code, but you can use the constantly growing (but still limited) repertoire of musical symbols the note editor exposes through its graphical interface. Wouldn't it be nice if someone came up with a ly2musicxml converter?

Is scorio.com a WYSIWYG editor for LilyPond?

The answer is yes and no, depending on how you look at it. The proper answer might be this: scorio.com offers a WYSIWYG note editor running in your browser using LilyPond to engrave your music. But scorio.com is not a WYSIWYG editor where you could start with an arbitrary piece of Ly code and then edit that code in a meaningful way. Currently I do not even foresee a way to import Ly code into scorio.com. The other way around however, i.e. exporting a score as Ly code is perfectly possible. If our users demand it we'll try to get this and other wonderful features onto our roadmap.

Some technical details

For tech savy readers I try to give some technical insights of how scorio.com has been realized.

HTML backend
One major task to do was writing an HTML backend to LilyPond. Looking into the socket backend I got an idea how one might do that. A LilyPond backend gets a bunch of so called Grobs (Graphical Objects). Each of these Grobs consists of one or more graphical elements that have to be placed at a certain position onto the page. The elements are of various types such as line (e.g. staffs), polygon (e.g. beams), bezier-sandwich (e.g. slurs), utf-8 string (e.g. composers' name), named-glyph (music symbols such as Clefs and note heads) and some more.

How might one put this into HTML? Our goal is to produce a solution that will run in any browser (including Microsoft's Internet Explorer and the iPad) without the need to install any special add-ons or plugins. I pondered different approaches such as Scaled Vector Graphics (SVG), HTML5, Flash and others. SVG was and is my favourite. But I wanted a solution working with any browser including Internet Explorer 7 and recently the iPad. IE7 and IE8 do not support SVG nor HTML5 (this is now changing with IE9) and the iPAD doesn't support Flash. Since all those wonderful technologies are not available I decided on using HTML images. Not really rocket science, but it works.

Graphical output of the simple Ly code { c'}

The simple Ly code { c' } produces the following Grob elements:

utf-8-string “ “ outside a Grob, don't really know what that one is for ledger line spanner with one line element for the note's ledger line
staff symbol with 5 line elements
time signature with the music symbol “c” for 4/4
clef with the music symbol for the violin clef
note head with the music symbol for a note head
utf-8-string “Music engraving by LilyPond …” for the tag line

Since HTML supports positioning objects at absolute pixel positions, I decided to create a div for every Grob element, i.e. every line, utf-8-string, music symbol and so on. The element divs are positioned absolutely according to the Grob's coordinates. Since I want to have an HTML rendering close to the LilyPond print out, I decided to produce an HTML image for every Grob element in perfect LilyPond quality.

A note head may serve as an example. The resulting HTML code is an absolutely positioned div (left/top) and within that div sits the note head as an HTML image. The title attribute holds the Ly code position (the note head is defined in line 1 at position 2) that will be displayed as tooltip when hovering with the mouse over the Grob element's HTML image. Note: Unfortunately not all Grobs do come with a source position, e.g. clefs.

src="http://weblily.net/app/service/Engraver ?gt=char&ff=Emmentaler&fs=40&ut=57654&co=0x000000">

As can be seen from code the image is not a static image but it's a call to an engraver servlet that will produce the image on demand. The parameters instruct the engraver servlet to produce a single character (gt=char) from the font Emmentaler (ff=Ementaler) at font size 40 (fs=40). The character to be produced has the Unicode code point 57769 (ut=57769) and the color is black (co=0x00000).

The conversion from LilyPond's coordinates to HTML pixels is straight forward

x-html-pixels = 10 * x-LilyPond-units
y-html-pixels = -10 * y-LilyPond-units (yes, y-directions differ)

One of the (unsolvable?) problems arising is that LilyPond uses floating point numbers for its coordinates, but in browsers we only have pixels at integer coordinates. This inevitably leads to rounding errors. Some times a beam and a note stem or a note stem and a note head do not fit tightly and you may note one pixel holes where they shouldn't be.

Another (solvable) problem is the positioning of characters and strings. As you note in the above example the image has a top position of -5px. Why? The reason is that LilyPond quite naturally uses the base line as a reference point. The div is positioned at the characters baseline. However, HTML images are positioned with respect to their upper left corner. There is no notion of a baseline in HTML. Since the note head rises 5 pixels above its base line (this is called the ascent of a character) we must move the image's position up north by that amount. Unfortunately the ascent information is not present in LilyPond's utf-8 callback for backends. So I had to add an additional parameter. This is one of the very few changes I had to apply to LilyPond's C/C++ source code.

Since every Grob element is an HTML image it is easy to use GWT to setup HTML callback functions to handle mouse events, to implement drag&drop, to insert new notes and to select/deselect notes for editor operations.

Performance
When using an interactive note editor such as scorio.com every click with the mouse may change the score's layout completely. Think about adding or deleting a note in the middle of your score or changing the time signature. Even though you may find some special cases where the effects of a click are strictly local you end up with the requirement to run LilyPond on the server for nearly each and every click. At first this sounds bizarre but after a while you get used to the thought, at least while working on little test examples. On weblily.net you can monitor the server performance in the lower right corner and see values around 0.6-0.8 seconds editing the simple above mentioned score.

Both project use the very same hardware: Intel XEON E5410, two cores 2.3 GHz, 1.7 GB RAM. On scorio.com some tuning efforts brought this down to 0.2-0.4 seconds.

But we may not forget to add network latency which easily adds 0.3 to 0.6 seconds. This isn't really snappy but since things on the web aren't that fast in many cases I think this is acceptable and makes for a good user experience.

(Solvable?) Problem: LilyPond's processing time grows exponentially with the size of the input score. Waiting 10 seconds for an update after clicking into a three page score is not really acceptable. So we had to resort to partial updates, processing just one system at a time. Thanks to LilyPond's skipTypesetting we found a good way of implementing that. So on scorio.com you will note, that after edit operations the following systems are grayed out, since only the current system is updated and their layout may have become invalid. Click on the gray area for a full score update when you are ready to update the page.

I have great hopes into LilyPond moving to Guile 2.0 which has a built-in compiler for Scheme code. But I am aware that this will make scorio.com snappier (single systems will engrave faster) but it will not solve the problem with larger scores. Maybe some reengineering of LilyPond could be done to provide incremental updates? I'd be happy to get into a discussion about that.

Links

http://www.lilypond.org/
http://www.scorio.com/
http://www.weblily.net/

Thanks to Johannes for this contribution!

So, what's coming next for "cloud-oriented" LilyPond applications? It certainly is too early to tell, but if history has taught us anything, it's that the LilyPond community closely and strikingly follows the general trends in computing and software development (both Free and proprietary), so we can safely assume that online music interfaces aren't going anywhere anytime soon. — A trend that's also taken by our friends at MuseScore. On the WYSIWYG side, I've also noticed with great interest that Jan Nieuwenhuizen's Schikkers List, which we introduced last year, is based on the GTK3 toolkit, that has an HTML canvas backend. Just sayin'.

Another point that shouldn't be overlooked, is of an ethical matter. Free software advocates have been warning about the dangers of "cloud computing" and entrusting any third-party (particularly of a private/commercial nature, with the degree of instability and unforeseen policy changes it may imply) with your data. Granted, a music score hardly contains as much personal confidential information as, say, your email account; nevertheless I for one wouldn't ever store my music scores on a webserver that isn't owned by me or a non-profit community I know and trust — not without a solid local backup, at least.

As I said above, a point certainly can be made that such Web-applications provide an excellent introduction to GNU LilyPond, and a great way of working together on music scores. That is, assuming people know the app they're using is powered by LilyPond: while it is mentioned on scorio (albeit not prominently), I couldn't find any mention whatsoever on EtudeApp.com. If anything, we must stay vigilant and make sure LilyPond remains freely and easily accessible other than through proprietary overlays; we also should encourage developers of such web applications to make use of Free licenses whenever they can, knowing that it won't drive people away but on the contrary gain support amongst Free software enthusiasts and the LilyPond community at large!

An addendum to LilyPond Report #23

In our previous installment, we mentioned that some rather simple syntax had been deprecated for several years. However, it turns our that current developments allowed for a further simplification of the LilyPond example presented in the article "Prelude #1 in Scheme". Check it out under this link.

That concludes the twentyfourth installment of the LilyPond Report. Our special guest next month will be Janek Warchoł, a young, enthusiastic LilyPond contributor who'll share with us his passion and ideas. Stay tuned!

Cheers,
Graham Percival, David Kastrup & Valentin Villenave.

The LilyPond Report #23

David Kastrup, Valentin Villenave — 2012-01-20T17:11:43Z

Table of contents

Greetings everybody, and welcome to this twenty-third issue of the LilyPond Report!

Another year, another Report. This month we're welcoming new LilyPond Report editor David Kastrup, who (in addition to being a talented developer) has been busy writing about some of the new, awesome features recently added to LilyPond... And speaking of awesomeness, don't miss his interview with composer/contributor Mike Solomon, whose work never ceases to amaze!
As always, you can post your comments at the bottom of the page, or send your contributions to the LilyPond Report's next issues.

Editorial

by David Kastrup & Valentin Villenave.

Greetings everybody, and a happy new year to all!

Yes, it's that time again, when the Report raises back from the dead and its editors make plans to keep it alive a bit further. This year we're back with a slightly larger team, since GNU veteran programmer David Kastrup has joined us to make the Report even deeper and more interesting!

That, however, will obviously not be enough on the long run. Thus, consider this editorial a call to arms: more than ever, the LilyPond Report needs more contributions from its readers! User testimonies, programmer complaints, nice stories (no matter how fluffy or useless), benchmarks and tests, anything that can make our community a more pleasant or a more efficient place. We're particularly interested, of course, in hearing from the various team involved: translators, bug squad, programmers, documentation writers, this Report is yours!

Not only is this (sort of) monthly newsletter a fun way to report the overall shape of the LilyPond project to the general public; it's also, for our community itself, a place of choice to call attention to a particular point or problem, that otherwise might get lost in the endlessly-flowing mailing lists.

For example, there may be a need to regain some interest in our reviewing procedures: many pretty good LilyPond programmers don't find the time to take part in reviewing patches. We need to make sure that what ends up in LilyPond can be used and understood by more than just its original author — which may not happen when things are left underdocumented, or when the contributor, coming from a music background, does not obey basic programming principles. Another matter in need of work and attention is our infrastructure: Graham Percival has recently posted a call for more tasks to be automated, in order to make the "routine maintenance" less unpleasant and frustrating.

Release news

by Valentin Villenave

We are now at version 2.15.26 (the previous unstable version 2.15.25 having been skipped due to build problems). This development release contains the usual number of bugfixes, and should in particular bring happiness to Microsoft Windows users, who have been previously unable to run lilypond-book due to a particularly annoying bug.

Please note, however, that a few critical bugs remain; this is but a development version — not even a release candidate — and ordinary users should preferably use the stable 2.14 version.

What's up with LilyPond?

by Valentin Villenave, David Kastup & Graham Percival

It has been a long time since the last Report has been published, so it is not surprising that a number of things have happened. In the free software tradition of scratch-your-itch, a few new contributors have appeared and worked on their favorite problems, but there has been less development activity these past few months than in the summer. Part of the reason might be that new procedures for committing changes have been tested and put into place, and occasional contributors might feel hesitant about what this means for them. In a nutshell, direct commits to master are discouraged; instead you should push to the staging branch. After automated testing, the staging branch will get committed to master automatically. The benefit of this procedure is that the little accidents that tended to make master unusable several times a month can be avoided.

Seasoned developers might feel insecure about what procedures to follow. But those are actually simple. Check out the summary for experienced developers, a very short but important document that should, alongside with the longer Contributor's Guide, tell you all there is to know with regard to our project development and current organization.

An Interview With Mike Solomon

by David Kastrup

Our special guest this month is Mike Solomon, a valuable LilyPond contributor and a rather unpredictable composer; after having studied and worked in California and Florida, he has been living in France for a couple of years. Our editor David Kastrup had this interview with him via email in December 2011.

The LilyPond Report — Hello Mike, thanks for agreeing to do this interview with the LilyPond Report. You have become one of the more prolific contributors to LilyPond recently, and that of course piques our readers' interest. Of course, we can hardly avoid talking LilyPond, but let's begin with the big picture: How would you describe yourself when music is not a topic?


A portrait of Mike Solomon: "I'm the one seated on the cellist, facing backwards."

Mike Solomon — That limits me to the 3% of my life that is not musical, which is mostly devoted to comedy writing and New York Jets football.

L.R. — Interesting; any links we could share?

Mike — There's not that much... But otherwise, comedy is a big part of my work, all of which is downloadable on my personal website.
As for the Jets, there's not much to link to - I am a die hard fan and I try to watch the games whenever I can.

L.R. — A quick back-of-the-envelope calculation makes me guess that you spent about 9 months on those endeavors. At what time of your life would you say that music did become a topic?

Mike — Rumor has it that my father gave me pre-natal piano lessons.

L.R. — Wouldn't your mother have preferred a smaller instrument for your first home teaching lessons? What would such a lesson have looked like?

Mike — Well, that's a picture you can't post on this website...

L.R. — Speaking of which, would you say that the photograph you sent me (thereabove) is typical for a gig of yours?

Mike — It's tame.

L.R. — What instruments do you play yourself, and when did you start with them? What was it that made each instrument important for you to learn?

Mike — I have played piano for 29 years and have sung for 23, in various ensembles, shows and solo gigs in America and Europe.


Das Zauberbuch (excerpt): M. Solomon, 2011 (Original file.) CC by-sa.

L.R. — Looking at the information so far, that would have made you start singing at the age of 5 or 6...

Mike — Piano was never really a choice - it was just always kinda around. I am not a very good player, but it is tough for me to conceive of my life without the piano - it's where my hands feel most at home.
Singing was thanks to the American public school system: the best for music education in the world.

L.R. — Considering the music education I received in Germany (my "Early Musical Education" classes started before primary school so that I actually learnt sight reading earlier than reading), the assertiveness of this statement rather piques my curiosity.
In what respects would you consider the American school system to have given you a better musical education than that you see pupils in France getting?

Mike — I can't speak for how music education is done in 2011, but the extent of most French twenty-somethings' musical education was learning to play the flute for a couple years unless their parents enrolled them in a conservatory. Several localities have gone well beyond this, and I have the good fortune of working for one of them (Dunkirk) that is great about bringing music into the public schools.

L.R. — Oh, and of course a question that every artist (and programmer of free software) can't avoid hearing occasionally: is that how you make a living?

Mike — Yes, and every score that has been part of the making of this living has been typeset using LilyPond.

L.R. — Looking at examples of your scores, one gets the impression that you focus on different things than one would usually expect from composers (I'd actually say that your code exhibits a similar style). Putting this focus to paper would, for that reason, appear to favor tools that have not specialized too much on standard tasks.
Can you tell us when LilyPond caught your interest, and what it was that made it appeal to you?

Mike — It's not that I found LilyPond to be the ideal tool suited to my ideas, but rather that my manner of creating work changed as I discovered LilyPond. When I was a graduate student at the University of Florida, the bottom 2/5 of my G4 PowerBook's monitor started to randomly black out. One day, while waiting for the bus, a cyclist riding on the sidewalk hit me in the back in full stride. I had my PowerBook in my backpack and this sealed the deal for the lower 2/5 of my monitor. So, not able to use a GUI (Finale) anymore, I turned to LilyPond (with which I had until then only experimented).

L.R. — It seems like the best prospects for recruiting high-profile LilyPond developers should then be in the Netherlands, the motherland of bicycling. Given your rather special requirements, what is your preferred editor for creating LilyPond scores?

Mike — Joe's Own Editor. I've tried them all and got speediest fastest with joe. I've since had no need to switch.
Using the tool helped me realize that the true value of free software is not that it lacks a purchase price, but that it gives you the freedom to do whatever you want. So, the "focus" of which you speak is something that, to a large extent, exists because of LilyPond.

L.R. — That's quite an inspiring endorsement for the founding principles of Free Software.
What made you start working on LilyPond rather than with it in earnest?

Mike — I had a gig coming up where I needed woodwind fingering charts and LilyPond didn't have them, so I made them.

L.R. — Now I am curious: what was it that you could express using a fingering chart that was not possible to convey musically using notes?

Mike — Multiphonics and bisbigliandi.


granini di luce beccucciati da uccelli di silenzio (excerpt): M. Solomon, 2009. (Original file) CC by-sa

L.R. — And you did not mention playing woodwind instruments yourself: so how have you acquired instrument-specific knowledge that is so special that its notation requires fingering charts?

Mike — Lots of reading - there are lots of great texts and websites out there with audio examples that provide solid information to start you off. But multiphonics vary a great deal from player to player and from instrument to instrument and I needed a tool that could make these diagrams fast as I got feedback from performers.

L.R. — I have to add that as a violinist I found it quite liberating to work with Urtext editions of Bach's solo partitas and sonatas: while Bach did not write down fingerings, they were inherent in the notes, and reading the mind of the composer over wide passages by playing the singularly anatomically reasonable fingering was actually quite an elating experience.
Do you think that exercising this kind of tight control over the performer and his instrument leads to results that better match your musical vision?

Mike — Absolutely. That's not to say that all forms of specificity lead to better results, but anything that gives more information about the mechanics of how to play an instrument in notationally-ambiguous situations seems almost universally welcome. Sometimes, the tight control doesn't line up with a musical vision as much as a "what would happen if I..." thought experiment. For example, the work six cercles for six players (takes a while to download - best viewed via Opera) is meant to garner a response from the performers that less precision would have a difficult time conjuring up. This work, like all of my recent work, is heavily indebted to LilyPond and other open source software.
The same holds true for footnotes and a few other features. Doing these things helped me to get to know certain corners of the LilyPond code base well, so each subsequent change is not that taxing. If I see a request for a feature, I can usually estimate how much time it'll take to implement and, if I have the time, I implement it.

L.R. — Would you say that the availability of LilyPond has had an influence on how you compose music? Or do create your compositions mostly independently from how you end up typesetting them? Have you changed decisions in your compositions because of typesetting problems?


Lucky Wok (excerpt): Mike Solomon, 2012. This is a short preview of a yet-to-be-published score.

Mike — I only ever conceive of (non-video) works in terms of how they will sound. I record myself singing mock-ups of the entire work before a single note is committed to the page and then transcribe it through LilyPond. The idea that something I do is not transcribable does not enter my head: I have confidence that if I want a sound, the only limitation in my getting it is my own failure to notate it, not LilyPond's failure to provide a means of notating it. I have never run into a typesetting problem that was not solvable through LilyPond, InkScape, or JavaScript. The only time I ever change my mind in terms of typesetting is a question of expediency: sometimes, I'll need to use a crude hack to get something done instead of finding an elegant and extensible solution. I try not to do this too often.

L.R. — Where would you want to see LilyPond heading? What areas in LilyPond are you comfortable working on? What areas would you like to see more work on? Are there areas where you would have wanted to do work on, but have found too hard to navigate?

Mike — There are certainly areas of LilyPond that I do not know at all. MetaFont, for example, is a magical black box out of which LilyPond's fonts appear. The build system is a black forest of commands that somehow assemble LilyPond. At a certain time, all of LilyPond appeared like this to me, but the code is well written and, if you follow the order of what things are called when, it can't not make sense.
I'd love to see more work on certain basic structural features of LilyPond that, by design, limit how fast it can evolve. For example, the way that non-musical objects (like markups) are laid out in LilyPond is underdeveloped compared to musical layout. I'd like to see this streamlined into a unified system that creates graphical objects and puts them on a given medium via various layout managers. I would also like to see LilyPond adopt linear programming so that certain constraints do not have to be solved one-after-the-other but instead can be solved in concord based on a given goal.

L.R. — Now being able to interface the inner linear programming blackboxes with user-defined typesetting tasks actually has been one pet peeve of mine with TeX, so if you figure out how to do this reasonably well and transparently, you'll certainly have my blessings.

Mike — Lastly, I'd like to represent graphical objects' boundaries with more precision so that collision avoidance can be more fine-tuned and uniform.
I see LilyPond becoming like SCORE with easier input syntax and more extensibility. I think that, if the program gets more modular and can drum up development teams around certain issues whose progress does not impede work on other issues, it can get there in the next five years. I, as well as several other LilyPond users and developers working in France, have been trying to integrate LilyPond into music pedagogy in hopes that students find it useful and, if so, that they contribute to the development of the program.

L.R. — Well, it certainly sounds like something that LilyPond could make use of, and perhaps this task will become easier as LilyPond progresses.
Thank you for this rather inspiring interview, and all the best for your work with and without LilyPond!

Mike — Thanks!

Feature story: Prelude #1 in Scheme

by David Kastrup

Now as an illustration of the impact of the recent work on LilyPond programming, I want to pick up an example that Nicolas Sceaux has made the topic of a LilyPond programming article on his website.

First thing to note is that his analysis and solution remain fully valid for now: after running convert-ly on his input file to account for some comparatively minor syntax changes since when it was written, the file runs and compiles.

Nicolas starts his analysis by showing the results of an evaluating several expressions at the Guile prompt in order to figure out the structure of his input expressions, as well as the structure of the output expressions he is going to create from them:

guile> (display-scheme-music #{ \notemode { c' e' g' c'' e'' } #})

The article does not actually mention how to arrive at such a prompt but the current version of LilyPond offers in its Scheme tutorial the advice to start

lilypond scheme-sandbox

for that purpose, and if it had been available at that time, it would have worked similarly. A noteworthy difference, however, is that

guile> (display-scheme-music #{ c' e' g' c'' e'' #})

is now sufficient (and does not produce unwanted additional output). As the next step in his tutorial, Nicolas defines a macro -> that is used as a syntactical shortcut to traverse through the layers of the SequentialMusic → EventChord → NoteEvent hierarchy. Now this actually touches a sore point in LilyPond: there are no established standards for conveniently deconstructing a music expression. LilyPond has ad hoc functions in several modules for doing this kind of operation. Some of these ways are more robust to changes of LilyPond (like if the intermediate layer of EventChord were to become optional, Issue 2070), some less so. But there is no published interface.

Addressing this deficiency should be a future target, but in the mean time, things have become quite more relaxed because music functions can recognize a lot more argument types, including pitches. Since pitches are readily available as music function arguments, there is no more need to deconstruct whole music expressions in order to have access to the actually required information, the pitch. As a consequence, the main task of converting five pitches to a one-measure phrase (distributed across three voices) can be done with the simple function

ph = #(define-music-function (parser location p1 p2 p3 p4 p5) (ly:pitch? ly:pitch? ly:pitch? ly:pitch? ly:pitch?) #{ \repeat unfold 2 { $p1 2 } | \repeat unfold 2 { r16 $p2 8. ~ $p2 4 } | \repeat unfold 2 { r8 $p3 16 $p4 $p5 $p3 $p4 $p5 } | #})

The other trick we are employing here is using "\parallelMusic" for distributing material across three voices. To be fair, \parallelMusic received a few tweaks upstream to cooperate better with music functions in the course of this article, but nothing that would not be equally desirable for a number of other things.

The last design decision leading to simple code is not to use an iterating construct over a single long music sequence, but rather call the above macro manually for each five-note pattern. While the automatic repetition did not add significant complexity to Nicolas' rendition of the problem, it does not really fit well with the simplicity we can now achieve.

All in all, the meat of this piece can now be presented as

\parallelMusic #'(low middle high) { \ph c' e' g' c'' e'' [...] \ph a c' e' g' c'' \voiceTwo | \change Staff = "down" \voiceOne | \oneVoice | \ph d a d' fis' c'' [...] \ph c, c g bes e' c,2~ c, | r16 c8. ~ c4 ~ c2 | r8 f16 a c' f' c' a c' a f a f d f d | c,2~ c, | r16 b,8. ~ b,4 ~ b,2 | r8 g'16 b' d'' f'' d'' b' d'' b' g' b' d' f' e' d' | c,1\fermata | c1 | 1\fermata \bar "|." | }

This sets up the three music variables low, middle, and high with the appropriate material for the respective voices.

Full code

ph = #(define-music-function (parser location p1 p2 p3 p4 p5) (ly:pitch? ly:pitch? ly:pitch? ly:pitch? ly:pitch?) #{ \repeat unfold 2 { $p1 2 } | \repeat unfold 2 { r16 $p2 8. ~ $p2 4 } | \repeat unfold 2 { r8 $p3 16 $p4 $p5 $p3 $p4 $p5 } | #}) \parallelMusic #'(low middle high) { \ph c' e' g' c'' e'' R1*7 | \skip 1*7 | \oneVoice R1*7 \voiceOne | \ph a c' e' g' c'' \voiceTwo | \change Staff = "down" \voiceOne | \oneVoice | \ph d a d' fis' c'' \oneVoice R1*21 \voiceTwo | \skip 1*21 | R1*21 | \ph c, c g bes e' c,2~ c, | r16 c8. ~ c4 ~ c2 | r8 f16 a c' f' c' a c' a f a f d f d | c,2~ c, | r16 b,8. ~ b,4 ~ b,2 | r8 g'16 b' d'' f'' d'' b' d'' b' g' b' d' f' e' d' | c,1\fermata | c1 | 1\fermata \bar "|." | } \score { \new PianoStaff << \new Staff = "up" { << \high \\ \middle >> } \new Staff = "down" { \clef bass \low } >> \midi { \context { \Score \with \settingsFrom { \tempo 4 = 80 } } } \layout { \context { \Score \with \settingsFrom { \compressFullBarRests } } \context { \Staff \with \settingsFrom { \accidentalStyle modern } } } }

One advantage of using an explicit macro \ph for each occurence of the default phrase is that we can just fudge in stuff like the staff change and the ending directly.

The spine of the piece can now be put together as

\score { \new PianoStaff << \new Staff = "up" { << \high \\ \middle >> } \new Staff = "down" { \clef bass \low } >> \midi { \context { \Score \with \settingsFrom { \tempo 4 = 80 } } } \layout { \context { \Staff \with \settingsFrom { \accidentalStyle modern } } } }

The context definitions introduce us to \settingsFrom, another addition. While the Midi definition looks like a step backwards from the original

\midi { \tempo 4 = 80 }

it turns out that this syntax has been deprecated since version 2.9.16 [UPDATE: see David's Post-Scriptum below] and would have required rewriting as:

\midi { \context { \Score tempoWholesPerMinute = #(ly:make-moment 80 4) } }


Præludium BWV 846: Download the complete LilyPond example.

Nicolas' original version did not set an accidental style, something we achieve score-wide with another use of \settingsFrom. Instead he manually forced the natural in measure 29. We cannot do this in input to \ph since forced accidentals are not a property of the pitch but rather of the note event, and thus are no longer accepted. But accepting a forced natural is not the same as actually interpreting it, and the original code failed to extract this information in addition to the pitch itself, and thus did not work as intended anyway.

So if we recapitulate in what respects the current version of LilyPond has made it easier to tackle this particular challenge, we can note the following points:

The actual information we wanted to process had been a sequence of pitches. Not having to extract them from single notes has been the major simplification of the original code.
\parallelMusic has been improved to a degree where we could employ it for this task. We have, however, been lucky that our basic phrase length has been a single measure: if it would have been half a measure, changing voices with a bar check would have resulted in warnings. Probably \parallelMusic should be changed to just remove the bar checks, and instead check that every set of fragments is consistent in its length.
\settingsFrom is a convenient tool for creating context modifications.
The addition of a simplistic (and documented) Scheme sandbox actually provides an environment where one can carry out the experiments that Nicolas showcased.

While user-level programming of LilyPond has a lot of potential for further improvements, this real-life example shows that we are on a good road for bringing the complexity down to a level where a user can actually expect to achieve useful results with a justifiable investment of learning.

Post-Scriptum: Prelude #1 in Scheme

Update from March 1., 2012:
In one of the examples above, we wrote that some syntax has been deprecated for several years; well it turns out that the upcoming version 2.15.32 will now allow writing the above \midi and \layout blocks just as

\midi { \tempo 4 = 80 } \layout { \accidentalStyle modern }

thus no longer requiring you to know which contexts actually require context modifications for the given functionality. This works with most property-setting music commands and makes it much easier to set global styles for music.
D.K.

Bug Report of the Report

Valentin Villenave

From its very first issue nearly four years ago, the LilyPond Report has always had a "favorite quote" or "favorite bug report" section. Of course, it's been a while since I've looked at our bug tracker; nevertheless, I've been keeping some quotes for a rainy day.

One of these doesn't come from an actual bug report, but from a patch which Mike Solomon (our special guest this month) proposed in May 2011: Allow LilyPond to ignore certain note-heads in a stem. Seeing Mike (then a rather new contributor) tackle such a useful work made our Project Manager Graham Percival... well, quite inspired:

I see trees of green, red roses too,
I watch them bloom for me and you and the broken input/regression/drums.ly
And I think to myself, what a wonderful world.

I hear babies cry, I watch them grow,
They'll learn much more, than I'll ever know, like a ton of programming

error: Grob direction requested while calculation in progress. continuing, cross fingers errors.

And I think to myself, ohh what a wonderful world

Got any public email, patch, request or bug report worth quoting? Then please do chip in!

That concludes the twenty-third issue of The LilyPond Report. Although we have no plans for a predictable publishing schedule yet, Our next installment is to be expected somewhere in February 2012. Lots of things are happening currently, so this Report will be outdated by then — which is a Good Thing™.
In the meantime, feel free to send us your contributions !

Cheers,
Graham Percival, David Kastrup & Valentin Villenave

The LilyPond Report #22

Valentin Villenave — 2010-11-03T18:01:31Z

This informal opinion column is about the GNU LilyPond project: its team, its world, its community. It is not meant to be an exhaustive documentation resource. Reader comments are, of course, welcome (see at the bottom of this page).

Table of contents

Welcome to this twenty-second issue of the LilyPond Report!

Here's a new Report, full of surprises and exciting news for our community! To be found in this issue is an up-to-date, complete list of all LilyPond mailing lists and forums around the world — and we do mean: complete. Also, for the very first time our special guest today is LilyPond's co-founder and core developper Jan Nieuwenhuizen, who has been busy these past months — read on to find out what for! As always, you are warmly invited to post comments at the bottom of the page, or even contribute to the LilyPond Report's next issues, by contacting Valentin or Graham.

Editorial

(by Valentin Villenave)

What's not to love with GNU LilyPond? Meaning: is it at all possible, either to mildly appreciate it, or perhaps even to hate the hell out of it?

On our reviews page, I recently stumbled upon Nicolas Sceaux's statement that he used to have "a love-hate relationship" with LilyPond. Coming from arguably the most skilled LilyPonder in the world, this is somehow surprising.

Thinking about it, however, I soon realized that my own involvement in the LilyPond project could accurately be described as some kind of a bipolar disorder: sometimes I can get very excited, eager to help, eager to please, and nothing seems to go fast enough. However, a contrecoup is always to be expected: at some point, things suddenly seem to go too fast for me to keep up, and I find myself not doing anything for months. Finally, on rare occasions, something gets me frustrated — and then, for a short time that feels like forever, everything is tainted with bitterness and rage.

Getting the Report out (almost) on time, in this regard, is always a challenge. So many exciting things to talk about... or, depending on how I look at it, so many saddening things that I would rather not talk about. This month we're having plenty of each, and therefore the LilyPond Report is unfortunately not getting any shorter; if you only have one thing to read though, please skip directly to our interview with Jan-, who has some pretty awesome news for all of us!

Release news

(by Graham Percival & Valentin Villenave.)

Work continues towards 2.14; the fourth alpha test, 2.13.38, is now online! This release still has one crash and two cases of suspicious spacing, so we cannot recommend it for daily production use. The current stable release is 2.12.3-2.

We have been experiencing several problems with our download mirror, but it seems to be fixed now. In case that happens again, and to alleviate the load of our bandwidth sponsor, it may be a good idea to set up a BitTorrent server/tacker somewhere (on LilyNet?), or to use an existing one (preferably free-software friendly). Does that sound appropriate to you? Would you be interested in downloading/sharing/seeding LilyPond binaries over the network? Please let us know!

A list of LilyPond communities?

(by Xavier Scheuer)

This month, Valentin suggested me to draw up an updated list of LilyPond communities for the LilyPond Report, but I decided to extend it to every LilyPond-related "Useful Addresses". This will also help keeping our contact page up-to-date; please help us complete this list if you know of any addresses that should be included here.

GENERAL, INTERNATIONAL "OFFICIAL" LILYPOND MAILING LISTS (ENGLISH-SPEAKINKG)

User mailing list: lilypond-user_at_gnu.org
Info & subscribe: http://lists.gnu.org/mailman/listin...
Archive: http://lists.gnu.org/archive/html/l...
Administrator: David R. Linn

Releases mailing list: info-lilypond_at_gnu.org
Info & subscribe: http://lists.gnu.org/mailman/listin...
Archive: http://lists.gnu.org/archive/html/i...
Administrator: David R. Linn

Developer mailing list: lilypond-devel_at_gnu.org
Info & subscribe: http://lists.gnu.org/mailman/listin...
Archive: http://lists.gnu.org/archive/html/l...
Administrator: David R. Linn

Bug mailing list: bug-lilypond_at_gnu.org
Info & subscribe: http://lists.gnu.org/mailman/listin...
Archive: http://lists.gnu.org/archive/html/b...
Administrator: David R. Linn

LOCALE, LANGUAGE SPECIFIC MAILING LISTS, FORUMS, GOOGLE GROUPS, ...

DE
Deutsches Lilypond Forum: http://www.lilypondforum.de/
Info & subscribe: http://www.lilypondforum.de/
Archive: http://www.lilypondforum.de/
Administrator: Till Paala

ES Lista de usuarios de LilyPond en español: lilypond-es_at_gnu.org
Info & subscribe: http://lists.gnu.org/mailman/listin...
Archive: http://lists.gnu.org/archive/html/l...
Administrator: Francisco Vila

FR
LilyPond French Users: lilypond-user-fr_at_gnu.org
Info & subscribe: http://lists.gnu.org/mailman/listin...
Archive: http://lilypond-french-users.129896...
Administrators: John Mandereau, Valentin Villenave

HU
Magyar LilyPond-felhasználók közössége: lilypond-user-hu_at_googlegroups.com
Info & subscribe: http://groups.google.com/group/lily...
Archive: http://groups.google.com/group/lily...
Administrator: Dénes Harmath(?)

NL
Nederlandstalig LilyPond forum: http://www.lilypondforum.nl/
Info & subscribe: http://www.lilypondforum.nl/
Archive: http://www.lilypondforum.nl/
Administrator: Wilbert Berendsen

PT (BR)
Lilypond-Brasil: lilypond-brasil_at_googlegroups.com
Info & subscribe: http://groups.google.com/group/lily...
Archive: http://groups.google.com/group/lily...
Administrator: Hugo Ribeiro

UNOFFICIAL LILYNET.NET MAILING LISTS

LilyPond Frogs: frogs_at_lilynet.net
Info & subscribe: http://lists.lilynet.net/frogs/
Archive: http://lists.lilynet.net/frogs/
Administrator: Valentin Villenave, Carl D. Sorensen

LilyPond's support for tablatures: tablatures_at_lilynet.net
Info & subscribe: http://lists.lilynet.net/tablatures/
Archive: http://lists.lilynet.net/tablatures/
Administrator: Valentin Villenave

LilyPond translations: translations_at_lilynet.net
Info & subscribe: http://lists.lilynet.net/translations/
Archive: http://lists.lilynet.net/translations/
Administrator: Valentin Villenave, John Mandereau

LilyPond Midi discussions: midi_at_lilynet.net
Info & subscribe: http://lists.lilynet.net/midi/
Archive: http://lists.lilynet.net/midi/
Administrator: Valentin Villenave, John Mandereau

LilyPond proposals list: proposals_at_lilynet.net
Info & subscribe: http://lists.lilynet.net/proposals/
Archive: http://lists.lilynet.net/proposals/
Administrators: Carl Sorensen, Graham Percival.

The GLISS project has begun!: syntax_at_lilynet.net
Info & subscribe: http://lists.lilynet.net/syntax/
Archive: http://lists.lilynet.net/syntax/
Administrator: Valentin Villenave, Graham Percival

[Editor's note: These last two lists are not active yet; of course the Report will let you know when/if they are.]

VERY (TOO?) UNOFFICIAL, INFORMAL

http://lists.lilynet.net/devel-chat/
Other "secret" mailing lists? ;p

Post-scriptum by V. Villenave.

The devel-chat mailing list is a public mailing list, that was officially announced a year ago. Subscription is freely allowed, and everybody's invited — provided that you don't expect serious discussions

A country is missing from that list: Italy. Our Italian contributor Federico Bruni, whom we asked about it, told us that he was considering creating an Italian LilyPond community as soon as he'd be able to translate some part of the Documentation. He does need help, so feel free to give him a hand on the translations mailing list!

Another point is to be made here. When Xavier asked around to find if he hadn't forgotten any mailing list, nobody bothered answering him [1]; it would be rude enough in itself, if it weren't for one annoying detail.
There actually is a secret mailing list.

The case of the Hidden Mailing List
The LilyPond community as I see it, is divided in two parts: there are those who know about -hackers, and those who don't. Oh, you can search everywhere on the Web, on our archives or anywhere else, for the lilypond-hackers mailing list: chances are you won't find anything. Its archives are, of course, well-kept; it is not even mentioned in the list of GNU mailing lists.
Earlier this month, it was merely alluded to on the developers' mailing list (read: the one developers' mailing list we do know about), and nobody seemed to take issue with the fact that important questions were admittedly discussed privately. — As a matter of fact, I do.

Full disclosure: I am not a member of this mailing list. Would it make me happy to be one? It certainly would, albeit in a very childish way. Would I deserve it? Obviously not. Does it make me feel sick? Hell yes.
However, whilst I have to acknowledge my own frustration not to be part of the Cool Kids Club™, I highly doubt I'd ever be comfortable being part of something I find abnormal, unethical and deceiving. I fail to see, indeed, how such practices are compatible with the openness, transparency, mutual respect that any community-based project requires. Any decent one, that is.

Granted, private mailing lists are not entirely uncommon in Free Software. Granted, if no such list existed, then the Cool Kids would probably exchange private e-mails on occasion, for "frank discussions that might otherwise hurt other people's feelings". Does this make it okay to open a whole frigging mailing list only dedicated to that purpose? This question should be open to debate, which it is not. If and when such lists exist, people usually know about it, know who takes part in these discussions, and why. Nothing comparable is to be found in the LilyPond project — yet one of the most friendly communities I've been involved with (or so I thought).

I first heard (by accident) about the lp-hackers mailing list a few years ago, and that made me bitter enough. By that time I was already spending most of my time contributing to the LilyPond community (albeit in my usual fluffy, unreliable, non-productive way). Discovering that there was a non-trivial part to it that I knew nothing about (and that contributors weren't deemed important enough to be told about it), made me consider leaving the project altogether. After a few weeks, as I was assured that this mailing list really was but a memory of the "old" development team, that nobody was using it any more, and that there was nothing to worry about, I eventually decided to suck it up, stay, try and make things better — one of the reasons why I started the LilyPond Report. Discovering, today, that it actually is still very much active and intended to stay that way, makes me wonder if that was the right thing to do.

Like I said, I certainly have a highly emotional point of view on this subject, and it doesn't make it easy to get a balanced perspective. Am I overreacting? Is there really nothing to talk about? I am looking forward to reading the community's reactions, the development team ~~excuses~~ reasons, and most importantly, what You, dear reader, might have to say about that!

[UPDATE: Please read the Post-Scriptum at the bottom of this issue, just above the comments.]

News from the Frog Pond

(by Graham Percival).

The Frogs are ordinary LilyPond users who have chosen to get involved in their favorite software's development. Fixing bugs, implementing new features, documenting the source code: there's a lot to be done, but most importantly: this is a chance for everyone to learn more about LilyPond, about Free Software, about programming… and to have fun!

All Frog action in the past month has centered around guile 2.0 work. The new major version of this GNU Free-Software Scheme interpreter (and now, compiler!) is expected within weeks, and LilyPond is getting ready to fully support it!

Hopefully once LilyPond 2.14 is out, we might get more new volunteers interested in improving our source code; if you are interested, remember that the word is: join the Frogs!

LilyPond's Companions (1)

(by Valentin Villenave)

Part One: The Quest of the Lost Interface

Up to this day, LilyPond is, has always been, and remains, a non-graphical program. No windows, no staves and notes that you can move around with the mouse, like in other music notation software (proprietary or not) you may be familiar with.

The idea of a LilyPond GUI has long been the Holy Grail of LilyPond users, for generations. Some see it as a myth, others keep hope of finding it someday, while others have been engaging in endless quests to build their own. Most newcomers to LilyPond often feel frustrated (myself included).

Eventually, everybody either gives up or gets used to writing LilyPond code on their own, and the closest thing to a GUI that we can think of is but an improved source-code editing environment, such as LilyPondTool or Frescobaldi (which we reviewed a few months ago). At best, even the most idealistic of us in their wildest dreams, can't imagine anything but to eventually ship LilyPond alongside with a lightweight editor and PDF viewer.

Little do we remember, however, that the founders of the LilyPond project themselves, developers Han-Wen Nienhuys and Jan Nieuwenhuizen haven't been exactly deaf to these calls for user-friendliness (and smooth shiny blingy interfaces). In 2004, Jan first tried and displayed music objects on a GNOME canvas. Then in 2005, Han-Wen launched his own project, codenamed Ikebana.

"Ikebana, Jan recalls, was created by Han-Wen in 2005, as a demo/proof of concept for integrating LilyPond (2.6 at the time) with a big 3rd-party commercial sequencer. Ikebana is 99,98732% Han-Wen's work. I just resurrected it."

In March 2009 indeed, Jan noted on his blog: "Pleased to find there's a new release of OOoLilyPond which makes it *much* easier to install. Now, who's going to resurrect and tie Ikebana up with this? Or better yet, build on Erik Sandberg's Music Streams work and start a true interactive GUI?"

It wasn't until this summer where Jan resurrected Ikebana, made it work with recent LilyPond versions (2.13.29 at the time), and started to seriously think about its possible future. However, he wasn't very optimistic: "Ikebana, he told us, was never intended as a LilyPond GUI effort. It is unsupported, does not work very well, won't be developed any further. But, it's very interesting, isn't it? Has been on my mind too."

A few days later, he added: "I've been talking with Wingo [Editor's note: Andy Wingo, lead developer of the guile-gnome bindings]. I'm thinking of maybe starting Ikebana afresh, in guile — for obvious reasons — with some major changes.

That was on September 22. Just the day after, Jan's new project was born...

LilyPond's companions (2)

(by Valentin Villenave)

Part Two: A man, a Scheme, a program: Schikkers List!
Interview with Jan Nieuwenhuizen.

The LilyPond Report — Greetings Jan! Wow, it is an honor for the Report to meet with you today; for years I guess I've just been too intimidated to suggest an interview with either you or Han-Wen...

Jan Nieuwenhuizen - - Hi Valentin! Yeah, I love the Report. Good, great!

TLR — Thanks! Let's get started then. What exactly is Schikkers List? Never mind — let me quote your README file for you:

README —

Schikkers List is a LilyPond GUI in the prototype phase. I have no idea if this approach could even result in a usable program. [...] The basic is to have LilyPond render only a small bit of the music: the bit that changed. [...] Simple LilyPond files can be entered and saved as .scm or exported as .ly files.

This is targeted at hackers and mainly presented as an invitation for comments and possibly for patches.

JN — That's right. There are so many bugs/missing features that I'll only be taking patches for now; at least until v0.1 or so.


Schikkers List's {iKLi} interface: courtesy of Jan Nieuwenhuizen.
Animated PNG: you need Firefox or Opera to see it...

TLR — Okay, now that you've made this clear, let me ask you the question everyone must be asking themselves by now: what's with the name ?? I mean, "Schikkers List"? Really?

JN — I know. "Schikkers List" is probably just as incapable of surviving as "Ikebana", but well

List is the dutch translation of "a guile scheme". It does not mean list[en], that word is lijst (Liste in German). Translating Ikebana into Dutch, you get Bloemschikken where bloem == flower. The person who performs Ikebana would be a Bloemschikker (arranger of flowers).

TLR — ...

JN — Wait, it gets worse: one such schikker/arranger had a guile scheme: only recompute the part that changed to arrange notes, and in contrast to Ikebana who used python, this one uses GUILE Scheme.

TLR — Er. Okay, we'll let our audience re-read this a couple of times... For the record: I didn't even know that "guile" was actually a word


Schikkers List's key bindings

TLR — (*cough*) Back on track. Obviously, the GNOME-Canvas plan has been on the table for some time now; I also can understand why you chose Guile/Scheme over python (since most of LilyPond itself is natively implemented in this language). What else inspired you?

JN — I was inspired first by Frescobaldi: how quick that was, but discouraged by how fully it is entangled with KDE, and it's still a text editor that will never compete with Emacs.

Then I was inspired by the gummi editor: I wrote a 20-line patch that made some sort of frescobaldi-like gui.

Then I was inspired by how fast and how usable Ikebana was.

TLR — Wait — "usable"??? How come all of us haven't been using it ever since then?

JN — Well, it wasn't just like that in 2005: it was neither fast, nor usable. Also, LilyPond has come a long way since 2004...

I want to learn programming in scheme, that seems such a nice language and I'm such a novice. I had to do all that alone because nobody can know about it, as I should be working on my open source consultancy startup company, and anything else is procrastination, which looks real bad.

I also think you should be have some real code before you start to talk vaporware, which is why I had to start alone. But I'm not going to do a project like this on my own, so if others are interested that would really help.

TLR — I'd be happy to help, except for one thing: I'm a KDE guy. There's no way I'll *ever* install GNOME on my system. So, I guess this is good-bye...

JN — Not at all! Schikkers List is not gnome dependent, just GTK. I started off using the pangocairo backend, which requires very recent builds of guile-gnome (later than 2.16.1). However, the GNOME canvas was not deprecated after all, so I'm using this as a backend now, which allows running with guile-gnome 2.16.1 (or possibly even 2.16).

TLR — Interesting. Now for the embarrassing part. For almost fifteen years, we've seen Han-Wen and you telling people that a GUI was just The Wrong Way, that LilyPond would never have a GUI because it wasn't meant to in the first place and it would be useless anyway. So I hope you won't mind me asking: who are you kidding, really? Was all that non-GUI motto just a pile of hypocrite bullshit?

JN — Keep it cool... LilyPond is not conceived, indeed, to be used in a GUI setting. Period.

There are so many good reasons why GUI programs in general, and a GUI for setting text or music specifically, are a bad idea! You get "expert users" and tricks and mouse-click video's on YouTube etc. Can't script it anymore.

However. LilyPond was created to free the world of badly typeset music and to help *all* musicians. Of course, we want everyone to enjoy the freedom of Linux and move there. However, this was the mind-shift that caused us to make the Windows/mingw and Apple/darwin ports. We cannot demand that people change their OS for their own good, happiness and health.

TLR — Well, I certainly would, but...

JN — The idea of my ikli interface is to do away with the text editor (there may be a structured text editor one day) because I think that the vast majority of potential users will never /start/ with lilypond if they have to enter text, even if there's visual feedback.

If the mere fact that LilyPond does not have a very good GUI, will mean that in the end we as musicians are still stuck (again stuck) with badly typeset music because "people /will/ use a GUI even if it handcuffs them", it's bad for their health and happiness etc., what's a hacker to do?

TLR — ... Well, you tell me!

JN — We/I have been "waiting" for years to get Eric Sandberg's work completed and have full-blown music streams in LilyPond. That would be the first step towards the remote possibility of a GUI. This did not happen and I do not foresee it to happen any time soon.

So, Schikkers List is another guile scheme to turn this dependency around. Instead of first waiting for boring, useless work to be done that should enable a gui, just make a gui first and when it grows, we can always decide that music streams are essential, integrate Schikkers List with LilyPond and just finish them.

On the other hand, it could well be that there will never be a v0.0.2 and this is just another silly pipe dream.

TLR — I sure hope not!


A real-world score: "Ständchen"

JN — If you get to use Schikkers List, you'll see that, as I said, LilyPond is not conceived for a GUI approach — however, it may just be good enough. I don't know. The big question here is: who can tell [me] what the very minimal set of requirements is that makes a gui useful, esp. for people who would never use text entry for music, and is there a path from this minimal version to more useful versions for real world music?

TLR — Well, perhaps our readers can help answering that question However, isn't there a risk that you may reinventing the wheel here? You do know that Denemo development has been revived for the past couple of years, and produces a nice interface already... Besides, many users appreciate the Frescobaldi/LilyPondTool "code on the left, preview on the right" paradigm; can you seriously believe that anyone will want to use a GUI program that hides the LilyPond code from the user?

JN — Let me be clear. Backed with the right amount of hacking effort, Schikkers List will just totally blow away all the other efforts.

Of course, the lily code will also be available in a separate window and we'll have some sort of structure-editor that feels like a text editor but is very annoying because it has to make sure you cannot enter invalid lily code We'll also have a music-tree view window where you can select/edit the underlying music structure.

For years to come however, Denemo will be a great GUI. As for Frescobaldi/LilyPondTool, these aim at people who want to write code — those are not real GUIs, are they? People who would benefit from Frescobaldi will probably also use lilypond without this nice preview...

TLR — Well, thank you Jan, and good luck: you'll need it! Anything we normal people can do to help, other than turning into programmer-demigods and sending you patches? How about suggesting a nice logo for Schikkers List?

JN — A logo. Man, this is a pile of rubbish, get real! Love, peace, thanks.

Many thanks to Jan Nieuwenhuizen for this interview.

Snippet of the Report

(by Valentin Villenave)

Let's be honest: for a while, this month's snippet tasted like a conflict of interest. I had written a snippet that — I thought — was just way too cool not to make it as the "Snippet of the month": Colored-background staves.

When browsing the LSR to retrieve my snippet's URL, however, I stumbled upon this other one — and realized I simply couldn't compete.

Coloring individual staff lines

Now, how cool is that? The worst part is, though, that we have no idea who posted this snippet: it doesn't include a %LSR comment, nor has it been announced on the mailing lists. If you wrote it and happen to read this, please do let us know!

[Update: unsurprisingly, Neil Puttock is actually to thank for this cool snippet. Cheers!]

That concludes the twenty-second issue of The LilyPond Report. Next instalment is expected on December 1st, and will be the last of the year 2010!

Cheers,
Graham Percival & Valentin Villenave

[1] Unanswered mails are probably not uncommon in any Free Software community, but that's not to say we shouldn't do our best not to let that happen. On a related note, I was flabbergasted that Jan's announcement about Schikkers List went totally unnoticed: how on Earth is that possible? — To be fair, I did not answer him either; I do hope that today's LilyPond Report feature will make it up to him...

Post-scriptum
by Valentin Villenave.

As was to be expected, my emotional testimony about the private lilypond-hackers mailing list raised a lot of strong reactions, mostly from the Report's co-editor Graham Percival, LilyPond's project manager, himself a member of this "secret" mailing list, and incidentally a friend of mine (which makes it all the more saddening).

If anything, I have to make absolutely clear that my goal was never to mock or insult any of the LilyPond developers, but to merely state how uncomfortable I feel, as a contributor, about this particular subject in an otherwise pleasant and enjoyable Free-Software community.

In addition to the comments below, a discussion susequently took place on the public lilypond-devel mailing list. Whilst the controversy is not solved yet, a first step has been taken thanks to our contributor Carl (not a member of lilypond-hackers), who added the first official mention (ever!) of this private mailing list, to our Documentation. Thanks Carl!

The LilyPond Report #21

Valentin Villenave — 2010-10-04T00:05:21Z

This short, informal opinion column is about the GNU LilyPond project: its team, its world, its community. It is not meant to be an exhaustive documentation resource. Reader comments are, of course, welcome (see at the bottom of this page).

Table of contents

Greetings everybody, and welcome to this twenty-first issue of the LilyPond Report!

Autumnal and microtonal: that's how one could describe this Report's issue. Microtonal notation support has indeed been at the center of a number of discussions recently: bug reports, feature requests, frogs contributions: read on to know all about it. On other news, we have been digging through the mailing lists archive to bring you some nice tricks (LilyPond made portable, anyone?) and overlooked announcements.
As always, you can post your comments at the bottom of the page, or even register and contribute to the LilyPond Report's next issues.

Editorial

Valentin Villenave


Paris, boulevard Diderot

As a beautiful Autumn is beginning here in Paris, France, it looks like the year 2010 will not end without some interesting events in the LilyPond community. Firstly, the upcoming stable 2.14 release is in good shape, as Graham will elaborate on below; our new website is now online at last, and even more interesting things are coming down the road: the Grand LilyPond Input Syntax Stabilization project, which we'll talk about extensively in months to come, our online AJAX-powered documentation (of which Reinhold already maintains a preview on his personal website), etc.

As far as the LilyNet.net community website is concerned, we'll be migrating to a more powerful server this fall, that will allow us to do quite a bunch of cool things (a lot of ideas are already floating around). Due to increasing spam, we may also be applying stronger rules to our Wiki. Your fluffy editor is also eagerly waiting to introduce some geeky pet projects of his... Anyways [1], stay tuned!

Release news

Graham Percival

The big news is that we've released the alpha milestone for 2.14 stable! So far we've had two "alpha test version", and users have been helpfully discovering odd cases where the test version produces questionable output. The latest such release is 2.13.35, which has 5 known regressions against the previous stable version.

Speaking of which, the current stable release is still 2.12.3-2. Despite all the excitement over the nearing end of 2.13 and the new stable version, all normal users should be engraving with 2.12.3 for any serious work.

News from the Frog Pond

The Frogs are ordinary LilyPond users who have chosen to get involved in their favorite software's development. Fixing bugs, implementing new features, documenting the source code: there's a lot to be done, but most importantly: this is a chance for everyone to learn more about LilyPond, about Free Software, about programming… and to have fun. If you're curious about any of it, then the word is: Join the Frogs!

Most of the "frog pond" work this month has focused on guile 2.0 compatibility issues, but one very interesting possibility arose in the middle of a discussion of microtonal music: Wol announced that he had written a "sample 'starting' engraver", which could be very valuable for other people if it was added to the Contributor's Guide. Hopefully we'll gave a patch that adds this soon, so that any future Frogs will have an easier time understanding lilypond internals!

Feature story: microtonal notation in LilyPond

What exactly do people mean when they're talking about microtonal notation? The LilyPond Report has asked some of our most specialized contributors (namely Graham Breed, Joe Wakeling and Hans Åberg) if they could elaborate on this topic. Hans was patient enough to answer our question thoroughly, and here is his contribution that we proudly publish below.

Why microtonality?
by Hans Åberg

Sometime after World War II, a common misconception starts to take hold: the basis of Western music is E12, the 12-equal temperament. In fact, it has never been, and despite the widespread use of E12 in popular music, jazz, and some, but certainly not all, fixed pitch instruments, CPP (Common Practice Period) orchestras have not shifted away from the adaptive Just intonation that they always have used. And if the music is not written with CPP harmonic rules, variable pitched instruments are likely to switch to something like Pythagorean tuning, even if the music is specifically called for being written in E12 (like some atonal music).

So even if one decides to stay within the realms of Western CPP derived music, there are good reasons for using at least some microtonality. If one wants to do some limited microtonality, what is required?

The to the modern mind surprising answer is: really nothing, because the staff system that LilyPond focuses on is perfectly capable of expressing different tunings. Our ancestors though, that devised this staff system at the end of the Renaissance, experimenting heavily with different tunings, might have been rather surprised to learn that we have forgotten about it.

If you want your music score to be retunable into say extended quarter-comma meantone tuning, which sets the major thirds exactly to the interval ratio 5/4, or Pythagorean tuning, all you have to do is making sure to not apply E12 enharmonic equivalence in ways that make the shift in pitch too obvious. So if used at all, only apply it at some sections in whole as a part of notational simplification. The overall pitch will shift, but typically only by a tonestep of E31, which is close to meantone, or E53, which is close to Pythagorean tuning, or between 20 to 40 cents.

And then you need a program that can translate this into correct pitches. And it is here where microtonality capabilities in LilyPond may play an important role. If implemented in full, writing complex microtonality should be no more difficult than any other type of music.

Beware, though, would you decide to walk down this road, that you may soon start to loath all those synth orchestras that only play in E12. And you may start to appreciate the adaptive pitch-work of real musicians a lot more. But those are side effect you probably can live with.

Let us now move beyond the simple retuning of a standard music score, to see why that might be needed. But first a bit of background:

During Medieval times, the Pythagorean tuning was common, built up by solely the octave interval ratio 2 and the pure fifth 3/2. This involves only the prime numbers 2 and 3, which is called the 3-limit. This tuning works perfectly fine if one not emphasizes the thirds harmonically strongly, and is still a good bet in for example Balkan music. Towards the Renaissance one wanted in West to add the interval major third 5/4, decreasing the beat rates of the Pythagorean major third (the difference between these two major thirds is the syntonic comma 81/80), and in the near East: Arab, Persia, and Turkey, one instead added an intermediate pitch, and in some descriptions these intervals involve the prime number 5. So all this is 5-limit music, though one should note that in the quarter-comma meantone tuning, the augmented 6th is very close to the interval 7/4, and if included, this is the 7-limit.

Above, the word "adaptive" was thrown in before Just intonation. If one adheres to CPP pivoting on the chords, then some chord sequences, for example I IV ii V I, will drift in pitch a small amount, called "comma pumps". So Just intonation may be impossible to play without some kind pitch adaptation. One can even out the small adaptation needed between all pitches played, which is called tempering, but a real musician playing a variable pitch instrument can do a little better; exactly what is needed is not known.

In addition, on variable pitched instruments, there is other pitch adaptation going on. This is well known in for example Arab music, where the tendency of moving some pitches together for musical expression is called the "principle of gravity". The exact description of this adaptation process is not well known, but probably helps giving the music life.

Here, we have so far stayed within the 5-limit. One might want to experiment with interval ratios involving more primes: the 7, 11 and perhaps the 13-limit. And we also implicitly assumed that it is better to be close to a rational interval. Being close helps up the harmonic flexibility, but in for example gamelan music, the beat rates typically around 8 Hz but up to 20 Hz, seem to be built into the tuning. In some music, the full scale, the diapason, may not have seven scale degrees, and one may also drop the requirement that the diapason is the interval ratio 2, the latter which happens in for example Georgian music and the Bohlen-Pierce scale.

Overall, the topic of different tunings may seem difficult and elusive, but with a way to easy produce these pitches accurately, it is more like choosing different stops on a pipe organ. The choice of tuning helps giving the music character, and different types of music and music styles will require different tunings. The feedback from being able to produce these tunings combined with the experience of the result will make the choice no more difficult than any other musical feature.

Bug Report of the Report

Graham Percival

Speaking of microtonality, I think the most interesting exchange this month was precisely about microtonal notation issue. We have an old issue 694 about microtonal accidentals, but the whole thing began with somebody who didn't know about the microtonal notation that was already present in lilypond. Discussion then veered off into transposition, year-old patches, and the result was a confusing mess. It wasn't obvious what, if anything, people were requesting (in terms of code, documentation, fonts, or what).

After almost deleting this issue, we had some discussion about what was actually sought, and came up with a new issue 1278. The new issue is great — there's a clear description of what's required, backed up by a strong reference in music notation (Kurt Stone's "Music Notation in the Twentieth Century"), and a two-line lilypond input segment (not counting braces and comments).

"Brevity is the soul of wit."

Remember folks, we have over 400 open issues, so if it takes longer than 30 seconds to understand what your pet issue is requesting, then chances are that nobody will look at it. Once you think you've finished writing a feature request, spend another 10 minutes staring at your request and try to figure out how it could be simplified.

On the mailing lists

Valentin Villenave

It is with great pleasure that I'm opening this new regular section of the LilyPond Report. The principle behind it is simple: our community is large and vibrant, with numerous mailing lists (even more so if you count the unofficial lilynet.net mailing lists, and many intelligent people producing many interesting discussions. Over time, some clever proposals or remarks are bound to get lost in the flow; this section aims to salvage some posts that might otherwise go unnoticed or get forgotten.

A good example is this announcement from our Brazilian contributor Alberto Simões, in March 2009:

I am working on a new Portuguese web-site. It is available at http://musica-liturgica.net/, and includes Portuguese liturgical music.

Probably it will not be that useful for most of the readers of this list, but I think I owe Lilypond a lot

Do you, too, know of a good LilyPond website we might not have heard of? Has one of your own posts or announcements gone unnoticed? Is there any project that our readers might be interested in discovering? Then there's a good chance the Report can help you; please let us know!

LilyPond tricks

Valentin Villenave

Running LilyPond run from a USB key on Windows
This question is frequently raised (and addressed) on the French-speaking list. Earlier this month, a user named Jean-Christophe D. posted a complete howto in French, that might interest other users as well.

As was requested, here are some instructions to make LilyPond portable, to whomever may find it useful.

Reminder: being written in DOS shell, these scripts will only work with Microsoft Windows®. [Note from the editor: GNU/Linux users may have quite a few other ways to make this work on their end, although this specific feature might be more needed amongst MS Winsows users...]

It works in a very simple way.

Installation:

1 - Install LilyPond on your computer (if not already done ;o))

2 - Copy the LilyPond directory (typically located in C:\Program Files\) on your USB key

There you go, it's installed!

Note: you may alternatively want to install LilyPond *directly* on your USB key. I haven't tried it, and therefore do not know if it's possible [Editor's note: It is, but not recommended — as it will make your computer, in the future, look for libraries that will not be on your main hard drive partitions.]

Usage:

1 - Download the latest version of the following small script (written by Eluze on the French list, or below by myself). It can be either a .exe or a .bat file; in the latter case, you may open it with the notepad if you want to read or modify the code.[Editor's note: this can be achieved easily through right-click>"modify".]

2 - Copy this script in the directory where your .ly files are located: it can be either on your Desktop, in a folder, on your USB key,...

3 - Instead of double-clicking on your .ly file to generate a PDF score, just drag-and-drop it on the script's icon. That's all!

Rationale:

The small script file is going to look for any file named LilyPond.exe on any drive (E:, F:, etc.). Whenever it has found one, it uses it to generate the requested PDF.

Below is the latest version of my script.

For advanced users: it can even recover BOTH log files (%1 and %2), and will delete the first one ONLY if empty!
Jean-Christophe's script

Well, thanks a lot to Jean-Christophe, Eluze, and all our users who helped establish and improve this method! As you can see, it is still a little hackish and involves moving/copying files around, but with a bit of luck the community will come up with more proper solutions in the future. In the meantime this will do the trick!

That concludes the twenty-first issue of The LilyPond Report. Next installment is expected on November 1st; please send us your contributions before October 22!

Cheers,
Graham Percival & Valentin Villenave

[1] As I was wondering whether anyways was a correct form or not, I stumbled upon this website which I think Graham might appreciate...

The LilyPond Report #20

Phil Holmes, Valentin Villenave — 2010-09-02T14:54:22Z

This short, informal opinion column is about the GNU LilyPond project: its team, its world, its community. It is not meant to be an exhaustive documentation resource. Reader comments are, of course, welcome (see at the bottom of this page).

Table of contents

Welcome to this twentieth issue of the LilyPond Report!

Editorial

by Graham Percival

Here's another LilyPond Report. Enjoy.

Release news

by Graham Percival

The current stable release is 2.12.3-2; all normal users should be engraving with this version. We have no plans on making any more 2.12 releases.

The current development release is 2.13.31. This release currently has 5 known Critical issues, but 3 of those were recorded after the previous Report. We're definitely making progress towards the first alpha release of 2.14. Finally, the infamous issue 989 has finally been fixed; many thanks to everybody who worked on it, especially Phil Holmes and James Lowe!

OMET: Online Music Editing Tools

Leveraging Ajax, the DOM and Lilypond for a web-based notation editor

Mike Blackstock wrote a nice review of his tool:

http://www.omet.ca/lilyreport/lilypondreport.html

News from the Frog Pond

The Frogs are ordinary LilyPond users who have chosen to get
involved in their favorite software's development. Fixing bugs, implementing new features, documenting the source code: there's a lot to be done, but most importantly: this is a chance for everyone to learn more about LilyPond, about Free Software, about programming… and to have fun. If you're curious about any of it, then the word is: Join the Frogs!

The Frogs have two new members: Wols (working on a capo option to the chordname engraver), and William (cleaning up our CSS files). Ian continues the difficult task of upgrading our scheme files to work with the upcoming guile 2.0.

Bug Report of the Report

by Phil Holmes

Patrick Schmidt reported some difficulties with \autoBeamOff when setting some vocal music. He reported that it has no effect on other voices, even when used in a global variable. He also noted some strange effects in combination with 16th notes and \partcombine.

Carl Sorensen replied that \autoBeamOff is simply an alias for \set autoBeaming = ##f, which means that, by default, it actually only applies in the Voice context. Given that this is not documented, it's not surprising that it's not obvious. I then found an almost identical bug report from 2003, and suggested it was time to fix the documentation. Ralph Palmer raised it as Issue #1227, and Carl provided a fix in build 2.13.31.

Carl also commented that the bug Patrick identified when using \partcombine is very challenging to try to fix. However, it can be worked around using set Staff.autoBeaming = ##f

Why have I selected this as bug report of the report? Well - I thought it was interesting to be reminded about how it's important to consider which context contains the property you're trying to change. And secondly, it was nice to be able to report that an untracked bug from 2003 has been fixed.

So remember to watch your contexts if you're using \autoBeamOff!

LSR Snippets

by Valentin Villenave

If you're a newcomer to LilyPond, you might not be familiar with the LSR yet; the LilyPond Snippet Repository is a user-contributed library of LilyPond tricks, ranging from simple yet helpful examples to sophisticated mind-challenging code magic.

Browsing through our mailing list archives, I recently found this nice trick that Trevor Daniels shared with us about two years ago; of course I immediately added it to the LSR:

Background color

This simple yet efficient snippet takes advantage of LilyPond's ability to manage different layers ans superimpose them through the, well, 'layer property. A very, very large colored rectangle is drawn in a markup object all over the page.

A few questions remain: how to automatically make the rectangle size match the page dimensions? How to use images instead of plain colors, for a nice watermark effect? how to find a better anchor point for the markup? etc.

If you have ideas to improve this method, please let me know and send us your nicely decorated music sheets!

That concludes the twentieth issue of The LilyPond Report. Next installment is expected on October 1st; please send us your contributions before September 22!

Cheers,
Graham Percival, Valentin Villenave

The LilyPond Report #19

Valentin Villenave — 2010-08-03T21:47:46Z

Table of contents

Welcome to this nineteenth issue of the LilyPond Report!

Let's open this issue with a disclaimer: yes, this installment was supposed to go online on July the 1st (per special request of Graham). Although it has been mostly ready for more than a month, one of the editors felt unhappy with the lack of long articles, thorough features and verbose investigations. (Which may give you a hint as to why said editor is regularly qualified as fluffy by his colleague Graham Percival.)

Anyway, my sincere apologies about the delay; let me wish you a (late) happy Canada Day, Independence Day, X day, to all!
Valentin "fluffy" Villenave

Editorial

by Valentin Villenave.

Here's that time again, where everything seems to take forever, when the LilyPond Report is unbelievably late, etc.

It is probably true of any Free Software community: time tends to lengthen and things tend to decay. Take any given deadline, forecast, roadmap or whatnot... but when you come back months later, chances are nothing has worked out according to plan.

This, of course, can be observed a lot more in volunteer-driven projects like ours, but the (somehow) good news is, we're not alone.

Look, for instance, at the Fedora project, arguably one of the top-three most interesting GNU/Linux distributions out there: in a recent blog post, its Program Manager John Poelstra has asked for "more discipline and fortitude", and presented as a "challenge" that the distribution's 14th release would ship on time... Time will tell.

On a similar note, a while back we had noted that Linus Torvalds was thinking about changing the Linux Kernel version numbering scheme: "instead of releasing 2.6.26, maybe we could have 2008.7 instead"
Two years later, we're at 2.6.35... and the old numbering scheme has never been better (thanks for asking).

To anyone who's been involved with the LilyPond project, such situations will probably look quite familiar. Our "stable" releases are a good example thereof: waiting for 2.10 to appear seemed like an eternity, but was nothing compared to the wait for 2.12 (as Mats alluded to), and whilst 2.14 was supposed to include a smaller number of new features, and be out of the door in no time, this release has yet to appear. Let's not even mention the next major version: as we already pointed out, LilyPond 3.0 has been discussed since 2004!

What causes such phenomena? Is there something we can do to improve the situation (hint: the answer is yes, obviously)? Quite a few answers are to be found in Graham's recent and insightful paper which we'll talk more about below.

Coming up next month...

What about the LilyPond Report? Can we expect to reach a more regular release cycle? Well, we're working on it. Starting next month, the Report will be run on a monthly basis, with a slightly revamped layout, and a new shorter, "low-fluff" writing style. The installments should be shorter, easier to read, and more to the point.

Next issue will be published on September the 1st, and it should include a feature about online LilyPond editors — no installation required; just surf to a website, type away, and view the output! Please contact us if you are working on such a system. All contributions are to be sent before August 22; anything that we'll receive after this point will be... preciously kept for our October issue!

Release news

by Graham

The current stable release is 2.12.3-2; all normal users should be engraving with this version. We have no plans on making any more 2.12 releases.

The current development release is 2.13.28. This release has 8 Critical issues (plus 7 affecting the new website), with patches pending for some of them. The good news is that half of these issues were added since the last Report, so we are making progress. The bad news is that issue 989 (ensure that no information is only in the regtests) is still open — but Phil Holmes and James Lowe have recently begun seriously working on this, so I'm optimistic that it will be resolved before the next Report.

Conference sightings!

by Valentin

The 2010 Libre Software Meeting

Unfortunately, none of your editors appears thereabove. But still, you get the picture.

Obviously, this summer's bigger event (with regards to the LilyPond community) was the 2010 Libre Software Meeting in Bordeaux, France, where our Graham "Grumpy" Percival had been officially invited (thanks to our French contributor John Mandereau, who had the brilliant idea to get in touch with the LSM staff).

This gave Graham a chance to discover some French oddities, and to meet with a few members of our French community, namely John, yours truly and — last but not least — Master Nicolas Sceaux.

As announced in the previous Report, Graham was to give a lecture entitled Sustainability in F/OSS: developers as a non-renewable resource. Here's the pitch:

The time and energy which developers spend on open-source projects is not an infinite resource. Developer effort can stall due to external demands on their time (such as family, career, or health), but also due to internal factors (such as a loss of motivation or interest). Long-term projects (5+ years old) should try to engage in sustainable development practices. How can we retain developer interest? How can we prepare for the inevitable loss of developers? How can we train the next generation of developers?

This talk draws upon experiences from GNU/LilyPond (a 14-year old sheet music typesetter), but makes general suggestions (and warnings!) for users, developers, and project leaders.

Graham's talk was quite interesting (however it had unfortunately not been given any visibility whatsoever, which caused Graham to give his talk to a merely empty classroom...). It did raise several questions and quite a few interesting answers; Graham was kind enough to put his slides on his blog, in a slightly improved and clarified version, so feel free to have a look!

To be honest, this kind of official, big-time events can get really boring really quick. However, it did allow us to make a few acquaintances: some people in this OFSET (Organization for Free Software in Education and Teaching) were very much interested in LilyPond, and we also had a meeting with a couple of MuseScore developers (whom we now know they do read the LilyReport! Hi guys! )

And of course, nothing can be compared to the experience of meeting the actual Graham Percival for the first time, face-to-face. However, let's not spoil it in case it could happen to you in the future, dear readers: this is one of these things one has to live by oneself.

News from the Frog Pond

The Frogs are ordinary LilyPond users who have chosen to get involved in their favorite software's development. Fixing bugs, implementing new features, documenting the source code: there's a lot to be done, but most importantly: this is a chance for everyone to learn more about LilyPond, about Free Software, about programming… and to have fun. If you're curious about any of it, then the word is: Join the Frogs!

The Frog Pond has been fairly still for the past two months; work has continued on preparing for our switch to guile 2.0, and mixing ties and harmonics in tablature.

Snippet of the Report

This month's snippet has been (re-)discovered by Laura Conrad, thanks to a post by Kieren MacMillan in March.

How to multiply (or divide) note values in Lily
by Laura Conrad

I use Lily mainly to transcribe Renaissance music, which I then play with the Cantabile Band, and publish on SerpentPublications.org. I believe that it's easier to play without the confusing barlines that modern editors insist on inserting. Sometimes I transcribe from the facsimiles as they were printed in the sixteenth or early seventeenth century, but often, I'm transcribing from a modern edition.

Besides inserting confusing barlines, modern editors often change the note values — if the composer was thinking in a whole note beat, and using quarter notes for runs and ornaments, a twentieth century editor will often turn the whole notes into quarter notes, and the quarter notes into sixteenth notes.

I originally started transcribing in ABC, which has a default note length, so doubling or quadrupling the note values is just a matter of changing all the L: statements to use a different default.

This is a feature I had missed in lilypond, but then last March there was a post that called my attention to this snippet.

So now I have a file called "slower.ly", which I include in my lilypond for pieces where I need to change the note values. It looks like this:

looksSlower = #(define-music-function (parser location music) (ly:music?) (let ((new-music (ly:music-deep-copy music))) (shift-duration-log new-music -1 0) new-music))

This doubles the note values. The first number in (shift-duration-log new-music -1 0) would be "-2" if I needed to quadruple them, or "1" if I wanted to halve them.

In my code that prints the part or the score, I have the line:

\include "slower.ly"

And when I've finished the proofreading where I have to keep referring back to my original source, I change the lines:

\context Staff="default" { \testnotes }

to:

\context Staff="default" { \looksSlower{\testnotes} }

So here you can see the same note entry producing both note values:

Bug Report of the Report

The big news about bugs is our vastly-expanded Bug Squad! We now have 8 Bug Squad members; one for each day of the week, with two on Friday. In addition, we have now caught up on all the old bug reports sent to bug-lilypond.

If you have reported a bug which was not fixed in 2.13.28 and is not in our issue tracker, please report it again by following the instructions on:

http://lilypond.org/website/bug-rep...

Please note that emails to lilypond-user or lilypond-devel do not consistute proper bug reports — we only have enough resources to deal with the dedicated bug-lilypond mailing list.

That concludes the nineteenth issue of The LilyPond Report. Next installment is expected on September 1st; please send us your contributions before August 22!

Cheers,
Graham Percival & Valentin Villenave