Greetings everybody, and welcome to this twenty-fourth issue of the LilyPond Report!
First things first: this slightly unusual issue features a rather important request from our co-editor David Kastrup, who also happens to be arguably the most involved and efficient LilyPond developer at the moment — and unarguably, our only full-time developer. Also featured in this issue is an overview of some aspects of the new, simplified syntax in LilyPond. At last, we’ll be discussing a long-overdue topic, to wit: LilyPond-based web applications, with a long-overdue contribution from scorio.com founder Johannes Feulner!
As always, you can post your comments at the bottom of the page, or send your contributions to the LilyPond Report’s next issues.
Editorial
by Valentin Villenave & Graham Percival.
The LilyPond Report was started exactly four years ago this month; although it has been a bumpy ride, with occasional long periods of inactivity, we hope it has proven a valuable medium for the GNU LilyPond project and the Free software world in general. It has given us a chance to meet dozens of interesting developers and contributors, to address various music-related questions and to provide a hindsight into LilyPond’s development process. This month however, we are contributing to the LilyPond project in a new, more serious way, upon request from our co-editor and colleague David Kastrup.
While several projects depending on LilyPond have managed to fund full-time developers, LilyPond development itself has not really seen significant monetary support from its users so far. Every so often, however, users ask if they can contribute financially to
LilyPond. Creating a single legal entity to handle such matters
would raise a number of legal and administrative problems which we
are not equipped to handle at the moment. However, we have a
specific place where developers can indicate their interest in
private arrangements:
http://lilypond.org/sponsoring.html
Currently the only developer on this page is David. As you can see from the git statistics, he has been very active over the past few months. Both his self-chosen projects as well as the expertise he has provided for work on our infrastructure and the projects of others have given LilyPond an important push forward.
As you can see in his plea below, David is in dire financial straits. Unless he receives money to support his LilyPond work, he will need to leave the project and find work elsewhere — which would be bad news for LilyPond.
Users of LilyPond, contributors and enthusiasts, you now have the chance to secure an important contributor for full-time work. If you are interested in supporting LilyPond financially, we warmly suggest that you sponsor David to fix bugs or add features that interest you.
On a personal note, both of us have unrelatedly sponsored David to add new features for us in the past, and in both cases his work was fast and high-quality.
Graham Percival, GNU LilyPond project manager;
Valentin Villenave, LilyPond newsletter editor.
An urgent request for funding
David Kastrup
Read this personal appeal from LilyPond developer David Kastrup.
Let’s begin with the most important bit. At the current point of time, my financial situation is in free fall with very few slowdowns. Its terminal velocity may be restrained by my rather basic lifestyle. But if you think me having to split wood at ten degrees below zero because of the price of oil heating is helping much, think again: that shaves off something like maybe €40 from the monthly bills at best. It merely spreads my reserves a bit thinner.
- Splitting wood.
- "As the temperature has risen above 5 degrees, we fortunately have stopped heating" — D. Kastrup, 2012
So impact is looming close. The common answer to that is ``Get a job!’’. If you take a look at the activities in the LilyPond repository in the recent months, you’ll find that I already have a full-time job. I am working for you, making LilyPond a greater tool to work with for you and other musicians. And much as I like this job and enjoy working for you, the time has come to talk about a raise.
Now I am not the only one working on LilyPond, and you may question why you should consider paying me instead of someone else for that work. That is a valid question as long as you are actually paying someone else rather than using it as an excuse not to do anything at all. Check out http://www.lilypond.org/sponsoring.html for your options at any given point of time. At the time of writing this article, there are no other requests for funding. I deliver excellent value, and there is a limited window of opportunity before I have to close down operations and am no longer available. So buy yourself in while you still can.
There are others working on LilyPond in their spare time (and some of them are doing a really fabulous job). Now for one thing, the rallying point that a full-time developer provides helps even them to be more effective at what they do and can achieve, even if this full-time developer would not be first pick for the actual project lead or other tasks focused on communication skills.
But for another, it turns out that my own situation is somewhat special, and so I have an additional personal interest in you making it possible to continue my work on LilyPond, going beyond a mere job search.
- Primary means of transportation (front and left).
- D. Kastrup, 2012.
And so that you know what you buy into and how to make the most of it, I am telling you.
After figuring out that there were some things affecting my productivity in a number of ways that don’t seem to affect other people nearly as much, I actually asked for professional opinion and got diagnosed with several medical conditions. Now "conditions’’ are things to take into account, "medical’’ means phenomena fitting a pattern so that one can focus on approaches that have shown to be effective in similar cases.
After lots of experimenting around, it turns out that in practice I mostly have to live with being David, and you have to deal with me being David, because that is quite solidly hard-wired. And this can actually be a rather good deal for you and others, even if some parts of it may seem unusual.
If we cut through the skinny of buzzphrases like "Asperger’s’’ and "ADHS’’, here are key points worth knowing about me:
- I can’t stand things being `wrong’ in a number of ways. This means that I am a good fit for working in a setting with ethical principles (call it "ideological’’ if you want) like Free Software. It also means that I am uncommonly good at finding and exterminating even more obscure bugs (like tracking down a GCC compiler bug that threatened to make LilyPond unusable on newer systems).
- I get irritated and distracted rather easily. That means that I can be spectacularly bad at boring but necessary tasks which is not really good when operating in normal employment situations. It actually is not really good when leading any kind of life, either, but that is a different topic.
- I see people react, but am bad at predicting it. If you meet me face to face, cues that are absent in non-interactive communication do a lot for avoiding misunderstandings and annoyances. Like with many other things, this is not unusual in itself but in the degree to which it happens.
While I do not really shine in human relations when doing `batch processing’, I am pretty good with computers, and indeed I started working with them at a time when the communication literally consisted of a batch of punch cards and, a few hours later, output in print.
The low frustration tolerance for `boring’ tasks still shows when working with computers, but I fare rather good at `hard’ and `impossible’ tasks as the `mind’ of computers tends to be more accessible to me than that of fellow human beings.
- Music is in the Air.
- D. Kastrup, 2012.
Now at the moment, LilyPond has captured my attention and provides a lot of motivation, and other developers and their needs and work put me back on track when I get distracted. That means that I am quite productive overall, though not necessarily according to any plan. My actual impetus had been extending LilyPond for typesetting music for accordion. I have not exactly been making lots of progress on that front, instead getting myself caught up in improving an increasing number of areas within LilyPond itself and the workflows focused around using and developing it, to a degree where I have, in my single-mindedness, not enough time and focus left doing another job.
As a result of my involvement, programming LilyPond has become quite more powerful and enjoyable: most of the rather extensive changes in LilyPond’s input and music processing during the 2.15 development phase are due to me, and its integration with its extension language Scheme has become more logical and powerful at the same time.
LilyPond has more areas than what I have been working on: there have been other large improvements particularly in the typesetting stage of LilyPond, and those improvements did not depend on work I have done. It is my impression, however, that the motivation of users on LilyPond’s user list to solve problems by writing Scheme code on their own and teaching others how to do it has picked up. I certainly hope to see this trend to accelerate after we manage to make the next stable release, 2.16, and that it will contribute to attracting new developers over time.
We don’t have a general LilyPond development fund at the current point of time, and it would pose organizational and legal problems that are not worth tackling before there is positive proof that enough users are willing to invest in the continued well-being of LilyPond, and put their money rather than their foot where their mouth is.
Fortunately, my current situation is the perfect litmus test for you to demonstrate that. It is hard to imagine a more convincing incentive to contribute more than just talk. I might add that even with a general fund, you could do worse than contribute to me in person, since funds play to their strengths when paying for systematically planned tasks, and I am by far most effective when working on stuff that just happens to annoy me at a given point of time.
- Patchy at work.
- D. Kastrup, 2012.
There would be some things I could apply for even if everything was run through a fund: the laptop I am currently working on has been paid by another developer, one who is already putting in more time and energy than anyone could demand. It is a second-hand system from 2007 and, in spite of being quite better to work with than the system from 2000 I had to use before, is tasked a lot with test runs for LilyPond since we are just now getting them streamlined enough to be able to offload them to less experienced developers with better computing facilities. I am holding a talk about LilyPond at a major GNU/Linux fair in Chemnitz (the home of the Chemnitzer Concertina) in a few weeks, and this incurs travelling and accommodation costs that are not yet accounted for.
But most of the stuff I do is hard to plan and account for: I do it because it feels necessary and is the right thing at a given point of time. And that is enough to compel me. I hope it is for you as well: if I have to turn to a different endeavor for keeping myself afloat, I will have to cut out working on LilyPond completely since it would continue being a distraction, and distractions are something I can deal with very badly.
My mail address dak[at]gnu[dot]org is registered at Paypal, and if you want to skip the middle man for regular or larger amounts of money, you can ask me for my banking details (I am living in Germany). People prefer not to think about details, so here are some payment plans. The idea is to contribute a fixed minimum, and if a specified target is not reached by all contributions, you contribute proportionally up to a cap. Of course, you are free to pick all three numbers yourself, but here are a few models:
- [Regular] €25 per month fixed, no cap. This is the payment plan to pick once everything is sailing smoothly and you don’t want to contribute unduly much or think about it unduly much.
- [Lifesaver] Minimum €0, cap €250 per month, monthly target €800. That means that if the target (which basically allows me to postpone my decision to work elsewhere) is reached with everybody’s minimum already, you are not billed. This is the option to pick if you don’t want to support a single person as much as keep the LilyPond project from losing me. You do what is necessary to avoid my leaving, but nothing else. Yes, it will be annoying if it turns out you have to pay the cap more than once, but it will also be annoying for me not even to afford survival in spite of highly qualified work.
- [Torchbearer] Minimum €50, cap €150 per month, monthly target €1200. This is a model aimed at being reasonably comfortable for you as well as for me if everything works out.
Since I have been surprised in the past by the percentage of people actually coming through with their expressed intentions, you would start payment with your cap, and I would inform you in time for your next payment how much of that could be carried over due to others pitching in. The more variable schemes are designed to be effective with a few highly dedicated people, but allow them to back off once more people join.
A tool like LilyPond, providing a simple, open way of describing music and typesetting it to high standards, is important not just for personal use, but also as a base to work with music that has already passed into the public domain and that forms a large part of our cultural heritage, and that LilyPond can help to make easier to access and cast into new forms.
I do not suggest that there are no other ways to invest money or energy for making LilyPond a better tool for working with music. But don’t take that as an excuse to do nothing at all. If you find that LilyPond would be better served by you working or investing on aspects not covered by me, then do it. Don’t merely think or talk about it: that may feel noble, but achieves nothing. Don’t postpone it. You have a chance of making the world a better place for music and, if you choose to do it through me, a limited window of opportunity. Don’t blow it.
Release news
by Valentin Villenave.
Although LilyPond’s next stable release, 2.16, is getting nearer every day, it will take a little more time and intermediate "development" versions. As we were hoping to release 2.16 in early March 2012, a number of new critical issues have been found, thereby delaying the release. The good news is, these issues are now solved and we’re ready to proceed with the development cycle — that is, at least one more Release Candidate will be needed.
Speaking of which, it is important to point out that, as 2.16 is looming, you should test the release candidate — by "you" we mean all users, not just the most adventurous ones: the more people test it and report any problem, regressions or unwanted behavior, the more our "stable" release will be worthy of that name. So, please download and install our release candidate!
What will this new stable version bring to the table? We’ll take an in-depth look later on, but here is a complete list of the most noteworthy changes, be sure to check it out and discover LilyPond’s new, exciting features!
Please also note that the LilyPond team is currently in need of a new Mac OS X maintainer. This LilyPond port definitely needs some love, and it has proven hard for regular developers to keep up with Apple’s latest broken incompatible absurd fancy new features. If you’re experienced enough (both with LilyPond build process and the Mac OS X environment) and feel up to the task, please don’t wait and get in touch!
LilyPond grammar update
Some LilyPond Parser Changes:
Recent Work and Plans
by David Kastrup.
LilyPond’s parser is written using the parser generator "Bison’’.
Bison generates parsers of the LALR(1) class which basically means
that it makes its parsing decisions by considering a set of rules like
property_operation: STRING '=' scalar
| "\unset" simple_string
| "\override" simple_string property_path '=' scalar
| "\revert" simple_string embedded_scm
(the LilyPond grammar can be found formatted in that way in the
Notation Reference). It gets "tokens’’ (such as integers,
inter-punctuation characters, complete words, note names) as input, and makes its decisions about which rule to apply next based on the
combination of its current state (of which there are about 700
distinct ones), the production on the top of a push-down stack, and
sometimes by looking at the next token. When it applies a rule, it
may execute actions associated with that rule, and when it has reduced
everything to the first rule (the start rule) it is finished.
For a compiler, the actions are commonly used for generating code, or as a first step to that purpose, a parse tree representing the input. In contrast, LilyPond works mostly as an interpreter. This means that it executes the actions associated with input as soon as it is able to exercise a rule.
Turning the LilyPond input language into a compiler would present
considerable challenges because of the design of the language: some
decisions, like assigning a syntactic class to an escaped
\keyword or an immediate Scheme expression such
as $(ly:make-pitch 0 0 0) are inherently dependent on the current state as evolved by the interpreter.
However, the purpose of the input language is almost exclusively limited to parsing and generating music lists; and actually transforming those lists into a printed representation via engravers and Midi via performers is what takes the bulk of processing time.
Parser developments
Optional music function arguments
Since the last stable versions (2.14), music functions with optional
arguments have made it into LilyPond. One result has been that a number of syntactic constructs hardwired into LilyPond’s grammar could be turned into music functions. For example, in LilyPond 2.14, the command \relative was parsed in the LilyPond
grammar with the rule
relative_music:
RELATIVE absolute_pitch music {
Pitch start = *unsmob_pitch ($2);
$$ = make_music_relative (start, $3, @$);
}
| RELATIVE composite_music {
Pitch middle_c (0, 0, 0);
$$ = make_music_relative (middle_c, $2, @$);
}
;
Now it is parsed using the following definition:
relative =
#(define-music-function (parser location pitch music)
((ly:pitch? (ly:make-pitch 0 0 0)) ly:music?)
(_i "Make @var{music} relative to @var{pitch} (default @code{c'}).")
(ly:make-music-relative! music pitch)
(make-music 'RelativeOctaveMusic
'element music))
For one thing, having fewer hard-coded exceptions makes the syntax more regular. For another, the functionality is now documented in the notation reference together with other music functions (the given documentation string is tunneled there in the process of creating the manuals). Another advantage is that changing the functionality does not require recompilation of LilyPond.
More importantly, users can now create functionality in Scheme (and thus as part of writing a LilyPond source file) that formerly required changing and recompiling the parser.
The workhorse #{ ... #}
In 2.14, you could employ the construct #{ ... #} for writing sequential music in Scheme. It has been slightly extended to also allow writing pitches, markups, scores, books and bookparts, single music expressions, postevents, context definitions, layout definitions and some other things. Basically, it has turned into a general interface for specifying details in LilyPond syntax when writing in Scheme, and, as one example, now provides a user-friendly alternative for the markup macro. markup is a tricky construct for writing markups with a syntax that is almost, but not quite, entirely unlike LilyPond.
Previously, there was a special interface using $ for
getting material from local variables into #{ ... #}. Some aspects of it have been turned into a general feature ("immediate’’ Scheme expressions) available throughout LilyPond, but more importantly, all Scheme expressions inside of #{ ... #} are evaluated in `lexical closure’, meaning that they can access the local variables of surrounding Scheme code. So #{ ... #} has turned into a quite more powerful feature, at the same time behaving more like `regular’ LilyPond.
If we take the above example for relative music and change it such
that the default will be to make the music relative to f
(which means that the first note will actually be written with
absolute pitch), we can call this \ablative (because it starts
out as absolute, and continues relative):
ablative =
#(define-music-function (parser location pitch music)
((ly:pitch? #{ f #}) ly:music?)
#{ \relative $pitch #music #})
We did a few things here typical for user-defined functions: there is
little point in a documentation string, and we generously employ #{ ... #}.
Actually, since \relative remains available, there is little point in accepting an optional pitch argument here, but it makes for a somewhat more interesting example.
Using #{ ... #} extensively makes it possible to employ Scheme mostly as glue code between LilyPond fragments.
Future developments
Further unification of argument types
At the current point of time, there are still two argument types
recognized separately in the grammar: ly:duration? and
ly:pitch?. As a result, the last example had to use an
immediate Scheme expression $pitch (as otherwise the argument would have been interpreted as type "Scheme’’), while it was possible to write #music since arguments of type music and of type Scheme are interchangeable already, relying only on evaluation of the predicate for type checking.
Removing those special argument types provides a bit of a challenge. Pitches are challenging because a pitch is a valid start of a music expression (and that implies a bit of incremental parsing to make them fit with the general scheme of things since one does not want to recognize pitches using lookahead in optional argument settings, and pitches can contain an arbitrary number of octave marks).
Durations are challenging in a similar manner because of being able to be confused with numbers, because of optional following dots, and partly because they can become part of a preceding music expression such as when preceded by a pitch.
Removal of "closed music’’ category
At the current point of time, music arguments before durations or
after skipped optional arguments need to be enclosed in braces or
other delimiters in order to avoid ambiguities in the interpretation
of things like c 8: is the 8 part of the music argument, or is it a duration following it? And an optional music function argument has to be first created, then checked against its predicate, and if that does not match, backed up in the grammar. And in a stateful parser like that of LilyPond, backing up is only possible
when the parser has not seen further tokens and/or has entered
into states depending on them.
Some of the more recent changes in the LilyPond parser have been in the area of resolving such ambiguities by looking at the context and potentially backtracking and/or parsing stuff incrementally.
The principal idea is that one parses first tentatively without lookahead, and checking possible expression types that can arise against the optional argument predicates, even though the actual expression might not be complete. This means that the argument predicates must not have side-effects, but just depend on their arguments, a reasonable assumption.
However, when the expression itself results from a music or scheme
function call, the amount of possible reparsing is limited since
requiring music or scheme functions to be without side-effect would be
excessive: for example the main point of \displayMusic is its side effect. So if
\displayMusic is in itself a potentially
optional argument of a music function, we either have to be able to
parse its argument without lookahead, or we won’t be able to back it
up again in case it is not accepted by the optional argument
predicate.
As one example, a \repeat construct at the current point of time is not closed music, because LilyPond needs to be able to parse an item without lookahead when it may or may not accept it as
an optional argument. And a \repeat construct may be followed by \alternatives, something that is impossible to see without lookahead. The solution is to first parse just up to the conclusion of the \repeat, and then call the optional argument predicate on the resulting music expression. If it is found acceptable, LilyPond can resume parsing after having made the decision that it will not skip this optional argument, and the resumed parsing can now work using lookahead tokens freely. If it turns out that the construct with alternatives is not acceptable
any more, we blame the user (meaning that we don’t deal with
surprising amounts of backtracking).
Removing the concept of "closed music’’ (or at least reducing its
impact) would reduce the number of things a user has to understand in
order to create and use music functions. Here is one example of a
set-back: at some point of time, \displayMusic was given an optional port argument before the music expression. Since only closed music expressions are allowed after a skipped optional argument, \displayMusic c'' ceased being valid input, a side effect discovered more than a week after making the change. The change has been reverted for now, but at one point of time, it should be possible to bring it back without such repercussions.
The idea is to have the user just see results equivalent to greedy and
backtracking parsing compatible with evaluating the given music
function predicates: this is conceptually easy to understand, explain,
and extend, but mapping this into code that can be tackled by an
LALR(1) parser generator is challenging.
LilyPond online editors
by Valentin Villenave.
Although some of our youngest readers may not remember, there actually was a time, not too many years ago, where people did rely on their computers to do their computing, where your phone’s primary use was to make phone calls, where using a webmail service was considered bad taste (and JavaScript, an unnecessary ugliness), where thin clients were regarded as subpar computers to be used exclusively in a corporate environment and usually meant your boss was kinda cheap; a time where the "cloud computing" buzzword — can you imagine — didn’t even exist!
One of LilyPond’s major downsides when it comes to dealing with inexperienced users, and probably one of the reasons why it hasn’t seen more widespread adoption amongst musicians, is its setup process that may seem peculiar to people unfamiliar with the notion of "source code". Installing LilyPond itself has become increasingly easy over the years; setting up your work environment (text editor, PDF preview, command-line invocation) is a bit more involved... assuming you can think about it: I can’t count the people around me to whom it has never occurred that they could actually be using something else than the default NotePad application for Windows™, or the LilyPad text editor we provide for Mac OS X.
In this regard, being given the ability to discover the full potential of LilyPond (including syntax-colored source code and score preview) in a Web browser without having to install anything whatsoever, may be an excellent introduction. As early as 2006, a notorious visionary — namely, myself — predicted that "LilyPond’s future might well be web-oriented". I wasn’t, to be fair, the first one: in January 2005, Sebastiano Vigna had introduced the LilyPond Snippet Repository that provided the very first public LilyPond server. The LSR was intended for documentation purposes and required a bit of work to get around its ERW-based interface, but it is still in use today.
Setting it up, to be honest, is quite an involved task and demands more than just courage, but also a bit of masochism and recklessness. By that I am not only referring to Sebastiano’s peculiar taste in terms of server-side Java, but the main point is that LilyPond, as a program and as a language, presents an extreme security risk for any server where you want to run it. Since it allows for embedded Scheme, that in turn can convey system calls, it can — and will eventually, if you leave it open to anyone on the interwebs — wipe out your system and kill your hamster! That’s not the only issue: even perfectly harmless code can eat up a considerable amount of resources, there’s also the problem of \include instructions that would require users to upload multiple files and thereby multiply injection risks, etc.
In short: children, do not do it at home!
But that kind of warnings has never stopped any self-respecting geek, has it?
Interestingly enough, all of the subsequent prototypes, based on Sebastiano’s work, weren’t conceived as standalone web-applications, but rather tried integrate LilyPond into existing Content Management Systems. In 2006 Chris Lamb came up with a Wordpress plugin; slightly later our developer and git expert Johannes Schindelin introduced a MediaWiki extension that raised a lot of eyebrows (man, LilyPond in Wikipedia! — hold your horses, more on that later). In 2007, Christophe Richard also turned it into a plugin for the SPIP cms, and it was even added as a plugin to the FCK editor. Last but not least, in 2009 Harmath Dénes integrated LilyPond into Google Wave® the brand-new disruptively-sexy Next Big Thing that was gonna revolutionize the world as we knew it.
In May 2007, I had introduced the first prototype of an online real-time syntax-highlighting and auto-indenting LilyPond editor. It was mostly a hack around the Codepress Javascript editor, but it was fun to make and it became part of a French effort to provide LilyPond-ready web applications (around the now-defunct lilyserv.net server, modeled after Sebastiano’s code). We were beginning to discuss a possible standalone web-application (a then new-and-shiny buzzword, as Gmail&Co were gaining traction). Laurent Ducos even suggested it should sport an online graphical interface such as Denemo or LilyComp, but that never went anywhere.
Then came what I’d refer to as the "second generation" of LilyPond-based online efforts: the era of standalone, user-friendly web applications. In this regard, the year 2010 certainly was some kind of a turning point. The iPhone® (and its useless brother the iPad®) was everywhere, and running LilyPond on it wouldn’t have been an option; taking advantage of that, a Daniel Glover released the proprietary, LilyPond-reliant Etude App, that is said to have sold quite well. For those of us who don’t have an Apple product and actually care about the open Web, the same year Mike Blackstock (after having tried to set up a LilyPond-friendly wiki — in vain ) announced the availability of a collection of online music editing tools, aptly named OMET. Then there was also LilyPondTool contributor Joshua Koo and his project (beware, that’s a Facebook link) of a "Google Docs for music" that became JabTunes (the homepage is ugly and contains some Flash, but check out his labs for some pretty nifty HTML5 experiments). Finally, the new kid on the block has been announced quite recently by Trevor Dixon as Lilybin, that looks quite promising. I’ve also seen other people (including Dominic Neumann, Eric Knapp, Philip Peterson) announce that they were working on online web interfaces for LilyPond; there definitely is a growing trend here.
As early as May 2009, an unidentified developer appeared to be working on a project called WebLily. It wasn’t made public until March 2010, as a LilyPond Report comment, although we still didn’t know (other than through whois) who was behind it. Learning about it made LilyPond project manager Graham Percival unusually enthusiastic:
Interesting! Never heard of it before. [...] I have to admit that at first I was expecting tons of advertizing (I have a nasty mind), but it all looks legit. The "clickable notation reference" is a neat idea.
(Evidently, Graham’s usual grumpy self took over soon thereafter, as he roasted "Mr. WebLily" for his alleged disregard of security risks. [Update: he then apologized, but his concerns were nothing if not legitimate in the first place.])
"Mr. WebLily" soon appeared to have a name, as he not only maintained his WebLily service but also launched a new, more commercial service: Scorio.com — although LilyPond isn’t credited prominently on this new website. Johannes Feulner — founder of WebLily and Scorio — also happens to be one of our readers, and he generously sent us the following contribution more than a year ago.
Scorio.com — An online WYSIWYG editor for LilyPond?
by Johannes Feulner
Creative Commons 3.0 BY-SA
Contact: johannes.feulner[at]scorio[dot]com
WYSIWYG? “Impossible”, “stupid idea”, “out of scope” might well be the first reaction of a passionate LilyPond user and LilyPond lover. However at scorio.com we tried to tackle the task and you are invited to see how we did.
scorio.com is a website where you can write, edit, publish and share sheet music right from within your web browser. No software installation is needed, it’s all HTML running in every browser. The music is engraved by LilyPond, giving you sheet music of its wonderful quality and beauty. Nonetheless, the editor comes with a graphical user interface completely hiding the user from the intricacies and subtle nuances of LilyPond’s mighty input language (Ly code).
In this way scorio.com makes LilyPond available to musicians who do not have programming skills or lack the time to acquire them. scorio.com is well suited for beginners and learners. But being a work in progress you may expect it to cover increasingly larger subsets of LilyPond’s endless possibilities to make it a useful tool for all sheet music writing musicians.
How it all began
In early 2009 I became excited about GWT, the Google Web Toolkit to write AJAX-Applications for web browsers in plain Java. The promises were many: No pains with JavaScript to produce effects like Drag&Drop and other AJAX effects, simple server side integration, Eclipse as development tool allowing to debug through client and server side in a single session, runs in any browser, multi language support and many more. Even though I had studied computer science, I’ve chosen the management track and had not written a line of code for many years. Thus the promise of ease of development was especially alluring to me. And it proved to be true.
Since I am also a professionally trained organist with great love for classical music (my top internet radio station is of course http://www.organlive.com) I came up with the wild idea of writing a score writer for the internet as a demo project. Wading through Google’s search I got the impression, that something like this was not yet available on the net. Only later I learned about Noteflight, a Flash based score writer, which might already have been in existence at that time.
But the best thing I found during my search was LilyPond. I was impressed and excited about this marvellous piece of software and wondered why I hadn’t found it before, because I really do not like the various proprietary “market leading” score writers I had been using. Their user interfaces are weird, the music layout they produce has quite some potential for improvement.
So my first idea was to write a web based graphical user interface as a frontend to LilyPond. I decided to stick to GWT (all Java based) and to run LilyPond on the server through system calls from my Java servlet.
The precursor: WebLily.net
After visiting the Google IO 2009 conference I was full of enthusiasm, filled up with positive energy and started to hack right away in evenings and on weekends. This project became WebLily.net, a website where you have an input window to type in your LilyPond code (things like
{ c e g }) and an output window where the graphical result is instantly shown in HTML.This is pretty much a light version of the Frescobaldi editor. However, the advantage is that the score display is updated while you are typing your Ly code. This is extremely useful for beginners starting to learn the syntax. The type-translate-review cycle is extremely fast for small examples.
- Screenshot of WebLily.net with textual input (left) and instant HTML rendering (right)
As a special feature I added the Clickable LilyPond Reference. Whenever you find an example in the reference you can simply click onto the example to start up WebLily’s editor. This is very helpful, when you learn about the various features of LilyPond and want to try them out right away.
Another useful feature is that you can click onto a note head and the position of that note head is marked in the Ly code window. To produce the HTML-output I wrote an HTML backend to LilyPond. See below for technical details.
Even though it works quite well and WebLily.net has attracted some actual users, I was not really satisfied. Because what I’ve really wanted in the first place, is an interactive Editor where I could change the pitch of a note by drag&drop and insert new notes by clicking on them. Doing drag&drop using GWT is not that difficult to realize. However, when you stop the drag&drop operation you must determine the note affected, compute the new pitch and then update the Ly code. WebLily.net uses the point-and-click feature of LilyPond which passes for most graphical objects their line number and character position into the backend. So when I click on a note head I can actually know where this note head has been defined e.g. in line 4 starting at character position 9.
The limitation of this approach is the difficulty to insert the edited changes back into the Ly code. A note head may be a single note (
c), part of a chord<a c d>, durations and octaves may be involved (c’2,<a,, c d’>4), there may be trailing parameters (c-.\ff), there may be tweaks (<a \tweak #'color #red e d>4) and due to the expressional power of LilyPond’s syntax (including Scheme expressions) there is no easy way to handle arbitrary inputs. This approach might work for a well defined subset of LilyPond. But then you loose many of the syntactic possibilities and advantages. So I stopped at this point and started thinking of a different solution to this problem.WebLily.net is free to use for all friends of LilyPond and I want to keep it that way as long as I can pay for the server. You may of course support the project by clicking on the wonderful ads beautifying the website.
The real thing: scorio.com
The WebLily.net project convinced me that an HTML-based WYSIWYG editor can be built but that it will take much more effort to do so. I got into contact with Karin Höthker and Dominik Hörnel, both experienced software developers with a strong background in music. We had worked together for several years at Karlsruhe University in a research project on Informational Structures in Music. We made neural networks learn how to create music in the styles of historic composers such as J.S. Bach and others. But then we lost sight of each other. Both were excited about doing a music related project and so we started building scorio.com.
Now our goal is to create a place on the web where musicians can write and edit music in an interactive score writer, publish their works if they want to and to provide a search engine to find the sheet music others have written. All you should need for this is a computer and a web browser. This portal has started in November 2010 with a first, still limited version of the score writer (note editor).
Having learnt from WebLily we decided to use LilyPond as engraver (layout engine) but not to use Ly code as internal representation for our scores. We rather selected MusicXML as internal format. Finding, inserting, updating notes is much easier with MusicXML, since Java provides powerful tools to read, write and modify XML-Documents and MusicXML comes with a (mostly) well defined semantic.
- Screenshot of scorio.com’s note editor
So on every click you do on scorio.com we update our MusicXML model (e.g. change a pitch), convert the score into Ly code, run LilyPond with our HTML backend on that code and then display the HTML output as update to the user. Even though this seems a lot to do on each single mouse click the performance is not all that bad (see below). Where WebLily attaches line numbers and character offsets to HTML images, scorio.com uses references into the XML-structure to backlink the HTML output to its corresponding input.
On scorio.com you can produce scores in perfect LilyPond quality. However, there is no way for direct input of Ly code, but you can use the constantly growing (but still limited) repertoire of musical symbols the note editor exposes through its graphical interface. Wouldn’t it be nice if someone came up with a ly2musicxml converter?
Is scorio.com a WYSIWYG editor for LilyPond?
The answer is yes and no, depending on how you look at it. The proper answer might be this: scorio.com offers a WYSIWYG note editor running in your browser using LilyPond to engrave your music. But scorio.com is not a WYSIWYG editor where you could start with an arbitrary piece of Ly code and then edit that code in a meaningful way. Currently I do not even foresee a way to import Ly code into scorio.com. The other way around however, i.e. exporting a score as Ly code is perfectly possible. If our users demand it we’ll try to get this and other wonderful features onto our roadmap.
Some technical details
For tech savy readers I try to give some technical insights of how scorio.com has been realized.
HTML backend
One major task to do was writing an HTML backend to LilyPond. Looking into the socket backend I got an idea how one might do that. A LilyPond backend gets a bunch of so called Grobs (Graphical Objects). Each of these Grobs consists of one or more graphical elements that have to be placed at a certain position onto the page. The elements are of various types such as line (e.g. staffs), polygon (e.g. beams), bezier-sandwich (e.g. slurs), utf-8 string (e.g. composers’ name), named-glyph (music symbols such as Clefs and note heads) and some more.How might one put this into HTML? Our goal is to produce a solution that will run in any browser (including Microsoft’s Internet Explorer and the iPad) without the need to install any special add-ons or plugins. I pondered different approaches such as Scaled Vector Graphics (SVG), HTML5, Flash and others. SVG was and is my favourite. But I wanted a solution working with any browser including Internet Explorer 7 and recently the iPad. IE7 and IE8 do not support SVG nor HTML5 (this is now changing with IE9) and the iPAD doesn’t support Flash. Since all those wonderful technologies are not available I decided on using HTML images. Not really rocket science, but it works.
- Graphical output of the simple Ly code
{ c'}The simple Ly code
{ c' }produces the following Grob elements:
- utf-8-string “ “ outside a Grob, don’t really know what that one is for ledger line spanner with one line element for the note’s ledger line
- staff symbol with 5 line elements
- time signature with the music symbol “c” for 4/4
- clef with the music symbol for the violin clef
- note head with the music symbol for a note head
- utf-8-string “Music engraving by LilyPond …” for the tag line
Since HTML supports positioning objects at absolute pixel positions, I decided to create a div for every Grob element, i.e. every line, utf-8-string, music symbol and so on. The element divs are positioned absolutely according to the Grob’s coordinates. Since I want to have an HTML rendering close to the LilyPond print out, I decided to produce an HTML image for every Grob element in perfect LilyPond quality.
A note head may serve as an example. The resulting HTML code is an absolutely positioned div (left/top) and within that div sits the note head as an HTML image. The title attribute holds the Ly code position (the note head is defined in line 1 at position 2) that will be displayed as tooltip when hovering with the mouse over the Grob element’s HTML image. Note: Unfortunately not all Grobs do come with a source position, e.g. clefs.
<div style="position: absolute; left: 229px; top: 150px;" title=”Note Head at 1:2” >
<img style="position: absolute; left: 0px; top: -5px;"
src="http://weblily.net/app/service/Engraver
?gt=char&ff=Emmentaler&fs=40&ut=57654&co=0x000000">
</div>As can be seen from code the image is not a static image but it’s a call to an engraver servlet that will produce the image on demand. The parameters instruct the engraver servlet to produce a single character (gt=char) from the font Emmentaler (ff=Ementaler) at font size 40 (fs=40). The character to be produced has the Unicode code point 57769 (ut=57769) and the color is black (co=0x00000).
The conversion from LilyPond’s coordinates to HTML pixels is straight forward
- x-html-pixels = 10 * x-LilyPond-units
- y-html-pixels = -10 * y-LilyPond-units (yes, y-directions differ)
One of the (unsolvable?) problems arising is that LilyPond uses floating point numbers for its coordinates, but in browsers we only have pixels at integer coordinates. This inevitably leads to rounding errors. Some times a beam and a note stem or a note stem and a note head do not fit tightly and you may note one pixel holes where they shouldn’t be.
Another (solvable) problem is the positioning of characters and strings. As you note in the above example the image has a top position of -5px. Why? The reason is that LilyPond quite naturally uses the base line as a reference point. The div is positioned at the characters baseline. However, HTML images are positioned with respect to their upper left corner. There is no notion of a baseline in HTML. Since the note head rises 5 pixels above its base line (this is called the ascent of a character) we must move the image’s position up north by that amount. Unfortunately the ascent information is not present in LilyPond’s utf-8 callback for backends. So I had to add an additional parameter. This is one of the very few changes I had to apply to LilyPond’s C/C++ source code.
Since every Grob element is an HTML image it is easy to use GWT to setup HTML callback functions to handle mouse events, to implement drag&drop, to insert new notes and to select/deselect notes for editor operations.
Performance
When using an interactive note editor such as scorio.com every click with the mouse may change the score’s layout completely. Think about adding or deleting a note in the middle of your score or changing the time signature. Even though you may find some special cases where the effects of a click are strictly local you end up with the requirement to run LilyPond on the server for nearly each and every click. At first this sounds bizarre but after a while you get used to the thought, at least while working on little test examples. On weblily.net you can monitor the server performance in the lower right corner and see values around 0.6-0.8 seconds editing the simple above mentioned score.Both project use the very same hardware: Intel XEON E5410, two cores 2.3 GHz, 1.7 GB RAM. On scorio.com some tuning efforts brought this down to 0.2-0.4 seconds.
But we may not forget to add network latency which easily adds 0.3 to 0.6 seconds. This isn’t really snappy but since things on the web aren’t that fast in many cases I think this is acceptable and makes for a good user experience.
(Solvable?) Problem: LilyPond’s processing time grows exponentially with the size of the input score. Waiting 10 seconds for an update after clicking into a three page score is not really acceptable. So we had to resort to partial updates, processing just one system at a time. Thanks to LilyPond’s skipTypesetting we found a good way of implementing that. So on scorio.com you will note, that after edit operations the following systems are grayed out, since only the current system is updated and their layout may have become invalid. Click on the gray area for a full score update when you are ready to update the page.
I have great hopes into LilyPond moving to Guile 2.0 which has a built-in compiler for Scheme code. But I am aware that this will make scorio.com snappier (single systems will engrave faster) but it will not solve the problem with larger scores. Maybe some reengineering of LilyPond could be done to provide incremental updates? I’d be happy to get into a discussion about that.
Links
http://www.lilypond.org/
http://www.scorio.com/
http://www.weblily.net/
Thanks to Johannes for this contribution!
So, what’s coming next for "cloud-oriented" LilyPond applications? It certainly is too early to tell, but if history has taught us anything, it’s that the LilyPond community closely and strikingly follows the general trends in computing and software development (both Free and proprietary), so we can safely assume that online music interfaces aren’t going anywhere anytime soon. — A trend that’s also taken by our friends at MuseScore. On the WYSIWYG side, I’ve also noticed with great interest that Jan Nieuwenhuizen’s Schikkers List, which we introduced last year, is based on the GTK3 toolkit, that has an HTML canvas backend. Just sayin’.
Another point that shouldn’t be overlooked, is of an ethical matter. Free software advocates have been warning about the dangers of "cloud computing" and entrusting any third-party (particularly of a private/commercial nature, with the degree of instability and unforeseen policy changes it may imply) with your data. Granted, a music score hardly contains as much personal confidential information as, say, your email account; nevertheless I for one wouldn’t ever store my music scores on a webserver that isn’t owned by me or a non-profit community I know and trust — not without a solid local backup, at least.
As I said above, a point certainly can be made that such Web-applications provide an excellent introduction to GNU LilyPond, and a great way of working together on music scores. That is, assuming people know the app they’re using is powered by LilyPond: while it is mentioned on scorio (albeit not prominently), I couldn’t find any mention whatsoever on EtudeApp.com. If anything, we must stay vigilant and make sure LilyPond remains freely and easily accessible other than through proprietary overlays; we also should encourage developers of such web applications to make use of Free licenses whenever they can, knowing that it won’t drive people away but on the contrary gain support amongst Free software enthusiasts and the LilyPond community at large!
An addendum to LilyPond Report #23
In our previous installment, we mentioned that some rather simple syntax had been deprecated for several years. However, it turns our that current developments allowed for a further simplification of the LilyPond example presented in the article "Prelude #1 in Scheme". Check it out under this link.
That concludes the twentyfourth installment of the LilyPond Report. Our special guest next month will be Janek Warchoł, a young, enthusiastic LilyPond contributor who’ll share with us his passion and ideas. Stay tuned!
Cheers,
Graham Percival, David Kastrup & Valentin Villenave.
