Post a message

Let’s begin with the most important bit. At the current point of time, my financial situation is in free fall with very few slowdowns. Its terminal velocity may be restrained by my rather basic lifestyle. But if you think me having to split wood at ten degrees below zero because of the price of oil heating is helping much, think again: that shaves off something like maybe €40 from the monthly bills at best. It merely spreads my reserves a bit thinner.

Splitting wood.

"As the temperature has risen above 5 degrees, we fortunately have stopped heating" — D. Kastrup, 2012

So impact is looming close. The common answer to that is ``Get a job!’’. If you take a look at the activities in the LilyPond repository in the recent months, you’ll find that I already have a full-time job. I am working for you, making LilyPond a greater tool to work with for you and other musicians. And much as I like this job and enjoy working for you, the time has come to talk about a raise.

Now I am not the only one working on LilyPond, and you may question why you should consider paying me instead of someone else for that work. That is a valid question as long as you are actually paying someone else rather than using it as an excuse not to do anything at all. Check out http://www.lilypond.org/sponsoring.html for your options at any given point of time. At the time of writing this article, there are no other requests for funding. I deliver excellent value, and there is a limited window of opportunity before I have to close down operations and am no longer available. So buy yourself in while you still can.

There are others working on LilyPond in their spare time (and some of them are doing a really fabulous job). Now for one thing, the rallying point that a full-time developer provides helps even them to be more effective at what they do and can achieve, even if this full-time developer would not be first pick for the actual project lead or other tasks focused on communication skills.

But for another, it turns out that my own situation is somewhat special, and so I have an additional personal interest in you making it possible to continue my work on LilyPond, going beyond a mere job search.

Primary means of transportation (front and left).

D. Kastrup, 2012.

And so that you know what you buy into and how to make the most of it, I am telling you.

After figuring out that there were some things affecting my productivity in a number of ways that don’t seem to affect other people nearly as much, I actually asked for professional opinion and got diagnosed with several medical conditions. Now "conditions’’ are things to take into account, "medical’’ means phenomena fitting a pattern so that one can focus on approaches that have shown to be effective in similar cases.

After lots of experimenting around, it turns out that in practice I mostly have to live with being David, and you have to deal with me being David, because that is quite solidly hard-wired. And this can actually be a rather good deal for you and others, even if some parts of it may seem unusual.

If we cut through the skinny of buzzphrases like "Asperger’s’’ and "ADHS’’, here are key points worth knowing about me:

• I can’t stand things being `wrong’ in a number of ways. This means that I am a good fit for working in a setting with ethical principles (call it "ideological’’ if you want) like Free Software. It also means that I am uncommonly good at finding and exterminating even more obscure bugs (like tracking down a GCC compiler bug that threatened to make LilyPond unusable on newer systems).
• I get irritated and distracted rather easily. That means that I can be spectacularly bad at boring but necessary tasks which is not really good when operating in normal employment situations. It actually is not really good when leading any kind of life, either, but that is a different topic.
• I see people react, but am bad at predicting it. If you meet me face to face, cues that are absent in non-interactive communication do a lot for avoiding misunderstandings and annoyances. Like with many other things, this is not unusual in itself but in the degree to which it happens.

While I do not really shine in human relations when doing `batch processing’, I am pretty good with computers, and indeed I started working with them at a time when the communication literally consisted of a batch of punch cards and, a few hours later, output in print.

The low frustration tolerance for `boring’ tasks still shows when working with computers, but I fare rather good at `hard’ and `impossible’ tasks as the `mind’ of computers tends to be more accessible to me than that of fellow human beings.

Music is in the Air.

D. Kastrup, 2012.

Now at the moment, LilyPond has captured my attention and provides a lot of motivation, and other developers and their needs and work put me back on track when I get distracted. That means that I am quite productive overall, though not necessarily according to any plan. My actual impetus had been extending LilyPond for typesetting music for accordion. I have not exactly been making lots of progress on that front, instead getting myself caught up in improving an increasing number of areas within LilyPond itself and the workflows focused around using and developing it, to a degree where I have, in my single-mindedness, not enough time and focus left doing another job.

As a result of my involvement, programming LilyPond has become quite more powerful and enjoyable: most of the rather extensive changes in LilyPond’s input and music processing during the 2.15 development phase are due to me, and its integration with its extension language Scheme has become more logical and powerful at the same time.

LilyPond has more areas than what I have been working on: there have been other large improvements particularly in the typesetting stage of LilyPond, and those improvements did not depend on work I have done. It is my impression, however, that the motivation of users on LilyPond’s user list to solve problems by writing Scheme code on their own and teaching others how to do it has picked up. I certainly hope to see this trend to accelerate after we manage to make the next stable release, 2.16, and that it will contribute to attracting new developers over time.

We don’t have a general LilyPond development fund at the current point of time, and it would pose organizational and legal problems that are not worth tackling before there is positive proof that enough users are willing to invest in the continued well-being of LilyPond, and put their money rather than their foot where their mouth is.

Fortunately, my current situation is the perfect litmus test for you to demonstrate that. It is hard to imagine a more convincing incentive to contribute more than just talk. I might add that even with a general fund, you could do worse than contribute to me in person, since funds play to their strengths when paying for systematically planned tasks, and I am by far most effective when working on stuff that just happens to annoy me at a given point of time.

Patchy at work.

D. Kastrup, 2012.

There would be some things I could apply for even if everything was run through a fund: the laptop I am currently working on has been paid by another developer, one who is already putting in more time and energy than anyone could demand. It is a second-hand system from 2007 and, in spite of being quite better to work with than the system from 2000 I had to use before, is tasked a lot with test runs for LilyPond since we are just now getting them streamlined enough to be able to offload them to less experienced developers with better computing facilities. I am holding a talk about LilyPond at a major GNU/Linux fair in Chemnitz (the home of the Chemnitzer Concertina) in a few weeks, and this incurs travelling and accommodation costs that are not yet accounted for.

But most of the stuff I do is hard to plan and account for: I do it because it feels necessary and is the right thing at a given point of time. And that is enough to compel me. I hope it is for you as well: if I have to turn to a different endeavor for keeping myself afloat, I will have to cut out working on LilyPond completely since it would continue being a distraction, and distractions are something I can deal with very badly.

My mail address dak[at]gnu[dot]org is registered at Paypal, and if you want to skip the middle man for regular or larger amounts of money, you can ask me for my banking details (I am living in Germany). People prefer not to think about details, so here are some payment plans. The idea is to contribute a fixed minimum, and if a specified target is not reached by all contributions, you contribute proportionally up to a cap. Of course, you are free to pick all three numbers yourself, but here are a few models:

• [Regular] €25 per month fixed, no cap. This is the payment plan to pick once everything is sailing smoothly and you don’t want to contribute unduly much or think about it unduly much.
• [Lifesaver] Minimum €0, cap €250 per month, monthly target €800. That means that if the target (which basically allows me to postpone my decision to work elsewhere) is reached with everybody’s minimum already, you are not billed. This is the option to pick if you don’t want to support a single person as much as keep the LilyPond project from losing me. You do what is necessary to avoid my leaving, but nothing else. Yes, it will be annoying if it turns out you have to pay the cap more than once, but it will also be annoying for me not even to afford survival in spite of highly qualified work.
• [Torchbearer] Minimum €50, cap €150 per month, monthly target €1200. This is a model aimed at being reasonably comfortable for you as well as for me if everything works out.

Since I have been surprised in the past by the percentage of people actually coming through with their expressed intentions, you would start payment with your cap, and I would inform you in time for your next payment how much of that could be carried over due to others pitching in. The more variable schemes are designed to be effective with a few highly dedicated people, but allow them to back off once more people join.

A tool like LilyPond, providing a simple, open way of describing music and typesetting it to high standards, is important not just for personal use, but also as a base to work with music that has already passed into the public domain and that forms a large part of our cultural heritage, and that LilyPond can help to make easier to access and cast into new forms.

I do not suggest that there are no other ways to invest money or energy for making LilyPond a better tool for working with music. But don’t take that as an excuse to do nothing at all. If you find that LilyPond would be better served by you working or investing on aspects not covered by me, then do it. Don’t merely think or talk about it: that may feel noble, but achieves nothing. Don’t postpone it. You have a chance of making the world a better place for music and, if you choose to do it through me, a limited window of opportunity. Don’t blow it.

Creative Commons 3.0 BY-SA
Contact: johannes.feulner[at]scorio[dot]com

WYSIWYG? “Impossible”, “stupid idea”, “out of scope” might well be the first reaction of a passionate LilyPond user and LilyPond lover. However at scorio.com we tried to tackle the task and you are invited to see how we did.

scorio.com is a website where you can write, edit, publish and share sheet music right from within your web browser. No software installation is needed, it’s all HTML running in every browser. The music is engraved by LilyPond, giving you sheet music of its wonderful quality and beauty. Nonetheless, the editor comes with a graphical user interface completely hiding the user from the intricacies and subtle nuances of LilyPond’s mighty input language (Ly code).

In this way scorio.com makes LilyPond available to musicians who do not have programming skills or lack the time to acquire them. scorio.com is well suited for beginners and learners. But being a work in progress you may expect it to cover increasingly larger subsets of LilyPond’s endless possibilities to make it a useful tool for all sheet music writing musicians.

How it all began

In early 2009 I became excited about GWT, the Google Web Toolkit to write AJAX-Applications for web browsers in plain Java. The promises were many: No pains with JavaScript to produce effects like Drag&Drop and other AJAX effects, simple server side integration, Eclipse as development tool allowing to debug through client and server side in a single session, runs in any browser, multi language support and many more. Even though I had studied computer science, I’ve chosen the management track and had not written a line of code for many years. Thus the promise of ease of development was especially alluring to me. And it proved to be true.

Since I am also a professionally trained organist with great love for classical music (my top internet radio station is of course http://www.organlive.com) I came up with the wild idea of writing a score writer for the internet as a demo project. Wading through Google’s search I got the impression, that something like this was not yet available on the net. Only later I learned about Noteflight, a Flash based score writer, which might already have been in existence at that time.

But the best thing I found during my search was LilyPond. I was impressed and excited about this marvellous piece of software and wondered why I hadn’t found it before, because I really do not like the various proprietary “market leading” score writers I had been using. Their user interfaces are weird, the music layout they produce has quite some potential for improvement.

So my first idea was to write a web based graphical user interface as a frontend to LilyPond. I decided to stick to GWT (all Java based) and to run LilyPond on the server through system calls from my Java servlet.

The precursor: WebLily.net

After visiting the Google IO 2009 conference I was full of enthusiasm, filled up with positive energy and started to hack right away in evenings and on weekends. This project became WebLily.net, a website where you have an input window to type in your LilyPond code (things like { c e g }) and an output window where the graphical result is instantly shown in HTML.

This is pretty much a light version of the Frescobaldi editor. However, the advantage is that the score display is updated while you are typing your Ly code. This is extremely useful for beginners starting to learn the syntax. The type-translate-review cycle is extremely fast for small examples.

Screenshot of WebLily.net with textual input (left) and instant HTML rendering (right)

As a special feature I added the Clickable LilyPond Reference. Whenever you find an example in the reference you can simply click onto the example to start up WebLily’s editor. This is very helpful, when you learn about the various features of LilyPond and want to try them out right away.

Another useful feature is that you can click onto a note head and the position of that note head is marked in the Ly code window. To produce the HTML-output I wrote an HTML backend to LilyPond. See below for technical details.

Even though it works quite well and WebLily.net has attracted some actual users, I was not really satisfied. Because what I’ve really wanted in the first place, is an interactive Editor where I could change the pitch of a note by drag&drop and insert new notes by clicking on them. Doing drag&drop using GWT is not that difficult to realize. However, when you stop the drag&drop operation you must determine the note affected, compute the new pitch and then update the Ly code. WebLily.net uses the point-and-click feature of LilyPond which passes for most graphical objects their line number and character position into the backend. So when I click on a note head I can actually know where this note head has been defined e.g. in line 4 starting at character position 9.

The limitation of this approach is the difficulty to insert the edited changes back into the Ly code. A note head may be a single note (c), part of a chord <a c d>, durations and octaves may be involved (c’2, <a,, c d’>4), there may be trailing parameters (c-.\ff), there may be tweaks (<a \tweak #'color #red e d>4) and due to the expressional power of LilyPond’s syntax (including Scheme expressions) there is no easy way to handle arbitrary inputs. This approach might work for a well defined subset of LilyPond. But then you loose many of the syntactic possibilities and advantages. So I stopped at this point and started thinking of a different solution to this problem.

WebLily.net is free to use for all friends of LilyPond and I want to keep it that way as long as I can pay for the server. You may of course support the project by clicking on the wonderful ads beautifying the website.

The real thing: scorio.com

The WebLily.net project convinced me that an HTML-based WYSIWYG editor can be built but that it will take much more effort to do so. I got into contact with Karin Höthker and Dominik Hörnel, both experienced software developers with a strong background in music. We had worked together for several years at Karlsruhe University in a research project on Informational Structures in Music. We made neural networks learn how to create music in the styles of historic composers such as J.S. Bach and others. But then we lost sight of each other. Both were excited about doing a music related project and so we started building scorio.com.

Now our goal is to create a place on the web where musicians can write and edit music in an interactive score writer, publish their works if they want to and to provide a search engine to find the sheet music others have written. All you should need for this is a computer and a web browser. This portal has started in November 2010 with a first, still limited version of the score writer (note editor).

Having learnt from WebLily we decided to use LilyPond as engraver (layout engine) but not to use Ly code as internal representation for our scores. We rather selected MusicXML as internal format. Finding, inserting, updating notes is much easier with MusicXML, since Java provides powerful tools to read, write and modify XML-Documents and MusicXML comes with a (mostly) well defined semantic.

Screenshot of scorio.com’s note editor

So on every click you do on scorio.com we update our MusicXML model (e.g. change a pitch), convert the score into Ly code, run LilyPond with our HTML backend on that code and then display the HTML output as update to the user. Even though this seems a lot to do on each single mouse click the performance is not all that bad (see below). Where WebLily attaches line numbers and character offsets to HTML images, scorio.com uses references into the XML-structure to backlink the HTML output to its corresponding input.

On scorio.com you can produce scores in perfect LilyPond quality. However, there is no way for direct input of Ly code, but you can use the constantly growing (but still limited) repertoire of musical symbols the note editor exposes through its graphical interface. Wouldn’t it be nice if someone came up with a ly2musicxml converter?

Is scorio.com a WYSIWYG editor for LilyPond?

The answer is yes and no, depending on how you look at it. The proper answer might be this: scorio.com offers a WYSIWYG note editor running in your browser using LilyPond to engrave your music. But scorio.com is not a WYSIWYG editor where you could start with an arbitrary piece of Ly code and then edit that code in a meaningful way. Currently I do not even foresee a way to import Ly code into scorio.com. The other way around however, i.e. exporting a score as Ly code is perfectly possible. If our users demand it we’ll try to get this and other wonderful features onto our roadmap.

Some technical details

For tech savy readers I try to give some technical insights of how scorio.com has been realized.

HTML backend
One major task to do was writing an HTML backend to LilyPond. Looking into the socket backend I got an idea how one might do that. A LilyPond backend gets a bunch of so called Grobs (Graphical Objects). Each of these Grobs consists of one or more graphical elements that have to be placed at a certain position onto the page. The elements are of various types such as line (e.g. staffs), polygon (e.g. beams), bezier-sandwich (e.g. slurs), utf-8 string (e.g. composers’ name), named-glyph (music symbols such as Clefs and note heads) and some more.

How might one put this into HTML? Our goal is to produce a solution that will run in any browser (including Microsoft’s Internet Explorer and the iPad) without the need to install any special add-ons or plugins. I pondered different approaches such as Scaled Vector Graphics (SVG), HTML5, Flash and others. SVG was and is my favourite. But I wanted a solution working with any browser including Internet Explorer 7 and recently the iPad. IE7 and IE8 do not support SVG nor HTML5 (this is now changing with IE9) and the iPAD doesn’t support Flash. Since all those wonderful technologies are not available I decided on using HTML images. Not really rocket science, but it works.

Graphical output of the simple Ly code { c'}

The simple Ly code { c' } produces the following Grob elements:

• utf-8-string “ “ outside a Grob, don’t really know what that one is for ledger line spanner with one line element for the note’s ledger line
• staff symbol with 5 line elements
• time signature with the music symbol “c” for 4/4
• clef with the music symbol for the violin clef
• note head with the music symbol for a note head
• utf-8-string “Music engraving by LilyPond …” for the tag line

Since HTML supports positioning objects at absolute pixel positions, I decided to create a div for every Grob element, i.e. every line, utf-8-string, music symbol and so on. The element divs are positioned absolutely according to the Grob’s coordinates. Since I want to have an HTML rendering close to the LilyPond print out, I decided to produce an HTML image for every Grob element in perfect LilyPond quality.

A note head may serve as an example. The resulting HTML code is an absolutely positioned div (left/top) and within that div sits the note head as an HTML image. The title attribute holds the Ly code position (the note head is defined in line 1 at position 2) that will be displayed as tooltip when hovering with the mouse over the Grob element’s HTML image. Note: Unfortunately not all Grobs do come with a source position, e.g. clefs.

<div style="position: absolute; left: 229px; top: 150px;" title=”Note Head at 1:2” >
     <img style="position: absolute; left: 0px; top: -5px;"
          src="http://weblily.net/app/service/Engraver
               ?gt=char&ff=Emmentaler&fs=40&ut=57654&co=0x000000">
</div>

As can be seen from code the image is not a static image but it’s a call to an engraver servlet that will produce the image on demand. The parameters instruct the engraver servlet to produce a single character (gt=char) from the font Emmentaler (ff=Ementaler) at font size 40 (fs=40). The character to be produced has the Unicode code point 57769 (ut=57769) and the color is black (co=0x00000).

The conversion from LilyPond’s coordinates to HTML pixels is straight forward

• x-html-pixels = 10 * x-LilyPond-units
• y-html-pixels = -10 * y-LilyPond-units (yes, y-directions differ)

One of the (unsolvable?) problems arising is that LilyPond uses floating point numbers for its coordinates, but in browsers we only have pixels at integer coordinates. This inevitably leads to rounding errors. Some times a beam and a note stem or a note stem and a note head do not fit tightly and you may note one pixel holes where they shouldn’t be.

Another (solvable) problem is the positioning of characters and strings. As you note in the above example the image has a top position of -5px. Why? The reason is that LilyPond quite naturally uses the base line as a reference point. The div is positioned at the characters baseline. However, HTML images are positioned with respect to their upper left corner. There is no notion of a baseline in HTML. Since the note head rises 5 pixels above its base line (this is called the ascent of a character) we must move the image’s position up north by that amount. Unfortunately the ascent information is not present in LilyPond’s utf-8 callback for backends. So I had to add an additional parameter. This is one of the very few changes I had to apply to LilyPond’s C/C++ source code.

Since every Grob element is an HTML image it is easy to use GWT to setup HTML callback functions to handle mouse events, to implement drag&drop, to insert new notes and to select/deselect notes for editor operations.

Performance
When using an interactive note editor such as scorio.com every click with the mouse may change the score’s layout completely. Think about adding or deleting a note in the middle of your score or changing the time signature. Even though you may find some special cases where the effects of a click are strictly local you end up with the requirement to run LilyPond on the server for nearly each and every click. At first this sounds bizarre but after a while you get used to the thought, at least while working on little test examples. On weblily.net you can monitor the server performance in the lower right corner and see values around 0.6-0.8 seconds editing the simple above mentioned score.

Both project use the very same hardware: Intel XEON E5410, two cores 2.3 GHz, 1.7 GB RAM. On scorio.com some tuning efforts brought this down to 0.2-0.4 seconds.

But we may not forget to add network latency which easily adds 0.3 to 0.6 seconds. This isn’t really snappy but since things on the web aren’t that fast in many cases I think this is acceptable and makes for a good user experience.

(Solvable?) Problem: LilyPond’s processing time grows exponentially with the size of the input score. Waiting 10 seconds for an update after clicking into a three page score is not really acceptable. So we had to resort to partial updates, processing just one system at a time. Thanks to LilyPond’s skipTypesetting we found a good way of implementing that. So on scorio.com you will note, that after edit operations the following systems are grayed out, since only the current system is updated and their layout may have become invalid. Click on the gray area for a full score update when you are ready to update the page.

I have great hopes into LilyPond moving to Guile 2.0 which has a built-in compiler for Scheme code. But I am aware that this will make scorio.com snappier (single systems will engrave faster) but it will not solve the problem with larger scores. Maybe some reengineering of LilyPond could be done to provide incremental updates? I’d be happy to get into a discussion about that.

Links

http://www.lilypond.org/
http://www.scorio.com/
http://www.weblily.net/