I was fortunate enough to be invited by Scott Reeves to speak about PulseAudio and specifically the KDE Integration work recently undertook. Scott works mostly on the SLED side of things (Enterprise Desktop) and PulseAudio falls within his remit. I'm pleased to say that the next version of OpenSuse's KDE flavour will follow the lead of Mandriva, Fedora and, most recently Kubuntu, and default to use PulseAudio under KDE. This is IMO a great step forward and will help continue the acceptance and appreciation of PulseAudio and I'm very happy to have played a part in making that possible.

Our talk itself went very well and seemed to be well received. It was also a pleasant surprise to finally meet both Takashi Iwai and Lennart Poettering who attended my talk and generally milled around. I've managed to miss meeting Lennart at various events over the years and while we've worked quite closely on PulseAudio in the past (to the extent he trusts me enough with managing the PA project while he works on the systemd project), it was nice to actually "talk" about PulseAudio! The subsequent BoF (Birds of a Feather) session on systemd was also very interesting (even if there was a little bit of a sticking point on defining preload vs. readahead which IMO was really labouring semantics to a degree and prevented getting to the more interesting stuff!). Takashi is another one of the few people who work on Linux Audio and it was nice to chat to him after exchanging several emails over the years on the ALSA mailing list.

The rest of the conference was also interesting. As someone who sits rather on the fence between KDE and Gnome it was interesting to see the developments in Gnome 3 demonstrated. There were various other titbits of interesting things too, but I wont bore the world with the details!

On the Saturday, I took the tram to the south of Nürmberg and visited their New Museum of Art and Design. It was very enjoyable. Almost deserted, which made it a rather nice experience, but also sad to see such a place under appreciated. Still it was quite early when I went, so maybe it hots up in the afternoon! My favourite photographs were by Gabriele Basilico and depicted a series of chairs with interesting seats: lattice, round holes, metal bars etc. and then accompanying photographs of a bottom that had clearly been sitting in said chair for a while. That quite tickled me An installation peace by Jeppe Hein was also very smart. Using light, mirrors and sensors to create interactive "rooms", including and empty one that just set off an alarm when you stepped into it (no doubt just to make people jump!). I then walked up through the centre checking out the churches (which would be a lot more interesting if they didn't have all that religious stuff inside them!) and the castle. A thoroughly pleasant day out!

When it was time to return home, my only regret was that I didn't get a chance to say goodbye to Scott, so as luck would have it I bumped into him at the airport in the queue going through security. It turned out he was on my flight to Amsterdam anyway so we were able to have a little blether on the way there.

What came as even more of a coincidence was the fact that I bumped into my mate Dom who happened to sit in front of me on the connecting flight to Edinburgh! He'd just been off climbing Kilimanjaro, which is arguably slightly more interesting than my week but he got his phone stolen at the "security" check in the airport (and the subsequent delay that it caused actually meant he was on my flight rather than an earlier one) so that evened things up!

But all in all, thanks to Novell, OpenSuse and Scott for inviting me along!

So how does PulseAudio route your audio and what the hell do you mean by "audio routing" anyway? Well, PA handles multiple audio devices ("sinks" for output, "sources" for capture). When an application wants to play sound it says "Hey PulseAudio, play this!!", and PA tries it's best to comply. The application will typically not care about which device to actually output too - they delegate this job to PA (and ultimately to the user via PA's tools such as pavucontrol, gnome-volume-control and (more recently) kmix/phonon too). These tools allow you to move the stream to your preferred sink - no need for an unfamiliar GUI inside a particular application for choosing which audio device to use; users know where to look in their desktop environment to control sound settings. From an HCI/usability perspective I think this is very important (although incumbent users need to shake of their natural assumption that apps should provide a way to configure audio device settings).

With a fresh user account PA will attempt to calculate a suitable default device to use. It does this by assigning each sink an internal score of appropriateness. This is just to determine our initial defaults so it doesn't matter too much if we get this wrong, but obviously it's nice to try and get things right first time! So once our default device is chosen, when a stream is played, it will use this sink. Simple huh? Well not quite. What if I change the default while a stream is playing? My stream is moved across to the new default right? No. Setting the default device does not do this. It acts as a fallback, it's not an "active" default. If I stop my app and play it again, it will be played on the new fallback, but it wont move when the stream is "live".

Now this is just the very beginning so if you've become overexcited by now, best take a cold shower (and probably look up psychiatrists in the phone book...) as I'm about to move on to look at the "stream restore" database...

Stream Restoration

So the stream restore database is handled by module-stream-restore (m-s-r). It's part of the default PA install so 99.9% of users are likely using it. What this module does is to track when a user specifically moves a stream to a new device. When the user uses e.g. pavucontrol or kmix to move a particular stream to a new sink, this triggers a mechanism inside m-s-r that causes the sink name to be saved for that application. If that application appears at any point in the future and tries to play sound, m-s-r will try it's best to ensure that this sink is used. If the device is not currently available, we will ultimately use the fallback/default instead. If this saved sink becomes available at some point during the lifetime of the stream, the stream will be moved across automatically (which differs from the behaviour when setting the default sink).

Now it gets complicated. Despite my terminology above, m-s-r does not use "applications" per-se when it saves the device choice. It will look at a stream and select a "stream restore id" for it based on several bits of metadata attached to the stream. Firstly it will look to see if the stream has a role. If it does, the role is used as the identifying factor, not the individual application. So e.g. for event sounds, the "stream restore id" will actually be "sink-input-by-media-role:event" regardless of which application actually produced it. As m-s-r is responsible for storing volume, mute and device preferences, this means that all event sounds will have the same volume and will be played on the same device. If the stream does not have a role, PA checks for a few other things to try and create it's "stream restore id": it checks for an "application id" (a specific property set AFAIK only by a select few applications), an "application name" (which 99% of PA clients provide) and finally it checks for a media name property (which is pretty unlikely IMO, although I may have missed a use case here). Once the "stream restore id" is chosen it is saved for that stream and we will always use this rule when updating our saved volume, mute and device settings thereafter.

When a user moves the stream (using one of the afore mentioned GUI apps - or via any app that makes use of the pa_context_move_sink_input_by_*()  APIs (also applies to source_outputs)), the "stream restore rule" they are ultimately changing will depend on the metadata the application has provided PA in the first place. So for example, Amarok will be tagged with a "music" role, and thus when you use kmix to move the Amarok stream to a different device you are not just moving Amarok, you are actually moving all streams tagged as "music". (Update 1) But wait! It does not actually move the currently playing "music" streams. They will only be moved the next time a PA stream is created (this could be when the track changes, or it could be when the app restarts - it depends on implementation), or when the stream restore database is written by some PA client which requests the "apply_immediately" flag set.

This whole approach can be summed up with two flow diagrams:

Device Manager

OK, so that about covers m-s-r, now I'll talk a bit about module-device-manager (m-d-m). This is a module I developed principally for integration with KDE. The GUI exposed for the Phonon preferences in KDE is such that it lists all devices that have ever been available on the device even if they are not currently available. This allows them to be listed and arranged into a priority list. Whenever the highest priority device is available, it will be used. It's a relatively simple UI and one that users can easily understand and work with. For a degree of flexibility, the GUI allows for different priority lists to be configured for different Categories (which in PulseAudio terminology are called Roles). So m-d-m provides an API and Protocol extension to implement such a routing policy directly in PA which turned the KDE GUI effectively into a frontend to PulseAudio (internally this is abstracted via Phonon but this is not really important). So this m-d-m will route your audio for you to the appropriate device. Due to how it is implemented, m-s-r will still take priority, so if you move a particular stream and the device choice is saved by m-s-r, it will take priority over the role-based priority list of m-d-m. I have now provided a way in kmix to delete any application specific rules in m-s-r such that the m-d-m routing (and thus what is displayed in Phonon preferences) will be used again should an override have been in place.

Critique

Right so that's how things work. Now let's rip it apart and moan a little about the shitty bits.

• A default that is not a default: First things first. This is a question we get a lot: "I've set my default device but <insert victim application here> doesn't want to use it". Now that you know how the routing works due to the excellent overview above, you can probably work out what's going on here. Either m-s-r has a specific device saved for that application's role or the application itself, and that overrides the "default" choice; the user was playing the stream when they set the default device but this wont be honoured until a new stream is created; or m-d-m is implementing a role-based priority list preference (which in all cases overrides the default button due to having a priority list for all cases - one for each role and one for the default case). All of the above is flexible but it sure as hell is not clear to the user what the hell is going on. There needs to be some kind of unification and automatic reaction here. Having a simple "default" button IMO makes sense. It's a very easy concept to grasp for users and we should try to make it work in the most case and explain more clearly in the various GUIs as to why it may not work in certain circumstances.
• Why, Why, WHY?: Things are not clear to the user what is going on. In a typical setup we have m-s-r and it remembers one thing. Unless you know how it works, you cannot configure a scenario when you have three or more devices and have it "just work" for you. For me three devices is not uncommon. At work I have set of USB speakers I use and at home I have Airtunes and network-based PA servers I use too. I want them to "just work" without me having to fire up a GUI every time I want to use them. A primary problem just now is that unless you've taken the time to read and understand the above descriptions, it's very unclear to users as to how the default device works and what m-s-r does and how it operates. Clarity and transparency is needed here, or some way of making it "just work" without needing such and explanation.
• You gotta role with it?: Yes that is a rather contrived "problem", but the point is valid. As we go down the path of encouraging applications to include as much metadata as possible in their streams, assuming we ever reach such a zenith, then the ultimate end result is that all streams have their role specified. When this happens a fundamentally useful part of PulseAudios design is effectively crippled in the default setup: m-s-r will always pick it's stream restore id of "sink-input-by-role:fooo". All application's will no longer be able to chose their own individual audio device; you will only be able to pick the device for that whole class of application. All music players must go to one device. All video players to another, but nothing in between. Some applications set their role generically but can be used for other purposes e.g. totem/dragon are video players but may be being used for just music in a given instance - I'd like to use my preferred music device not my preferred video device but I don't want to affect all video players with this choice. Obviously it's better if a multi-purpose app can work out it's role dynamically based on content but the principle point still stands - you cannot move streams anymore (with the exception of the afore mentioned caveat for currently playing streams when the move is initiated), only whole classes of streams. Some may argue that this is OK/acceptable. Personally I don't think so.
• I don't want an override anymore: Say you've picked an override for your stream/role with m-s-r. Say you've now decided you don't want that override and just want to use the default device now? None of the GUI tools out there just now allow you to reset this (with the exception of the latest kmix in my Git Repository at the time of writing).

How do you solve a problem like Myrouting?

So the above problems basically make things a bit too much like a black box. It's not clear what's going on and it doesn't offer enough flexibility in many cases either. How do we solve this? I'll now outline what I think is the best possible implementation of a routing system and how it will work.

Overall Operation

Firstly we need to remove the internal concept of a single "default" device. As we've discussed before within the PA community, we will move to a priority list concept. A significant difference to the KDE approach of showing the device priority list for users to tweak, we will ensure that exposing this list is not a strict requirement for operation. In order to service this requirement we will still offer the existing UIs for setting the default device, but the internal behaviour will change. Setting a device as default would simply move it to the top of the priority list. This will ensure that simpler UIs can still operate and whenever the user prefers a device above other others they just set it as default. In order to get the perfect priority list, they may have to click the "default" button a couple of times with a given setup, but it will stabilise for them very quickly (for reference, this default priory list will be the equivalent of the current m-d-m priority list with the psuedo-role: "none").

As well as the default priority list, we will also implement similar lists for each role (again very similar to the current m-d-m role-priority lists). When a stream has a role, we will simply use this list rather than the overall default one. If a given role does not have a priority list of it's own, the default one will be used (in KDE the phonon GUI will likely enforce that a list exists for each role, but has GUI techniques to keep it in sync).

In addition to the above, each individual application will also have it's own priority list. Again this is fully optional. If such a list does not exist, the role-based list will be used instead, ultimately falling back to the default list if no role is present.

When a new device appears it will be added to the default priority list only. It is open for debate as to where this device should appear in this list - top or bottom, but this could ultimately be a user preference. The routing logic should ultimately check the most appropriate list first, try to find the highest priority device in the list that is currently available. If none are available it will use the next most appropriate list ultimately falling back to the default list which will always contain an available device.

New APIs

New APIs will be made available (either as extensions or baked in - details TBC). These APIs will allow the querying and editing of these priority lists (thus facilitating GUIs such as the one in KDE). When editing these lists the changes they reflect will always be immediately applied to any running streams. APIs will also be exposed to delete a  role-based list or an application specific list (NB the application list could potentially also be cleared via the existing APIs by passing PA_INVALID_INDEX or an empty or NULL device name).

Existing APIs

The existing APIs for setting the default device and moving a stream to a given device will still operate as just now but will operate differently internally.

As mentioned previously, setting a default device will update the overall default priority list. This change will be propagated to any currently running stream immediately. Thus a fresh user account with no priority lists saved, the default priority list will be used for all streams. Clicking on the default device will move any running streams to that new device. And thus the very basic use case will operate in an intuitive way.

If a user moves a specific stream to a new device via the (pa_context_move_sink_input_by_*()  style APIs) this will trigger the update of the application specific priority list. This is a stark change to the existing behaviour which will update the details for the currently used device choice mechanism. If the priority list for particular role is to be updated by a given GUI application, then the new APIs above should be used to achieve this  result. If a stream is moved and it does not yet have it's own priority list one will be created for it automatically (containing only the chosen device initially.

GUIs

So what can the GUIs show? Well there are a lot of options now:

• A very simple GUI which just shows a list of devices and allows a single default to be shown.
• A more complex GUI that shows all devices in the default list and allows the user to order them.
• One that shows both the default and the known roles and allows a default to be chosen for each (or expose the full list for user ordering)
• etc. etc.

(FWIW although it's possible I don't think it would be necessary to ever expose the per-application list although there should be a method for indicating that such a list exists and is in use and allow the user to delete it and user a higher level priority list).

Caveats

With regards to event sounds specifically, we want to make sure that these are handled differently to a regular stream produced by an application. For example I may want to use rhythm box to output sound to my Airtunes device and implement this via an application specific priority list. Any event sounds for this should not be pushed to this device. I'm not sure what the best solution to this conundrum is, but I suspect that a simple property set on the stream that says "do not allow per-application priority list updating" would be sufficient. This still allows streams to be moved to the device, but this wont be saved for later restoration (obviously moving an event sound is really hard due to their duration, so this logic is more for the principle rather than the practicality).

(Update 1) As I propose to apply changes whenever they happen if you run e.g. two paplay processes at the same time and change one to use a specific device, my logic dictates that both streams would be moved. This is a regression on the current behaviour where only when the stream is restarted will it be moved, but to be honest I don't think it's much of a problem. If it is desired to have specific control over a given instance of an application, then we need to either a public API to not save the move (much like our internal one), or support a proplist property that suppresses this (e.g. no API change but a loose "standard" for performing this type of operation. If you want to run an application in a specific way, then you can always set a property for the e.g. the "application id" as mentioned above which would make it appear as a unique application compared to running it normally. So flexibility still exists here (albeit rather complex), but we probably do want to deal with the default cases well and make the more niche ones possible via some degree of extra work.

Overall

OK, so this is quite an exhaustive proposal and I've not nailed down the exact best way to implement it, but I think that this approach is ultimately the most flexible possible that supports all of the interfaces desired and still exposes the flexibility of the PulseAudio system. I have not given much thought to the storage of volume/mute status (which m-s-r currently tracks along with it's single device preference).  My gut feeling here is a fairly similar approach where you should be allowed to set the default volume, a per-device volume and a per-application volume. The former two being used as defaults and as soon as the per-application volume is changed via any API call, we save it with that specific application for future reuse. APIs can set the default volume and the per-role volumes and this will be propagated to applications that do not have their own specific override in place.

So in summary, this is how I think things should work. I don't think it's actually as complicated to implement as I've made it sound (would probably only take about three or four times as long as it's taken me to write about it!!) and I believe it keeps the power and flexibility of the system, still allows for minimal control interfaces yet allowing more exposed and complex ones if required.

Updates:

[1] 16th Feb: Add notes about how moving a stream with a role does not move any currently live streams and how the proposed solution would prevent running two instances of the same app and moving the device independently for each (as one would follow the other) unless the application id was overridden for one of the instances.

I've spent quite a lot of time refining the initial support I added a few weeks back. What follows is a brief summary of the changes/improvements/bugfixes.

Firstly, some of the tabs (namely "Capture Streams" were not shown unless an application was actively capturing while kmix started... somewhat unlikely really :p This is now fixed.

Then there was a problem with Profiles. These would be created automatically when you went to the "Configure Channels..." dialog. There were actually several problems relating to profiles which ultimately lead to controls going AWOL - with no way in the GUI to see them even if the backend itself was actually aware of their presence. The concept of GUIProfiles does not really tie in too well with the concept of "Dynamic Mixers" (where controls come and go), but I think I've managed to get it all working nicely now.

And finally we come to something I said I wasn't going to do. In my last post I explained that moving streams to different devices was something best left to the System Settings -> Multimedia dialog. Lots of people commented on that post saying that individual stream moving is still needed. Well, not wanting to disappoint the public, I have now added relevant patches to enable this in KMix. The UI is not ideal yet, but you can access it via the context menu of Playback and Capture streams by right clicking on the sliders and selecting an option from the "Move" submenu.

There were several other small cosmetic fixes (like using the right icons for devices/applications in the GUI) too and a really annoying race condition due to a singleton class taking too long to start up that got fixed too.

All in all I'm rather pleased with the outcome for now. It's not perfect by any means (if the PA server dies there is not much feedback), but it's certainly a good start. Failing any massive problems with the build I've just submitted to Mandriva Cooker, I'll sync this to subversion trunk for everyone else to play with.

I had to make more changes to the core of kmix than I originally intended. I've tried to ensure I always remain backwards compatible but some subtle differences in profile loading certainly exists. I'm not sure if these are avoidable (I don't think so) so this may be a sticking point... time will tell.

Have fun

Quick screenie before a more detailed description:

What does this mean?

Well it means you will typically get three or four tabs in KMix that represent (in order), "Playback Devices", "Capture Devices", "Playback Stream" and "Capture Streams".

All the physical (and virtual) cards' output "sinks" will appear under "Playback Devices". I'm sure the names are pretty obvious so I wont explain further!

The functionality is close to pavucontrol, but with three important exceptions:

1. There is no equiv of the "Configuration" tab to change "Profile" for a given card.
2. There is no way to change "Ports" for a given Sink/Source (not all sound hardware supports this and it is intended to make this automatic in some cases - i.e. Headphone Jack Sensing - once it becomes reliable upstream in ALSA, but sometimes it will still be user choice (e.g. Amplified vs. Non-Amplified))
3. There is no way to move an given application from one output to another.

For 1 and 2 I intend to (eventually) provide a KCM to go alongside the Multimedia tab. For 3 you can move "categories" of applications to different devices via the Phonon settings in System Settings -> Multimedia already thanks to my previous patches.

Where can I get it?

It's already in Mandriva Cooker or you can grab it from my Git repo here: http://colin.guthr.ie/git/kdemultimedia/log/?h=pulse

The master branch is upstream svn. The pulse branch is my changes. Do a clone then issue:

 git diff master..pulse >mypatch.patch

or just build it directly.

Hopefully it'll be committed to trunk soon and if all goes well it can be easily backported to Mandriva 2010 too.

Caveats

Every time a new slider appears it is added to the "Shortcuts" system. This is OK for hardware devices, but for every application it can get a bit much.... I'll try and find a way to disable this (see "What's Left" below).

PulseAudio supports a pretty crazy limit on the number of channels a device or stream can have. Kmix only has support for a fairly standard set of elements. In come cases not all channels of a given device/stream may be shown in kmix due to this limitation. Stereo -> 5.1 setups should work fine tho'.

You Suck, I use PA but I want a Real Man's ALSA mixer!

Whatever floats your boat baby!

 $  KMIX_PULSEAUDIO_DISABLE=1 kmix

Knock yourself out!

What's Left?

I have two remaining issues that I do not think are show stoppers:

1. Everytime a new device shows up a new Global Shortcut dialog appears. I don't think this is any different to ALSA but as I now have per-app volume control, this dialog is also shown everytime a new application plays sound. It only happens the first time a given application plays a sound, but it could still be considered annoying by some.
2. If there are no capture streams running at startup the tab for that will never be displayed - likewise if the stream restore module in PA is not loaded (unlikely) the "Playback Streams" tab will never be displayed.
3. There are various things that could be more efficient (e.g. refreshing the GUI for a new device or application current redraws all tabs, not just the one that has changed when a new application appears or disappears)
4. Make KMix dumber! KMix is pretty clever and it tries to do some smart things like saving and restoring volume for you. But when PA is used, it knows better, so I need to ensure that KMix does not do any saving/restoring of actual volume values.
5. Use application icons for per-application streams. Just for eye candy, it would be nice to use the applications own icon in the GUI.

This article follows on from my previous articles here and here. If you've not read them, then it would make sense to give them a quick glance (also as this was composed on my phone while on a flight, please forgive me any silly spelling errors I didn't spot when reviewing it!!)

First of all it's probably worth explaining what happens and what the phonon backend itself needs to be aware of and how it can interoperate with pulse.

So when libphonon is used in an application, a connection to PulseAudio will be attempted. If that connection fails everything will work as currently and nothing much changes. If the connection succeeds, we establish if "module-device-manager" is loaded. This is a module specifically written to implement the routing policy (an ordered priority list of devices for each category of sound produced - e.g. Notifications, Music or Video). If this module is not detected we offer a reduced/cut-down PulseAudio integration where the user will only see a single, virtual "PulseAudio" sound "device" listed in the KDE configuration system's multimedia section. If m-d-m is detected however, users will get the full rich experience. All of the devices ever seen by PulseAudio (I.e. Built in sound cards, PCI devices, USB speakers and headsets, Bluetooth devices and Apple Airtunes devices etc.) will be listed. If the device is not currently available, it will be shown greyed out as expected and you can of course reorder them to suit your own preferences.

So all that is well and good from a user perspective but "how does it work internally?" I hear you scream!

The Nitty Gritty

When m-d-m is used the device preference order is not actually stored in phonon itself as is the case when PulseAudio is not used, it is instead passed across to PulseAudio which takes over responsibility for keeping this priority list up to date. Why do it this way? Well while the preferences stored in KDE work fine for phonon apps they do not cover any non-phonon ones (which still make up the vast majority of sound producing applications on Linux). As someone involved with sound on Linux I've often been asked by confused users why their preferences are not honoured in e.g. Audacious or other non-phonon apps. As PulseAudio is integrated lower down the sound stack than phonon, ensuring it knows your device order preferences also ensures that it can route your audio appropriately according to your preferences for all apps! A clear bonus

Now as you may know phonon has it's own API calls to move streams to specific devices. This is something also handled in PulseAudio so it makes sense to simply wrap up the phonon API around these PA call which is exactly what I do! However, there is a complication. As phonon has pluggable backends, we don't actually speak to pulse directly to do the sound output (our connection to the PulseAudio daemon is only for querying and control purposes) so we don't know which stream is ours when the audio is actually playing. To get around this obscurity introduced by the pluggable backends, we take advantage of a facility in PulseAudio to attach arbitrary metadata to our streams by way of setting special environment variables. When phonon is instructed to start outputting audio, we set both the PulseAudio "media.role" property (this is the equivalent of the Phonon Categories system and we happily have an easy 1:1 mapping) and we generate a GUID for our stream and assign it as the "phonon.streamid" property. Our control connection sees all audio streams played in PulseAudio and can use this streamid to map back the PulseAudio streams to the phonon streams. This then allows us to process the stream move requests! It's a bit of a round about road to get this capability but it is the only way we can skirt around the obscurity provided by a pluggable backend system in a general way (the opaque interfaces provided by e.g. Xine, GStreamer and other backends would not expose this info directly anyway).

"But wait!" I hear those of you paying close attention scream. "Didn't you say above that PulseAudio will do the routing for us according to our preferences? Why not let it do that for phonon streams as well as non-phonon streams and forget about the whole GUID thing?" Well, if truth be told, I did start off doing it this way but it had two major issues. The first was the "Test" button not working properly - it always resulted in the highest priority device being used, no the one the user had selected! Not really ideal and more than a little confusing I'm sure you'll agree! The second problem related to visual feedback. When you reorder your devices or plug in a device of higher priority when sound is currently playing, phonon will visually inform the user of the device change. Handing over wholesale to PulseAudio meant that this feedback was lost. Again this is certainly not what I'd call seamless integration!

So at present the stream will be routed to the correct device by PulseAudio and then Phonon will also issue a move command which should in most cases be a NOOP. Likewise when devices are hotplugged (or unplugged!) PulseAudio will route them correctly and phonon will follow this up with a specific move request which will again end up being a NOOP. While this may seem pointless it does allow for the phonon API to be use properly (as needed by the test button) and give visual feedback)

Backends

So what does it mean for a backend? Well the backend needs to be somewhat aware of PulseAudio. Thankfully this is pretty trivial thanks to the PulseSupport class I've written. The backend should ask PulseSupport if pulse is detected if so it should check for and enable it's own PulseAudio output system. If this cannot be fond or activated it can tell PulseSupport that it's a no go and everything should fall back to how things work without any PulseAudio support. There are a couple other bits and bobs that a backend needs to do to work fully but that's the bulk of it.

So I hope I've provided a good and in-depth overview of how this support works. Please keep any comments on topic. If you want to post comments about PA in general, please do so on (and after reading!) this article.

This support is included in the upcoming Mandriva 2010.0 Release which will be hitting the streets very soon.

I only saw one 64 bit package so I figured I'd convert it. It's for Ubuntu, but alien -r skype.deb soon converts that to an RPM. Sadly it turns out that this is not a 64 bit app at all, it's just 32 bit but presumably it's got the necessary dependancies cooked into the package. So overall, the Fedora package will probably run fine on Mandriva.

Once it was installed I fired it up and under sound preferences it just says "PulseAudio" all the way Happy Days.

I found that Skype seemed to try and change volumes a bit and that seemed to trigger an assert in pulse so I disabled the relevant setting in Skype for now and things seem a bit more stable since then.

Now, PulseAudio has some nice features, like corking (pausing if the app supports it) music playback when a call comes in. In order to do this, PulseAudio has to know that Skype is a telephony application. In theory this is just a matter of adding "Telephony" to the Categories in the skype.desktop file. Sadly this didn't seem to work for me... not sure why, but I didn't debug for too long. Instead I added a

X-PulseAudio-Properties=media.role=phone

That did the trick. Pulse now knows that Skype is a VoIP app

One other neat feature (in addition to the music pausing) is that when I connected my Bluetooth headset half way through a call, PulseAudio moved my call across to the headset automatically! Nice.

So the conclusion? Well, there does still seem to be a few issues with the volume setting but overall, this is a great step forward!

So what next? Well work out why the asserts happen on volume changes and then work out why the Telephony category is not parsed from skype.desktop.

ALSA vs OSS

A lot of the comments of late have been discussing things such as how amazingly brilliant OSS is. Personally I don't buy it. I've never really played overly much with OSS and as such this is probably a slightly ill-informed view - although that's not to say it's not accurate of course . Most of these kind of comments are made by people who don't really understand ALSA and are bought over by the "ALSA API is overly complex" type comments. Yes, the ALSA client library is rather complex and has numerous pitfalls - so much so that there exists now an unofficial "safe" ALSA Subset API. But what people invariably fail to comment on (and thus fully understand) is that ALSA comes in two parts: the kernel driver and the userspace library. ALSA differs from OSS in that all access to the kernel layer is performed via a userspace library. I don't know of any ALSA clients that communicate directly with the kernel layer without going through libasound. What this means is that the kernel interface has the freedom to be re-factored and improved at any time, provided the userspace library is developed in parallel. For this reason, the kernel layer is actually quite clean and well defined. The rather rigorous quality control that goes on in the kernel is testament to the fact that on the kernel side of things, ALSA is doing pretty well. Of course there can (and will be) improvements in this area in the future, but this side of things is certainly not in a poor state as people seem to assume.

The "too complex" argument relates to the ALSA userspace API. In order to remain backwards compatible, the userspace API has undergone several refinements. As will anything not designed from the top down, some parts of it are rather confusing and have sometimes been misinterpreted (the classic example here being the confusion over snd_pcm_delay() - it's documentation hinting at a hardware based implementation that subsequently lead to some project (i.e. WINE) assuming that this function will eventually return 0 which is not true; fortunately this problem is behind us now, but with a new API call added that does return the info the WINE guys (and others) needed).

So yes, the ALSA userspace API could use a complete top-down redesign, but in order to do that, we would immediately break compatibility with 90% of the apps out there: Not a great idea all in all. Retaining backwards compatibility is a pain, but it's also quite important!

But Sound Servers Suck!

What is in a name? That which we call a rose by any other name would smell as sweet. Some people seem to have some sort of built in hatred of "sound servers" as a concept without really thinking through what this means. Yes, there have been some pretty awful experiences with some sound servers in the past (EsounD and aRTs being the immediate examples that spring to mind), but that doesn't mean the concept itself is flawed. You may drive a couple of shit cars but doesn't mean we should all abandon the roads. In addition, the sound servers of old were really just mixers. In the old days most hardware was not capable of doing hardware mixing and thus couldn't produce sound from multiple apps at the same time, so a mixer was an essential component. Nowadays, software mixing is the norm rather than the exception, even on high end hardware, and ALSA itself has a pretty solid sofware mixing in the form of DMIX, thus obsoleting large parts of the previous sound server functionality - certainly making the additional features they did offer seem disproportionate to the hassle they introduced. In the early days DMIX was just another sound server. Apparently this has changed these days, no longer needing an additional process. While it achieves the job of software mixing very well, it's not as fast or as flexible as other solutions can offer.

Modern Multi-user Desktop

So, these days a modern, multi-user desktop is quite a different beast to what it once was. Components such as Console Kit track which users are currently active (e.g. when more than one user is logged in simultaneously) and tells udev to write appropriate ACLs to enforce this policy. Users also want to use network attached sound systems, such as Apple Airtunes (RAOP) devices and UPnP media renderers etc. not to mention Bluetooth devices. All of this is much further up the sound stack than the low level driver level and has to deal with various permission and authentication schemes. This obviously needs a userspace component to govern this interaction. Something has to be responsible for this and a "sound server" of some sort obviously fits the bill perfectly.

PulseAudio

So enter PulseAudio. It's had it's fair share of bad publicity, but ultimately this important part of the Linux sound stack is taking on several roles that are important in a modern desktop. It's dealing with several different things:

• Software mixing
• Independent (per-application) volume control
• Dealing with permissions (is the user allowed to access the sound device?)
• Dealing with Bluetooth devices
• Dealing with Network based devices (UPnP, Apple Airtunes, Native PulseAudio etc).
• Handling the moving of streams between outputs.
• Handling sound from remote applications run via X11 over a network.
• Dealing with routing policy (Music goes to USB speakers, Desktop sound events to built in speakers, VoIP to Bluetooth headset)
• Effects to promote HCI (e.g. positional event sounds  - button clicks etc, coming out louder on the left hand speaker when triggered from the left hand side of the desktop)
• Power Consumption and Efficient savings.
• Reduces risk of buffer under-runs.

So the people who talk about OSSv4 and how it can do mixing and per-app volume control and how this means that ALSA and PulseAudio are not needed are totally underestimating what's needed in a modern audio stack. There still needs to be some kind of userspace daemon to govern these other sound systems and deal with multiple users. This is a non-trivial job and no other system out there is currently aiming to implement these capabilities.

One of the often overlooked advantages of PulseAudio is the "glitch free" system. This is an approach that ultimately disabled interrupt driven audio and instead relies on system timers. Modern kernels can provide these timers easily and reducing the number of interrupts and using larger buffers allows you to greatly reduce the number of CPU wake-ups thus saving power. This is actually a very important technique to implement when dealing with modern mobile platforms.

Reuse

It's obviously important to ensure efficient code reuse. It doesn't make sense for all sound producing applications to implement direct support for "exotic" sound systems such as Bluetooth, UPnP and Apple Airtunes etc. To do so is very inefficient (there are some exceptions to this - e.g. a media player that targets Win/Lin/Mac will maybe need to implement direct support if it is to be available across the board). Keeping the implementation centralised and having a single app->sound system API is essential here.

Consistency of UI

One of my big problems with many applications is inconsistent UI. This is a problem on Windows as much as on Linux, but it's something OSX has done mostly right. Users got to a central GUI to configure their sound and which device is currently active/in use. In Linux land all sound producing apps have their own config GUI for selecting sound devices. This is insane. Non-technical users don't know that you have to go to Tool->Preferences->Advanced->Sound in App A and Edit->Settings->Audio in App B. Sure, those of us who are reasonably technical will generally find the options (that's how we use applications - we click and look at all the settings pretty early on!), it's going to be less than obvious for a massive number of users. Keeping the preferences centralised so the user always know where to look is important and for a general purpose application that outputs sound, there should be no reason to provide any config option relating to this to the user - it should "just work"(tm).

Incompatibility

Some users have complained that some proprietary applications have stopped working with PulseAudio, Skype being an oft mentioned example. Well, I'm sorry but that's just tough. If a closed source application does not implement an API cleanly and does bizarre things, there is nothing we can do to fix it. The problems Skype has experienced with PulseAudio would also be experienced by any other plugin to ALSA. I'm sorry to say it, but in order to move forward, some applications have to suffer and/or be forced into action. By not allowing the people who care about this stuff the right to improve things themselves you're taking on the responsibility to do this yourself and you need to live up to your responsibilities. Considering the last version of Skype for Linux was released more than one and a half years ago, it's hard to consider it as anything more than abandon-ware at present. Will there be more pain like this? Yes, probably but that's just way things are - Free Software only truly works if the whole bundle is Free, if you mix and match you, as a user, have to accept this state of affairs. I do.

Desktop environments need to ensure they integrate nicely with PulseAudio. GNOME is obviously doing this, but KDE is lagging behind. I do hope to rectify the latter situation personally, and have a pretty clear roadmap to making this happen - it's just a matter of finding the time to do it!

Conclusion

So, with all this in mind, the sound stack has to be more than just a driver layer. It needs a persistent userspace layer that can run and keep track of various permission problems, deal with network connections and generally govern things. At present PulseAudio is fitting the bill pretty nicely and is continuing to add support for additional constructs in the Linux stack. As things stand all the major Linux distributions are now using PulseAudio with commercial interest from Nokia, Intel and Palm among others.

Future

So the future? Well, the drivers in ALSA need to be further debugged and developed to ensure the accuracy of the timing information that has so far plagued the "glitch free" system in PulseAudio. Nothing has pushed the ALSA drivers to such limits before, but the benefits of the glitch free mode are clearly worth the pain. Applications using the ALSA API need to ensure that they are using it correctly and sticking to the safe subset whenever possible (thus ensuring compatibility with PulseAudio's ALSA plugin). In addition, applications such as media players need to deal properly with latencies. It's a bit of a myth that low latencies are needed by such applications - higher latencies will ensure better battery life on mobile players and depending how the user wants to route their sound (e.g. to the Bluetooth enabled hi-fi system) latencies will be something beyond the control of the application in any event. It's therefore important to deal with this correctly and appropriately to ensure A/V sync. It's only been about half a year that the ALSA level limitations on buffer sizes were lifted after lobbying from the PulseAudio maintainer. Intel are even experimenting with 10 second buffers (that's not the same as latency!) in order to save power!

Every day more and more applications are tightening up their ALSA implementations. Every day the constructs of the Linux desktop are becoming more stable and solidified, offering a truly joined up multi-user and network aware experience. I think this is particularly impressive considering the fact that (as far as I know) only three people are employed to look after the Linux sound stack: Takashi Iwai andJaroslav Kysela on the ALSA side and Lennart Poettering on the PulseAudio side. While there are numerous other contributors, this is still pretty impressive progress with the resources at hand. It's also worth noting that two of the three are employed by RedHat, the other by Novell.

While Mandriva will still provide an easy way to disable PulseAudio if you feel it's not right for you (just untick the box - it's not hard!!) or need to use these closed applications such as Skype, I believe that this will not be necessary in the not too distant future.

So where is Sound on Linux? In my opinion it's in a pretty good state - there are still lots of things to do, and that will never change, but there is a firm and solid framework out there now and it's getting better every day.

ALSA Application

So first off, an application using the ALSA API tries to open the "default" device. Assuming we've configured this default device to be the PulseAudio plugin for ALSA, it will basically act like every PulseAudio client application. We're now into the land of the PulseAudio configuration.

PulseAudio Client

PulseAudio adopts a client/server model that is very similar in principle to that of the X11 system. It is the server that actually outputs the audio and the client app that tells the server what to play. While this approach can be inefficient, resulting in the copying of audio data around, PulseAudio goes to great lengths to ensure that data copying and other latency-prone operations are kept to a minimum. In the common use case of both client and server running on the same machine, PulseAudio uses SHM (Shared Memory) to ensure that data sent from the client to server is not copied across the wire. The core of the PulseAudio server itself is "zero copy" meaning that references to the data are passed around without actually copying the data itself.

The first thing a PulseAudio client has to do is connect to a server. In order to do this, it checks various variables and configuration files to determine precisely to which server it will connect!

Initially, the PulseAudio client library looks for a PULSE_SERVER environment variable. If found, this variable can define a list of servers to which the client should connect. These servers can be specified as local UNIX sockets or DNS names/IP addresses for a TCP connection.

If this is variable does not exist or is empty, PulseAudio will then check for X11 properties on the root window. These properties are much like environment variables, but will be available remotely if you SSH to another machine with X11 forwarding. I'll speak about this more later. You can see a list of PulseAudio related properties by doing:

xprop -root | grep PULSE

The variables names used are the same as those used in the environment, so PulseAudio will look for a property called PULSE_SERVER.

Assuming it's still not got a server to connect to yet, PulseAudio will check for a default-server configuration in it's client.conf file. This file is located in either /etc/pulse/ or ~/.pulse/. Only one of the client.conf files is parsed. So if the user has their own one, the system one will not be parsed at all (this is a point I tried to make in vein on PulseAudio bug #606 - it took quite a while for this to sink in as you can see!).

So we've tried three ways to find a server. If we've still not found one, we just resort to defaults - i.e. connecting to a local, personal daemon and a local system-wide daemon (system-wide use is generally not recommended, but is supported for certainly circumstances - typically embedded systems). If we still cannot connect, the client.conf file can specify whether or not we will try to automatically start a personal daemon. Since PulseAudio 0.9.11, this is the default behaviour and allows console applications to work out of the box without starting a PulseAudio deamon beforehand.

So, in the unlikely event that all that fails, we will ultimately not be able to play sound, but we've done pretty much everything we can to make it work! In order to better visualise this, let's look at a couple of typical scenarios and run through the above process.

Console Application

So, under a default install, we've booted to runlevel 3 and logged into the terminal. We don't set any special variables and start an application that plays sound via ALSA. Here is what happens.

1. App opens default device
2. ALSA PulseAudio plugin (like any PulseAudio client) checks it's config for a server and finds none.
3. It tries to connect to a local server but fail as it is not running.
4. It then starts a PulseAudio server automatically and then connects to it.
5. The application then plays audio via the ALSA API functions and this is ultimately played by the PulseAudio daemon.
6. The client application finishes doing it's thing, and exits.
7. The PulseAudio daemon stays around for a while just incase another app wants to play sound in the near future.
8. After a while, the PulseAudio daemon will go in the huff because no one loves it and kill itself :p

So that's it. It's quite simple. Let's have another example.

X11 Application

Under X11 things are a little bit different, but the same basic principles are followed.

1. During X11 initialisation, modern desktops that support XDG Autostart ultimately run the script start-pulseaudio-x11. This script ensures the PulseAudio daemon is started and some extra X11 related modules loaded into it. These modules ensure that, unlike a console application, the X11 PulseAudio daemon will not exit after an idle timeout - it will instead stick around for as long as the X11 session exists. It als ensures that the X11 Properties mentioned earlier are set - the reasons for will will become apparent in the next example.
2. When any PulseAudio client (be it an ALSA app via the ALSA PulseAudio plugin, or a native PulseAudio) is started it goes through it's "find a server routine". It will now stop this process when it reaches the X11 properties and use the info therein and connect to the server.
3. The client application then plays sound as before and ultimately finishes and extis.
4. The PulseAudio daemon doen't exit/kill itself as the X11 session is still going.

So again, things are actually quite simple.

Remote X11 Application

One of the handiest things with X11 is the ability to connect to another machine on your network and run GUI applications and have them display on your local display. With PulseAudio, the sound is also heard on the local machine. Out of the box (for security reasons), remote connections are not enabled. To enable them, run paprefs and enable the option Enable network access to local sound devices. This is the only option needed for this example. It loads an additional module into the server that listens on TCP port 4713 for incoming connections. Obviously, it goes without saying that any firewall on the machine must allow connections to this port!

1. User starts a normal X11 session and the start-pulseaudio-x11 script ensures the X11 properties are set as in the previous example.
2. User then connects to another machine on his network via SSH and starts an audio player (e.g. RhythmBox, Amarok, etc.)
3. Even tho' the app is running remotely the visual display will be local.
4. When the app starts playing audio, the PulseAudio client portion will find the X11 properties that have been forwarded through the SSH connection.
5. The PulseAudio client will then connect over TCP to the PulseAudio server running on the user's local machine.
6. The User revels in the local display and audio the application provides.

So as you can see, the use of the X11 properties has allowed us to piggy back on top of the X11 forwarding. It's not a totally clean connection as under SSH the X11 data will actually be tunnelled over a secure link handled by SSH itself, whereas all we are doing is telling the PulseAudio client where to connect directly, outside of any SSH tunnels. This means that while the display can work over a NATed system, the sound will not. This is fairly easily addressed, but we'd have to teach SSH about PulseAudio for this to work. The reason it works for X11 is because SSH is aware of, and has specific support for, X11. We're just piggy backing on this. That said, the current arrangement is "good enough" for most use cases.

So I hope this article has demystified how the PulseAudio client and server interact and the various configuration files/variables that come into play. If you have any questions, please ask in the comments and I'll endeavour to update the article.

First of all we should probably explain a little bit about how this works. ALSA is the Advanced Linux Sound Architecture. It is a replacement for OSS (Open Sound System) that had several problem on Liunx and which has ultimately been supplanted by ALSA as the only sound system in the Kernel. Despite a relaxation of license terms and further development on OSSv4, OSS is unlikely to replace ALSA in the mainline kernel.

ALSA also has a userspace component, libasound, that acts as a primary interface to the driver layer. Most complaints about the complexity of the ALSA API actually relate to this userspace component, not the kernel layer which, as you'd expect, is much more rigorously controlled. It does not suffer from the same need to remain backwards compatible with various userspace applications (the ALSA kernel drivers only need to work with libasound which can obviously be developed in parallel) thus leading to a cleaner design than the userspace layer itself which does have to remain backwards compatible.

In addition to interfacing with the kernel layer and interacting with sound hardware physically installed, ALSA also has a plugin architecture. This plugin system allows for devices to be faked/emulated, in various interesting ways. It allows, for example to create a null device that sends all audio to /dev/null, it allows for bluetooth headsets to be used (note this is considered a legacy way these days), and it allows all audio to be routed through PulseAudio which I'll discuss further in a subsequent article.

So, let's talk about ALSA configuration files. Now most of the ALSA config files live in:

/usr/share/alsa/

Arguably these are not really "config" files in the classic sense, i.e. you are not meant to change them as a user according to your own preferences and whims - they are really more like source files and define the structure of various ALSA plugins and multichannel configurations. Unless you are developing/hacking on ALSA, you probably shouldn't tweak the vast majority of the files in this folder.

The main file

/usr/share/alsa/alsa.conf

defines a list of additional files to parse and the order in which to parse them. In order to incorporate PulseAudio in an elegant and configurable way, we make a couple changes to the list of additional files:

--- alsa-lib-1.0.15rc3.lennart/src/conf/alsa.conf	2007-10-17 18:28:03.000000000 -0400
+++ alsa-lib-1.0.15rc3/src/conf/alsa.conf	2007-10-17 18:33:10.000000000 -0400
@@ -8,6 +8,8 @@
 	{
 		func load
 		files [
+			"/usr/share/alsa/pcm/pulseaudio.conf"
+			"/etc/alsa/pulse-default.conf"
 			"/etc/asound.conf"
 			"~/.asoundrc"
 		]

The first file allows us to define a named "device" for ALSA called "pulse". This is always present even if the user ultimately decides not to use PulseAudio by default on their machine. This would allow them to e.g. define a remote PulseAudio server (via Environment variables or client.conf setup - see the next article in this series, linked below) and tell ALSA apps to use this "device" specifically. Arguably this is a corner case, but there is no reason not to support this all the same

The second file allows us to turn on PulseAudio by default. This is a pretty important as offering users a simple way to enable/disable PulseAudio is hightly desirable. PulseAudio roll out is not been completely free from problems and giving users the ability to quickly and easily disable it is essential. I firmly believe that everyone will ultimately use PulseAudio, but it'll take time for every app to fully support it (by removing non-"safe" ALSA API usage) and for driver issues to be worked out properly.

In this second file, we either comment out the file completely to disable PulseAudio by default, or leave it uncommented to enable PulseAudio. This is handled by draksound so the users just see a simple GUI which is a simple but effective solution. That said, I'll probably change this for Mandriva 2010.0, switching to an "Alternatives" driven system although this doesn't really matter in the overall scheme of things!

So when an ALSA application startups up, it parses all these files and ultimately works out how to route your audio. 99% of the time, the "default" device is used (and I use "device" in the loosest possible sense). For a standard Mandriva install, the default device is actually PulseAudio (via the PulseAudio plugin for ALSA).

So what happens next? In the next article, I'll go on to talk about how an ALSA app (or any PulseAudio client) works when connecting to PulseAudio.

KDE has a fairly robust audio framework these days. It's still got several kinks and missing features, but it's a good basis for apps going forward. Sadly pulse has had a pretty rough ride here. There are two "backends" in common usage with Phonon on Linux (it has configurable backends originally intended to allow cross platform support - e.g. DirectX on windows and QT on Mac OS). Personally, I think that this is a bad idea - while I support diversity, having a single backend that is subsequently developed by everyone and made rock solid is IMO more important than diversity - certianly in the initial stages. There are also VLC and mplayer backends too... I think this just muddies the water, regardless of the quality of these specific parts.

QT originally developed a GStreamer based backend for use on Linux. This is, for me, the right approach as the GStreamer library seems ideally suited for this. Unfortantely the KDE community didn't really pick up on this and rather than develop and improve the GStreamer backend (which was pretty early stage and could definitely done with improvement), a Xine based backend was created instead. While this works fine for audio decoding, I don't think that this is the right approach longer term, but such is life.

The sad thing is that Pulse support in Xine is not that great right now and causes a lot of problems for people. Improving this support will go a long, long way to gaining better acceptance of PulseAudio by the KDE community.

Speaking of which, this is where the rest of my interest lies - improving KDE integration.

Phonon has a settings GUI that lists various output devices, including those not currently connected, and allows the user to rank the devices in order of preference. These "devices" also include output special output drivers such as the PulseAudio plugin. Now as pulse is a complete management system, it makes very little sense to include it like any other physical device and order them - when a user uses pulse on their system, they want to use it for everything (of course they may choose not to use pulse at all and that's fine too - even if I wouldn't recommend it!). In Mandriva we've long since patched the settings GUI to detect if the user wants to use pulse and simply hide all the other devices - the user is directed to use pavucontrol to move their streams around. This works OK, but it's clearly not the ultimate solution! Any self respecting KDE zealot would never run pavucontrol (it's a GTK app!) and while my own desktop is a horrible mix of just about every DE under the sun, I appreciate many people want to be "pure"!

So what do I want to see? Well, when it's in use, I'd like to off load pretty much all the handling of audio devices to pulse. It should take care of handling the device preferences and the automatic switching to higher priority devices when they become available and/or are reordered. The routing system in pulse is alreay setup to do a lot of funky things routing wise (like automatically moving music streams to RAOP devices, or VOIP streams to Bluetooth headsets), although a manual priority list is not currently supported. I am aiming to create such support for a priority list in pulse and expose it via a protocol extension. In the Gnome case, it's likely that this priority list either completely masked from the user and the existing routing stuff is used in it's stead, or that said existing routing stuff is evolved to just become a manager for managing the priority list internally. The final design decisions here are not finalised, but there will be some sort of working solution that will present itself!

In the mean time, I've started development on "module-device-manager" which acheives the rather simple goal of remembering all the past and present devices pulse has seen and allow querying of that list. As a neat little side effect, it can be used to edit the descriptions given to devices so for shits and giggles I implemented this renaming support in pavucontrol.

Here is a quick screenie: