As well as taking part in various discussions, I was able to spend some time cooking up a UI to control the configuration of PulseAudio and the various cards attached.When adding PulseAudio support to the various parts of KDE that need it (Phonon, KMix), there was an important part of the puzzle missing: a card profile selector, and a sink/source port selector. I had always intended to include this functionality somewhere, but the KMix framework didn't really allow for it neatly (I could have created a separate dialog of course but it didn't quite feel right).

The eagle eyed readers may have seen a sneak preview of this feature when looking at the KDE-specific help page I wrote up on the PulseAudio website. So I give you this: the Speaker Setup tab in System Settings.

Speaker Setup GUI

The little icon in the middle is your user icon, so you'll see this differently unless your a weird stalker type and have set your icon to mine :s

The various drop downs allow fully control over all cards that are attached to the system. The buttons on the main pane allow you to test each speaker separately. In order to test the speakers, libcanberra is used. libcanberra is an implementation of the Free Desktop Sound Theme specification. It allows this test to be implemented with minimum hassle and I'm not personally interested in reinventing the wheel, hence the use of this library and the additional dependency.  Some people dislike using libcanberra in KDE (as was apparent from some discussions at the Multimedia Sprint), but I believe the reasons were often personal ones and not related to the usefulness of the code. If someone really wants to factor this out, they can but I have no interest in doing so and will prefer to use the existing implementation whenever possible (and this would include any future implementation I may or may not do with regards to support the Sound Theme Specification in KDE (or maybe Qt) directly)

When a 5.1 surround system is presented, the GUI is obviously a bit more advanced:

Speaker Setup GUI (5.1)

For those naysayers, I've tried my best to ensure that the compile will work fine without PulseAudio installed. It should also degrade gracefully when it is compiled with PA support, but PA is not configured to be used at runtime. The whole tab will simply not be available.

There is still one small problem left in that I don't handle disconnection/reconnection yet. This causes the GUI to crash if the PA server is stopped and restarted. This is not typically something that happens, but it's still something I will fix shortly all the same.

This code is now in trunk (r1154776) so feel free to try it out and report other bugs etc. This GUI is also included in Mandriva Cooker (I did want to include it prior to 2010.1 release, but the timing didn't work out - tho' it probably would have been OK considering the delays that cropped up in the release process). I expect this functionality to be included in any updated/backported versions of KDE for 2010.1.

For reference, in case you didn't spot it yourself, this GUI was heavily inspired by the gnome-speaker-setup utility.

Sadly the GStreamer backend was plagued with some problems. I wont go into the details (primary because I don't really know them all!) but many KDE developers felt that it just wasn't appropriate and decided to write a Xine backend for Phonon instead (NOTE: This is not actually correct. Please read comment by Kevin Kofler below). This matured quickly, but it was always had it's own set of problems that resulted in many hacks to be introduced at the application level. This was very bad as it meant that those of us (Mandriva included) who stuck with and helped fix the GStreamer backend were sometimes left a little out in the cold due to these hacks. Ultimately this story has concluded with the Xine backend being more or less abandoned now, and renewed focus being put on the new VLC backend. VLC, or more accurately "VideoLAN", is a multimedia framework, very similar to GStreamer. It has a very well maintained code base and support for numerous codecs and formats. It is also totally cross platform, working with Windows, OSX and of course Linux and several embedded OSs to boot. Having a single Phonon backend with uniform capabilities and support across all platforms is very desirable, so progress with this is quite rapid and an official release of the Phonon-VLC backend will be available in a matter of weeks.

But this is all about Phonon, not Qt Multimedia or Qt Mobility. So where does this all fit in? Well, Phonon has ultimately not really been developed much by Qt. That's not to say they have totally forgotten about it. I am assured that bugs reported will be looked at, time permitting, and that patches from downstream will be tested and merged, but it's fairly clear that it will not be actively developed further and no new features will come from the Qt side. Their new babies are Qt Multimedia and Qt Mobility.

Qt Multimedia, and arguably to a larger extent, Qt Mobility has been developed with a strong focus on mobile usage. Now that Qt is owned by Nokia this is not surprising! So what was the motivation for this and how does it differ from Phonon? Well, like Phonon it also makes use of existing projects to provide the core media decode capabilities. It will continue to use GStreamer for most "linuxy" systems and others for e.g. Windows and OSX. It differs however with regards to focus. It encompasses several additional features regarding the various needs for a typical mobile environment, such as still image and video capture (think cameras on a mobile phone).

There are several other key differences too but I wouldn't like to go into detail just now (I'm not really familiar enough with either Qt MM or Qt Mob to comment in any great depth), but suffice to say that this general abandonment of innovation in Phonon in favour of a new project has left several people in the KDE multimedia community feeling rather uncertain about what to do next; which horse to back!

Here at the KDE Multimedia Sprint in Randa, Switzerland, we were lucky enough to discuss the various issues with the Qt Multimedia/Mobility guys in Brisbane, Australia via a video conference. Well it was supposed to be a video conference, but despite it being between two groups of multimedia geeks, video completely failed us (various firewall issues for one solution and the lack of admin rights at the remote end for another!), meant we had to make do with a normal phone call! Such is life! Anyway, they were kind enough to give us an overview of Qt Multimedia/Mobility and let us know their approximate roadmap.

The outcome of this meeting was more or less that KDE Multimedia focus for the short term will remain with Phonon but the Qt guys will try and be more open about their plans and try and work better with the community longer term. At present, this is an uphill struggle as the VLC project has been very active in the KDE community for the last little while and seems to fit the needs in both the short and relatively long term of KDE, albeit in a way that is not currently exposed via Phonon API. This previous lack of communication is something that will be hard to overcome, but using a Qt integrated and supported solution definitely has advantages in the future - if they really mean it this time! The fact that the code is also used on Nokia mobile platforms give a fairly good reassurance that this will indeed be the case at least!

The rather obvious question of "Why not extend Phonon?" was also raised. While it was difficult to hear exactly the responses, the general reasoning seemed to be that there were some design mistakes in Phonon that basically meant that each individual feature that needed to be added, actually needed to be done at the backend level first (for each and every backend) and then exposed out through the API. With the Qt Mobility approach, more of the core features and functionality can be implemented once, with it only reaching out the the underlying platform specific system for specific and defined operations.

We enquired about the possibility of having a single Qt Multimedia backend written for Phonon, that allowed the Phonon API to continue in use at the application level while using the underlying systems/framework provided by Qt Mobility and/or Multimedia and deprecating all other Phonon backends. This is attractive in the sense that applications to not need to change horses mid stream and can gradually move over to using a pure Qt Multimedia API if indeed that is determined to be the most desirable outcome. That said it is also unattractive in some ways too. This adds yet another layer of abstraction to a system that many people argue is a layer of abstraction too far in itself! i.e. Application uses Phonon which uses Qt Multimedia which uses GStreamer; Why not just Application uses GStreamer? (there are of course many reasons to add a layer of abstraction; cross platform support being a primary one; but two layers is arguably not the best situation, especially on mobile platforms where wasted CPU cycles really hurt). That said, it would still be an acceptable stepping stone for most people. Regardless, there was no actual commitment to this eventuality from the Qt side. It was seen as a good idea but it's likely not something that there will be time (aka budget) for developing in the near term, which is understandable.

Many of the problems with Phonon still remain however. Qt Mobility will still require the setup of different underlying support libraries on different platforms. Just as before, GStreamer will be required for it to work on Linux and downstream (distributions and, to a lesser extent, users) will need to ensure that this is all configured and working correctly. As with the original Phonon, this backend will be different on different platforms, which is not ideal for the downstream.

So in the short to medium term (e.g. the next one to two years at least), Phonon will continue to be the primary media framework on KDE, and development on the Phonon-VLC backend will be seen as the best way forward to provide a standard experience across all platforms. GStreamer as a phonon backend will also continue, although the mplayer and xine backends should be considered deprecated.

With regards to Qt Multimedia and Mobility development itself, the Qt guys will try to be more open; opening a private mailing list to more access and generally being more communicative.

Those of you who regularly read this blog will be no doubt wondering where the PulseAudio connection is. Well, from what I gather, this is a primary difference of Qt Mobility and Qt Multimedia. Qt Mutlimedia seems to use ALSA directly and thus is not ideally suited for mobile situations (PulseAudio's timer based scheduling now being pretty much universally accepted as being the preferred approach on mobile platforms). Qt Mobility does actually use PulseAudio indirectly via GStreamer, but it does not seem to do anything special with regards to the "buffer-time" or "latency-time" attributes of the pulsesink (not that pulsesink is actually referred to directly in the appropriate place anyway from what I can tell). These attributes map directly to the tlength and minreq attributes of the buffer metrics in a PulseAudio stream. These are very important when it comes to mobile environments as the general aim is to provide as high a latency as possible. Higher latency means lower power usage (and for those wondering, this does not necessarily affect A/V sync on video playback - it's just about how much data you pump into the audio buffers before sleeping until you know it'll be almost empty). For a system that deals with audio playback on a mobile device, this is very important.

Now Phonon is also similarly ignorant to these properties when using PulseAudio. The the help of folk from Intel we're going to look at increasing the defaults in Phonon when used with GStreamer pulsesink to see if higher latencies would bring benefits without any drawbacks at the application level (I don't think any of the uses of Phonon will have problems with this approach), but we'll also have to consider how best to expose this via the Phonon API to allow the application level to state more explicitly if this matters. When capture APIs become possible on Phonon this will become important. The whole question of how much thought has gone into this kind of power saving is Qt Mobility is still very much at the forefront of my mind. I hope to be able to discuss things in more depth with the developers in due course, hopefully influencing them to extend the API in a similar was as described above.

More generally, those of us in the KDE community will try to get involved with Qt Multimedia/Mobility but until such times as it is easily configurable on the Desktop or there is a Phonon backend based on it, it will be hard to get involved too deeply.

Overall I think everyone has an open mind, but with the current focus on VLC and the functionality it provides, it will be the most interesting bits for most KDE Multimedia guys for the short term.

With regards to all of the above, I hope I've made fair and levelled comments. If you feel this is not the case, please tell me so in the comments below. I will happily retract or rephrase or reconsider any point if a suitable argument is made. I do not intend to represent anyone else in the KDE community in a way they do not feel is accurate so I'll be happy to both comment and edit the article with suitable annotation if this is deemed necessary.

Edits:

1. 27th May: Clarification of my incorrect statement of when/why Xine backend was introduced. Thanks to those who pointed it out.

Firstly, it was pointed out to me that the use of a QEventLoop was not the best approach when connecting to PulseAudio from a KDE app. I used this technique in Phonon and while it seemed to work fine in testing it did trigger some problems in some unlikely places: namely this bug as reported against Okular! So I changed how the connection test to PulseAudio is run to make it blocking. This has so far appeared to fix the problems \o/

Now on to something interesting I've recently added support in Phonon for "hijacking" the volume change requests to allow them to be pushed directly to PulseAudio, totally avoiding the backend itself. In additional it is possible to listen to changes from PulseAudio and then emit the relevant Phonon signal to let applications know the volume has changed. This allows volumes changed in a PulseAudio mixer (pavucontrol and now KMix too!) to be reflected in the Application GUI (should it present it to the user).

Phonon provides it's own VolumeSlider widget that applications can use. I had to make some changes to this to avoid some complicated race conditions. As volume changes in PulseAudio are asynchronous, the change request may be followed some time later by the actual update notification. For this reason the it was not possible to use the "traditional" blockSignals(true) approach. I settled on two flags to achieve the necessary protection and it seems to be working well so far in my testing.

Then I tested Amarok and found it didn't work at all This is because Amarok has it's own volume slider widget and does not listen for any volume changes from the Phonon layer. I was able to make the necessary changes to support this without too much trouble, but in doing so I had to change the mute behaviour slightly. Due to (I'm told) legacy reasons, when the volume is muted, Amarok would also set the volume to 0. This was reflected in the GUI by the slider showing 0. While it would be possible to keep this behaviour it's not really desirable these days (the OSD always said "Volume: 0% (muted)" when muted which made it rather redundant to show the %age!) when mute is supported properly. If GUI feedback of mute status on the slider itself is desired (the image does change to a mute symbol already) then it would likely be more appropriate to change the slider colour to grey or similar (which is also how other media players - e.g. VLC represent things).

Anyway, with these issues fixed and suitable support for the feedback race conditions added to the Amarok slider code too, I've now got a rather nicely integrated solution

Here is a quick screencast I recorded to show you how all these changes: Phonon, KMix & Amarok all work together

(direct link)

As always, these latest changes are in Mandriva Cooker. The Amarok changes are not in the 2010.1 Beta 1 just recently released, but they will be part of Beta 2

So how does PulseAudio route your audio and what the hell do you mean by "audio routing" anyway? Well, PA handles multiple audio devices ("sinks" for output, "sources" for capture). When an application wants to play sound it says "Hey PulseAudio, play this!!", and PA tries it's best to comply. The application will typically not care about which device to actually output too - they delegate this job to PA (and ultimately to the user via PA's tools such as pavucontrol, gnome-volume-control and (more recently) kmix/phonon too). These tools allow you to move the stream to your preferred sink - no need for an unfamiliar GUI inside a particular application for choosing which audio device to use; users know where to look in their desktop environment to control sound settings. From an HCI/usability perspective I think this is very important (although incumbent users need to shake of their natural assumption that apps should provide a way to configure audio device settings).

With a fresh user account PA will attempt to calculate a suitable default device to use. It does this by assigning each sink an internal score of appropriateness. This is just to determine our initial defaults so it doesn't matter too much if we get this wrong, but obviously it's nice to try and get things right first time! So once our default device is chosen, when a stream is played, it will use this sink. Simple huh? Well not quite. What if I change the default while a stream is playing? My stream is moved across to the new default right? No. Setting the default device does not do this. It acts as a fallback, it's not an "active" default. If I stop my app and play it again, it will be played on the new fallback, but it wont move when the stream is "live".

Now this is just the very beginning so if you've become overexcited by now, best take a cold shower (and probably look up psychiatrists in the phone book...) as I'm about to move on to look at the "stream restore" database...

Stream Restoration

So the stream restore database is handled by module-stream-restore (m-s-r). It's part of the default PA install so 99.9% of users are likely using it. What this module does is to track when a user specifically moves a stream to a new device. When the user uses e.g. pavucontrol or kmix to move a particular stream to a new sink, this triggers a mechanism inside m-s-r that causes the sink name to be saved for that application. If that application appears at any point in the future and tries to play sound, m-s-r will try it's best to ensure that this sink is used. If the device is not currently available, we will ultimately use the fallback/default instead. If this saved sink becomes available at some point during the lifetime of the stream, the stream will be moved across automatically (which differs from the behaviour when setting the default sink).

Now it gets complicated. Despite my terminology above, m-s-r does not use "applications" per-se when it saves the device choice. It will look at a stream and select a "stream restore id" for it based on several bits of metadata attached to the stream. Firstly it will look to see if the stream has a role. If it does, the role is used as the identifying factor, not the individual application. So e.g. for event sounds, the "stream restore id" will actually be "sink-input-by-media-role:event" regardless of which application actually produced it. As m-s-r is responsible for storing volume, mute and device preferences, this means that all event sounds will have the same volume and will be played on the same device. If the stream does not have a role, PA checks for a few other things to try and create it's "stream restore id": it checks for an "application id" (a specific property set AFAIK only by a select few applications), an "application name" (which 99% of PA clients provide) and finally it checks for a media name property (which is pretty unlikely IMO, although I may have missed a use case here). Once the "stream restore id" is chosen it is saved for that stream and we will always use this rule when updating our saved volume, mute and device settings thereafter.

When a user moves the stream (using one of the afore mentioned GUI apps - or via any app that makes use of the pa_context_move_sink_input_by_*()  APIs (also applies to source_outputs)), the "stream restore rule" they are ultimately changing will depend on the metadata the application has provided PA in the first place. So for example, Amarok will be tagged with a "music" role, and thus when you use kmix to move the Amarok stream to a different device you are not just moving Amarok, you are actually moving all streams tagged as "music". (Update 1) But wait! It does not actually move the currently playing "music" streams. They will only be moved the next time a PA stream is created (this could be when the track changes, or it could be when the app restarts - it depends on implementation), or when the stream restore database is written by some PA client which requests the "apply_immediately" flag set.

This whole approach can be summed up with two flow diagrams:

Device Manager

OK, so that about covers m-s-r, now I'll talk a bit about module-device-manager (m-d-m). This is a module I developed principally for integration with KDE. The GUI exposed for the Phonon preferences in KDE is such that it lists all devices that have ever been available on the device even if they are not currently available. This allows them to be listed and arranged into a priority list. Whenever the highest priority device is available, it will be used. It's a relatively simple UI and one that users can easily understand and work with. For a degree of flexibility, the GUI allows for different priority lists to be configured for different Categories (which in PulseAudio terminology are called Roles). So m-d-m provides an API and Protocol extension to implement such a routing policy directly in PA which turned the KDE GUI effectively into a frontend to PulseAudio (internally this is abstracted via Phonon but this is not really important). So this m-d-m will route your audio for you to the appropriate device. Due to how it is implemented, m-s-r will still take priority, so if you move a particular stream and the device choice is saved by m-s-r, it will take priority over the role-based priority list of m-d-m. I have now provided a way in kmix to delete any application specific rules in m-s-r such that the m-d-m routing (and thus what is displayed in Phonon preferences) will be used again should an override have been in place.

Critique

Right so that's how things work. Now let's rip it apart and moan a little about the shitty bits.

• A default that is not a default: First things first. This is a question we get a lot: "I've set my default device but <insert victim application here> doesn't want to use it". Now that you know how the routing works due to the excellent overview above, you can probably work out what's going on here. Either m-s-r has a specific device saved for that application's role or the application itself, and that overrides the "default" choice; the user was playing the stream when they set the default device but this wont be honoured until a new stream is created; or m-d-m is implementing a role-based priority list preference (which in all cases overrides the default button due to having a priority list for all cases - one for each role and one for the default case). All of the above is flexible but it sure as hell is not clear to the user what the hell is going on. There needs to be some kind of unification and automatic reaction here. Having a simple "default" button IMO makes sense. It's a very easy concept to grasp for users and we should try to make it work in the most case and explain more clearly in the various GUIs as to why it may not work in certain circumstances.
• Why, Why, WHY?: Things are not clear to the user what is going on. In a typical setup we have m-s-r and it remembers one thing. Unless you know how it works, you cannot configure a scenario when you have three or more devices and have it "just work" for you. For me three devices is not uncommon. At work I have set of USB speakers I use and at home I have Airtunes and network-based PA servers I use too. I want them to "just work" without me having to fire up a GUI every time I want to use them. A primary problem just now is that unless you've taken the time to read and understand the above descriptions, it's very unclear to users as to how the default device works and what m-s-r does and how it operates. Clarity and transparency is needed here, or some way of making it "just work" without needing such and explanation.
• You gotta role with it?: Yes that is a rather contrived "problem", but the point is valid. As we go down the path of encouraging applications to include as much metadata as possible in their streams, assuming we ever reach such a zenith, then the ultimate end result is that all streams have their role specified. When this happens a fundamentally useful part of PulseAudios design is effectively crippled in the default setup: m-s-r will always pick it's stream restore id of "sink-input-by-role:fooo". All application's will no longer be able to chose their own individual audio device; you will only be able to pick the device for that whole class of application. All music players must go to one device. All video players to another, but nothing in between. Some applications set their role generically but can be used for other purposes e.g. totem/dragon are video players but may be being used for just music in a given instance - I'd like to use my preferred music device not my preferred video device but I don't want to affect all video players with this choice. Obviously it's better if a multi-purpose app can work out it's role dynamically based on content but the principle point still stands - you cannot move streams anymore (with the exception of the afore mentioned caveat for currently playing streams when the move is initiated), only whole classes of streams. Some may argue that this is OK/acceptable. Personally I don't think so.
• I don't want an override anymore: Say you've picked an override for your stream/role with m-s-r. Say you've now decided you don't want that override and just want to use the default device now? None of the GUI tools out there just now allow you to reset this (with the exception of the latest kmix in my Git Repository at the time of writing).

How do you solve a problem like Myrouting?

So the above problems basically make things a bit too much like a black box. It's not clear what's going on and it doesn't offer enough flexibility in many cases either. How do we solve this? I'll now outline what I think is the best possible implementation of a routing system and how it will work.

Overall Operation

Firstly we need to remove the internal concept of a single "default" device. As we've discussed before within the PA community, we will move to a priority list concept. A significant difference to the KDE approach of showing the device priority list for users to tweak, we will ensure that exposing this list is not a strict requirement for operation. In order to service this requirement we will still offer the existing UIs for setting the default device, but the internal behaviour will change. Setting a device as default would simply move it to the top of the priority list. This will ensure that simpler UIs can still operate and whenever the user prefers a device above other others they just set it as default. In order to get the perfect priority list, they may have to click the "default" button a couple of times with a given setup, but it will stabilise for them very quickly (for reference, this default priory list will be the equivalent of the current m-d-m priority list with the psuedo-role: "none").

As well as the default priority list, we will also implement similar lists for each role (again very similar to the current m-d-m role-priority lists). When a stream has a role, we will simply use this list rather than the overall default one. If a given role does not have a priority list of it's own, the default one will be used (in KDE the phonon GUI will likely enforce that a list exists for each role, but has GUI techniques to keep it in sync).

In addition to the above, each individual application will also have it's own priority list. Again this is fully optional. If such a list does not exist, the role-based list will be used instead, ultimately falling back to the default list if no role is present.

When a new device appears it will be added to the default priority list only. It is open for debate as to where this device should appear in this list - top or bottom, but this could ultimately be a user preference. The routing logic should ultimately check the most appropriate list first, try to find the highest priority device in the list that is currently available. If none are available it will use the next most appropriate list ultimately falling back to the default list which will always contain an available device.

New APIs

New APIs will be made available (either as extensions or baked in - details TBC). These APIs will allow the querying and editing of these priority lists (thus facilitating GUIs such as the one in KDE). When editing these lists the changes they reflect will always be immediately applied to any running streams. APIs will also be exposed to delete a  role-based list or an application specific list (NB the application list could potentially also be cleared via the existing APIs by passing PA_INVALID_INDEX or an empty or NULL device name).

Existing APIs

The existing APIs for setting the default device and moving a stream to a given device will still operate as just now but will operate differently internally.

As mentioned previously, setting a default device will update the overall default priority list. This change will be propagated to any currently running stream immediately. Thus a fresh user account with no priority lists saved, the default priority list will be used for all streams. Clicking on the default device will move any running streams to that new device. And thus the very basic use case will operate in an intuitive way.

If a user moves a specific stream to a new device via the (pa_context_move_sink_input_by_*()  style APIs) this will trigger the update of the application specific priority list. This is a stark change to the existing behaviour which will update the details for the currently used device choice mechanism. If the priority list for particular role is to be updated by a given GUI application, then the new APIs above should be used to achieve this  result. If a stream is moved and it does not yet have it's own priority list one will be created for it automatically (containing only the chosen device initially.

GUIs

So what can the GUIs show? Well there are a lot of options now:

• A very simple GUI which just shows a list of devices and allows a single default to be shown.
• A more complex GUI that shows all devices in the default list and allows the user to order them.
• One that shows both the default and the known roles and allows a default to be chosen for each (or expose the full list for user ordering)
• etc. etc.

(FWIW although it's possible I don't think it would be necessary to ever expose the per-application list although there should be a method for indicating that such a list exists and is in use and allow the user to delete it and user a higher level priority list).

Caveats

With regards to event sounds specifically, we want to make sure that these are handled differently to a regular stream produced by an application. For example I may want to use rhythm box to output sound to my Airtunes device and implement this via an application specific priority list. Any event sounds for this should not be pushed to this device. I'm not sure what the best solution to this conundrum is, but I suspect that a simple property set on the stream that says "do not allow per-application priority list updating" would be sufficient. This still allows streams to be moved to the device, but this wont be saved for later restoration (obviously moving an event sound is really hard due to their duration, so this logic is more for the principle rather than the practicality).

(Update 1) As I propose to apply changes whenever they happen if you run e.g. two paplay processes at the same time and change one to use a specific device, my logic dictates that both streams would be moved. This is a regression on the current behaviour where only when the stream is restarted will it be moved, but to be honest I don't think it's much of a problem. If it is desired to have specific control over a given instance of an application, then we need to either a public API to not save the move (much like our internal one), or support a proplist property that suppresses this (e.g. no API change but a loose "standard" for performing this type of operation. If you want to run an application in a specific way, then you can always set a property for the e.g. the "application id" as mentioned above which would make it appear as a unique application compared to running it normally. So flexibility still exists here (albeit rather complex), but we probably do want to deal with the default cases well and make the more niche ones possible via some degree of extra work.

Overall

OK, so this is quite an exhaustive proposal and I've not nailed down the exact best way to implement it, but I think that this approach is ultimately the most flexible possible that supports all of the interfaces desired and still exposes the flexibility of the PulseAudio system. I have not given much thought to the storage of volume/mute status (which m-s-r currently tracks along with it's single device preference).  My gut feeling here is a fairly similar approach where you should be allowed to set the default volume, a per-device volume and a per-application volume. The former two being used as defaults and as soon as the per-application volume is changed via any API call, we save it with that specific application for future reuse. APIs can set the default volume and the per-role volumes and this will be propagated to applications that do not have their own specific override in place.

So in summary, this is how I think things should work. I don't think it's actually as complicated to implement as I've made it sound (would probably only take about three or four times as long as it's taken me to write about it!!) and I believe it keeps the power and flexibility of the system, still allows for minimal control interfaces yet allowing more exposed and complex ones if required.

Updates:

[1] 16th Feb: Add notes about how moving a stream with a role does not move any currently live streams and how the proposed solution would prevent running two instances of the same app and moving the device independently for each (as one would follow the other) unless the application id was overridden for one of the instances.

I've spent quite a lot of time refining the initial support I added a few weeks back. What follows is a brief summary of the changes/improvements/bugfixes.

Firstly, some of the tabs (namely "Capture Streams" were not shown unless an application was actively capturing while kmix started... somewhat unlikely really :p This is now fixed.

Then there was a problem with Profiles. These would be created automatically when you went to the "Configure Channels..." dialog. There were actually several problems relating to profiles which ultimately lead to controls going AWOL - with no way in the GUI to see them even if the backend itself was actually aware of their presence. The concept of GUIProfiles does not really tie in too well with the concept of "Dynamic Mixers" (where controls come and go), but I think I've managed to get it all working nicely now.

And finally we come to something I said I wasn't going to do. In my last post I explained that moving streams to different devices was something best left to the System Settings -> Multimedia dialog. Lots of people commented on that post saying that individual stream moving is still needed. Well, not wanting to disappoint the public, I have now added relevant patches to enable this in KMix. The UI is not ideal yet, but you can access it via the context menu of Playback and Capture streams by right clicking on the sliders and selecting an option from the "Move" submenu.

There were several other small cosmetic fixes (like using the right icons for devices/applications in the GUI) too and a really annoying race condition due to a singleton class taking too long to start up that got fixed too.

All in all I'm rather pleased with the outcome for now. It's not perfect by any means (if the PA server dies there is not much feedback), but it's certainly a good start. Failing any massive problems with the build I've just submitted to Mandriva Cooker, I'll sync this to subversion trunk for everyone else to play with.

I had to make more changes to the core of kmix than I originally intended. I've tried to ensure I always remain backwards compatible but some subtle differences in profile loading certainly exists. I'm not sure if these are avoidable (I don't think so) so this may be a sticking point... time will tell.

Have fun

Quick screenie before a more detailed description:

What does this mean?

Well it means you will typically get three or four tabs in KMix that represent (in order), "Playback Devices", "Capture Devices", "Playback Stream" and "Capture Streams".

All the physical (and virtual) cards' output "sinks" will appear under "Playback Devices". I'm sure the names are pretty obvious so I wont explain further!

The functionality is close to pavucontrol, but with three important exceptions:

1. There is no equiv of the "Configuration" tab to change "Profile" for a given card.
2. There is no way to change "Ports" for a given Sink/Source (not all sound hardware supports this and it is intended to make this automatic in some cases - i.e. Headphone Jack Sensing - once it becomes reliable upstream in ALSA, but sometimes it will still be user choice (e.g. Amplified vs. Non-Amplified))
3. There is no way to move an given application from one output to another.

For 1 and 2 I intend to (eventually) provide a KCM to go alongside the Multimedia tab. For 3 you can move "categories" of applications to different devices via the Phonon settings in System Settings -> Multimedia already thanks to my previous patches.

Where can I get it?

It's already in Mandriva Cooker or you can grab it from my Git repo here: http://colin.guthr.ie/git/kdemultimedia/log/?h=pulse

The master branch is upstream svn. The pulse branch is my changes. Do a clone then issue:

 git diff master..pulse >mypatch.patch

or just build it directly.

Hopefully it'll be committed to trunk soon and if all goes well it can be easily backported to Mandriva 2010 too.

Caveats

Every time a new slider appears it is added to the "Shortcuts" system. This is OK for hardware devices, but for every application it can get a bit much.... I'll try and find a way to disable this (see "What's Left" below).

PulseAudio supports a pretty crazy limit on the number of channels a device or stream can have. Kmix only has support for a fairly standard set of elements. In come cases not all channels of a given device/stream may be shown in kmix due to this limitation. Stereo -> 5.1 setups should work fine tho'.

You Suck, I use PA but I want a Real Man's ALSA mixer!

Whatever floats your boat baby!

 $  KMIX_PULSEAUDIO_DISABLE=1 kmix

Knock yourself out!

What's Left?

I have two remaining issues that I do not think are show stoppers:

1. Everytime a new device shows up a new Global Shortcut dialog appears. I don't think this is any different to ALSA but as I now have per-app volume control, this dialog is also shown everytime a new application plays sound. It only happens the first time a given application plays a sound, but it could still be considered annoying by some.
2. If there are no capture streams running at startup the tab for that will never be displayed - likewise if the stream restore module in PA is not loaded (unlikely) the "Playback Streams" tab will never be displayed.
3. There are various things that could be more efficient (e.g. refreshing the GUI for a new device or application current redraws all tabs, not just the one that has changed when a new application appears or disappears)
4. Make KMix dumber! KMix is pretty clever and it tries to do some smart things like saving and restoring volume for you. But when PA is used, it knows better, so I need to ensure that KMix does not do any saving/restoring of actual volume values.
5. Use application icons for per-application streams. Just for eye candy, it would be nice to use the applications own icon in the GUI.

There were a few teething problems due to some last minute merges I did (which I clearly didn't test properly - my bad) and which I then went on to mis-interpret which led me to commit two rather silly things in phonon (a revert and then a revert of that revert!). What can I say... I need more caffeine obviously!

Update: PulseAudio 0.9.21 is now out and has all the necessary fixes to make the below patches unnecessary.

The slightly annoying news is that the support needed for this to work is not yet in a released version of PA. I do however have a simple patch for you here, that distro packagers are welcome to stealliberate from the Mandriva SVN. If you do this, you'll also need to apply this minor patch to phonon:

diff --git a/phonon/CMakeLists.txt b/phonon/CMakeLists.txt
index c5ce57a..50a3851 100644
--- a/phonon/CMakeLists.txt
+++ b/phonon/CMakeLists.txt
@@ -8,7 +8,6 @@ endif (PHONON_BUILD_EXAMPLES)

 add_subdirectory(experimental)

-set(PULSEAUDIO_MINIMUM_VERSION "0.9.21")
 macro_optional_find_package(PulseAudio)
 macro_log_feature(PULSEAUDIO_FOUND "PulseAudio" "A cross-platform, networked sound server." "http://www.pulseaudio.org" FALSE "" "Allows audio playback via the PulseAudio soundserver when it is running")
 macro_optional_find_package(GLIB2)

Just so that it recognises your PA build.

There don't seem to be any specific bugs yet, although I am dealing with something relating to streams going AWOL with Xine, but I'm not sure the problem is in this code. Further testing and feedback is much appreciated. Feel free to report problems to me directly or assign bugs to my kde (at) <this domain> account in b.k.o.

Have fun!

Myself and the rest of the Mandriva Developers and Contributors have put in a lot of work this time round. I'm pretty happy with the PulseAudio->Phonon integration work I did for KDE which builds on our previous approaches which were not quite as functional (although did at least hide potential configuration problems from users unlike on some distros! (for which the usual "solution" was a urpme/yum remove/apt-remove pulseaudio rather than actually finding the real cause!)

Anyway, anyone looking for a change or wanting to see what other distros have to offer, I really encourage you to take it for a spin!

This article follows on from my previous articles here and here. If you've not read them, then it would make sense to give them a quick glance (also as this was composed on my phone while on a flight, please forgive me any silly spelling errors I didn't spot when reviewing it!!)

First of all it's probably worth explaining what happens and what the phonon backend itself needs to be aware of and how it can interoperate with pulse.

So when libphonon is used in an application, a connection to PulseAudio will be attempted. If that connection fails everything will work as currently and nothing much changes. If the connection succeeds, we establish if "module-device-manager" is loaded. This is a module specifically written to implement the routing policy (an ordered priority list of devices for each category of sound produced - e.g. Notifications, Music or Video). If this module is not detected we offer a reduced/cut-down PulseAudio integration where the user will only see a single, virtual "PulseAudio" sound "device" listed in the KDE configuration system's multimedia section. If m-d-m is detected however, users will get the full rich experience. All of the devices ever seen by PulseAudio (I.e. Built in sound cards, PCI devices, USB speakers and headsets, Bluetooth devices and Apple Airtunes devices etc.) will be listed. If the device is not currently available, it will be shown greyed out as expected and you can of course reorder them to suit your own preferences.

So all that is well and good from a user perspective but "how does it work internally?" I hear you scream!

The Nitty Gritty

When m-d-m is used the device preference order is not actually stored in phonon itself as is the case when PulseAudio is not used, it is instead passed across to PulseAudio which takes over responsibility for keeping this priority list up to date. Why do it this way? Well while the preferences stored in KDE work fine for phonon apps they do not cover any non-phonon ones (which still make up the vast majority of sound producing applications on Linux). As someone involved with sound on Linux I've often been asked by confused users why their preferences are not honoured in e.g. Audacious or other non-phonon apps. As PulseAudio is integrated lower down the sound stack than phonon, ensuring it knows your device order preferences also ensures that it can route your audio appropriately according to your preferences for all apps! A clear bonus

Now as you may know phonon has it's own API calls to move streams to specific devices. This is something also handled in PulseAudio so it makes sense to simply wrap up the phonon API around these PA call which is exactly what I do! However, there is a complication. As phonon has pluggable backends, we don't actually speak to pulse directly to do the sound output (our connection to the PulseAudio daemon is only for querying and control purposes) so we don't know which stream is ours when the audio is actually playing. To get around this obscurity introduced by the pluggable backends, we take advantage of a facility in PulseAudio to attach arbitrary metadata to our streams by way of setting special environment variables. When phonon is instructed to start outputting audio, we set both the PulseAudio "media.role" property (this is the equivalent of the Phonon Categories system and we happily have an easy 1:1 mapping) and we generate a GUID for our stream and assign it as the "phonon.streamid" property. Our control connection sees all audio streams played in PulseAudio and can use this streamid to map back the PulseAudio streams to the phonon streams. This then allows us to process the stream move requests! It's a bit of a round about road to get this capability but it is the only way we can skirt around the obscurity provided by a pluggable backend system in a general way (the opaque interfaces provided by e.g. Xine, GStreamer and other backends would not expose this info directly anyway).

"But wait!" I hear those of you paying close attention scream. "Didn't you say above that PulseAudio will do the routing for us according to our preferences? Why not let it do that for phonon streams as well as non-phonon streams and forget about the whole GUID thing?" Well, if truth be told, I did start off doing it this way but it had two major issues. The first was the "Test" button not working properly - it always resulted in the highest priority device being used, no the one the user had selected! Not really ideal and more than a little confusing I'm sure you'll agree! The second problem related to visual feedback. When you reorder your devices or plug in a device of higher priority when sound is currently playing, phonon will visually inform the user of the device change. Handing over wholesale to PulseAudio meant that this feedback was lost. Again this is certainly not what I'd call seamless integration!

So at present the stream will be routed to the correct device by PulseAudio and then Phonon will also issue a move command which should in most cases be a NOOP. Likewise when devices are hotplugged (or unplugged!) PulseAudio will route them correctly and phonon will follow this up with a specific move request which will again end up being a NOOP. While this may seem pointless it does allow for the phonon API to be use properly (as needed by the test button) and give visual feedback)

Backends

So what does it mean for a backend? Well the backend needs to be somewhat aware of PulseAudio. Thankfully this is pretty trivial thanks to the PulseSupport class I've written. The backend should ask PulseSupport if pulse is detected if so it should check for and enable it's own PulseAudio output system. If this cannot be fond or activated it can tell PulseSupport that it's a no go and everything should fall back to how things work without any PulseAudio support. There are a couple other bits and bobs that a backend needs to do to work fully but that's the bulk of it.

So I hope I've provided a good and in-depth overview of how this support works. Please keep any comments on topic. If you want to post comments about PA in general, please do so on (and after reading!) this article.

This support is included in the upcoming Mandriva 2010.0 Release which will be hitting the streets very soon.

The Test button now selects the right device. The way of achieving this is a little convoluted, but it seems to work well. We now generate a UUID for every phonon stream and push this into the PulseAudio stream proplist via an environment variable. We then look for all the streams in pulse and match up our UUID such that we can match up the pulse stream index to our phonon stream. We can then use this to move the stream to the appropriate device. Due to the async nature of PulseAudio, phonon sometimes requests the move before the stream is actually setup, so we have to queue up the move requests and process them when a stream appears. This approach also has the effect of causing the specific sink to be saved in module-stream-restore database, but this shouldn't matter in practice as phonon will always move it, so if the priority list changes, then we will always set the right device one way or the other even if the whole rigmarole could be avoided. But like I say, the practical outcome is fine, so let's just leave that for now.

The category is also passed through to PulseAudio now, so the the only thing that is not handled is when PulseAudio disconnects/restarts. Thankfully this is now getting less and less often as stability improves