New features for the Hildon Application Manager
===============================================

* Clean up pre-defined catalogues mechanics [victor]

The pre-defined catalogues of the Application manager should be
handled according to the new packaging policy (doc/packpol-conf.txt).

** Spec

Roughly, this means that we need a strict division into catalogue
entries that are completely 'owned' by packages, and catalogue entries
that are competely owned by the user.

All the pre-defined catalogues should be owned by packages and can not
be edited by the user.  They are not backed up.

However, the enabled/disabled flag for each catalogue entry is owned
by the user: the user can still disable a pre-defined catalogue.  This
flag is backed up.

** Implementation hints

Right now, all catalogue information is stored in a single file,
/etc/h-a-m/catalogues.  This should be changed so that only user
catalogues are stored there.  Packages should be able to drop files
into /usr/share/hildon-application-manager/catalogues/.

When reading the catalogue information, the xexps from
/etc/h-a-m/catalogues and /usr/share/h-a-m/catalogues/ should be
merged.  Concretely, the /etc/h-a-m/catalogues file contains the
enabled/disabled flags for the catalogues in
/usr/share/h-a-m/catalogues directory.  For this, packaged catalogue
entries need to be identified reliably: a new "id" attribute should be
used for this.

It is probably a good idea to remember where a catalogue has been read
from: after reading its xexp from a file
/usr/share/h-a-m/catalogues/foo, we add a <file>foo</file> element to
the xexp.  Then catalogues are identified by their "file" plus "id"
attributes, making it easier to choose unique ids.

User catalogues have no "file" attributes.

When saving the catalogues after the user has edited them, only the
/etc/h-a-m/catalogues file should be touched.  That is, the xexps for
a catalogue without a "file" attribute are written to that file, for
ther xexps, only the "file", "id" and "disabled" attribute are
written.

These rules should also be followed by the
hildon-application-manager-config utility.  Changes made with that
tool count as user edits.

* Better handling and reporting of dependency and conflict issues. [mvo]

We need to get better at explaining the reasons why a given operation
can not be performed so that the user has a chance of taking
corrective actions.

Also, since we now have a visible package that represents the
operating system, we can be more agressive with making changes to
invisible packages: the operating system package will make sure (via
its dependencies) that we don't kill the device by accidentally
uninstalling or upgrading important packages.

The general principle that this feature aims to implement is that the
user-visible packages are managed completely manually by the user, and
the user-invisible packages are managed completely automatically.

Thus, the general behavior of the Application Manager should be the
following:

 1 When the user requests to install or remove a package, the AM will
   plan to do any necessary (and possible) changes to all the packages
   to satisfy the request of the user.  These changes can include
   installing additional packages, removing already installed
   packages, or upgrading installed packages to a newer version.

   The AM should try at least as hard as apt-get to find a solution,
   but it should favor upgrading over removing.

   For example, instead of removing the OS meta package in order to
   upgrade some invisible packages, it should prefer to upgrade the OS
   meta package (and pull in the complete OS update).

 2 Once a solution has been found, and it requires changes to user
   visible packages, the user has to confirm these changes.

 3 When no solution has been found, the only reasons for this can be
   that the OS package would need to be removed, or that packages are
   needed that are not available from any repository.  This can be
   explained easily, I'd assume.  We should probably not even list the
   individual packages that are missing, but just say that the
   catalogues are broken.

The following steps might be appropriate:

- Investigate which algorithms apt-get and aptitude use for "install
  <pkg>" and "remove <pkg>" and make them available in the apt-worker
  (preferably aptitudes).  Don't remove the existing algorithms.

- Add a red-pill setting that makes the apt-worker use those
  algorithms.

- Tune the algorithms to favor upgrading over removal, as explained
  under point 1 above.

- Add suitable problem detection and reporting algorithms that
  implement points 1 and 2 from above.  But instead of letting the
  user confirm the planned changes to visible packages, we just report
  these changes as errors, or do them automatically:

  - If a visible package A would be removed and the user is doing a
    "uninstall B" operation, we report "B is needed by A."

  - If a visible package A would be removed and the user is doing a 
    "install B" operation, we report "B conflicts with A."

  - If a visible package A would be newly installed or upgraded and
    the user is doing a "install B" operation, and package A is not
    the OS, we just do the upgrade.

  - If package A from the last paragraph is the OS, we report "B needs
    A to be installed first".

  - If a visible package A would be newly installed or upgraded and
    the user is doing a "uninstall B" operation, we report "B is
    needed by A".

  The reports should always include the respective versions of the
  packages.

- Experiment!  Make a decision whether we can ship with these
  algorithms enabled by default or whether we can get the next step
  done as well:

- Show the thing to the UI people so that we can come up with the
  right way to do it: letting users confirm the changes to the visible
  packages.

* Avoiding invisible interaction flows [mvo]

[ needs second pass to be sure we caught enough cases ]

Currently, it is possible to start an interaction flow (using
start_interaction_flow, say) without immediatly popping up a dialog
that would represent this interaction flow.

The delays used to be shortish, but they can now be quite long because
of big OS updates.

For example, opening the "Details" dialog for a "OS2008 Reloaded"
update can take a couple of seconds and there is therefore a long gap
between activating the button and the dialog opening.

Two things need to be done: the preparing of large operations needs to
be sped up, and potential gaps in the UI flow need to be plugged.
This feature is about pluggin the gaps in the UI.

First, the gaps need to be identified: look for uses of
start_interaction_flow etc and make sure that no operation is
performed between calling this start_interaction_flow function and
showing the first dialog.

No apt-worker request can be assumed to be quick (not even things like
APTCMD_GET_CATALOGUES that don't require significant processing) since
the apt-worker might be busy with something else before being able to
process a quick request.

If information from the apt-worker is needed to construct the dialog,
the code could be changed to show an 'empty' dialog with insensitive
buttons and a "updating" progress banner.  When the information
arrives, the dialog should be filled and the buttons should be made
sensitive.

Please prepare a list of these cases so that we can quickly see
whether this approach needs to be refined in some cases.

The details dialog is already a bit special: do not make the "Close"
button insensitive while waiting for more information from the
apt-worker.  It should be possible to close the dialog at any time.
The apt-worker request does not need to be canceled in that case, but
the eventual reply should be safely ignored, of course.

Maybe some apt-worker requests need to split in two so that the first
half can finish quickly.
