notes/android.md

title
Android

Binder

Reference material

• Deep Dive into Android IPC/Binder Framework

• AnDevCon IV: Android Binder IPC Framework

• Digging into Android System Services

• Binder transactions in the bowels of the Linux kernel

Notes from deep dive

• An IPC framework for developing object oriented OS services.

• Built in reference counting mechanism.

• Identifying senders to receivers via UID/PID.

• Unique object mapping across process boundaries.

• Ability to send file descriptors across process boundaries.

• Android supports a simple form of IPC via intents and content providers. Uses binder behind the scenes. Asynchronous.

• Messenger IPC

  • Messenger represents a reference to Handler that can be send to a remote process via an Intent.
  • A reference to the Messenger can be sent via an Intent using the previously mentioned IPC mechanism.
  • Messages send by the remote process are delivered to the local handler.
  • Messages are like Intents in that they designate the operation and data. Still asynchronous.

• Parcel is a container for a message (data and object references) that can be sent through an IBinder. A unit of transactional data - one for the outbound request and another for the inbound reply.

• Marshalling/Unmarshalling

  • Marshalling is a procedure for converting higher level app data structures i.e request response parameters into parcels for the purposes of embedding them into Binder transactions.

  • Unmarshalling is a procedure for reconstructing higher level app data structures i.e request response parameters from parcels received through Binder transactions.

• Proxy is an implementation of the AIDL interface that un/marshals data and maps method calls to transactions submitted via a wrapped IBinder reference to the Binder object.

• Stub is a partial implementation of the AIDL interface that maps transactions to Binder service method calls while un/marshalling data.

• ContextManager or servicemanager, a special binder object with a known handle (registered as handle 0) that's used as a registry/lookup service for other Binder objects.

• Most low level operations and data structures i.e Parcel are abstracted by libbinder at the native level which is what the clients and services use.

• Clients and services don't want to know anything about the Binder protocol and libbinder, so they makes use of proxies and stubs.

• Proxy's job is to take the high level Java or C++ request, and convert it into one of Parcels and then submit an ioctl transaction to the binder driver. Stub's is to listen to the binder driver callback, and then upon receiving a callback unmarshal the Parcel into something the service can understand and call the appropriate callback in the service. Java based proxies and stubs can be automatically generated by aidl tool for services described with AIDL. Service needs to implement the binder interface.

• Manager sits between the Client and Proxy. ContextManager sits between the binder driver and the service.

• Anything you pass as an IBinder to the other side gets passed as a reference.

Binder object reference mapping across process boundaries

• A binder object reference is one of the following

  • An actual virtual memory address to a binder object in the same process

  • An abstract 32-bit handle to a binder object in another process

• On every transaction, the binder driver automatically maps local addresses to remote binder handles and remote binder handles to local addresses

• This mapping is done on:

  • Targets of binder transactions

  • IBinder object references shared across process boundaries as a parameter or a return value (embedded in transaction data)

• For this to work

  • The driver maintains mappings of local addresses and remote handles between processes as binary tree per process so that it can perform this translation

  • References embedded in transaction data is discovered based on effects that the client provies when it submits its transaction and then rewritten in place

Thumbnail extraction

Framework Hardening

• Abstract class Thumbnailer in MediaProvider. Note that MediaProvider includes ThumbnailUtils. A separate directory for thumbnails is used .thumbnails. Creates an audio thumbnail for music directory, video thumbnail for movies directory and image thumbnail for pictures directory. Bitmap format is used for the thumbnails. Note that these utilities functions have a note mentioning that these should be used only if direct access to it's available. If it's media hosted outside app, use ContentResolver#loadThumbnail which is in ContentResolver.java. This seems to be the way MediaStore uses it, though the function where it's used seems to be deprecated.

• Ensures that thumbnails collection on the given storage volume can be used with the given database. If the database UUID doesn't match the UUID found on disk, then all thumbnails are considered stale and deleted. Seems to get called in onIdleMaintainance.

• Primarily three functions are seen which are called in MediaProvider from ThumbnailUtils.

  1. createAudioThumbnail

  2. createVideoThumbnail

  3. createImageThumbnail

• These create an instance of MediaMetadataRetriever and then call methods on this instance. For example, createVideoThumbnail calls extractMetadata, which then calls in to the nativeExtractMetadata. The native implementation itself calls the implementation in MediaPlayerService through a Binder call. See the constructor in mediametadataretriever.cpp.

• MediaProvider seems to primarily use the invalidate and ensure thumbnail functions. The Thumbnailer instances created for audio, video and image primarily call ensureThumbnail from the Thumbnail abstract class. This function has a ParcelFileDescriptor.

• FileDescriptor objects, representing raw Linux file descriptor identifiers can be written and ParcelFileDescriptor objects returned to operate on the original file descriptor.

• The flow in MediaProvider is openFile -> openFileCommon -> ensureThumbnail -> mAudio/Video/ImageThumbnailer.ensureThumbnail.

• The actual extraction of the thumbnail happens in getBitmapFromVideoFrame in the native code.

Summary

MediaProvider uses the MediaMetadataRetriever which eventually relies on native code via JNI to extract the thumbnail. This seems to be allowed only because the provider here is kind of owner so to speak and has direct access to the media library?. Elsewhere like MediaStore or other applications the appropriate interface would ContentResolver.

Examples

• https://cgit.freedesktop.org/gstreamer/gst-plugins-base/tree/tests/examples/snapshot/snapshot.c

• https://stackoverflow.com/questions/15789652/how-to-create-video-thumbnails-with-python-and-gstreamer

ModernMediaScanner

• Implements the MediaScanner interface below.

  public interface MediaScanner {
      public static final int REASON_UNKNOWN = MEDIA_PROVIDER_SCAN_OCCURRED__REASON__UNKNOWN;
      public static final int REASON_MOUNTED = MEDIA_PROVIDER_SCAN_OCCURRED__REASON__MOUNTED;
      public static final int REASON_DEMAND = MEDIA_PROVIDER_SCAN_OCCURRED__REASON__DEMAND;
      public static final int REASON_IDLE = MEDIA_PROVIDER_SCAN_OCCURRED__REASON__IDLE;
  
      public Context getContext();
      public void scanDirectory(File file int reason);
      public Uri scanFile(File file, int reason);
      public Uri scanFile(File file, int reason, @Nullable String ownerPackage);
      public void onDetachVolume(String volumeName);
  }
  

• It starts by populating metadata based on file attributes and then overwrites with any valid metadata found using MediaDataRetriever, ExifInterface, XmpInterface.

• ModernMediaScanner is called by MediaProvider and gets passed a Context. getContext just returns this Context when called.

  /**
   * Map from volume name to signals that can be used to cancel any active
   * scan operations on those volumes.
   */
  @GuardedBy("mSignals")
  private final ArrayMap<String, CancellationSignal> mSignals = new ArrayMap<>();
  
  /**
   * Holder that contains a reference count of the number of threads
   * interested in a specific directory, along with a lock to ensure that
   * parallel scans don't overlap and confuse each other.
   */
  private static class DirectoryLock {
      public int count;
      public final Lock lock = new ReentrantLock();
  }
  
  /**
   * Map from directory to locks designed to ensure that parallel scans don't
   * overlap and confuse each other.
   */
  @GuardedBy("mDirectoryLocks")
  private final Map<Path, DirectoryLock> mDirectoryLocks = new ArrayMap<>();
  

• scanDirectory and the two scanFile functions call the private class Scan. See ContentProvider, ContentResolver and ContentInterface.

• ScanItemAudio and ScanItemVideo use MediaMetadataRetriever.

Working of `Scan` which is a public class

• Get the content provider client using the content resolver

• The primary function is run

  • This calls walkFileTree. Acquire directory lock at the start of operation and release at the end. This uses the existing walkFileTree from Files. The rough way this works is, it's recursive. The file visitor function visitFile gets called for every file and depending upon the FileVisitResult the operation continues or stop. visitFile is what calls scanItem which in turn calls the media specific scan functions. Of the media specific scan functions, only the audio and video ones use MediaMetadataRetriever, the other uses XMP, ISO, EXIF or standard file utilities. Here something called as ContentProviderOperation exists. It represents a single operation to be performed as part of a batch of operations. See the ContentProviderOperation section below.

  • Then reconcileAndClean. This talks to the database and some operations on the database entries. It cleans up unknown entries and update the generation associated with a file column.

  • resolvePlaylists. Validates the playlist members?

Media modules documentation

Media Modules

MediaStore

• It's the contract between the MediaProvider and applications. Contains definitions for the supported URIs and columns.

• A generation value is associated with a given volume. Generation numbers are useful for apps that are attempting to identify exactly which media items have been added or changed since a previous point in time. Generation numbers are monotonically increasing over time, and can be arithmetically compared.

MediaProvider

• The MediaProvider module optimizes indexed metadata (audio, video, and images from SD cards and USB devices) and makes that data available to apps through the MediaStore API.

• In this case, MediaProvider extends ContentProvider.

• Some of the modules which reference MediaProvider are MediaService, ExternalStorageServiceImpl, PermissionActivity, FuseDaemon and IdleService.

• For permissions, see checkUriPermission.

Notes on threading

• Some of the functions seemed to be called in JNI context and have the note Called from JNI in jni/MediaProviderWrapper.cpp. These functions seem to have the current thread in their JNIenv.

  void MediaProviderWrapper::ScanFile(const string& path) {
    JNIEnv* env = MaybeAttachCurrentThread();
    scanFileInternal(env, media_provider_object_, mid_scan_file_, path);
  }
  

Notes on security

• Media Provider

• Scoped Storage

• To maintain privacy, the MediaProvider module enforces the scoped storage security module.

ContentProvider

• Content providers are on of the primary building blocks of Android applications, providing content to applications. They encapsulate data and provide it to applications through the single ContentResolver interface. A content provider is only required if you need to share data between multiple applications. For example, the contacts data is used by multiple applications and must be stored in a content provider. If you don't need to share data amongst multiple applications you can use a database directly via android.database.sqlite.SQLiteDatabase.

• MediaProvider extends ContentProvider.

• See more on Content Providers in the below link.

  • Content Provider Basics

  • Content Providers

• When a request is made via a ContentResolver the system inspects the authority of the given URI and passes the request to the content provider registered with the authority. The content provider can interpret the rest of the URI however it wants.

• Requests to ContentResolver are automatically forwarded to the appropriate ContentProvider instance, so subclasses don't have to worry about the details of cross-process calls.

• See permissions based on Intent.

Data access via intents

Intents can provide indirect access to a content provider. You allow the user to access data in a provider even if your app doesn't have access permissions, either by getting a result intent back from an app that has permissions, or by activating an app that has permissions and letting the user do work in it.

You can access data in a content provider, even if you don't have the proper access permissions, by sending an intent to an app that does have the permissions and receiving back a result intent containing "URI" permissions. These are permissions for a specific content URI that last until the activity that receives them is finished. The app that has permanent permissions grants temporary permissions by setting a flag in the result intent:

• Read permission: FLAG_GRANT_READ_URI_PERMISSION

• Write permission: FLAG_GRANT_WRITE_URI_PERMISSION

ContentProviderOperation

Used for batch access, see the preceding content-provider-basics article link.

Better Performance with ContentProviderOperation

MediaMetadataRetriever

• For retrieving meta data when it comes to audio and video files, an instance of this is used by ModernMediaScanner. It seems to call into the native media extractor via JNI.