sanchayanmaity/notes
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
notes/android.md

260 lines
15 KiB
Markdown
Raw Permalink Blame History

---
title: Android
---
# Binder
## Reference material
- [Deep Dive into Android IPC/Binder Framework](https://www.youtube.com/watch?v=Jgampt1DOak)
- [AnDevCon IV: Android Binder IPC Framework](https://www.youtube.com/watch?v=hiq3mGfLOtE)
- [Digging into Android System Services](https://www.youtube.com/watch?v=M6extgmQQNw)
- [Binder transactions in the bowels of the Linux kernel](https://www.synacktiv.com/en/publications/binder-transactions-in-the-bowels-of-the-linux-kernel.html)
## Notes from deep dive
- An IPC framework for developing object oriented OS services.
- Built in reference counting mechanism.
- Identifying senders to receivers via UID/PID.
- Unique object mapping across process boundaries.
- Ability to send file descriptors across process boundaries.
- Android supports a simple form of IPC via `intents` and `content providers`. Uses binder behind the scenes. Asynchronous.
- Messenger IPC
- `Messenger` represents a reference to `Handler` that can be send to a remote process via an `Intent`.
- A reference to the `Messenger` can be sent via an `Intent` using the previously mentioned IPC mechanism.
- `Messages` send by the remote process are delivered to the local handler.
- `Messages` are like `Intents` in that they designate the operation and data. Still asynchronous.
- `Parcel` is a container for a message (data and object references) that can be sent through an `IBinder`. A unit of transactional data - one for the outbound request and another for the inbound reply.
- `Marshalling/Unmarshalling`
- Marshalling is a procedure for converting higher level app data structures i.e request response parameters into parcels for the purposes of embedding them into Binder transactions.
- Unmarshalling is a procedure for reconstructing higher level app data structures i.e request response parameters from parcels received through Binder transactions.
- Proxy is an implementation of the AIDL interface that un/marshals data and maps method calls to transactions submitted via a wrapped IBinder reference to the Binder object.
- Stub is a partial implementation of the AIDL interface that maps transactions to Binder service method calls while un/marshalling data.
- `ContextManager` or `servicemanager`, a special binder object with a known handle (registered as handle 0) that's used as a registry/lookup service for other Binder objects.
- Most low level operations and data structures i.e `Parcel` are abstracted by `libbinder` at the native level which is what the clients and services use.
- Clients and services don't want to know anything about the `Binder` protocol and `libbinder`, so they makes use of `proxies` and `stubs`.
- `Proxy's` job is to take the high level Java or C++ request, and convert it into one of `Parcels` and then submit an `ioctl` transaction to the binder driver. `Stub's` is to listen to the binder driver callback, and then upon receiving a callback unmarshal the `Parcel` into something the service can understand and call the appropriate callback in the service. Java based proxies and stubs can be automatically generated by `aidl` tool for services described with AIDL. Service needs to implement the binder interface.
- `Manager` sits between the `Client` and `Proxy`. `ContextManager` sits between the binder driver and the service.
- Anything you pass as an `IBinder` to the other side gets passed as a reference.
## Binder object reference mapping across process boundaries
- A binder object reference is one of the following
- An actual virtual memory address to a binder object in the same process
- An abstract 32-bit handle to a binder object in another process
- On every transaction, the binder driver automatically maps local addresses to remote binder handles and remote binder handles to local addresses
- This mapping is done on:
- Targets of binder transactions
- `IBinder` object references shared across process boundaries as a parameter or a return value (embedded in transaction data)
- For this to work
- The driver maintains mappings of local addresses and remote handles between processes as binary tree per process so that it can perform this translation
- References embedded in transaction data is discovered based on effects that the client provies when it submits its transaction and then rewritten in place
# Thumbnail extraction
[Framework Hardening](https://source.android.com/devices/media/framework-hardening)
- Abstract class `Thumbnailer` in `MediaProvider`. Note that `MediaProvider` includes `ThumbnailUtils`. A separate directory for thumbnails is used `.thumbnails`. Creates an audio thumbnail for music directory, video thumbnail for movies directory and image thumbnail for pictures directory. Bitmap format is used for the thumbnails. Note that these utilities functions have a note mentioning that these should be used only if direct access to it's available. If it's media hosted outside app, use `ContentResolver#loadThumbnail` which is in `ContentResolver.java`. This seems to be the way `MediaStore` uses it, though the function where it's used seems to be deprecated.
- Ensures that thumbnails collection on the given storage volume can be used with the given database. If the database UUID doesn't match the UUID found on disk, then all thumbnails are considered stale and deleted. Seems to get called in `onIdleMaintainance`.
- Primarily three functions are seen which are called in `MediaProvider` from `ThumbnailUtils`.
1. `createAudioThumbnail`
2. `createVideoThumbnail`
3. `createImageThumbnail`
- These create an instance of `MediaMetadataRetriever` and then call methods on this instance. For example, `createVideoThumbnail` calls `extractMetadata`, which then calls in to the `nativeExtractMetadata`. The native implementation itself calls the implementation in `MediaPlayerService` through a Binder call. See the constructor in `mediametadataretriever.cpp`.
- `MediaProvider` seems to primarily use the `invalidate` and `ensure` thumbnail functions. The `Thumbnailer` instances created for audio, video and image primarily call `ensureThumbnail` from the `Thumbnail` abstract class. This function has a `ParcelFileDescriptor`.
- `FileDescriptor` objects, representing raw Linux file descriptor identifiers can be written and `ParcelFileDescriptor` objects returned to operate on the original file descriptor.
- The flow in `MediaProvider` is `openFile -> openFileCommon -> ensureThumbnail -> mAudio/Video/ImageThumbnailer.ensureThumbnail`.
- The actual extraction of the thumbnail happens in `getBitmapFromVideoFrame` in the native code.
### Summary
`MediaProvider` uses the `MediaMetadataRetriever` which eventually relies on native code via JNI to extract the thumbnail. This seems to be allowed only because the provider here is kind of owner so to speak and has direct access to the media library?. Elsewhere like `MediaStore` or other applications the appropriate interface would `ContentResolver`.
## Examples
- https://cgit.freedesktop.org/gstreamer/gst-plugins-base/tree/tests/examples/snapshot/snapshot.c
- https://stackoverflow.com/questions/15789652/how-to-create-video-thumbnails-with-python-and-gstreamer
# ModernMediaScanner
- Implements the `MediaScanner` interface below.
``` java
public interface MediaScanner {
public static final int REASON_UNKNOWN = MEDIA_PROVIDER_SCAN_OCCURRED__REASON__UNKNOWN;
public static final int REASON_MOUNTED = MEDIA_PROVIDER_SCAN_OCCURRED__REASON__MOUNTED;
public static final int REASON_DEMAND = MEDIA_PROVIDER_SCAN_OCCURRED__REASON__DEMAND;
public static final int REASON_IDLE = MEDIA_PROVIDER_SCAN_OCCURRED__REASON__IDLE;
public Context getContext();
public void scanDirectory(File file int reason);
public Uri scanFile(File file, int reason);
public Uri scanFile(File file, int reason, @Nullable String ownerPackage);
public void onDetachVolume(String volumeName);
}
```
- It starts by populating metadata based on file attributes and then overwrites with any valid metadata found using `MediaDataRetriever`, `ExifInterface`, `XmpInterface`.
- `ModernMediaScanner` is called by `MediaProvider` and gets passed a `Context`. `getContext` just returns this `Context` when called.
``` java
/**
* Map from volume name to signals that can be used to cancel any active
* scan operations on those volumes.
*/
@GuardedBy("mSignals")
private final ArrayMap<String, CancellationSignal> mSignals = new ArrayMap<>();
/**
* Holder that contains a reference count of the number of threads
* interested in a specific directory, along with a lock to ensure that
* parallel scans don't overlap and confuse each other.
*/
private static class DirectoryLock {
public int count;
public final Lock lock = new ReentrantLock();
}
/**
* Map from directory to locks designed to ensure that parallel scans don't
* overlap and confuse each other.
*/
@GuardedBy("mDirectoryLocks")
private final Map<Path, DirectoryLock> mDirectoryLocks = new ArrayMap<>();
```
- `scanDirectory` and the two `scanFile` functions call the private class `Scan`. See `ContentProvider`, `ContentResolver` and `ContentInterface`.
- `ScanItemAudio` and `ScanItemVideo` use `MediaMetadataRetriever`.
## Working of \`Scan\` which is a public class
- Get the content provider client using the content resolver
- The primary function is `run`
- This calls `walkFileTree`. Acquire directory lock at the start of operation and release at the end. This uses the existing `walkFileTree` from `Files`. The rough way this works is, it's recursive. The file visitor function `visitFile` gets called for every file and depending upon the `FileVisitResult` the operation continues or stop. `visitFile` is what calls `scanItem` which in turn calls the media specific scan functions. Of the media specific scan functions, only the audio and video ones use `MediaMetadataRetriever`, the other uses XMP, ISO, EXIF or standard file utilities. Here something called as `ContentProviderOperation` exists. It represents a single operation to be performed as part of a batch of operations. See the `ContentProviderOperation` section below.
- Then `reconcileAndClean`. This talks to the database and some operations on the database entries. It cleans up unknown entries and update the generation associated with a file column.
- `resolvePlaylists`. Validates the playlist members?
## Media modules documentation
[Media Modules](https://source.android.com/devices/media/media-modules)
## MediaStore
- It's the contract between the `MediaProvider` and applications. Contains definitions for the supported URIs and columns.
- A generation value is associated with a given volume. Generation numbers are useful for apps that are attempting to identify exactly which media items have been added or changed since a previous point in time. Generation numbers are monotonically increasing over time, and can be arithmetically compared.
## MediaProvider
- The MediaProvider module optimizes indexed metadata (audio, video, and images from SD cards and USB devices) and makes that data available to apps through the `MediaStore` API.
- In this case, `MediaProvider` extends `ContentProvider`.
- Some of the modules which reference `MediaProvider` are `MediaService`, `ExternalStorageServiceImpl`, `PermissionActivity`, `FuseDaemon` and `IdleService`.
- For permissions, see `checkUriPermission`.
### Notes on threading
- Some of the functions seemed to be called in JNI context and have the note `Called from JNI in jni/MediaProviderWrapper.cpp`. These functions seem to have the current thread in their `JNIenv`.
``` java
void MediaProviderWrapper::ScanFile(const string& path) {
JNIEnv* env = MaybeAttachCurrentThread();
scanFileInternal(env, media_provider_object_, mid_scan_file_, path);
}
```
### Notes on security
- [Media Provider](https://source.android.com/devices/media/media-provider)
- [Scoped Storage](https://developer.android.com/training/data-storage#scoped-storage)
- To maintain privacy, the `MediaProvider` module enforces the `scoped storage security module`.
## ContentProvider
- Content providers are on of the primary building blocks of Android applications, providing content to applications. They encapsulate data and provide it to applications through the single `ContentResolver` interface. A content provider is only required if you need to share data between multiple applications. For example, the contacts data is used by multiple applications and must be stored in a content provider. If you don't need to share data amongst multiple applications you can use a database directly via `android.database.sqlite.SQLiteDatabase`.
- `MediaProvider` extends `ContentProvider`.
- See more on Content Providers in the below link.
- [Content Provider Basics](https://developer.android.com/guide/topics/providers/content-provider-basics)
- [Content Providers](https://developer.android.com/guide/topics/providers/content-providers)
- When a request is made via a `ContentResolver` the system inspects the authority of the given URI and passes the request to the content provider registered with the authority. The content provider can interpret the rest of the URI however it wants.
- Requests to `ContentResolver` are automatically forwarded to the appropriate `ContentProvider` instance, so subclasses don't have to worry about the details of cross-process calls.
- See permissions based on `Intent`.
### Data access via intents
Intents can provide indirect access to a content provider. You allow the user to access data in a provider even if your app doesn't have access permissions, either by getting a result intent back from an app that has permissions, or by activating an app that has permissions and letting the user do work in it.
You can access data in a content provider, even if you don't have the proper access permissions, by sending an intent to an app that does have the permissions and receiving back a result intent containing "URI" permissions. These are permissions for a specific content URI that last until the activity that receives them is finished. The app that has permanent permissions grants temporary permissions by setting a flag in the result intent:
- Read permission: `FLAG_GRANT_READ_URI_PERMISSION`
- Write permission: `FLAG_GRANT_WRITE_URI_PERMISSION`
## ContentProviderOperation
Used for batch access, see the preceding `content-provider-basics` article link.
[Better Performance with ContentProviderOperation](https://www.grokkingandroid.com/better-performance-with-contentprovideroperation/)
## MediaMetadataRetriever
- For retrieving meta data when it comes to audio and video files, an instance of this is used by `ModernMediaScanner`. It seems to call into the native media extractor via JNI.
Reference in a new issue View git blame Copy permalink