Added: python bindings, example videos, shared memory buffers, Activi…

…ties as dialogs

.gitignore

@@ -13,3 +13,6 @@
 .externalNativeBuild
 .cxx
 local.properties
+/bindings/Python/src/termux_gui.egg-info/
+/bindings/Python/dist/
+/bindings/Python/src/termuxgui.egg-info/

Protocol.md

@@ -2,6 +2,9 @@
 
 If you don't want to implement a custom library for using the protocol, you can skip to the [protocol methods](#methods)<!-- @IGNORE PREVIOUS: anchor -->
 
+### WARNING: The protocol is not yet stable and could change anytime
+
+
 ## Connection
 
 
@@ -40,11 +43,12 @@ Available methods:
 
 Activity and Task control:  
 These methods control Android [Activities](https://developer.android.com/reference/android/app/Activity) and [Tasks](https://developer.android.com/guide/components/activities/tasks-and-back-stack).
-- newActivity: Launches a new Activity and return the activity id.
+- newActivity: Launches a new Activity and returns the activity id. In case of an error, -1 is returned.
   - Parameters:
     - tid: the task in which the Activity should be started. If not specified, a new Task is created and the id is returned after the Activity id.
     - flags: Flags to set when launching the Activity via the Intent.
-    - isDialog: boolean value. If true, the Activity will be launched as a dialog.
+    - dialog: boolean value. If true, the Activity will be launched as a dialog.
+    - pip: boolean. Whether or not to start the Activity in [Picture-in-Picture mode](https://developer.android.com/guide/topics/ui/picture-in-picture). Default is false. This should only be used to create Activities in a new Task.
 - finishActivity: Closes an Activity.
   - Parameters:
      - aid: The id of the Activity to close.
@@ -62,50 +66,90 @@ These methods control Android [Activities](https://developer.android.com/referen
     - windowBackground
     - textColor
     - colorAccent
-- setTaskIcon: Sets the Icon of the Task. The background color of the icon will be the current primary color.
+- setTaskDescription: Sets the Icon of the Task, the label and set the primary color to the one specified with setTheme. The background color of the icon will be the current primary color.
   - Parameters:
     - aid: The id of an Activity in the Task you want to set the icon.
     - img: The image data in PNG or JPEG format.
+    - label: The new Task label.
+- setPiPParams: Sets the PiP parameters for the Activity. PiP parameters are only available on Android 8+, on earlier versions this is a noop.
+  - Parameters:
+    - aid: The id of an Activity in the Task you want to set the icon.
+    - num: Numerator of the desired aspect ration.
+    - den: Denominator of the desired aspect ration.
 
 
 Layout and View control:  
 These methods create and Manipulate [Views](https://developer.android.com/reference/android/view/View), [Layouts](https://developer.android.com/guide/topics/ui/declaring-layout) and the Layout hierarchy.
-- create*: Creates a View and places it in the Layout hierarchy. It returns the View id.
+- create*: Creates a View and places it in the Layout hierarchy. It returns the View id or -1 in case of an error.
   - The following Views and Layouts are supported:
     - [LinearLayout](https://developer.android.com/guide/topics/ui/layout/linear)
     - [RelativeLayout](https://developer.android.com/guide/topics/ui/layout/relative)
     - [TextView](https://developer.android.com/reference/android/widget/TextView)
     - [EditText](https://developer.android.com/reference/android/widget/EditText)
     - [Button](https://developer.android.com/reference/android/widget/Button)
-    - [Canvas](https://developer.android.com/reference/android/graphics/Canvas)
+    - [ImageView](https://developer.android.com/reference/android/widget/ImageView)
     - [Space](https://developer.android.com/reference/android/widget/Space)
     - [NestedScrollView](https://developer.android.com/reference/androidx/core/widget/NestedScrollView)
     - [RecyclerView](https://developer.android.com/guide/topics/ui/layout/recyclerview)
+    - [AutocompleteTextView](https://developer.android.com/reference/android/widget/AutoCompleteTextView)
+    - [RadioButton](https://developer.android.com/guide/topics/ui/controls/radiobutton)
+    - [RadioGroup](https://developer.android.com/reference/android/widget/RadioGroup)
+    - [Checkbox](https://developer.android.com/guide/topics/ui/controls/checkbox)
+    - [ToggleButton](https://developer.android.com/guide/topics/ui/controls/togglebutton)
+    - [ImageButton](https://developer.android.com/reference/android/widget/ImageButton)
+    - [Spinner](https://developer.android.com/guide/topics/ui/controls/spinner)
+    - [GridLayout](https://developer.android.com/reference/android/widget/GridLayout)
+    - [ProgressBar](https://developer.android.com/reference/android/widget/ProgressBar)
+    - [ViewPager2](https://developer.android.com/guide/navigation/navigation-swipe-view-2)
+    - [TabLayout](https://developer.android.com/reference/com/google/android/material/tabs/TabLayout)
+    - [Tab](https://developer.android.com/reference/com/google/android/material/tabs/TabLayout.Tab)
+    - [WebView](https://developer.android.com/reference/android/webkit/WebView)
   - Parameters:
     - parent: The View id of the parent in the Layout hierarchy. if not specified, this will replace the root of the hierarchy and delete all existing views.
     - aid: The id of the Activity in which to create the View.
     - text: For Button, TextView and EditText, this is the initial Text.
     - vertical: For LinearLayout, this specifies if the Layout is vertical or horizontal. If not specified, vertical is assumed.
+    - All possible parameters to set the Layout parameters, depending on the parent Layout.
+- setLinearLayoutParams: Sets the LinearLayout parameters for a View in a LinearLayout.
+  - Parameters:
+    - parent: The View id of the parent in the Layout hierarchy.
+    - aid: The id of the Activity the View is in.
+    - weight: Sets the Layout weight.
+- setRelativeLayoutParams: Sets the RelativeLayout parameters for a View in a RelativeLayout.
+  - Parameters:
+    - parent: The View id of the parent in the Layout hierarchy.
+    - aid: The id of the Activity the View is in.
+    - 
+- setVisibility: Sets the visibility of a Vew.
+  - Parameters:
+    - id: The id of the View.
+    - aid: The id of the Activity the View is in.
 - deleteView: Deletes a View and its children from the Layout hierarchy.
   - Parameters:
     - id: The id of the View to delete.
-    - aid: The id of the Activity in which the View is in.
-- setMargin: Sets the Margin of a View.
+    - aid: The id of the Activity the View is in.
+- setMargin: Sets the margin of a View.
   - Parameters:
     - id: The id of the View.
-    - aid: The id of the Activity in which the View is in.
-    - margin: The margin value as an integer in [dp](https://developer.android.com/guide/topics/resources/more-resources.html#Dimension)
+    - aid: The id of the Activity the View is in.
+    - margin: The margin value as an integer in [dp](https://developer.android.com/guide/topics/resources/more-resources.html#Dimension).
     - dir: can be "top", "bottom", "left", "right" to set the margin for one of those directions. If not specified. the margin is set for all directions.
-- setLayoutWeight: Sets the Layout weight for Layouts that support it.
+- setPadding: Sets the padding of a View.
   - Parameters:
     - id: The id of the View.
     - aid: The id of the Activity the View is in.
-    - weight: The Layout weight.
+    - padding: The padding value as an integer in [dp](https://developer.android.com/guide/topics/resources/more-resources.html#Dimension)
+    - dir: can be "top", "bottom", "left", "right" to set the padding for one of those directions. If not specified. the padding is set for all directions.
 - setText: Sets the text of the View.
   - Parameters:
     - id: The View id of a TextView, Button or EditText.
     - aid: The id of the Activity the View is in.
     - text: The text.
+- setTextSize: Sets the text size.
+  - Parameters:
+    - id: The View id of a TextView, Button or EditText.
+    - aid: The id of the Activity the View is in.
+    - size: The text size in [sp](https://developer.android.com/guide/topics/resources/more-resources.html#Dimension).
 - getText: gets the text of the View.
   - Parameters:
     - id: The View id of a TextView, Button or EditText.
@@ -114,6 +158,34 @@ These methods create and Manipulate [Views](https://developer.android.com/refere
   - Parameters:
     - aid: The id of the Activity the View is in.
     - parent: The View id of the parent in the Layout hierarchy. if not specified, this will replace the root of the hierarchy and delete all existing views.
+- generateIDs: Generates and reserves View ids. Use them to set the ids for the JSON Layout in inflateJSON.
+  - Parameters:
+    - n: The number of ids to generate
+- setImage: Sets the image for an ImageView.
+  - Parameters:
+    - id: The View id of a ImageView.
+    - aid: The id of the Activity the View is in.
+    - img: The image data in PNG or JPEG format.
+- addBuffer: Adds a buffer to be used for ImageViews. Returns to id of the generated buffer. Returns -1 in case of an error. This should be a file descriptor to a shared memory file. You can then write to the shared memory and use the image in the plugin without having to transmit it. When successful, it transfers a file descriptor via SCM_RIGHTS. That file descriptor can then be mapped. This feature is only supported on Android 8.1+, on earlier versions this always fails.
+  - Parameters:
+    - format: the Image buffer format. Supported: "ARGB888". The order in memory will be rgba with one byte each.
+    - w: The width of the buffer.
+    - h: The height of the buffer.
+- deleteBuffer: Deletes a shared buffer. You have to do this when you aren't using a buffer anymore (eg. it is to small and you want a larger one), because Buffers consume a lot of memory. Only call this once no ImageView uses this buffer anymore, by setting another buffer, an image or removing the ImageView.
+  - Parameters:
+    - bid: id of the buffer.
+- blitBuffer: Copies the content from the Shared memory buffer into the buffer.
+  - Parameters:
+    - bid: id of the buffer.
+- setBuffer: Sets the ImageView to have a buffer as a source.
+  - Parameters:
+    - id: The View id of a ImageView.
+    - aid: The id of the Activity the View is in.
+    - bid: id of the buffer.
+- refreshImageView: This should be called after blitting an ImageView's buffer to show the new contents.
+  - Parameters:
+    - id: The View id of a ImageView.
+    - aid: The id of the Activity the View is in.
 
 
 
@@ -135,30 +207,42 @@ Event control:
 
 Once you have opened an Activity and placed all Views and configured it, like mit GUI applications you have to wait for user input.  
 Events arrive on the event socket.  
-Certain events are send automatically, like Activity lifecycle events and click events for buttons.  
+Events that are enabled by default:
+- click for Buttons
+- 
 
 Event types:  
 - [Input Events](https://developer.android.com/guide/topics/ui/ui-events#EventListeners) :
+  - values for all:
+    - id: The id of the View that fired the event.
+    - aid: The id of the Activity the View is in.
   - click
+    - Additional values for ImageView: x and y position in the view
+    - Additional values for RadioButton and checkbox: set: whether the View is set or not
   - longClick
   - focusChange
+    - Additional value: focus: whether or not the View now has focus.
   - key
+    - Additional value: key: the key that was pressed.
   - touch
+    - Additional values x,y: Coordinates in the View.
+  - refresh: Refresh triggered in a SwipeRefreshLayout
 - [Activity Lifecycle](https://developer.android.com/guide/components/activities/activity-lifecycle) :
+  - values for all:
+    - aid: The id of the Activity.
   - start
-    - value: null
   - resume
-    - value: null
   - pause
-    - value: null
+    - value: whether or not the Activity is finishing
   - stop
-    - value: null
+    - value: whether or not the Activity is finishing
   - destroy
     - value: whether or not the Activity is finishing
 - Custom events:
   - recitem: The plugin needs an item for a RecyclerView
     - value:
       - rec: the RecyclerView View id
+      - The id of the Activity the View is in.
       - it: the index of the item required
 
 

README.md

@@ -0,0 +1,61 @@
+# Termux:GUI
+
+This is a plugin for [Termux](https://github.com/termux/termux-app) that enables command line programs to use the native android GUI.  
+
+In the examples directory you can find demo videos and sample code.
+
+[Installation notes](https://github.com/termux/termux-app#installation)  
+There is currently no release on f-droid, so the only method to install this plugin is to compile it and termux yourself.  
+Releases on f-droid will be provided as soon as possible.  
+
+Protocol.md describes the Protocol used and the available functions you can use.
+Using this plugin requires a bit of knowledge of the android GUI system. You can read about it in detail in the [official documentation](https://developer.android.com/guide).  
+Relevant documentation is also linked in Protocol.md for more specific subjects.  
+Here is a crash-course:
+
+#### Tasks
+
+Tasks are the individual screen you can switch between. Each Task has a stack of Activities, called the back stack, with the top one being visible.
+
+#### Activities
+
+Activities are the individual screens of apps, like a home screen, a settings screen etc.  
+When an Activity finishes itself, it is removed from the Activity stack of its Task, showing the underlying Activity.  
+An Activity can also launch another Activity in the Task, adding it to the back stack on top of itself.  
+To not let the device to unneeded work like drawing Elements that aren't visible, the System informs Activities of certain changes.  
+These are the Activity lifecycle events.
+
+#### Activity Lifecycle
+
+The most important lifecycle events for you will be onDestroy, onStart and onStop.
+When an Activity is started, it is visible. An Activity is destroyed if the user finishes it (by dismissing it), when it finishes itself or when a configuration change occurs.  
+The plugin will tell you if the Activity is finishing when onDestroy is fired, so you know the Activity will be closed.  
+Configuration changes are handled for you, so you don't need to worry about onDestroy if the Activity is not finishing.
+
+#### Views
+
+Views are the Elements that are displayed in Activities.  
+They are divided into Views and Viewgroups, which can themselves contain Views or Viewgroups.  
+That results in a hierarchy of Elements, with one at the top.  
+The Viewgroups are Layouts that position the contained Views according to your configuration.  
+
+The most used Layout is LinearLayout. It simply displays Views in a horizontal or vertical list, giving each Element the same space by default.  
+TextViews can display text.  
+EditText can get text input from the user.  
+Buttons can be clicked.  
+ImageViews can display images.  
+
+With these fundamentals you can go ahead and use this plugin.  If you need more sophisticated Views or Layouts look into Protocol.md for what's available.
+
+
+## Language Bindings
+
+### Python
+
+
+
+
+### C/C++
+
+
+

TODO

@@ -1,16 +1,64 @@
 
 
-- test if requireActivity works while the activity is being recreated
-
-
+[ ] implement the rest of the already specified methods
 
 
+[ ] Activity lifecycle events
+[ ] support for locking orientation: Activity.setRequestedOrientation(ActivityInfo.SCREEN_ORIENTATION_LOCKED);
+[ ] PiP events (enter/exit, onUserLeaveHint)
+[ ] system overlay window support
+[ ] Action Bar with elements
+[ ] App widgets
+    [ ] when the user places a widget, generate a ID and put it in the clipboard. The Widget can then be manipulated with the ID.
+[ ] Notifications with custom Layout
+[ ] method to get the screen resolution, to change layout if needed
+[ ] on screen orientation change event
+[ ] support for divider between elements in LinearLayouts an RecyclerViews
+[ ] Gesture events
+[ ] Multitouch
+[ ] setting visibility to hidden or gone
+[ ] Lockscreen activities: https://stackoverflow.com/questions/60732308/android-displaying-view-over-the-lockscreen-like-google-maps
+    [ ] demo program: Write notes in an EditText, they get placed in a file.
 
 
 
+[ ] finish Python bindings when the plugin is available on f-droid and upload them to pypi
 
+Implement Views:
+[ ] RelativeLayout
+[ ] Button
+[x] ImageView
+[x] Space
+[ ] NestedScrollView
+[ ] RecyclerView
+    [ ] use the paging library to get good performance. When a new page is needed, send an event to the program.
+        Then the program has to send back a list of json layouts to populate the view.
+        To update existing items, the program can send a page unrequested.
+[ ] RadioButton
+[ ] RadioGroup
+[ ] Checkbox
+[ ] ToggleButton
+[ ] ImageButton
+[ ] Spinner
+[ ] SwipeRefreshLayout
+[ ] GridLayout
+[ ] ProgressBar
+[ ] ViewPager2
+[ ] CardLayout
+[ ] TabLayout
+[ ] Tab
+[ ] WebView
 
 
+Implement Events:
+[ ] Methods to set and unset Listeners
+[ ] Click
+[ ] LongClick
+[ ] FocusChange
+[ ] Key
+[ ] Touch
+[ ] Refresh (SwipeRefreshLayout)
+[ ] 
 
 
 

app/build.gradle

@@ -4,16 +4,17 @@ plugins {
 }
 
 
+
 android {
     compileSdk 31
 
     defaultConfig {
         applicationId "com.termux.gui"
-        minSdk 26
+        minSdk 24
         targetSdk 31
         versionCode 1
         versionName "1.0"
-        
+
     }
     signingConfigs {
         debug {
@@ -39,10 +40,11 @@ android {
 dependencies {
 
 
-    implementation 'com.google.code.gson:gson:2.8.8'
+    implementation 'com.google.code.gson:gson:2.8.9'
     implementation 'androidx.appcompat:appcompat:1.3.1'
-    implementation 'org.jetbrains.kotlinx:kotlinx-coroutines-android:1.5.0'
-
+    implementation 'org.jetbrains.kotlinx:kotlinx-coroutines-android:1.5.2-native-mt'
+
+
     implementation 'com.github.flipkart-incubator.proteus:proteus-core:5.0.1'
     implementation 'com.github.flipkart-incubator.proteus:gson-adapter:5.0.1'
     implementation 'com.github.flipkart-incubator.proteus:cardview-v7:5.0.1'

app/src/main/AndroidManifest.xml

@@ -20,7 +20,14 @@
 
         <activity
             android:name=".GUIActivity"
-            android:exported="false" />
+            android:exported="false"
+            android:supportsPictureInPicture="true"
+            android:configChanges="screenSize|smallestScreenSize|screenLayout|orientation"/>
+        <activity
+            android:name=".GUIActivityDialog"
+            android:exported="false"
+            android:theme="@style/TermuxAlertDialogStyle"
+            android:excludeFromRecents="true"/>
 
         <service
             android:name=".GUIService"