Screentext Sensor
The screen reader, known as the “screentext” sensor, captures a variety of text information during user interactions. This includes the text that users browse, the coordinates of this text on the screen, and the actions they perform while interacting with it. It’s important to highlight that the screen reader doesn’t solely focus on the text visible on the screen but also extends its monitoring to text within structured widgets that might not be currently displayed. This sensor does not record password inputs.
A Checklist of Password Detection for Screentext Sensor:
Application passwords may be captured by screen readers if the show-password functionality is enabled by the smartphone user. We have tested this functionality on the following widely-used applications, and the results are as follows:
Safe (Password will not be captured by screen reader when show-password is enabled)
Temu, myGov, WhatsApp, ChatGPT, Shenin, Microsoft, Paramount+, BWS, CapCut, Netflix, Commonwealth Bank, Duolingo, DoorDash, Uber, Afterpay, PayPal, LinkedIn, Seek, Officeworks, Woolworths, Zoom, Optus, Discord, Airbnb, eBay, Virgin, NAB, X, ATO, 9Now, AF App, Westpac, ANZ
Not Safe (Password will be captured by screen reader when show-password is enabled)
Instagram, TikTok, Messenger, Facebook, Amazon, Reddit, Google, Spotify, DiDi
This list provides an overview of which applications protect user passwords from screen readers when the show-password feature is activated and those that do not.
Settings
- Aware_Preferences.SCREENTEXT: true or false to activate or deactivate the sensor.
- Aware_Preferences.PACKAGE_SPECIFICATION: a value specified to collect the data in inclusive applications, exclusive applications or default all applications. Value 0 means to only track data of the inclusive applications; value 1 means to only track data except the exclusive packages; value 2 means to collect data from all applications
- Aware_Preferences.PACKAGE_NAMES: Package names for either inclusive or exclusive applications. The package names for the applications should be separated by a comma or space.
ScreenText.ACTION_SCREENTEXT_DETECT: text detected on the browsing page shown on the screen.
ScreentextData.CONTENT_URI
content://com.aware.provider.screentext/screentext
| Table field | Field type | Description |
|---|---|---|
| _id | INTEGER | primary key, auto incremented |
| timestamp | REAL | unixtime milliseconds since 1970 |
| device_id | TEXT | AWARE device UUID |
| class_name | TEXT | the name of the widget class |
| package_name | TEXT | the package name for the application that is shown in the foreground |
| text | LONGTEXT | Basically, the text on the screen. The screen displays various text elements from different widgets, each of which is presented in a structured format that combines the text and its corresponding coordinates. This representation encompasses not only the visible on-screen text but also text elements within the same widget tree structure that may not currently appear on the screen. Each such combination of text and coordinates is referred to as a “text pack” and multiple text packs are linked together using ||. For instance, the string “Privacy StatementRect(129, 1802 – 330, 1601)||Website Terms and Conditions***Rect(346, 1802 – 684, 1601)” consists of two distinct text packs. |
| user_action | INTEGER | the type of user action triggered by the user. 0=ACTION_DOWN, 1=ACTION_UP, 2=ACTION_MOVE,3=ACTION_CANCEL. More can be found at: MotionEvent |
| event_type | INTEGER | the type of event triggered by the user. 1=TYPE_VIEW_CLICKED, 2048=TYPE_WINDOW_CONTENT_CHANGED, 4096=TYPE_VIEW_SCROLLED,32=TYPE_WINDOW_STATE_CHANGED, 64=CONTENT_CHANGE_TYPE_STATE_DESCRIPTION. More can be found at: TYPE_VIEW_CLICKED |