In MSAB, working with all widely used apps is part of our job. To make sure that our software product is able to decode all extracted information stored in devices seized by police forensics department, we have to run tests to guarantee that we can handle all released versions for a certain app.
App Decoder Test Procedure:
Here is how we run a test manually, when we try to test for a new app version.
Step 1: Generate test data in a device
Step 2: Extract data from device
Step 3: Verify XRY is working well for decoding extracted data
To automate this test, we have written scripts to do the job.
For step 2 and 3, the automation work is relatively easy, for those steps are more standardized.
For step 1, we are facing challenges.
General speaking, a program that can automate Step 1 should be able to use an app as a human. When using a chat app, the program can find out where the text box is to type and send text messages, switch view and press buttons to make calls. Here, we are not requiring the program to talk as a human. This is too advanced. We rather just need a program to learn how to use an app to generate test data in devices.
It would be easy if an app never change its UI. We could just use some basic image recognition techniques to recognize buttons, text box, contact names that we need to handle. Of course, we also need to know how to programmatically press buttons on a device screen, type words, verify data has been received by other device, etc. Most of these problems have already been solved.
Assuming, for an app’s latest version, a button icon changed slightly. Then we need a more robust image recognition algorithm. Then the problem is, in that case, what threshold value we need to choose for matching images. Will our experience-based threshold be always working?
A possible solution is an AI that can learn how to use an app by itself.
Why is this possible?
- To read a device screen for an AI is not too hard, comparing to other real life objects. We have started some image recognition work.
- We have historical app installation files, which store icon images with its meaningful file name. With historical installation files, we can also generate training data.
- By our previous work, we already know how to programmatically press buttons on a device screen, type words and verify data that has been received by another device.
- A little demo of our bot to show why it’s possible.
Using image recognition create an program that can automatically test apps.
We are looking for an engaged student who is in his or her final year of his Bachelor studies or equivalent.
Programming skills in C++, Python or similar. Machine learning Knowledge. Image processing Knowledge
Proposed timeline for the Thesis period is 10 weeks or 20 weeks of full-time study, equivalent to 15 or 30 academic points respectively, culminating in a final report.
Okay, I am interested. What do I do now?
Do not hesitate to contact us if you are interested in exploring this opportunity.
Please contact firstname.lastname@example.org should you have any questions.