Welcome to EQ2430/EQ2440 Android Lecture

1. Welcome toEQ2430/EQ2440 Android Lecture March 20, 2014 Per Zetterberg School of Electrical Engineering

2. The project course goes Android! Features used in our projects: • Display : plots and text. • Microphone. • Loudspeaker. • Magnetometer measurements. • WiFi RSSI measurements. • Send data using WiFi. • Accelerometers. • Touch screen. • Camera

3. FrameWork FrameWork is a java project (program) in Eclipse. FrameWork is a Android 4.2.1 project that runs (at least) Samsung S3. FrameWork is a generic skeleton which can be used as the starting point for many student projects (including yours!). FrameWork has an object StudentCode which is where you implement your prototype. Application name: StudentFrameWork (Change this) Creators: Martin Ohlsson and Per Zetterberg

4. FrameWork: User interface PlotView: Graphs and/or Camera picture TextView: Textual output.

5. Downloading FrameWork Go to EQ2440 web (KTH social) => Android support Download: FrameWork.zip Unzip and move the code to workspace directory. Do “file->import->”existing project into workspace”->browse to FrameWork (the directory)

6. StudentCode (i.e your code) Parameters: useSensors, loggingOn, logSound, useCameraGUI, useMessaging, messageServer, processInterval, introText, messageGroups,userInputString, StudentMessage (class), userInputString useSensors (bitfield): “xor” the following GPS,CAMERA,CAMERA_RGB,SOUND_OUT,SOUND_IN,WIFI_SCAN,ACCELEROMETER,MAGNETIC_FIELD,PROXIMITY,TIME_SYNC, PROXIMITY,LIGHT,SOUND_IN,CAMERA, GYROSCOPE

7. Called by StudentCode 1(3) sound_out(short[] buffer, int length): Send data to DAC=>loudspeaker. message_out(StudentMessage message): Send message to all the other phones. message_out(StudentMessage message, String groupIdString): Send message to the phones in group defined by “groupIdString. streaming_buffer_out(short[] buffer, int length, String groupIdString): Send buffer of int16 to the phones in group defined by “groupIdString”.

8. Called by StudentCode 2(3) set_output_text(String text): Set the text too be displayed in TextView. plot_camera_image(Canvas plotCanvas, byte[] image, int imageWidth,int imageHeight, int width, int height): Display a camera image. plot_camera_image_rgb(…): Display color … write_string_on_logfile(String): No ”\n” or ”;”. load_sound_file(String fileName) + play_sound_file(int loop): Repeatedly plays the wav-file specified by fileName. No timing gitter (loop=-1 means forever) stop_sound_file(): Stops the above.

9. Called by StudentCode 3(3) save_rgb_frame_on_file(String filename,byte[] frame,int width,int height,boolean append) : Saves an RGB picture (=frame) on file. public byte[] read_rgb_frame_from_file(int image_number, String filename) : loads an RGB picture (=frame) from file. public byte[] read_data_from_file(String filename) : Reads binary data from file.

10. Defined by StudentCode: Called by FrameWork 1(2) You fill these functions with code! init(): Do all your initialization. start(): Called when menu button ”start” is pressed. stop (): Called when menu button ”stop” is pressed. message_in(…) Received message from other phone. gps(…) …… position estimate magnetic_field(…) …… magnetic field measurement light(…) : ….Light intensity (lux) event (trigger when changed) proximity(…): Proximity state event (trigger when changed) accelerometer(…) accelerometermeasurement process(),Periodicprocessing with interval set by processInterval parameter. The processInterval parameter can be changed during runtime.

11. Defined by StudentCode: Called by FrameWork 2(2) All are inputs except plot_data! You fill these functions with code! gyroscope(…) gyroscope measurement sound_in(…) …sound input samples wifi_ap(…) ...signal strength of surrounding WiFi APs screen_touched(…) position event streaming_sound_in(…) buffer of samples (int16) received from other phone camera_image(…) image received from camera (black and white) camera_image_rgb() image received from camera (color) plot_data(): Screen update (make plots with Canvas). list_ips(): Return a list (in string format) of the IP:s of the phones in your messaging group. stringFromUser(): Called when the user provides input.

12. Your code structure plot_data(Canvas plotCanvas, { … present results} public void init() { … set parameters initialize} Data structures algorithm state camera_image_rgb() { … save latest image new_image=true; public message_in () { … process … save} public void process() { … e.g heavy calculations }

13. The GUI Start FrameWork App-> calls init() introText Press menu button Press Start “Greeting message” Start() is called Sensors switched +on etc. set_output_text(String text): Set the text too be displayed in TextView. Stop plot_data(Canvas plotCanvas,…) Press menu + Stop stop() is called Start Kill String

14. Get a string from the user 1(2) Start Kill Menu if userInputString=true: userInputStringTitle=“Title” userInputStringMessage=“Message” userInputStringMenuItem=“String” Pressing String gives: Message Title Start Kill String OK Cancel Menu if userInputString=false:

15. Get a string from the user 2(2) User types ”orange” Orange User presses OK The input provided is sent to the Function stringFromUser inside StudentCode.java Title void stringFromUser(String user_input){ } OK Cancel

16. Browse for filename To use it, somewhere inside StudentCode.java do: browseForFile(); This opens a dialog where you can browse for a file (quite intuitive). When the user selects a file and presses ”Done” the function stringFromBrowseForFile is called back. You can capture the filename inside it. stringFromBrowseForFile(String filename) Only the variable filename is set in filename not the path. However, we will only use the sdcard directory.

17. Threads (internals) “Parent thread”: Indirectly calls init(). studentProcessThread: Calls process(). ipListenerThread: Calls message_in() guiTriggerThread: Renders what you provide with set_output_text() and plot_data(). t1: Calls sound_in(…) wifiScanThread: callswifi_ap(…) ipSoundListenerThread: calls streaming_buffer_in FrameWork implements OnRecordPositionUpdateListener: Calls gps() FrameWork implements SensorEventListener accelerometer(), magnetic_field(), light(), proximity() FrameWork extends Activity: sreen_touched() streaming_sound_in(…)buffer of samples (int16) sent from other phone

18. How communication between phones works 1(2) • A phone (or a Linux computer) acts as a server (the address of which is defined by messageServer). • When using a phone as server, that phone is starts FrameWork first. • When FrameWork starts (on the other phones), it contacts the server and asks for the ipadress of one of the other phones in the project (a project is defined by the projectName string e.g. ”yellow”). • The phone then contacts that phone and gets a list of all other phones in the project. • When one phone calls message_out(message) the message is sent to all other phones in the project. • The other phones receives the message through a call of message_in (”callback”). • The message is of type StudentMessage and contains the fields that you decide. • Debugging hint: set_output_text(set_list_ips) lists current configuration.

19. How communication between phones work 2(2) • Second form of message_out: message_out(message, groupIdString) • Where groupIdString defines a group of users e.g. ”data_receivers” which should receive the message. • Which groups exist ? You define the groups with the ”messageGroups” parameters. • When FrameWork starts, it asks the user which group the phone belongs to. • When sending buffers of int16 use the following version: streaming_buffer_out(buffer,length, groupIdString) • This data is received through: streaming_buffer_in( buffer, int length, senderId)

20. How to use communication between phones 1(2) Which message group do this phone belong to ?: messageGroups[myGroupID] Set useMessaging = true; If youneedgroups: • String temp[] = {"MICROPHONE","SPEAKER"}; • messageGroups = temp; Otherwise: • messageGroups = null; In bothcases: • messageServerType = LINUX_MESSAGE_SERVER; • messageServer = "192.168.0.68"; • DefineStudentMessage (defined in StudentMessage.java)

21. How to use communication between phones 2(2) public class StudentMessage extends StudentMessageBase { // Add your message fields here // do not use primitive types that begins with lower case letters Float dummy=0.0F; /* Example of student message data */ Boolean is_sync_message; Boolean is_distance_message; Float value1; Float value2; Float value3; Float value4; } message_out(my_message,SPEAKER”); Same definition in all phones! In transmitter In receiver (SPEAKER) public void message_in(StudentMessage message) { If (sync_message) correlation=value1; …

22. Logging on file 1(2) • loggingOn = true; • Logs measurements from all active sensor exceptcamera • … and youowndebug messages: • write_string_on_logfile(“correlation=“+correlation) • Creates a measurement file called sensorlogYYYYMMDDHHMM.csv on the SDCARD. The measurementfile is ASCI withmeasurentsin clear text. Howretrieve file: method1: Copy usingwindowsexlporer. methods: Usingourmatlabfunctions.

23. Logging on file 2(2): matlab utilities • Copy file to PC-harddisc (matlab command) • copy_file_from_sdcard_to_working_directory.m • Obtain the data from the log-file into matlab: • get_log_data_from_FrameWork.m • Extract data from somecertain sensors: • extract_acceloremeter_from_log_data.m, • extract_sound_from_log_data.m, • extract_magnetic_from_log_data.m • Other sensors: • Do it yourself. Problem: SOUND_IN fluds measurement log. Solution: logSound=false;

24. State-machine and functions Functions that can be tested against matlab implementation using harness

25. Test harness All, non-trivial functions should be tested like this. Purpose: Test implementation of math-intensive functions/algorithms in an isolated and repeatable way. Method: Run your matlab- and java- (with android environment) -implementations side by side with exactly the same input . How? • Matlab produces data, and pushes it on the sdcard • FrameWork includes a function called ”test_harness()”. • Inside this function • the data can be read from the sdcard, • the function to be tested is called, • and the results are saved on the sdcard. • Matlab then pulls the data from sdcard. • Finally, the results from the matlab- and java-implementation are compared. What do I needto do ? The example test_harness_example.m shows how to automate the process. This code can be used as a starting point, however, you need to customize the code to the function you are going to test.

26. Using sound (input) useSensors = SOUND_IN | SOUND_OUT; sampleRate = 44100; 4096 samples 4096 samples 4096 samples time sound_in(long time, short[] samples, int length) sound_in(long time,…

27. Sound output 1(2) sound_out(short[] buffer, int length) sound_out(short[] buffer, int length) sound_out(short[] buffer, int length) -2^15,2^15-1

28. Sound output 2(2) wavwrite(y,44100,16,’bing.wav’); copy_file_from_working_directory_to_sdcard(‘bing.wav’); load_sound_file("bing.wav"); play_sound_file(-1); stop_sound_file(); Only in init(), process() and stop() Constant gaps

29. Create sound in matlab and play it on phone In matlab sound: [-1,1]. In android FrameWork: [-2^15,2^15-1] In Matlab • [y,Fs] = wavread('C:\Downloads\lga.wav'); • create_file_of_shorts('lga.dat', y(20001:1e6,1)*1000 ); • copy_file_from_working_directory_to_sdcard('lga.dat' ) On the Phone: Seelines in in StudentCode.java with the word ”sound”

30. RGB Camera: Image format void camera_image_rgb(byte[] image, int width, int height) 0 1 3 image={r0,g0,b0,r1,g1,b1,r2,g2,b2,…} Byte 0-255

31. RGB Camera: File format File format (”.rec”): Frame Frame N Frame 2 Frame 1 Always stored in /sdram/recordings Filename (example): video201303181626.rec width height r0,g0,b0,r1,g1,b1,….

32. RGB_Camera: GUI Stop Play Rec Start void camera_image_rgb(byte[] image, int width, int height) Log to file: /recordings/videoyyyyMMddHHmm.rec Browse for file and playback: Calls camera_image_rgb(byte [] image, int width, int height) Stop recording Alt1: useSensors = CAMERA_RGB; useCameraGUI=true;

33. RGB_Camera:No GUI • Camera runs continuously. void camera_image_rgb(byte[] image, int width, int height) ….is called…. save pictures to file using call: save_rgb_frame_on_file(String filename,byte[] frame,int width,int height,boolean append) Alt1: useSensors = CAMERA_RGB; useCameraGUI=false;

34. Load RGB picture from file int read_rgb_frame_width_from_file(int image_number, String filename) int read_rgb_frame_height_from_file(int image_number, String filename) byte[] read_rgb_frame_from_file(int image_number, String filename)

35. Display picture (example) Updates ≈10Hz plot_data(Canvas plotCanvas, int width, int height) { if((latestRGBImage != null) && ((useSensors & CAMERA_RGB) == CAMERA_RGB)) { plot_camera_image_rgb(plotCanvas, latestRGBImage,ImageWidth,imageHeight,width,height); };};

36. Pictures Matlab interface A 768x432x3 matrix (:,:,1)=red (:,:,2)=green (:,:,3)=blue list=list_video_recordings_on_sdcard(time0) copy_video_recording_file_from_sdcard_to_working_directory(filename) rgb_video= extract_video_recording_from_file(filename); image(rgb_video.frames{1}.rgb) create_video( filename, rgb_video ) rgb_video.frames{1}.rgb

37. Display pictures another way void plot_data(Canvas plotCanvas, int width, int height) { plotCanvas.drawLine(… plotCanvas.drawRect(… … } More info: http://developer.android.com/reference/android/graphics/Canvas.html

38. ZXING QR decoder “ZXing (pronounced "zebra crossing") is an open-source, multi-format 1D/2D barcode image processing library implemented in Java” To test: useSensor=CAMERA_RGB; uncomment code in StudentCode.java between ”zxing start color” and ”zxing stop color” Direct camera towards QR code Source code located under ExternalSrc.com.google.zxing

39. Give your App a unique identity Right click on project folder (Top level folder named FrameWork) within the Package Explorer window Select Android Tools/Rename Application Package Add a unique name after FrameWork like: se.kth.android.FrameWork.GroupRed2013 Press Finish in the Refactoring window Confirm update of Launch Configurations Voila, you now have a separately installable version of your app!

40. Phone configuration Samsung S3 ”Settings” -> ”developer options” -> tick ”USB debugging” ”Settings” -> ”Security and lock screen” -> tick ”Unknown sources” HTC Desire ”Settings” -> ”Applications” -> ”Development” -> tick ”USB debugging” ”Settings” -> Applications” -> tick ”Unknown sources” When connecting to PC: choose ”charge only” Eduroam (both): http://www.lan.kth.se/eduroam/phones/phones_mobile_guide.html

41. code versioning systems CVS, SVN, git, …. purpose: Keep track of created textfiles (e.g. C++ code). Versions. Helps merge changes by multiple users. We will use git: distributed versioning system … but we will used in a centralized manner

42. Set up local repository (=local copy) From the git bash command line: >git clone https://github.com/pzett/blue.gitblue_alice >cd blue_alice >gitconfig –-local user.emailalice@kth.se >gitconfig –-local user.name alice Import the projectintoeclipse: 1) File->Import->Existing AndroindCodeInto Workspace -> Browsetoblue_alice/FrameWork and select it 2) Right clickFrameWorkPackage Explorer in eclipse -> buildpath -> source tab ->Add folder - > Mark ExternalSrc-> OK ok Reading: http://toroid.org/ams/git-central-repo-howto

43. How we will work Repository blue file1.cpp file2.cpp file1.hpp file2.hpp Alice: * Implements or changes file1.cpp and file2.cpp Updates repository: git add file1.cpp file2.cpp git commit -m “comment” Bob: * Implements or changes file1.hpp and file2.hpp Updates repository: git add file1.hpp file2.hpp git commit -m “comment” git push git push git pull git pull file1.hpp file2.hpp file1.cpp file2.cpp

44. Conflict resolution • Alice and Bob have clean updated local repositories. • Alice modifies her file1.cpp • Bob modifies his file1.cpp • Bob does: • git add file1.cpp • git commit -m “change initialization of i1” • git push • Alice does: • git add file1.cpp • git commit -m “I changed initialization of i1 to 2” • git push CONFLICT (content): Merge conflict in file1.cpp Automatic merge failed; fix conflicts and then commit the result.

45. Conflict resolution continued • Alice opens file1.cpp: <<<<<<< HEAD i1=2; ======= I1=1; >>>>>>> 55f223d4ab24ae40f19ae551d51205b69a23177e • Alice and Bob discusses and finds that Alice was right. Alice removes Bobs code and pushes it. Alice> git add file1.cpp Alice> git commit -F .git/MERGE_MSG Alice> git push Bob> git pull • Alice and Bob now has identical local repositories with Alice's code. Alice's code Bob's code

46. Assignments • Write letter ”G” and ”H” in plot_data using “draw” functions. • Switch between the letters of your group color every two seconds. • Record a sound and play it with two echoes. The sound should be read continuously like a reverb i.e. not record->pause->play. • Make an Appthat transmits a sinusoid. The Appshouldfirst ask for the frequency – and then transmit a sound withthatfrequencyusing the loudspeaker. • Make an AppsimilartoAudalyzer (butsimpler). The Appshoulduse a bank of filters centered around 200Hz,300Hz,400Hz,500Hz, and 600Hz. The Appshouldcontinously listen tothesefrequencies (audio input) and make a bar plotof the power at the output of the filters. • Make a test-harness for a non-trivial algorithm which is relevant for your project. Deadline : Mid-term evaluation.

47. Hints • Android software developers kit: http://developer.android.com/sdk/index.html • Killing tasks. Download and install “Advanced Task Killer Free (ATKF)”. To kill e.g. “StudentFrameWork” open ATFK press “StudentFrameWork” until a new menu appears where you can select “Kill” • Error message: “ActivityManager: Warning: Activity not started, its current task has been brought to the front” => Change a single line in the source files code and try again. • Error message:FrameWork_empty_130319] Re-installation failed due to different application signatures => Do “abd uninstall se.kth.android.FrameWork” from command line prompt on PC. • InstallAudalyzerto be abletoview in the input sound.