Sound effects and music

1. Sound effects and music Java Sound API: javax.sound.sampled • 8- or 16-bit samples • from 8,000Hz to 48,000Hz • mono or stereo • sound file formats: AIFF, AU, WAV (Examples: 16-bit, mono, 44,100Hz, WAV)

2. Loading and playing a sound try { File file = new File(“…”); AudioInputStream stream = AudioSystem.getAudioInputStream(file); AudioFormat format = stream.getFormat(); } catch(Exception ex) { // IOException or UnsupportedAudioFileException } try { DataLine.Info info = new DataLine.Info(SourceDataLine.class,format); line = (SourceDataLine) AudioSystem.getLine(info); line.open(format,bufferSize); } catch(Exception ex) { // LineUnavailableException }

3. try { int numBytesRead = 0; while (numBytesRead != -1) { numBytesRead = stream.read(buffer, 0, buffer.length); if (numBytesRead != -1) { line.write(buffer, 0, numBytesRead); } } } catch (Exception ex) { // IOException }; line.drain(); line.close();

4. Sound filter Examples: • echo filter • simulated 3D sound filter SoundFilter class (abstract class): • filter(byte[] buffer, int offset, int length) – Filters an array of samples. • getRemainingSize() – Gets the remaining size, in bytes, that this filter can play after the sound is finished. • reset() – Resets the filter so that it can be used again on a different sound.

5. FilteredSoundStream public class FilteredSoundStream extends FilterInputStream { … public int read(byte[] samples, int offset, int length) throws IOException { int bytesRead = super.read(samples, offset, length); if (bytesRead > 0) { soundFilter.filter(samples, offset, bytesRead); return bytesRead; }; if (remainingSize == REMAINING_SIZE_UNKNOWN) { remainingSize = soundFilter.getRemainingSize(); remainingSize = remainingSize / 4 * 4; };

6. if (remainingSize > 0) { length = Math.min(length, remainingSize); for (int i=offset; i<offset+length; i++) { samples[i] = 0; }; soundFilter.filter(samples, offset, length); remainingSize-=length; return length; } else { return -1; }; } }

7. Delay Original sound First echo Second echo Echo filter • delay: numbers of samples to delay (44,100Hz sound, 1 sec delay = 44,100 samples) • decay: value from 0 to 1 • 0 means no echo • 1 means the echo is the same volume as the original sound

8. Emulating 3D sound Many different effects are used to create 3D sounds. • Make sound diminish with distance so the farther away a sound source is, the quieter it is. • Pan sounds to the appropriate speaker. • Apply room effects so sound waves bounce off walls, creating echoes and reverberation. • Apply the Doppler effect so a sound source movement affect its pitch.

9. SoundManager The SoundManager class has the following features: • provides a common interface for loading and playing sounds (including filters), • extends the ThreadPool class, • each thread in the thread pool has its own buffer and line object (thread-local variables).

10. Playing music Java Sound API provides MIDI sound capabilities in javax.sound.midi • synthesizes MIDI music through the use of a soundbank, • Java SDK includes a minimal-quality soundbank, • higher-quality soundbanks from http://java.sun.com/products/java-media/sound/soundbanks.html To play MIDI music, you need two objects: • Sequence object containing the data • Sequencer sending the Sequence to the MIDI synthesizer

11. Sequence sequence = MidiSystem.getSequence(new File(“…”)); Sequencer sequencer = MidiSystem.getSequencer(); sequencer.open(); sequencer.setSequence(sequence); sequencer.start(); Adding or taking away an instrument (track): sequencer.setTrackMute(trackNum, true);

12. Network programming Socket-based communication (java.net.*) • server sockets (class ServerSocket) • each server socket listens at a specific port • the server must be running before its clients initiate contact • after the sever socket is contacted by a client, a connection can be established • constructor ServerSocket(int port) • client sockets (class Socket) • on the client side, constructors • Socket(String host, int port) • Socket(InetAddress address, int port) • on the server side • returned by the accept method of the server socket • have an input and output stream

13. Example (Server) ... try { ServerSocket server = new ServerSocket(10997); while (running) { Socket client = server.accept(); InputStream in = client.getInputStream(); OutputStream out = client.getOutputStream(); ... // handle the client ... in.close(); out.flush(); out.close(); client.close(); }; server.close(); } catch (Exception e) { e.printStackTrace(); }; ...

14. Example (Client) ... try { Socket socket = new Socket(host,10997); InputStream in = socket.getInputStream(); OutputStream out = socket.getOutputStream(); ... // Send and receive data ... in.close(); out.flush(); out.close(); socket.close(); } catch (Exception e) { e.printStackTrace(); }; ...

15. Network programming (cont’d) The server in the previous example handles one client at a time. • just one client accepted at a time, • operations on streams are blocking, Solution • use multiple threads (one for each client)

16. Example (Server) ... try { ServerSocket server = new ServerSocket(10997); while (running) { Socket client = server.accept(); new ClientHandler(client).start(); }; server.close(); } catch (Exception e) { e.printStackTrace(); }; ...

17. public class ClientHandler extends Thread { private Socket client; public ClientHandler(Socket client) { this.client = client; } public void run() { try { InputStream in = client.getInputStream(); OutputStream out = client.getOutputStream(); ... // handle the client ... in.close(); out.flush(); out.close(); client.close(); } catch (Exception e) { e.printStackTrace(); }; } }

18. Example public class ChatterServer { public static Vector clients = new Vector(); public static void main(String[] args) throws Exception { try { ServerSocket server = new ServerSocket(10997); while (true) { Socket client = server.accept(); ClientHandler clientHandler = new ClientHandler(client); clients.add(clientHandler); clientHandler.start(); }; } catch (Exception e) { e.printStackTrace(); }; } }

19. Example public class ClientHandler extends Thread { private Socket client; private String clientName; private BufferedReader in; private PrintWriter out; public ClientHandler(Socket client) { this.client = client; try { in = new BufferedReader(new InputStreamReader( client.getInputStream())); out = new PrintWriter(new OutputStreamWriter( client.getOutputStream())); } catch (Exception e) { e.printStackTrace(); }; }

20. Example public String getClientName() { return clientName; } public synchronized void sendMessage(String msg) { if (out != null) { out.println(msg); out.flush(); }; } public void broadcastMessage(String msg) { for(Iterator it = ChatterServer.clients.iterator(); it.hasNext();) { ClientHandler ch = (ClientHandler) it.next(); if (ch != this) ch.sendMessage(msg); }; }

21. Example public void run() { if (in != null && out != null) { ... while (true) { String str = in.readLine(); if (str == null) break; if (str.trim().equals("BYE")) break; broadcastMessage("From " + clientName + ": " + str); }; broadcastMessage("User " + clientName + " has closed connection."); client.close(); ChatterServer.clients.remove(this); ... }

22. Class InetAddress This class represents internet addresses: • InetAddress addr = InetAddress.getByName("sandcastle")); • byte[] b = {(byte) 139, (byte)57, (byte)96, (byte)6}; InetAddress addr = InetAddress.getByAddress(b)

23. JDK 1.4 NIO Libraries Channels (java.nio.channels.*) • Interfaces • Channel public void close(); public boolean isOpen(); • ReadableByteChannel and WritableByteChannel public int read(ByteBuffer dst); public int write(ByteBuffer src); • ByteChannel • GatheringByteChannel and ScatteringByteChannel long write(ByteBuffer[] srcs); long write(ByteBuffer[] srcs, long offset, long length); long read(ByteBuffer[] dsts); long read(ByteBuffer[] dsts, long offset, long length);

24. NIO (cont.) • Classes • ServerSocketChannel ServerSocketChannel sSockChan; sSockChan = ServerSocketChannel.open(); sSockChan.configureBlocking(false); InetAddress addr = InetAddress.getLocalHost(); sSockChan.socket().bind(new InetSocketAddress(addr,PORT)); • SocketChannel • SocketChannel has all the methods of ServerSocketChannel • methods for reading and writing • methods for managing the connection

25. NIO (cont.) sSockChan.accept(); (server) SocketChannel.open(SocketAddress address); (client) or channel = SocketChannel.open(); channel.connect(SocketAddress address); • DatagramChannel • using UDP (User Datagram Protocol) instead of TCP • UDP is an unreliable protocol.

26. mark limit 0 1 2 3 4 5 6 7 8 9 position capacity NIO (cont.) Buffers • Classes • ByteBuffer • CharBuffer • DoubleBuffer • FloatBuffer • IntBuffer • LongBuffer • ShortBuffer • MappedByteBuffer • methods for querying and manipulating the capacity, position, limit, and mark of the buffer

27. limit 0 1 2 3 4 5 6 7 8 9 position capacity limit 0 1 2 3 4 5 6 7 8 9 position capacity limit 0 1 2 3 4 5 6 7 8 9 position capacity NIO (cont.) FileChannel.read(ByteBuffer src, int o, int length); SocketChannel.write(ByteBuffer src); • clear() – preparing a buffer for filling • flip() – preparing the buffer for draining

28. limit 0 1 2 3 4 5 6 7 8 9 position capacity limit 0 1 2 3 4 5 6 7 8 9 position capacity NIO (cont.) • rewind() – preparing the buffer for another draining • compact() – moves the elements between the current position and the limit to the beginning of the buffer • direct versus nondirect buffers

29. Selector and SelectionKey Provide a mechanism for multiplexing access to channels. • register a channel (along with a set of operations that the selector should watch for) chan.register(readSelector, SelectionKey.OP_READ, new StringBuffer()); • OP_ACCEPT • OP_CONNECT • OP_READ • OP_WRITE • watching the channels readSelector.selectNow(); • select() blocks until at least one channel has activity • select(long timeout) as select() with timeout • selectNow() returns immediately

30. Selector (cont.) • accessing the channels readSelector.selectedKeys(); • returns a Set of SelectionKeys • use an Iterator to get the keys • key.channel() returns the channel

31. Example: ChatterBox ChatterServer is a multi-user chat application that allows for any number of users to connect to the server and send text messages to one another. The server needs to perform the following main functions: • Accept client connections • Read messages from clients • Write messages to clients • Handle disconnections (graceful or otherwise)

32. ChatterServer • Important attributes: private ServerSocketChannel sSockChan; private Selector readSelector; private LinkedList clients; private ByteBuffer readBuffer; private ByteBuffer writeBuffer; private CharsetDecoder asciiDecoder; private Logger log = Logger.getLogger(ChatterServer.class);

33. private static final int PORT = 10997; private void initServerSocket() { try { // open a non-blocking server socket channel sSockChan = ServerSocketChannel.open(); sSockChan.configureBlocking(false); // bind to localhost on designated port InetAddress addr = InetAddress.getLocalHost(); sSockChan.socket().bind( new InetSocketAddress(addr, PORT)); // get a selector for multiplexing the client channels readSelector = Selector.open(); } catch (Exception e) { log.error("error initializing server", e); } }

34. private void acceptNewConnections() { try { SocketChannel clientChannel; while ((clientChannel = sSockChan.accept()) != null) { addNewClient(clientChannel); log.info("got connection from: " + clientChannel.socket().getInetAddress()); sendBroadcastMessage("login from: " + clientChannel.socket().getInetAddress(), clientChannel); sendMessage(clientChannel, "\n\nWelcome to ChatterBox, there are " + clients.size() + " users online.\n"); sendMessage(clientChannel, "Type 'quit' to exit.\n"); } } catch (IOException ioe) { log.warn("error during accept(): ", ioe); } catch (Exception e) { log.error("exception in acceptNewConnections()", e); } }

35. private void addNewClient(SocketChannel chan) { // add to our list clients.add(chan); // register the channel with the selector // store a new StringBuffer as the Key's attachment for // holding partially read messages try { chan.configureBlocking( false); SelectionKey readKey = chan.register(readSelector, SelectionKey.OP_READ, new StringBuffer()); } catch (ClosedChannelException cce) { } catch (IOException ioe) { } }

36. private void sendMessage(SocketChannel channel, String mesg) { prepWriteBuffer(mesg); channelWrite(channel, writeBuffer); } private void sendBroadcastMessage(String mesg, SocketChannel from) { prepWriteBuffer(mesg); Iterator i = clients.iterator(); while (i.hasNext()) { SocketChannel channel = (SocketChannel)i.next(); if (channel != from) channelWrite(channel, writeBuffer); } } private void prepWriteBuffer(String mesg) { writeBuffer.clear(); writeBuffer.put(mesg.getBytes()); writeBuffer.putChar('\n'); writeBuffer.flip(); }

37. private void channelWrite(SocketChannel channel, ByteBuffer writeBuffer) { long nbytes = 0; long toWrite = writeBuffer.remaining(); // loop on the channel.write() call since it will not necessarily // write all bytes in one shot try { while (nbytes != toWrite) { nbytes += channel.write(writeBuffer); try { Thread.sleep(CHANNEL_WRITE_SLEEP); } catch (InterruptedException e) {} } } catch (ClosedChannelException cce) { } catch (Exception e) { } // get ready for another write if needed writeBuffer.rewind(); }

38. private void readIncomingMessages() { try { readSelector.selectNow(); Set readyKeys = readSelector.selectedKeys(); Iterator i = readyKeys.iterator(); while (i.hasNext()) { SelectionKey key = (SelectionKey) i.next(); i.remove(); SocketChannel channel = (SocketChannel) key.channel(); readBuffer.clear(); long nbytes = channel.read(readBuffer); if (nbytes == -1) { log.info("disconnect: " + channel.socket().getInetAddress() + ", end-of-stream"); channel.close(); clients.remove(channel); sendBroadcastMessage("logout: " + channel.socket().getInetAddress() , channel); }

39. else { StringBuffer sb = (StringBuffer)key.attachment(); readBuffer.flip( ); String str = asciiDecoder.decode(readBuffer).toString(); readBuffer.clear( ); sb.append( str); String line = sb.toString(); … sendBroadcastMessage(channel.socket().getInetAddress() + ": " + line, channel); … } }

40. ChatterClient The ChatterClient needs to perform three tasks: • Connect to the server • Send messages • Receive messages The client uses two threads, one for console input/writing and the main execution thread.

41. private void connect(String hostname) { try { readSelector = Selector.open(); InetAddress addr = InetAddress.getByName(hostname); channel = SocketChannel.open(new InetSocketAddress(addr, PORT)); channel.configureBlocking(false); channel.register(readSelector, SelectionKey.OP_READ, new StringBuffer()); } catch (UnknownHostException uhe) { } catch (ConnectException ce) { } catch (Exception e) { } }