Ionic, Cordova, Android

The Ionic docs describing development on Android are a bit out of date.
  1. Assuming Java, JDK, and Ant are installed with `ANT_HOME` set to the directory containing `bin/ant`, and `JAVA_HOME` is set.
  2. Install Android Studio.
  3. Run Android Studio, select the `Configure` drop down, select `SDK Manager`, use it to install the relevant SDK (1.7.1 for me) and the `Android API`.
  4. Copy the `Android SDK Location` displayed at the top of the that Stuio window (eg `C:\Users\$username\AppData\Local\Android\Sdk`) into the env var `ANDROID_SDK`.
  5. Add to `PATH` the values `$ANDROID_SDK/tools` and `$ANDROID_SDK/platform-tools`.
  6. Sset mobile to Debugging mode (`Settings` > `About Phone`, and tap `Build number` seven times). Turn on `USB Debugging`.
  7. Connect phone to PC via USB. Grant access rights and install Windows drivers for the mobile phone when prompted/donwload drivers from vendor’s site.
  8. On PC, run `adb devices` and check the mobile is listed, if not visit SO.
  9. Finally, either launch the app by either:
    1. In MS VSC, add `Cordova Tools` and add debug targets in `launch.json`
    2. ionic cordova run android --livereload

HTML5 PCM Waveform Visualisation

The oud riff was recorded at home through an 01X, visualised by some JavaScript code knocked-up in an hour one Monday morning.

One of the many wonderful features of the HTML5 Web Audio API is the ability to access the PCM data of an audio file. Combined with the speed of contemporary machines, this enables an HTML5 web page running in WAA-compatible browser to display the waveform of the playing file in real time – and it is really, really easy to use.

Access to the PCM data is via an ‘analyser’, which is inserted into the audio node chain, and provides an array of floats or unsigned integers for each audio ‘frame.’ The API allows the user to specify the number of samples to return in a frame, and for the purpose of this illustrative example, the default 2048 was replaced with 512, which is plenty for the non-professional eye, and allows for a relatively fast canvas refresh rate.

Access this data at regular intervals, render it to a canvas, and you’re done.

To make things a little more interesting, I thought to maintain a stack of the past n PCM waves, and render them disappearing into the distance. My maths skills are not advanced, and I spent some time messing around with scaling and translation factors applied directly to the stored co-ordinates, before smacking my head for my own stupidity when remembering that the canvas context supplies ‘scale’ and ‘translate’ functions to do this: for each generation in the frame stack, the origin is shifted up, and the scale is decreased, to create a perspective effect.

Update

Deep joy – requestAnimationFrame actually works, so finally setTimeout/.periodical is ditched. I found this shim, and slotted it into place within seconds:

window.requestAnimFrame = (function(){
  return  window.requestAnimationFrame       || 
		  window.webkitRequestAnimationFrame || 
		  window.mozRequestAnimationFrame    || 
		  window.oRequestAnimationFrame      || 
		  window.msRequestAnimationFrame     || 
		  function( callback ){
			window.setTimeout(callback, 1000 / 60);
		  };
})();

Frankly, I see no performance difference, but I am excited the possibilities for games programming.

Next

When dabbling with the scaling factors, I was tempted to shift to WebGL. I may try this next, using three.js. May also render an opaque polygon between generations, to create mountains. Or maybe I’ll just increase the number of generations – and hence their visual density – until the fan gets too loud.

Links

Plonk – Like Plink, in two and a half mornings

Sunday

When looking for something to impplement using HTML5 Web Sockets,
other than a chat server, I cam across , by the Swedish compnay,
DinahMoe, and so ripped-off the idea.

When watching Plink, it seemed obvious to implement it as … a
chat server, in as much a number of users are linked to a
persistent-state server, that relays each users messages to all other
users, in real time. The interesting thing about Plink is that the
messages are musical sounds.

Monday: Relay Server

The first task was to write or find a
Web Socket relay server. Node.js seemed the obvious choice, but
support for Web Sockets in Node.js is patchy, and as I wanted to play
with the latest version of the specification, I had only the
web-socket library to look at. Unfortunately, this failed to build on
OS X, and the support response was ‘your need Xcode,’ which I have
had since buying the Mac.

I thought of trolling through the Maven
repositories for something suitable, but on my way there checked what
Perl had to offer. After getting side-tracked by the usual Perl
half-baked or badly documented implementations, I found that
Mojolicious has a simple WS daemon that can run out of the box. It
took literally five minutes to install the package and have the relay
up and running – a fraction of the time it took to find the
library.

#!/usr/bin/perl 
use strict;
use warnings;

use Mojolicious::Lite;
# use Data::Dumper;

my $clients_tx = {};
my $clients_cursors = {};

websocket '/' => sub {
	my $self = shift;
	my $client_id = sprintf "%s",$self->tx;
	$clients_tx->{$client_id} = { tx =>$self->tx };

	$self->on(message => sub {
		my ($self, $msg) = @_;
		# warn $msg;
		my ($x, $y, $pitch) = split/,/, $msg;
		if (defined $y){
			$clients_cursors->{ $client_id }->{xy} = [$x+0, $y+0];
			$clients_cursors->{ $client_id }->{pitch} = $pitch ne ''? $pitch+0 : undef;
			# warn Dumper( $clients_cursors );
			for my $i (keys %$clients_tx) {
				$clients_tx->{$i}->{tx}->send(
					Mojo::JSON->new->encode({ 
						cursors => $clients_cursors
					})
				);
			}
		}
	});

	$self->on(finish => sub {
		delete $clients_tx->{$client_id};
		delete $clients_cursors->{$client_id};
	});
};
app->start; 

__END__

The code simply receives a CSV of three
numbers representing the cursor position and selected pitch, which it
stores for each connected client. Every time a client sends this
information, the server responds with the latest copy of the
information for all clients.

Prototype 1

 

Screenshot
It’s not easy to take a screenshot whilst playing with two mice.

The first stage of the prototype was boiler plate – setting up
an Apache virtual server, creating directories to hold JS and CSS,
and a basic HTML5 document to pull in Mootools, Modernizer, the blank
application CSS file, and a fresh Mootools class definition, the
latter linked to an element in the body of the page into which the
app could insert itelf.

 

Capturing mouse movement events is straight forward, and once
their offsets are removed, I was ready to send the co-ordinates to
the server. When jotting down the server code, above, I had already
decided to send both co-ordinates and pitch information, as I wasn’t
quite sure how I would implement the functionality of the app, and
wanted to leave my options open. I’ve been a victim of premature
optimisation in the past.

Next I had to find the Web Socket API. There are many examples of
this lying around the net, and it took only a minute to implement the
necessary callbacks, and have the server logging connections and
messages, and the client console logging what it was sending and
receiving.

After adding to my websocket .message callback some code to render
a circle at the co-ordinates specified in the message, I could see a
mess under my cursor whenever I moved it.

Next came my only implementation error – at least, the only one
I have noticed.

Plink shows the current note the user is playing as a circle on
the screen, but also shows past notes as scrolling horizontally away
from the cursor: very pretty. I remember implementing horizontal
scrolling for games on a Vic-20 in the early 1980s, and the fastest
way of doing it then was to shift a block of memory by the number of
bytes to be scrolled, an operation easily and quickly done even then,
by a ten-year old.

HTML5 is not so simple, and the canvas element still lacks a
built-in scrolling function. This left four choices: two involved
dropping the canvas, in favour of either SVG or WebGL; I’ve played
with both, and found basic operation in both to be similarly
straight-forward: in both cases, I would for every ‘note’ create an
object that could be rendered. The other option was to do this for
the canvas, and wipe it between every rendering, or to attempt to
copy the canvas element, wipe the original, and render the copy one
pixel to the left.

My thoughts were that the latter would be the simplest, and
closest to the pattern I had learnt as a kid.

scroll: function(){

var destinationCanvas = this.canvas.clone()

destinationCanvas.cloneEvents( this.canvas, 'mousemove');

var destCtx = destinationCanvas.getContext('2d');

destCtx.drawImage(

this.canvas,

(this.options.scrollPx)*-1,

0

);

destinationCanvas.replaces( this.canvas );

this.canvas.destroy();

this.canvas = destinationCanvas;

this.ctx = destCtx;

},

I hooked the above method up to a timer, via Mootools .periodial
method, and was mildly pleased to see a trail from cursor,
disappearing off the edge of my screen.

Prototype 2

Before the day ended, I wanted to hear sounds, and as I browsed
for API notes and implementation examples, I noticed my fan buzzing,
and Mozilla getting sluggish. I flicked, slowly, through my tabs, and
found the Plink rip-off crawling. I killed the server, closed the
tab, and everything was fine. I still don’t know exactly what the
problem was: I tried minimising the rate messages were sent to the
server, and the frequency of canvas renderings, but in the end went
to have tea, and gave up for the day.

Tuesday

For some reason I felt an attachment to the canvas copying method,
but dropped it all the same, and had the .message handler push the
latest batch of JSON objects onto a stack. I then rewrote the
periodical scroll method to render everything on the stack, making
the latest object appear on the right, decrementing everything by one
cursor width, and dropping from the stack anything that would fall
the screen. This removed the performance problem, and reinforced my
desire for a native canvas.scroll method.

Web Audio API

In my experiments writing an audio sequencer for SoundCloud files,
I learn the limitations of the HTML5 audio element – it is to the
Audio API as the video element is to the Web Graphics Library. So, I
found a decent introduction to the Audio API, on HTML5Doctor, and had
sound within a few minutes, hooked up in a new periodical method. I
had the .send method calculate pitch by computing within which of a
number zones the vertical position of the mouse fell, and then spent
an hour find sounds in Logic, and exporting individual notes as wav
files, loaded into buffers stored in arrays, whose indexes could be
accessed by the ‘pitch’ index sent to the server.

Plink has a percussion track, so I extended the sound-firing
method to play the drums as appropriate:

options: {
…
percussion: {
kick: [[4,1]],
snare: [[8,1], [16,2]],
hat_closed: []
}

}

self.percNames.each( function(instrument){

self.options.percussion[instrument].each( function(i){

console.log( instrument +' '+ self.pulseNumber +' '+ i[0] +' '+i[1]);

if (self.pulseNumber % i[0] == i[1]){

self.playNow( self.percBuffers[instrument] );

self.scaleCursor += self.options.cursorScaleIncrement;

}

});

});

Notice that routine also changes the cursor size, as in Plink, so
the visual echo of the sound reflects the unerlying rhythm.

My children then took over the development machine for beta
testing, and I joined in on the old media centre PC, on which I had
to install Chrome, which seems to be the only limitation of the
project, other than my inability to find a free host for my
web-socket relay code: my current ISP is too cheap to provide this.

I was quite pleased with the effort, and dropped a line to the
address Dinahmoe publicised for job applicants, but no word yet:
perhaps they realise Plink isn’t that hard to do. Whilst I was there
I had a quite look at the Plink source code, and was only mildly
horrified, mainly by the amount of hard-coding, lack of framework,
and lack of white-space.

Wednesday

I then added the ability to change patches, as in Plink, as well
as some options to have the horizontal cursor position equate to
volume and panning – two options the children found far from
intuitive, and quite intrusive into their play.

Also added the ability for patches to only sound if pre-required,
much in the manner of the percussion track.

Node.js

Node.js is beautiful: clearer and more natural than Perl,
faster to write and install than Java, and performs as well
as both. The following Node.js websocket server took ten minutes
to write, based of the stub code that came with the module. The
only change made to the Plond.js client was to add a sub-protocol
argument to the WebSocket instantiation.

#!/usr/bin/env node
var WebSocketServer = require('websocket').server;
var http = require('http');

var cursors = {};

var server = http.createServer(function(request, response) {
    console.log((new Date()) + ' Received request for ' + request.url);
    response.writeHead(404);
    response.end();
});
server.listen(3000, function() {
    console.log((new Date()) + ' Server is listening on port 3000');
});

wsServer = new WebSocketServer({
    httpServer: server,
    autoAcceptConnections: false,
    maxReceivedFrameSize: 50
});

function originIsAllowed(origin) {
  // put logic here to detect whether the specified origin is allowed.
  return true;
}

wsServer.on('request', function(request) {
    if (!originIsAllowed(request.origin)) {
      // Make sure we only accept requests from an allowed origin
      request.reject();
      console.log((new Date()) + ' Connection from origin ' + request.origin + ' rejected.');
      return;
    }

    var connection = request.accept('sec-websocket-protocol', request.origin);
    console.log((new Date()) + ' Connection accepted.');

	var id =  connection.socket._peername.family
		+':'+connection.socket._peername.address
		+':'+connection.socket._peername.port;

    connection.on('message', function(message) {
    		if (message.type === 'utf8') {
            // console.log('Received Message: ' + message.utf8Data);

            var csv = message.utf8Data.split(',');
            cursors[id] = {
            		userId: 		 csv[0],
            		xy: 			 [ parseInt(csv[1]), parseInt(csv[2])],
            		scaleCursor: parseInt(csv[3]),
            		gain:		 parseInt(csv[4]),
            		pain:		 parseInt(csv[5]),
            		patch:		 parseInt(csv[6]),
            		pitch:		 parseInt(csv[7]),
            };
            connection.sendUTF(JSON.stringify({ cursors: cursors} ));
        }
        else if (message.type === 'binary') {
            console.warn('Ignoring received Binary Message of ' + message.binaryData.length + ' bytes');
            connection.close();
        }
    });

    connection.on('close', function(reasonCode, description) {
        console.log((new Date()) + ' Peer ' + connection.remoteAddress + ' disconnected.');
    		delete cursors[id];
	});
});

Finally, I moved the patches into objects, along with their individual pulses,
for ease of legibility and maintenance.

In hindisght, the cross-platform features of MooTools were unncessary,
as the code will only run where Web Audio is available,
which seems to be only Safari and Chrome. Still, I think it does allow
for more readable code, and has no performance impact (unless it was
responsible for that canvas copying slow-down).

Next

  • Have the event loops execute in separate WebWorker threads – as yet, I’ve
    no idea what the threading model is, but I expect it to be as
    frustrating as threading in Perl.
  • Maybe switch to WebGl, just for fun.
  • Find a way of playing like this for money.

Links