SteveX Compiled

My son attends an elementary school that's fairly advanced in what they're offering in the way of classroom technology. The school allows students to bring in their own laptops, has a number of activities that make use of computers, and seems to be doing a good job of integrating this technology into the school environment.

But there's a problem. The school encourages students to bring in their own laptop computers (and I think they have a supply of simple netbooks for any student that doesn't want to bring in their own) but the school board refuses to help students get on the wireless network.

On top of that, the school board requires that student computers have up-to-date antivirus software (if Windows; Mac OS computers are exempt from that requirement). The board has configured the network to require a WPA key to get online, and to use a proxy server for access to the Internet (which isn't unreasonable).

The proxy server is configured to check that the computer meets the antivirus requirement before letting the user log in to the network. This check is done by way of a software agent that the student must download and install onto their own computer. The agent listens on a number of TCP ports and reports to the proxy server whether the computer is "safe" and can be allowed onto the network.

Once the proxy server has verified that the computer's antivirus is working and up to date, the proxy server lets the request get a little farther and prompts for a login. Each student has their own login, so that network traffic is traceable back to the originator. Once the student logs in - presto, they're online.

Now, if you're not already, think to yourself, "how many ways could this possibly go wrong?"

The principal mentioned that they were having some technology troubles and I volunteered to help. I've been by the school a few times now and each time there are more laptops waiting for me to take a look at. I'm amazed at the number of ways that this configuration can go wrong.

For example. Did you catch that the board requires that the computers have up-to-date antivirus software? And did you catch that the board requires that the student download a program and run it so that the proxy server can check to make sure the antivirus software is running? Some of the students had problems where the antivirus software had quarantined the security agent, and prevented the students from running it.

The network uses a proxy autodiscovery script to inform the clients where to locate the proxy server. But the autodiscovery pointed at the wrong server, so most computers needed to override it. This is a misconfiguration on the part of the school board and I'm hoping that can be resolved without too much trouble.

Some students would download the 32 bit version of the agent instead of the 64 bit version, or vice versa, and get an error message that they didn't understand.

One system I looked at would bluescreen when trying to install the agent. The second time through, it installed, and the system got online no trouble after that. Strange.

Another system has a problem where the wireless status shows that it's online, but the computer doesn't have an IPv4 address. Windows seems to think the IPv4 stack isn't bound to the wireless adapter. Reinstalling the network driver may fix that, but I'm afraid to try it without knowing for sure that the wireless driver will reinstall correctly - I don't want to break anyone's computer.

Another system, the local firewall was blocking the proxy's request back to the security agent. The agent apparently requires ports 8192, 8193, and 8194 be open.

This is bad.

I can understand the school board's reluctance to take ownership of problems on student laptops. But the combination of the proxy server, the antivirus requirement and the security agent requirement has created a system that users won't encounter anywhere else and aren't going to be able to resolve on their own.

But you know, I'll bet there are a lot of techies out there that wouldn't mind spending an hour at a school helping kids with computer problems. Many schools have equipment that they need help using, or could be making better use of if they only knew how. The question is, how to connect those dots.

I was hand-editing some JSON today in BBEdit, and wanted to keep it valid and nicely formatted. Copying the text and pasting it into jsonlint.com is no fun, so I went looking for a solution.

I found this post on a creating a Python script and dropping it into the right place to get JSON validation. It almost worked, but I had to make two minor changes. One for BBEdit 10 compatibility, and one to get actual line numbers when there's a failure validating.

BBEdit 10 moved where text filters live, so instead of dropping the script into [cci]~/Library/Application Support/BBEdit/Unix Filters[/cci], it needs to go into [cci]~/Library/Application Support/BBEdit/Text Filters[/cci].

The text of the script is:

[cc lang="python"]
#!/usr/local/bin/python
import fileinput
import json
if __name__ == "__main__":
jsonStr = ''
for a_line in fileinput.input():
jsonStr = jsonStr + '\n' + a_line.strip()
jsonObj = json.loads(jsonStr)
print json.dumps(jsonObj, sort_keys=True, indent=2)
[/cc]

The only change I made is replacing the ' ' with '\n' in the line that's appending the string, so the JSON parser is aware of what line number you're on. That way when it fails parsing it can give you a line number, not just a column.

I haven't reviewed a non-tech product in a while, but I've been introduced to a new appliance that I think is worth mentioning. I bought a water boiler.

It all started with "I should drink more tea". Why don't I drink more tea? Because I'm impatient and I don't like hanging around waiting for the kettle to boil. I was looking for some sort of turbo-kettle when I ran across this Zojirushi Water Boiler.

Rather than boiling water on demand, these products (and there are a lot of them - Amazon has over a dozen) boil the water once and then keep it hot in an insulated pot.

The one I have lets you set the temperature you want the water at, and I keept it set at 208 degrees, but if you want to make specific types of tea that need water at different temperatures, you can do that.

The cool thing about having a water boiler isn't that you can make tea with it, but all the other things you can do with it. Here are a couple of the uses I've found.

Warming up baby formula. Babies, especially little fussypants babies like ours (my wife doesn't read my blog so I can say that), want their formula a little bit warm. We've been using powdered formula since she was two months old, and what we do is fill the bottle mostly with cold bottled water, and then top it off with water from the boiler. This warms up the overall temperature to something she likes.

Instant Oatmeal. It's great to have hot "instant" oatmeal that really is instant. No waiting for the water to boil.

Instant Coffee. Same with the coffee. Those Starbucks instant packages are pretty good, but I have some regular instant coffee around for emergencies.

Melting butter for popcorn. Normally this would be the microwave's job, but I find that for small amounts of butter I generally end up leaving it in a few seconds too long and boiling the butter. What I do here is put a little bowl inside a bigger bowl, and then pour some boiling water into the bigger bowl. The butter melts while I'm making the popcorn.

I'm sure there are more uses for these things that I haven't come up with yet, so I'd love to hear comments on how other people are using them.

Funny thing? I still don't drink much tea.

There's a saying, "Security is a process, not a product". But companies don't ship processes, they ship products.

Now, when you buy some products, you're also buying the process. If you're running Windows, Mac OS, iOS, the Xbox 360, or a PS3 (to cite a few examples), then assuming your device is online and keeping up with updates, the vendor is producing updates and you are downloading and installing them. That's the process.

There are a few problems with this, but the one I've been thinking about lately is the proliferation of devices that are coming with network connections, and the lack of attention they get from the vendors. They shipped a product with a set of features, and their solution to a problem in the current version is to fix it in next year's model.

I'm thinking specifically of something like my TV. It supports Netflix and YouTube and a number of other online services, and it's plugged into my network.

Is whatever OS is running in the TV secure? I have no idea, but I know from experience that the software embedded in this sort of device generally doesn't get much security scrutiny. It may be running an embedded Linux or some other OS that is reasonably secure, or was when it was burned into the firmware, but new vulnerabilities are found all the time. Did the device patch itself against the zlib vulnerabilities found a few years ago? I doubt it.

It might sound like the stuff of science fiction but more and more devices are getting connected and running software. And any one of these can be exploited.

If my TV was compromised, how would I even know it? Maybe it's a part of some massive TV botnet. Smart hackers can get your devices to do their bidding without you knowing it, and it's quantity, not the power of each individual node, that makes for a dangerous botnet.

Think about all the Android phones sold by vendors that aren't issuing updates for those devices. There are millions of them out there. Give an evil organization a million machines on a botnet and they can take down a good chunk of the Internet.

Scary stuff.

The only solution I can think of is a system where a machine has to have some sort of valid "allowed to access the network" ticket that is automatically invalidated when an exploit is found for that machine. Packets would carry that ticket and upstream network providers (like maybe your home gateway, and then your ISP) would drop packets that came from exploitable devices (whether they'd been exploited yet or not).

Maybe this would force the process on companies that just want to sell a product.

I was working on a spec in a wiki based editor yesterday and I noticed how uncomfortable I was working on this document in a browser.

This is a long, fairly important document I was working on, and I was working on it in a tool designed for easy skipping from page to page, deigned browsing content but not really designed for creating it. If I accidentally hit the wrong hotkey, or click in the wrong place, the browser will whisk me off to another page, leaving the content I'm editing behind.

Some browsers will warn you before you do this (detecting that you've edited something on the page and not yet done a server round-trip to commit the change) but others don't, and some web apps will save drafts or interim copies of what you're working on, but others won't. It's a dangerous way to work.

Now, I'm not knocking web apps. I like apps that run in a browser; I just don't like running them in a browser.

One of the interesting things that iOS has done is convince web developers to create an app versions of their site to run on the phone. Many times the "app" is just the HTML version of the site, bundled up with all the style sheets and JavaScript code and whatnot, combined with a tool like PhoneGap into an iPhone App that you get through the app store. Or it could be a Flash site turned into an AIR application.

There's some space between a native app for the platform you're using and a web app, and tools that let you turn your web app into a platform app with its own chrome and more importantly behaviours around what happens when a user clicks on a link or quits the app (an opportunity to save, for example) are a huge improvement over running the same apps in a browser.

One point I think is interesting here is that often in these cases your app will be running on the phone and making API requests to the server, not fetching the whole site from the server. This means your "web app" is running under the radar of sites that give out mobile browser usage stats. When I'm using the Facebook app on my iPhone, does that count as a user running a mobile browser to access Facebook? Probably not. But what's the difference?

Ideally I'd love to see a common "app runtime" that knew how to run a web app in some chrome other than a browser. Chrome is trying to provide this, I believe, but what you get still feels too much like a web app with the inherent apparent lack of concern for the content you're editing on a page.

AIR provides a great desktop runtime for Flash apps; PhoneGap is another option. But either way, if you're building a website today, expect to be building an app. That's what users want on their mobile devices. Not mobile-optimized websites. Apps.

I think the industry is completely underestimating the impact that iCloud and Apple's cloud storage implementation is going to have.

Applications synchronising data between devices is going to just work, without the user having to do anything to set it up. No accounts, no paying for separate services, nothing. Just buy the app in iTunes and the app appears automatically on any compatible devices that the user has. Use the app on one device, and the data you create is immediately accessible on other iOS devices or from Mac apps.

The revolutionary part of this is that it's no work for the user to set up and very little work for the developer (almost no work if you use the right APIs to manage your documents), so it's going to get used. A lot.

Apple's good at solving hard problems in ways that make the solution seem simple.

And because of the commercial ecosystem Apple has created, they can afford to offer a limited (but adequate for many users) amount of iCloud storage for free. That's going to be tough for anyone else.

I've been writing some code that talks to Jackrabbit, and wanted to write some tests.

Testing this sort of thing has traditionally been fairly difficult, because so much of your code is just calls into Jackrabbit. You'd need to create a lot of mock objects to mock the Jackrabbit side of the test, often leading to tests that are bigger than the code being tested!

Enter Mockito.

So you want to test some code that walks the children of a node in the repository? Simple. Here's an example:

[cc lang="java"]
// Create a mock repository, and when login is called with any
// SimpleCredentials instance, return a mock session.
Repository mockRepo = mock(Repository.class);
Session mockSession = mock(Session.class);
when (mockRepo.login(any(SimpleCredentials.class))).thenReturn(mockSession);

// Mock the root node, and have the session return it on request.
Node mockRootNode = mock(Node.class);
when (mockSession.getNode(eq("/content/myroot"))).thenReturn(mockRootNode);

// Mock two child nodes
Node mockNodeOne = mock(Node.class);
when (mockNodeOne.getName()).thenReturn("NodeOne");
Node mockNodeTwo = mock(Node.class);
when (mockNodeOne.getName()).thenReturn("NodeTwo");

// Mock an iterator that returns these two nodes
NodeIterator mockNodeIterator = mock(NodeIterator.class);
when (mockNodeIterator.hasNext()).thenReturn(true).thenReturn(true).thenReturn(false);
when (mockNodeIterator.nextNode()).thenReturn(mockNodeOne).thenReturn(mockNodeTwo);

// When getNodes() is called on the root node, return the iterator
when (mockRootNode.getNodes()).thenReturn(mockNodeIterator);

// Now call into the code to be tested with the mock Repository.
codeToBeTested(mockRepo);
[/cc]

And if you're using Maven to build (I've become a huge Maven fan) then you can integrate Mockito simply by adding a dependency to your pom. Details are on the Mockito site.

Implementing an OSGi service using Maven isn't hard, but there are a bunch of little pieces to pull together the first time through the process.

Let's create a SlingPostProcessor.

A SlingPostProcessor is called by Sling just before POSTed changes are written into the repository. It provides a mechanism for validating what goes into the repository.

Our goal isn't to create just an example, but an example that's ready to use in a larger software development project. This means we're developing in Eclipse and building with Maven.

So to get started, make sure you've got a recent Eclipse and the M2Eclipse plugin installed.

Once you're there, start with File / New Project and the Maven 2 Project Creation Wizard. I'm going to create a simple Java project named "my-sling-validator".

If everything is working right, you should have a Java project that builds both in Eclipse, and from the command line (via "mvn package"). But it's a Java project, not an OSGi bundle.

The M2Eclipse plugin provides an editor for the pom.xml file (Maven's project object model), but I find it's easier to edit the xml file directly, and you're going to want to become familiar with the format of the pom.xml file since it's a fundamental part of working with Maven, so let's do it there.

First we need to turn our Java project into a project that produces an OSGi bundle. There's a plugin for that - and there's a plugin for producing the service descriptor file that describes the service that our bundle will provide, so let's add both those plugins right now.

Edit the pom.xml file and add, as a child of the <project> node, the following:

[cc]
org.apache.felix
maven-bundle-plugin
2.2.0
true


net.stevex.examples.sling.validator

org.apache.felix
maven-scr-plugin
1.7.0


generate-scr-scrdescriptor

scr



[/cc]

Paste this into your pom.xml file. Save it, and Eclipse should immediately go hunting for the two plugins, downloading them and installing them into your local Maven repository if required. The plugins give your project the ability to produce an OSGi bundle, but to actually get a bundle, you need to change the project packaging from the default value of 'jar' to 'bundle'. (For this simple change you can try using the M2Eclipse pom.xml editor if you like. The value you're changing is:

[cc] ... bundle ... [/cc]

Now if you package your project, you'll get an OSGi bundle. To package, right-click on your project, and on the Maven 2 menu select Package Artifact.

Refresh the 'target' folder in your project and you'll find a Jar file. Look at the META-INF/MANIFEST.MF file inside that jar and you'll see it has the bundle properties that identify it as an OSGi bundle.

Manifest-Version: 1.0
Export-Package: net.stevex.examples.sling.validator
Bundle-Version: 1.0.0.SNAPSHOT
Build-Jdk: 1.6.0_24
Built-By: stevex
Tool: Bnd-1.15.0
Bnd-LastModified: 1308567248570
Bundle-Name: my-sling-validator
Bundle-ManifestVersion: 2
Created-By: Apache Maven Bundle Plugin
Bundle-SymbolicName: net.stevex.my-sling-validator

Next, we need to create a class that implements the SlingPostProcessor service. To do this we need some dependencies on the ClassPath so we have access to the interface we need to implement. Edit the pom.xml file again and add some dependencies:

[cc] ..

..

org.apache.felix
org.apache.felix.scr.annotations
1.4.0

org.apache.felix
org.osgi.core
1.4.0

org.apache.sling
org.apache.sling.servlets.post
2.1.0
provided

org.apache.sling
org.apache.sling.api
2.2.0

..

.. [/cc]

Add these dependencies, save, and let Maven download the required dependencies. Once that's done, you can use the Eclipse class wizard to extend the SlingPostProcessor:

Now the next step is exporting this service so the OSGi container will find it.

[cc lang="java"]
@Component(metatype=false, immediate=true)
@Services({
@Service(value=org.apache.sling.servlets.post.SlingPostProcessor.class)
})
public class MyPostProcessor implements SlingPostProcessor {

@Override
public void process(SlingHttpServletRequest arg0, List arg1)
throws Exception {
System.out.println("MyPostProcessor.process called!");
}

}
[/cc]

There are two ways that you can specify the SCR annotations: as JavaDoc comments, or as annotations. I prefer annotations because they seem more like part of the language, even though the comment syntax is shorter and simpler.

Now that the project implements a service, take a look at the jar file generated when you invoke the Maven package step. You'll see it has an OSGI-INF folder with some files that describe the service the jar implements.

At this point, we've got a complete working OSGi bundle that implements the SlingPostProcessor service. Let's see if it works!

If you've got CRX installed on localhost, you should be able to access the Felix (OSGi) console at http://localhost:4502/system/console/bundles (or whatever port number you're running it on). On that page, click the "Install/Update..." link at the top right, browse to your jar file, and upload it. Once it's uploaded you may need to click the Reload button to see it show up in the list but once it does, you can click the little triangle at the left of the name to see details.

Click the "play" button at the right of the list and start the bundle.

To see if the service registration was successful, take a look at the Services tab in the Felix console. You should be able to find your service in the list. And the final test: watch stdout.log and invoke curl to POST some content:

$ curl -F "foo=bar" http://localhost:7402/content/foo

You should see the line we're printing in the process function show up in stdout.log.

(Note that cheesy stdout logging is not something you'd want to use, but it saved me from having to get into how to set up logging).

Notice that we didn't need to register with the SlingPostProcessor? That's because the SlingPostProcessor is listening to the service registry and when it sees a SlingPostProcessor register itself, it gets added to the list of post processors. Stop your bundle and Sling will automatically remove the post processor.

What if something goes wrong, where do you look? Here are some troubleshooting steps.

Make sure the bundle is loading. If you click the "play" button on the Bundles tab for your bundle, and then reload, and it's not showing as Active, then something went wrong starting the bundle. Check the logs (there are a few logs) in the crx-quickstart/logs folder. (A good way to do this is to "tail -f *" in the logs folder, then attempt to start your bundle, and scroll back through all the output to see what went wrong).

Make sure you're importing the classes you need. Our example project doesn't import any packages but a real project would. You'll need to specify these in the configuration/instructions section of the maven-bundle-plugin plugin configuration (by adding an Import-Package next to the Export-Package directive that we did specify).

Did the bundle load but the service isn't showing up on the services tab? Something is wrong with your service registration.

If you need to see more debugging information in the log, go to the the Sling Log Support tab in the Felix console, and click the little configuration wrench next to stderr.log. Change the level from 'info' to 'debug' and you'll get more detailed logs.

CRX is a content repository built by Day Software, now Adobe. It's a fascinating product, consisting of a few major pieces of software working together to provide a whole new way of building web apps.

At its core is Jackrabbit, a Java content repository with no web interface. On top of JackRabbit is Sling, a web framework that uses JackRabbit for storage and processes web requests using various scripting languages.

One of the great features about Sling is the ease of which you can build content driven applications, with little or no coding. On a default CRX installation, you can create an HTML form that will POST content right into the repository without having to write a line of code on the server. Sling's tagline is "Bringing Back the Fun", and it seems fitting.

But this post (actually this series of posts) is about security. On the Internet you don't want users POSTing content right into the repository without some careful consideration of the contents of that content and where it's being stored.

Let's take a look at the CRX blog example.

I'm going to assume for this post that you've installed CRX and you have run through the blog example and have it working on a CRX server on localhost. If you haven't run through the blog example, here's the end result - the blog app and content - as a CRX package that you can import through Package Share: crx-blog-example.zip.

CRX gives you a lot of features for free - for example, a JSON interface to retrieving content. With the blog sample installed on a CRX instance running on localhost, this link shows the blog posts for March 2010; this link shows the same content in JSON format. And this link does the same for XML.

Another feature you get for free is the ability to manage content using a REST interface, through the SlingPostServlet.

With no extra coding or configuration, you can post new blog entries to the server, update them or remove them. For example, here's how you'd post a new blog entry (using curl):

curl -v -Ftitle="3rd March 2010" -Fbody="It's lovely today." http://localhost:7402/content/myBlog/2010/March/03March2010

If you're not familiar with curl, it's a simple command-line tool that lets you submit HTTP requests. In this case you're asking curl to POST a form with two values, 'title' and 'body', to the URL supplied. -v just asks it to be verbose, so you can see the header values.

This will post a new blog entry into March 2010. You can see the Location: header in the curl output that shows the location of your newly posted content:

http://localhost:7402/content/myBlog/2010/March/03March2010.html

Want to remove a blog post? No problem, with :operation=delete.

curl -v -F":operation=delete" http://localhost:7402/content/myBlog/2010/March/03March2010

Obviously we want to lock this down - we can't have random Internet users deleting our blog posts. Use the CRX Content Explorer to access the User Manager, and add a new user. Call the user 'blogger', password 'password'. We'll use this as the user that's allowed to post to the blog, and lock the system down so it's not modifiable by anyone else.

To do this, in Content Explorer, select the /content/myBlog node. This is the root of the content for the blog, and is a good place to lock the whole thing down. On the tool bar, click Security, then Access Control Editor.

The ACL editor shows you that there are no "local" access control policies, so the "effective" policy is inherited from the root. A default CRX installation grants "everyone" the right to do anything except edit the access control lists.

A nice feature of CRX is the ability to test ACLs. In the ACL editor, click the Test button. You'll see a page that shows the effective privileges for the node you were editing, with some controls at the bottom for picking a path, and picking a user.

With a path of "/content/myBlog" and a principal of "anonymous", you should see this:

jcr:read
rep:write
jcr:removeNode
jcr:modifyProperties
jcr:addChildNodes
jcr:removeChildNodes
jcr:nodeTypeManagement
jcr:versionManagement
jcr:lockManagement
jcr:lifecycleManagement
jcr:retentionManagement

Anonymous is way too powerful. Use the ACL editor to remove everything but jcr:read by going back to the ACL editor, clicking the checkbox next to the ACL policy in the list at the top, then click Set selected policies. Click New ACE, and create an entry to deny jcr:all to anonymous. Now create an ACE to grant jcr:read to anonymous. Apply, then test.

With this change, a link to a blog post should work fine, but attempting to use curl to make a new post should fail:

// this should fail with an HTTP 403 response (Access Forbidden)
curl -v -F":operation=delete" http://localhost:7402/content/myBlog/2010/March/03March2010

This keeps everyone from posting content - even the blogger. That's no good, so let's create a blogger user (call it "blogger", password="password"), and grant blogger write access to /content/myBlog.

Now, if we attempt to delete while logged in as the blogger user:

curl -v -F":operation=delete" http://blogger:password@localhost:7402/content/myBlog/2010/March/03March2010

The operation succeeds.

> POST /content/myBlog/2010/March/03March2010 HTTP/1.1
> Authorization: Basic YmxvZ2dlcjpwYXNzd29yZA==
> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3
> Host: localhost:7402
> Accept: */*
> Content-Length: 151
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=----------------------------41e17a7038f6
>
< HTTP/1.1 100 Continue 
< HTTP/1.1 200 OK 
< Connection: Keep-Alive 
< Server: Day-Servlet-Engine/4.1.12 
< Content-Type: text/html;charset=UTF-8 
< Date: Thu, 09 Jun 2011 03:22:56 GMT 
< Transfer-Encoding: chunked  

Another feature of the blogging app is commenting. From any page within the content hierarchy, you can link to a comment form that will let you write a comment. The comment will be uploaded to the server and placed alongside the blog entry.

The example shows how easy it is to add this feature, but there are a number of security holes opened by allowing anonymous users to post content directly into the repository. When we locked down the site so that only the blogger could post content, we also locked out commenters.

In the next post, I'm going to talk about how to allow anonymous users to post content in a safer manner.

I've spent some time recently working on the RTMP protocol implementation of our Objective C iOS LCDS Messaging and Remoting client. (You can see a video of it in action here).

The RTMP protocol was designed for streaming multimedia with the ability to interleave other kinds of content. It's is a low-level binary protocol, so I've spent a lot of time over the last few months looking at dumps of bytes sent over the wire and decoding them to determine that we're sending the right bytes and that they match what our other clients are sending and expecting.

For anyone else implementing RTMP, or just interested in it, I've cleaned up and annotated some parts of a typical RTMP session below for your reading pleasure.

This RTMP stream represents a client connecting to the server, and sending a request to subscribe to a messaging destination. This example is how a client would start a data feed from the Market Data LCDS sample. The subscribe message s an instance of "flex.messaging.messages.CommandMessage", and looks like this:

[cc]
operation = multi_subscribe
clientId = null
correlationId = null
destination = market-data-feed
messageId = 3300FE5C-F850-451F-847C-B7D257E6D224
timestamp = 1296478297471
timeToLive = 0
body = null
hdr(DSMaxFrequency) = 1000
hdr(DSAddSub) =
[
ADBE_;_
]
[/cc]

And on to the protocol decoding. Bytes being sent to the server from the client are denoted with > (greater than) and bytes coming the other direction are denoted < (less than).

The first thing that happens is the client opens a socket connection to the RTMP server, and sends:

[cc]
> 03
[/cc]

This is the client telling the server "I support protocol version 3".

[cc]
> (1536 bytes of random data)
[/cc]

The client sends 1536 bytes of random data to the server. This is part of a sequence of exchanges that the server and client use to make a guess as to their available bandwidth.

[cc]
< 03 [/cc] This is the server responding with the protocol version that it is using. All is well; both report version 3.

[cc]
< (1536 bytes of random data) [/cc] The server also sends 1536 bytes of random data.

[cc]
< (Echo of the 1536 bytes that the client sent to the server) [/cc] The server sends back the data that the client sent it. This gives the client a way of guessing not only at bandwidth but a pretty good measure of latency. [cc] > (Echo of the 1536 bytes that the server sent to the client)
[/cc]

The client does the same.

Now the initial handshake is done and we can start talking RTMP.

Our goal is to request a subscription to an LCDS messaging destination, so we need to send the message to do this. This starts out with a chunk header (Type 0, section 6.1.2.1 of the RTMP spec):

[cc]
> 00 00 00 00 00 3D 11 00 00 00 00
[/cc]

This decodes as a timestamp of 0 (the first 3 bytes), message length of 61 bytes (next 3 bytes), message type ID 17 (decimal, 0x11), and a stream ID of 0 (the last 4 bytes). The message type is 17, which means an AMF3 encoded message follows.

Here's what the subscribe message looks like:

[cc]
> 0x00 (msg format amf3)
> 0x02 0x00 0x07 5f726573756c74 (string "_result")
> 0x05 (trxID = null - 0x05 is null in AMF0 and we're still in AMF0 mode)
> 0x05 (cmdData = null)
> 0x11 (AvmPlusObjectType (17) = an object follows)
[/cc]

Writing the CommandMessage object:

[cc]
> 0x0a // Type of what follow is an object, AMF3ObjectType, see section 3.1 of the AMF3 spec
[/cc]

Next, internally we have a function that walks the object's properties and serializes it. The object's properties are written first, then the values. So here's what the object's properties look like on the wire:

[cc]
> 0x81 0x13
[/cc]

This is a variable-length 29-bit integer encoding (see 1.3.1 in the AMF3 spec) of the value 147 (0x93) which is: 0x03 (U290-traits) | 0x08 (dynamic) | number of member names that follow (9 in this case).

Next we have the class name:

[cc]
> 0x4d // ((utflen << 1) | 1), utflen=38 > 66 6c 65 78 2e 6d 65 73 73 61 67 69 6e 67
> 2e 6d 65 73 73 61 67 65 73 2e 43 6f 6d 6d
> 61 6e 64 4d 65 73 73 61 67 65
[/cc]

This is a string serialization of the 38 byte class name, "flex.messaging.messages.CommandMessage".

Next we have the 9 property names, each represented as a UTF-8 variable length string:

[cc]
> 0x13 6f 70 65 72 61 74 69 6f 6e "timestamp"
> 0x0f 68 65 61 64 65 72 73 "headers"
> 0x09 62 70 65 72 61 74 69 6f 6e "operation"
> 0x09 62 6f 64 79 "body"
> 0x1b 63 6f 72 72 65 6c 61 74 69 6f 6e 49 64 "correlationId"
> 0x13 6d 65 73 73 61 67 65 49 64 "messageId"
> 0x15 74 69 6d 65 54 6f 4c 69 76 65 "timeToLive"
> 0x13 74 69 6d 65 73 74 61 6d 70 "timestamp"
> 0x11 63 6c 69 65 6e 74 49 64 "clientId"
> 0x17 64 65 73 74 69 6e 61 74 69 6f 6e "destination"
[/cc]

Now that the 9 names have been written, the 9 values are written (in the same order).

[cc]
> 5 42 72 de 1c d6 76 0 0
[/cc]

0x05 is a double, always written as 8 bytes

[cc]
> 0x0a object marker for 'headers'
[/cc]

Next up is 'headers', a map of name/value pairs that's written as a custom object. This takes almost the same form as the object we're currently in the middle of serializing, except that it's an anonymous class.

[cc]
> 0x43 // 4<<4 | 3 = embedded object (not a reference), 3 properties > 0x1 // object name (empty string in this case - 0x01 is the marker for a null)
[/cc]

Next the traits for the headers:

[cc]
> 0x1d 44 53 4d 61 78 46 72 65 71 75 65 6e 63 79 "DSMaxFrequency"
> 0x9 44 53 49 64 "DSId"
> 0x11 44 53 41 64 64 53 75 62 "DSAddSub"
> 15 44 53 45 6e 64 70 6f 69 6e 74 "DSEndpoint"
[/cc]

Next, the values for the headers:

[cc]
> 0x04 87 68
[/cc]

DSMaxFrequency header. 0x04 is an integer, and the next 2 bytes are a UInt29 representation of the integer value 1000. 1000 is ((7<<7) | 0x68).

[cc]
> 0x06 0x49 34 34 36 41 31 32 38 37 2d 42 31 41
> 46 2d 30 44 33 38 2d 43 35 31 41 2d
> 38 41 45 32 36 31 43 46 44 41 39 42
[/cc]

DSId header. 0x06 is a string; 0x49 is the length of the string << 1 with bit 0 unset. If bit 0 was not set, then it would be a reference into a string table. The string length is 36 bytes (it's a GUID) << 1 | 1.

[cc]
> 0x09 0x03 0x01 0x06 0x0f 41 44 42 45 5f 3b 5f
[/cc]

DSAddSub header. This is an array (0x09) of 1 element (bit 1 is always set, count is shifted left one), with no name (0x01 indicates a null string). The one element is a string (0x06) of length 7 (0x0f = 7 << 1 | 1), containing the bytes "ADBE_;_". This is what we're subscribing to.

[cc]
> 0x06 0x0f 6d 79 2d 72 74 6d 70 "my-rtmp"
[/cc]

DSEndpoint header. The endpoint name, "my-rtmp".

(At this point we're done with the embedded headers array and we're back to serializing the properties of the CommandMessage, continuing with the 'operation' property).

[cc]
> 0x09 0xb
[/cc]

The 'operation' property is an integer, value 11. Operation 11 is the multi-subscribe operation. (You can see the operations in the BlazeDS documentation for CommandMessage). http://livedocs.adobe.com/blazeds/1/javadoc/constant-values.html#flex.messaging.messages.CommandMessage.MULTI_SUBSCRIBE_OPERATION

[cc]
> 0x01 (body = null)
> 0x01 (correlationId = null)
> 0x06 0x49 37 35 37 35 44 42 45 34 2d 42 30 31
> 32 2d 34 41 45 46 2d 42 43 37 41 2d
> 41 37 34 44 30 36 43 46 37 42 46 44
[/cc]

The messageId is a GUID, sent similarly to the DSId header above.

[cc]
> 0x05 0 0 0 0 0 0 0 0
[/cc]

timeToLive is of type double (0x05) value zero (doubles are always 8 bytes).

[cc]
> 0x01 (clientId = null)
> 0x06 0x21 6d 61 72 6b 65 74 2d 64 61 74 61 2d 66 65 65 64 (market-data-feed)
[/cc]

Finally, the name of the feed that we're subscribing to, "market-data-feed".

And that's that.

As you can tell RTMP and AMF3 are incredibly efficient binary protocols - the decoded streams above show the efficiency of the protocol in general but don't show one of the other big gains, the tables that the client and server maintain so that if a string or object has already been sent once during a session, all that needs to be sent over the wire is a reference to the string, not the string itself. This shaves many more bytes off the session.

And why is that important? Well, especially in mobile apps, users are paying for bytes. There is typically less bandwidth. And even if your clients don't mind the traffic, think of the bytes you're saving on the server side when you've got a few thousand clients connected and receiving streaming updates. It can be significant.

Looking for more information about RTMP? The spec is here. The AMF3 spec is here, and it builds on the AMF0 spec which is here.