Android + CircleCI + versionCode

December 9th, 2018

Automatically incrementing build numbers in CI is always a hassle. Here’s a simple way of using a generated version number in your CI, based on the current time in 10 second intervals. This is based on this post by esplo but with the missing piece of how to read the version in the build.gradle.

Add this to your CircleCI .circleci/config.yml file:

- run:
name: Write versionCode to version.properties
command: |
U=$(expr $(date +%s) / 10) # increase by 1 in each 10 seconds
echo "versionCode=${U}" >> app/version.properties
cat app/version.properties

Add this to the top of your app’s build.gradle:

def Properties versionProperties = new Properties()
versionProperties.load(new FileInputStream(file('version.properties')))

And then later in your app’s build.gradle, to read the versionCode:

versionCode versionProperties['versionCode'].toInteger()

This way, the CircleCI build will write out versionCode to a version.properties file, for example:

versionCode=154435338

And then the Gradle build will read it while building the app.

React Native or Electron: Why?

December 5th, 2018

Many developers hate Electron apps, and have a distaste for React Native apps. There are some good technical reasons for this. They’re inelegant. Inefficient. Bloated, slow, pigs. That’s the reputation.

But to the users? They’re fine. They’re good enough. I don’t see users dropping Slack in favour of Skype because Skype is native. That’s just now how users make these decisions.

But native is better right?  Big, smart, rich companies are choosing to build software using React Native and Electron. Why would they do this, given that the products will be, by some measures, sub-par?

It’s not about the raw cost savings of using one code base. It’s about the organization.

Software development is more than just developers. It’s QA, HR, IT, CI, Design. UX.

Companies get addicted to their own branding. Most big, successful apps use their own look & feel, rather than adopting the platform conventions. Adobe apps don’t look like Mac apps or Windows apps, they look like Adobe apps. Slack looks like Slack. Facebook looks like Facebook.  Word is going to come down that all of these apps need adopt the same look and feel on all platforms. In doing so, you’re giving up the platform familiarity that users have with native apps.

There are 5 major platforms to build for: Mac, Windows, iOS, Android, and the web. If you’re going to build a “native” app for each of these platforms, that’s five times more design work to do. Five teams to manage. Five teams that you need to coordinate whenever you want to make a simple change.

If the 5 native apps could evolve on their own timeframe, it might be easier, but each platform has both internal and external factors that feed into development schedules. Releases need to be coordinated, feature sets coordinated, look & feel coordinated. 

Cross-platform development isn’t about saving money on development; it’s about coordinating development across all the platforms. Even if it cost *more* to build an Electron app than building separate native apps, I think for many companies it would be preferable. It’s just easier to manage.

Sending RabbitMQ Message from JavaScript with async / await

November 22nd, 2018

I’m doing a lot of JavaScript development these days, and most of the example JS code I find on the web uses callbacks. I much prefer async/await, and sometimes the translation isn’t obvious.  

Here’s how to send a message using amqplib to RabbitMQ using async await.

The interesting bit here is the waitForConfirms method. You’d think you could await sendToQueue, and that would wait for the send to complete. However, the way amqpplib implements this, the call returns once the message is queued for sending. Without the waitForConfirms, the process.exit will run before the message is sent.

Building a Slack bot in Node

November 16th, 2018

I was looking into how to build a simple Slack bot. I ran across various bot kits and frameworks and whatnot, that seemed to really overcomplicate the issue. Went back to the Slack API and discovered that it’s actually quite straightforward.

This particular code listens for messages in a private channel called ‘dev’. Private channels are called ‘Groups’ in the Slack API.

The bot listens for messages, looks fo the text ‘ping’ and responds ‘pong’.  The rest is up to you!

Local Siri

May 17th, 2018

Here’s my WWDC prediction for 2018:

Siri will be significantly replaced with a new, on-device Siri.

A few attributes of this new Siri that would make it interesting, and give it a shot of coexisting in a world with Google Assistant, Alexa and Cortana:

  • Requests will run directly on your device. No more waiting for your voice to be uploaded and the response downloaded, no more spotty Siri when your network connectivity is down. This will significantly increase response time and reliability.
  • Siri will treat your devices as a single cloud, fetching information from other devices or sending a request to the device best suited to process it.

This goes a step beyond where Siri is today regarding your privacy, and from that perspective it’s totally in sync with Apple’s privacy stance. But does doing things locally mean worse results?

I don’t think it has to. Some questions will require a network request, like asking how long it would take to drive somewhere, but if you’re an Apple user who’s all-in with iCloud, all your data is already on your device. Your documents, your calendars, messages, emails, can all be scanned by machine learning right on your device. Spotlight already does this for indexing.

So asking “What movies are playing nearby” will result in an internet request, but “Open the garage door” would not, since that can be completely handled inside your house on your local LAN. If you’re out, it will send a message to one of your other devices that is at home, like the Apple TV or HomePod. This is how HomeKit works today.

This is what I’d like to see, but since I’ve never been right with my WWDC predictions, I’d suggest not getting your hopes up.

Formatting CVaListPointer

April 9th, 2018

So you’re using libtiff in Swift, and need to implement the TIFFErrorHandler.

Or, for whatever reason, you have a CVaListPointer (the args parameter, below) and a format string, and you want to format. Here’s how:

let errorHandler:TIFFErrorHandler = {
(moduleNamePtr, formatPtr, args) in
guard let formatPtr = formatPtr else {
// No format string - this is unexpected and there's
// not much we can do here, perhaps log that it happened.
return
}

// Convert the format pointer into a Swift String
let formatStr = String(cString:formatPtr)

// Create the output buffer
var buff = Data.init(count: 1024)

// Accessing the buffer's bytes...
buff.withUnsafeMutableBytes({ (ptr) -> () in
// Format into them
vsnprintf(ptr, 1024, formatStr.UTF8CString, args)
})

guard let formatted = String(data: buff, encoding: .utf8) else {
// This would only happen if the logging produced an invalid string,
// we don't have a string we can display
return
}

NSLog("TIFF error: \(formatted)")
}

TIFFSetErrorHandler(errorHandler)

				

The Amazon Web Services Platform

October 19th, 2017

I’ve been working with Amazon Web Services as part of a new job, and I was surprised to see how many rich services Amazon is offering under the AWS banner now.  

 

I made a point of looking at every one of them (as of October 2017) and reading enough about it to get what it’s for. The scope of these services is incredible, and I get what the “serverless” trend is about now.  You can build and deploy complete application back ends in AWS, start to finish. No surprise to many, I’m sure, but last time I looked at AWS it was mostly storage and VMs.

 

Here’s my cheat sheet.

 

Code Hosting

EC2

VMs in the cloud.

EC2 Container Service

Docker containers in the cloud.

Lightsail

Cheap, simple VMs in the cloud. Backed by EC2. Check out this StackOverflow post for some details.

Elastic Beanstalk

App hosting. You supply the your application code (Java, PHP, .NET, Node, Tomcat, Ruby, etc), and they run it on a managed server.

Lambda

Function hosting. You write backend logic code in JS, C#, Java or Python, and it hooks up to an endpoint that you can call or trigger from other AWS activity. “Serverless”.

Batch

Manages running large jobs in parallel Docker containers across many EC2 instances.


File Storage

S3

Simple file storage with an HTTP API. Pay for the amount of storage and transfer you use.

EFS

Provides filesystem semantics on top of S3.

Glacier

Slow, low-cost storage, for backups and archiving.

Storage Gateway

Software you run that acts as a local cache for your EFS data. Mount as iSCSI or as a share on your network.


Database

RDS

Hosted SQL databases (hosting MySQL, PostgreSQL, or a number of others).

DynamoDB

Amazon’s own MongoDB style noSQL database.

ElastiCache

Hosted redis / memcached compatible in-memory caching.

Amazon Redshift

“a fast, fully managed, petabyte-scale data warehouse solution”

 

Artificial Intelligence

Lex

Speech recognition and natural language understanding. Build a smart Slack bot.

Polly

Text to speech.

Rekognition

Imagine analysis and object detection.

Machine Learning

Machine learning. Upload models, run them on data.

 

Messaging

Simple Queue Service

Message queueing service. Create queues, add messages to them, and have various things pull messages and act on them.

Simple Email Service

Send and receive email. SMTP gateway or API.

Simple Notification Service

Pub/Sub messaging, for delivering notifications to different parts of your application, and SMS notification service. (Those two don’t seem that similar to me).

 

Developer Tools

CodeStar

Sort of a New Project wizard for creating projects that use Amazon services, in a large number of languages. Generates a Git repository and the provisions the services your project type needs.

CodeCommit

Amazon does GitHub.

CodeBuild

Amazon does continuous integration. 

CodeDeploy

Automated deployment of built applications to EC2 instances.

CodePipeline

Visual workflow tool for orchestrating builds and deployments.

X-Ray

Log and trace collection and analysis for distributed applications.

 

Migration

Database Migration Service

Move your MySQL data into Amazon’s Aurora SQL database.

Server Migration Service

Move your physical PC into an EC2 VM.

Snowball

Move your local data into and out of Amazon by copying it to physical devices and shipping them around. 

 

That’s the core stuff. I’m not going to list every other service, just the ones that I think might be of interest to app developers.

 

CloudFront

CDN. Make your content available quickly worldwide.

Route53

Amazon sells domains?

CloudWatch

Monitoring for your AWS services.

CloudFormation

Create templates for sets of services that you typically deploy together, with a visual editor for adding services and relationships.

CloudTrail

Audit trail for things that happen across your AWS services.

TrustedAdvisor

Watches your AWS services and makes security, cost and other recommendations.

Inspector

Automated security analysis of your services.

Certificate Manager

Free SSL certificates for your AWS-hosted services.

Cognito

User account management services for your apps. Good blog post here.

WAF

Web application firewall.

Kinesis

Collect, process and analyze streaming data.

GameLift

Online game server hosting, with game engine integration.

Mobile Hub

Helps you provision AWS services and integrate them into your mobile app. Not really a service itself.

Device Farm

Deploy your app to real physical devices for testing. “AWS Device Farm tests are run on real, non-rooted devices”

Pinpoint

Spam your users with push notifications. I’m sorry, I mean “Engage Your Audience with Messaging Campaigns”. Also includes app usage analytics.

Step Functions

High-level service that lets you create workflows by stringing together Lambda functions.

SWF

Another workflow service, which on the surface seems pretty similar to Step Functions. 

AppStream

Stream desktop applications to a web browser.

Elastic Transcoder

Video format conversion.

API Gateway

Define your own external API to give to your developers as a front end for the services you’ve built in AWS.

 

There are also services under the AWS banner, like WorkDocs, WorkMail, WorkSpaces and Chime, that seem more like Office365 type user services than developer services.

Parsing Hex Strings in Swift

August 12th, 2017

I want to turn the string "0x02 0x03 0x04" into an array of bytes ([UInt8]). Seems simple enough right?

let hex = "0x02 0x03 0x04"
let components = hex.components(separatedBy: " ")
let array = components.map { UInt8($0) }

But the UInt8 constructor that takes a string doesn’t understand the “0x” prefix to indicate hex strings. There’s a handy UInt8("3F" radix:16) constructor that does understand hex but we have to remove the prefix first.

Swift doesn’t have an obvious way to remove the first 2 characters from a string.

This works:

let hexStr = "0x3F"
let hexValue = UInt8(hexStr.replacingOccurrences(of: "0x", with: ""), radix: 16)

But this will scan the entire string (admittedly only two more bytes in this case). But is there an easy way to just skip the first two bytes?

String has substring(from:String.Index) which seems like what we want, but "0x3F".substring(from:2) doesn’t compile. The String.Index is a type that’s managed by the String, to account for Unicode glyphs that are longer than one byte.

hexStr.substring(from: hexStr.index(hexStr.startIndex, offsetBy: 2)) will give us the correct substring but this is a mouthful – it’s hard to tell from scanning that line that all we want to do is remove the first two characters.

So we have these two options to do this with Swift string methods:

let hex = "0x3F"
let hex1 = hex.replacingOccurrences(of: "0x", with: "")
let hex2 = hex.substring(from: hex.index(hex.startIndex, offsetBy: 2))

As expected, a quick benchmark shows the second version is about 4 times faster than the first version.

There’s also Scanner, which can take the hex string with the 0x prefix and return a hex value, but it works with UInt32 and UInt64 types so there’s some extra gymnastics to get the result into a UInt8. Scanner benchmarks to be almost as fast as using substring, but using a separate class for this operation feels like more overhead than I want.

One of the great things about Swift is we can drop down to C code, and for this particular operation, there’s a C function that fits the bill exactly.

let hex3 = UInt8(strtoul(hex, nil, 16))

The C strtoul function converts a string given a radix, and for radix 16, ignores the prefix. And it’s 3x faster than the Swift substring version above.

For readability let’s wrap this up into a UInt8 extension:

extension UInt8 {
 static func from(hexString: String) -> UInt8 {
   return UInt8(strtoul(hex, nil, 16))
 }
}

With that in place we finally have:

let hex4 = UInt8.from(hexString: "0x3F")

Since making it easy to apply to other int sizes veers off into generics, I’ll leave that as an exercise for the reader.

Always-On iOS

June 23rd, 2017

The iPad is a perfectly capable computer, but one thing you can’t use the iPad for are jobs that take a long time.

If you’re on a desktop or laptop PC and you need to kick off a job that’s going to take a half hour, such as rendering a 3D video, you just do it. The machine sits there and does the job until it’s done. The computer may go to sleep, which will pause the job, but you can control if and when this happens. The laptop may run out of power, but plug it in and the job continues.

On iOS, you can kick off a long-running job, but you have to keep that app in the foreground until it completes. You can’t hit the Home button and go surf the web until it’s done, because iOS will kill the backgrounded app after 10 minutes, whether it’s done or not. Need to upload a large file to an FTP server? Better hope you can get it done in 10 minutes, or make sure you switch back to the FTP tool before the 10 minutes is up, or the app is killed.


This also means you can’t use an iOS device as a server. The hardware is more than capable of acting in a server role, listening for connections and responding to them, but because hitting the Home button would wind up eventually killing your server, you can’t reliably use an iOS device this way.

iOS has the ability to run background jobs. Photos, for example, downloads and processes your photos in the background. iCloud does this for documents. But no third party applications are allowed to do this.

Suppose I wanted to build an app that would monitor photos from a webcam over wifi, do image detection with the vision API, and take some action depending on what objects are detected. Totally possible and totally reasonable app to build, but the app would have to be launched interactively by the user and kept in the foreground. If the device were to lose power, a user would have to tap on the app to re-launch it.

iOS does seem like an odd platform for this sort of server application, but it’s the platform where all the innovation is happening. Some Apple technologies, like CoreML and HomeKit, are only supported on iOS.

I don’t mind that iOS is so aggressive with background processes when the device is running on battery power. That seems like the right tradeoff. But when plugged in, apps should be able to request, and be granted, the ability to run in the background as long as they need to.

OperationQueue and dynamic priorities

May 7th, 2017

I had a question about OperationQueue that I couldn’t find an easy answer to: Can you modify the priority of an operation once it’s been added to the queue?

My situation is that I have some background operations that are executing, but the user may choose to view one of these items interactively, and at that point, I’d like to bump that item’s priority so the user doesn’t have to wait for it to get its turn in the queue. I want that item to jump to the head of the line.

Many of the StackOverflow responses I looked at said that the queuePriority had to be set before the operation was added to the queue, but I couldn’t find a definitive answer in Apple’s documentation, so I put together a short playground to test it. Here’s the code:


import UIKit

let queue = OperationQueue()

// Serial queue, start suspended
queue.isSuspended = true
queue.maxConcurrentOperationCount = 1

// Create three operations
let operations = [
BlockOperation() {
print("o1")
},
BlockOperation() {
print("o2")
},
BlockOperation() {
print("o3")
}
]

// Add all the operations
operations.forEach { queue.addOperation($0) }

// Resume (one of the items will start executing immediately)
queue.isSuspended = false

// Now dynamically change the priority of one of the items
operations[2].queuePriority = .high

// Must wait for competion or you won't see the result
queue.waitUntilAllOperationsAreFinished()

Play with which operation you’re adjusting the priority on and you can see the that the priority changes are respected. You can adjust the priorities of operations on a queue.