Automation considered harmful

Intro
I've had a particularly unproductive couple of days in the office so far this week.  Two quite separate projects which my current project depends on have had enough of an impact to prevent me from testing code changes, or checking in merged content.

We are still investigating one of the problems, so I will just focus on the problem that has been fully identified and resolved, saving the continuous integration issue for another post.

Background
I have a development machine which sits under my desk and acts as a local content management system server.  I use this to try out functionality and present demonstrations without fear of scheduled or unscheduled downtime.

To ensure that this system doesn't fall out of sync with new functionality that is being developed by another team, I periodically call on the automation magic of chef to obtain the latest binaries and configuration.

A week or so ago the chef update failed partway through.  This wasn't a major problem as the application would still run, however I was not in a position to identify the cause of the problem or how to fix it.

The problem
This week I decided to try the chef update again, as a colleague had agreed to address the earlier problem.  This seemed like a good idea at the time, but resulted in the chef update failing - and leaving the content management applications unresponsive.

Here we go again I thought, except now it's a higher priority issue for me as this week's development needed this all to be running.

Thankfully the remote team responsible for managing the chef configuration had some availability to look into this issue.  Unfortunately the individual involved didn't seem to have enough context to appreciate my non-virtualized setup etc.  So it was time for me to take another dig around the chef-managed setup.

Ultimately it came down to the usual diagnostic approach of checking the log files for the various applications involved.  One of the content management server processes failed to initialise with a duplicate column error.

Like many modern extensible software products, this particular content management system automatically manages the structure of an underlying relational database.  When new properties need to be represented, a developer can specify that in a configuration file - which will ultimately trigger an alteration to a database table.

Tracing back through git commits showed up what was special about this duplicate column - it was actually a special case of renaming the column with a different - non-camel - casing.

Summary
I wouldn't criticise any of the technological approaches involved:

• It made sense for the content management system to flag up the unusual database structure change
• It made sense to use chef for managing updates to the binaries and configuration

It just feels quite strange that I have started off simply being a consumer of a web application or web service with my own local installation, but ended up having to delve into multiple relational databases to rename some columns.

To tie back to the cheesy considered harmful title, using tools such as chef without understanding what they are doing or without having access to examine their effect can result in problems and delays.  This ties back to a concept that I keep coming across sometimes you don't know what you don't know.

Sample Java code for obtaining an authorisation token for your Twitter application

Here is an example of how to obtain an authorisation token from Twitter, using your registered application's API key and API secret.

The relevant REST API documentation can be found at: https://dev.twitter.com/docs/api/1.1/post/oauth2/token

For the HTTP client I have made use of the ever popular Apache HTTP client (v4).

The response should include a JSON document containing a property keyed by access_token.

There are a few potential gotchas involved with including this in a multithreaded application, so I'd recommend that you read Twitter's documentation carefully.

        final String basicAuthentication = TwitterCredentials.API_KEY + 

                ":" + TwitterCredentials.API_SECRET;

        String base64EncodedAuthentication = Base64.getEncoder().encodeToString(basicAuthentication.getBytes(StandardCharsets.UTF_8));

        HttpClient client = new DefaultHttpClient();

        HttpHost httpHost = new HttpHost("api.twitter.com", 443, "https");

        HttpPost httpRequest = new HttpPost("https://api.twitter.com/oauth2/token?grant_type=client_credentials");

        Header authenticationHeader = new BasicHeader("Authorization", "Basic " + base64EncodedAuthentication);

        httpRequest.addHeader(authenticationHeader);

        httpRequest.addHeader("Content-Type", "application/json; charset=utf-8");

        HttpResponse httpResponse = client.execute(httpHost, httpRequest);

        HttpEntity entity = httpResponse.getEntity();

        String responseBody = EntityUtils.toString(entity);

Twitter referrals in Google Analytics

Twitter has its own short url for tweets, which will show up in Google Analytics reports as referrer starting with http://t.co

To trace back to the tweet we can use Twitter's search with the url as the search term.

I recently tweeted a link to a blog post, and sure enough the subsequent day included a referrer from a URL below http://t.co

The search result can be seen from:

https://twitter.com/search?q=http%3A%2F%2Ft.co%2FcqWPRVyJ9l

Chances are that this will stop working by the time you get around to reading this, as the search time range is rather limited (a few days?)

Getting up and running with Google Analytics Core Reporting API 3.0

Update: Google have recently improved their documentation so it should be straight-forward to get set up:  Core Reporting API - Developer Guide


Back in late 2013 I struggled to get anything to work with Google's Java client code for version 3 of their analytics API.

The sample code and documentation seemed to be somewhat lacking compared to what was then available for version 2.

A week or so ago I decided that it might be worthwhile to have another crack at making something work.  After much trial and error I now have some code which will successfully send requests to and receive responses from Google's Analytics service.

Before you get too excited, I'll just add the caveat that this particular setup involves the application triggering a browser to open so that a Google account associated with the Analytics account can be prompted to authorise the access to the data.

If this code isn't useful to you, then so be it but I may find myself Googling for this information in the future so here goes.

Dependencies

- at least one includes the dreaded alpha in the version, so expect it to require updating in the future:

• com.google.apis:google-api-services-analytics:v3-rev83-1.17.0-rc
• com.google.api-client:google-api-client-jackson2:1.17.0-rc
• com.google.api.client:google-api-client-auth-oauth2:1.2.3-alpha
• com.google.oauth-client:google-oauth-client-java7:1.16.0-rc
• com.google.oauth-client:google-oauth-client-jetty:1.17.0-rc

Sample code for authentication

Specify your own CLIENT_ID, CLIENT_SECRET and APPLICATION_NAME values.

        HttpTransport transport = new NetHttpTransport();

        JsonFactory jsonFactory = new JacksonFactory();

        GoogleClientSecrets secrets = new GoogleClientSecrets().set(

                "client_id", CLIENT_ID).set("client_secret", CLIENT_SECRET);

        HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();

        JsonFactory jsonFactory = JacksonFactory.getDefaultInstance();

        DataStoreFactory dataStoreFactory = new MemoryDataStoreFactory();

        GoogleAuthorizationCodeFlow.Builder builder = 

                new GoogleAuthorizationCodeFlow.Builder(httpTransport, jsonFactory, 

                        CLIENT_ID,

                        CLIENT_SECRET,

                        Collections.singleton(AnalyticsScopes.ANALYTICS_READONLY));

        GoogleAuthorizationCodeFlow flow = builder.setDataStoreFactory(dataStoreFactory).build();

        final Credential credential = new AuthorizationCodeInstalledApp(flow, 

                new LocalServerReceiver()).authorize("user");

        Analytics analytics = new Analytics.Builder(transport, jsonFactory, credential)

                .setApplicationName(APPLICATION_NAME).setHttpRequestInitializer(

                        new HttpRequestInitializer() {

                    @Override

                    public void initialize(HttpRequest httpRequest) throws IOException {

                        credential.initialize(httpRequest);

                        httpRequest.setConnectTimeout(30000);

                        httpRequest.setReadTimeout(60000);

                    }

                }).build();

Sample code for request on API

Specify your own profileId value.

        Analytics.Data data = analytics.data();

        Analytics.Data.Ga ga = data.ga();

        // External search

        Analytics.Data.Ga.Get get = ga.get("ga:" + profileId,

                "2013-01-01",              // Start date.

                "2013-12-31",              // End date.

                "ga:pageviews")  // Metrics.

                .setDimensions("ga:keyword").setSort("-ga:pageviews");

        GaData results = get.execute();

Do what you like with the results object - probably some iterating :)

What next

Because my intended use of the API will ultimately involve some server side application, I will need to come up with a different approach to authentication.

Let me know if you found this post useful.

Game logic

During a job interview a few years ago I was asked how I would go about implementing a simple game of noughts and crosses.

I was fine with defining the objects, but when it came to the logic for determining when someone had won I found myself overthinking the situation, considering pre-loading the system with won states and using some kind of map lookup, or navigating the game state with some kind of tree of neighbouring cells based on the last move and following a backtracking algorithm.

In the case of Noughts and crosses there are only 8 possible winning combinations per player, with each combination involving checking the state of 3 positions on the playing grid.

A brute force approach would be to check every possibility until finding a win or running out of possibilities.

A more elegant solution would only consider the combinations that involve the most recently played move. This would introduce a requirement of being able to determine which winning states are related to the new move's grid position.

* Footnote: The interview was so long ago that I don't recall the company - or the interview.  I found this post awaiting publication so have updated the wording accordingly.

Playing with Activator

I've had a bit of time to myself at work recently, so have taken the opportunity to download and try out the latest available technology stack from my department's preferred provider as a test client for some services that I've been developing.

So far I have found Typesafe Activator to be a surprisingly usable rapid prototyping system.

There are a few typos in the documentation and the occasional minor background process failure indicates that it is still a work in progress, but I have managed to develop a simple web application which consumes a JSON/REST web service with about 20 lines of code (excluding imports).

This has been my first time trying to use Scala and Play for anything potentially useful in real life, so I was pleasantly surprised at how productive the development cycle was - within a web browser.

Being able to add some code, see the syntax colouring highlight any newbie errors, trigger a compile and then hit my controller with a request without leaving the browser was borderline fun.

For my current project this stack should be sufficient, but once my Scala confidence gets a bit higher I expect to move on to trying out Akka and Spray.