Better Code Snippets with org-capture

30. April 2018

A question on /r/emacs inspired me to improve the template I was using for saving code snippets with org-capture. Previously, I’d been using the %^{prompt} expansion to manually enter the language of the snippet; this poster made me realise that could be automated.

They also introuduced me to which-function, which is particularly helpful when you’re only saving a portion of a larger method. Click through for the elisp I wrote.

read more

A Brief Introduction to Literate Analytics With org-babel

30. April 2018

As an engineer on an analytics and data infrastructure team, I spend quite a bit of time exploring and performing ad-hoc analyses of data scattered across SQL databases; Hive and Spark clusters; and blob storage. These investigations can be a chore – a single instance might involve a half dozen ETL steps, ad-hoc querying to assess whether data needs to be cleaned, and moving code between a REPL and a program destined for production.

Once an initial conclusion has been reached, I need to go back and document my process for posterity, remembering exactly what worked along the way. This is a recipe for mistakes and omissions, so why not document as you go?

What I need to do is:

  1. Log each command
  2. Document what it does and why it’s necessary
  3. Execute and examine the results as I go

org-mode and org-babel provide an amazing environment for this sort of polyglot, multi-modal series of tasks. Prose and code can be freely intermingled in a literate programming style, any references used during the process can be cited, and the final product can be exported into human- or machine-readable formats. If you’re keen to see the final product of an analysis, here’s an example I recently created for Tea With Strangers, reporting on growth in recent months that demonstrates these capabilities.

read more

mkDocs on Azure App Service + CD with VSTS

10. April 2018

Our team uses mkDocs as an internal wiki; a simple static site generator, it’s easy to host anywhere. At Microsoft, Azure is our go-to for internal services like this, and so we use Azure App Service as our deployment target. Though a PaaS offering, it works quite well for static sites as well - files deployed to the server root are served up by the IIS proxy that sits in front of the site.

Our mkDocs builds and deployments are managed by Visual Studio Team Services using the new YAML Builds. Every time a commit is made to the master branch, a new build is triggered; if succesful, the contents of the site directory on the build server are pushed to our docs site, ensuring it’s always up-to-date.

This is how we got everything up and running.

read more

A Better bash Prompt

1. April 2018

Recent work has included a significant amount of time spent ssh‘d into different hosts; to avoid confusion about which machine I was executing commands on I decided to update my default bash prompt to include some additional information. If everything’s going well, my prompt looks like this:

[1722][Nick@prozess: ~]$

Though simple at first glance, it’s quite information dense. The first bit, [1722] tells me this the 1722nd command in my bash_history, meaning I can easily rerun it with !1722.

The colour coding of the next square bracket delimited section means that the last command I ran executed successfully ($0 == 0), and I also know my username, my hostname, and my relative path. If I ssh into my bounce box or my Windows build machine, it’s now immediately clear which environment I’m operating in. Interested in trying it yourself?

read more

Capturing Regions with Org Capture

28. February 2018

I’m an avid user of org-capture for recording TODOs, code snippets, and reference material without interrupting my train of thought. A quick C-c c t lets me rattle off a note-to-self while capturing a link back to the buffer I’m in for later reference.

I recently found myself frequently repeating the pattern of calling kill-ring-save on a region to yank it into my org-capture buffer. By taking advantage of org-capture-templates and a little elisp, this pattern can be reduced to simply highlighting the target region before invoking capture.

read more

Remote Debugging Spark Jobs

11. January 2018

As we move past kicking the tires of Spark, we’re finding quite a few areas where the documentation doesn’t quite cover the scenarios you’ll run into day-after-day in a production environment. Today’s exemplar? Debugging jobs running in our staging cluster.

While taking a functional approach to data transformations allows us to write easily testable and composable code, we don’t claim to be perfect, and being able to set breakpoints and inspect values at runtime are invaluable tools in a programmer’s arsenal when println just won’t do.

The what of debugging JVM application in production is simple, and among the most powerful (and impressive!) capabilities of the Java platform. The Java Debug Wire Protocol (JDWP) enables attaching to a remote server as if it were on your machine. Just launch the JVM for your Spark driver with debug flags and attach. Easy, right?

Not quite.

read more