For a few years, predating my adoption of emacs, I’ve kept a Quotes file with phrases I want to remember or come back to. Originally formatted as Markdown, converting it to org was easy enough with macros (pandoc would’ve been a great alternative if the structure was more complex).

read more

I was recently writing a paper where the submission requirements mandated using custom LaTeX class but omitting the title page typically produced when doing so.

After unsuccesfully experimenting with various org options to try and remove the title page, I realised you could drop down to LaTex and redefine the class’ \maketitle definition instead:

read more

Our team uses mkDocs as an internal wiki; a simple static site generator, it’s easy to host anywhere. At Microsoft, Azure is our go-to for internal services like this, and so we use Azure App Service as our deployment target. Though a PaaS offering, it works quite well for static sites as well - files deployed to the server root are served up by the IIS proxy that sits in front of the site.

Our mkDocs builds and deployments are managed by Visual Studio Team Services using the new YAML Builds. Every time a commit is made to the master branch, a new build is triggered; if succesful, the contents of the site directory on the build server are pushed to our docs site, ensuring it’s always up-to-date.

This is how we got everything up and running.

read more

Recent work has included a significant amount of time spent ssh‘d into different hosts; to avoid confusion about which machine I was executing commands on I decided to update my default bash prompt to include some additional information. If everything’s going well, my prompt looks like this:

Though simple at first glance, it’s quite information dense. The first bit, [1722] tells me this the 1722nd command in my bash_history, meaning I can easily rerun it with !1722.

The colour coding of the next square bracket delimited section means that the last command I ran executed successfully ($0 == 0), and I also know my username, my hostname, and my relative path. If I ssh into my bounce box or my Windows build machine, it’s now immediately clear which environment I’m operating in. Interested in trying it yourself?

read more

It’s a common pattern in web development to use environment variables for app configuration in different environments. My team wanted to use the same pattern for our Spark jobs, but documentation on how to set environment variables for an HDInsight cluster was hard to come by, but we eventually found a solution for HDInsight 3.6.

read more

I’m an avid user of org-capture for recording TODOs, code snippets, and reference material without interrupting my train of thought. A quick C-c c t lets me rattle off a note-to-self while capturing a link back to the buffer I’m in for later reference.

I recently found myself frequently repeating the pattern of calling kill-ring-save on a region to yank it into my org-capture buffer. By taking advantage of org-capture-templates and a little elisp, this pattern can be reduced to simply highlighting the target region before invoking capture.

read more

As we move past kicking the tires of Spark, we’re finding quite a few areas where the documentation doesn’t quite cover the scenarios you’ll run into day-after-day in a production environment. Today’s exemplar? Debugging jobs running in our staging cluster.

While taking a functional approach to data transformations allows us to write easily testable and composable code, we don’t claim to be perfect, and being able to set breakpoints and inspect values at runtime are invaluable tools in a programmer’s arsenal when println just won’t do.

The what of debugging JVM application in production is simple, and among the most powerful (and impressive!) capabilities of the Java platform. The Java Debug Wire Protocol (JDWP) enables attaching to a remote server as if it were on your machine. Just launch the JVM for your Spark driver with debug flags and attach. Easy, right?

Not quite.

read more