Author Archives: logicgrimoire

About logicgrimoire

I'm the author of 'Jelec: the White Bear, or, Beware an Encounter with a Raven and his Friends.' There is a second book in the series on the way.

Announcing flycheck-pod.el, Flycheck support for Perl 5 POD files

Recently I was writing some Plain Old Documentation (aka POD) for one of my Perl modules, and I thought it would be convenient to have Flycheck support while writing docs.

Since I couldn’t find the Elisp code already written to do this, I whipped something up: flycheck-pod.

As of right now, it supports error checking in Emacs’ various POD and Perl modes using podchecker in the background. I doubt it covers all of the features of POD::Checker though — patches are certainly welcome. See the source code for details.

(Image courtesy Kenneth Lu via Creative Commons license.)

Advertisements

How to use Locate from Emacs on Windows

If you are like me, you like to:

  • Live in Emacs as much as possible to avoid context-switching
  • Set up Emacs so your environment abstracts the OS as much as possible

Being able to sit down at any of my computers and type M-x locate in Emacs is a requirement for me, even if It’s running Windows underneath.

In this post I’ll describe how to set up a locate(1) command on Windows 10, and how to access it from Emacs.

Step 1. Install Locate32

Download and install locate32 on your machine. It doesn’t have an installer, it just gives you a directory full of things, including the locate.exe binary. I put mine in "C:/Users/rml/Programs/locate32/", and added that location to my Windows %PATH%.

Step 2. Tell Emacs where to find it

In Emacs, set the value of the locate-command variable to wherever you ended up putting it. Here’s where it is on my machine:

(setq locate-command "c:/Users/rml/Programs/locate32/locate.exe")

Step 3. Locate all the things

Now when you run the M-x locate command from inside Emacs, it should give you a Dired buffer of results, the same way it does on other systems. Because it’s Dired, you can hit enter on a filename to visit it or mark files in various ways and then operate on them.

Here’s what it looks like on my Windows 10 laptop if I search for the text “svn”:

How to use CockroachDB with Emacs SQL Mode

old skool

(Image courtesy Sajith T S under Creative Commons license.)

In this post I’ll describe how to hack up the Postgres config for sql.el to work with CockroachDB (aka “CRDB”).

In the future I’d like to submit code to sql.el to make CRDB a fully-supported option, but for now this is what I’ve been using.

Note that these instructions assume you are running a secure local cluster. If you are running insecure, you can avoid all of this and just M-x sql-postgres and use most of the defaults, modulo port numbers and such. It uses psql on the backend which works because CRDB speaks the Postgres wire format.

However, once you get up and running for real and want to use a secure network connection, it’s easier to use the cockroach sql client. That’s what we’ll configure in this post.

(It may be possible to configure psql to use the CRDB certs, but I don’t know since I haven’t looked into it. Also, keep in mind that I have not tested this setup over the network yet – only on my local machine.)

Step 1. Modify basic config

Since the client is invoked by comint as two “words”, cockroach sql, you have to mess with the options a bit.

First set the cockroach binary:

(setq sql-postgres-program "cockroach")

Then invoke the SQL client with the first arg, and pass options in with the rest. The certs directory is where your encryption certificates are stored. Since this is an ephemeral local cluster I’m using the temp directory.

(setq sql-postgres-options
      '("sql" "--certs-dir=/tmp/certs"))

Finally the login params are pretty standard. My local clusters are not usually long-lived, so I just use the “root” user. This would not be recommended on real systems of course.

(setq sql-postgres-login-params
      '((user :default "root")
        (database :default "")
        (server :default "localhost")
        (port :default 26500)))

Step 2. Modify the Postgres “product” to work with CRDB

Sql.el calls each of its supported databases “products”. for whatever reason.

In any case, here’s how to modify the Postgres product to work for CRDB.

First we need a new function to talk to comint.el. (See the bottom of this post for the definition since it’s longer and not interesting.)

(sql-set-product-feature 'postgres
                         :sqli-comint-func #'sql-comint-cockroach)

The usual comint prompt regexp things. This one isn’t that well tested but works on my machine ™ … so far.

(sql-set-product-feature 'postgres
                         :prompt-regexp "^[a-z]+\@[a-zA-Z0-9\.-_]+:[0-9]+/\\([a-z]+\\)?> ")

I don’t really know what this does. The CRDB prompt is not necessarily of a fixed length so it doesn’t really apply. It seems to have no effect, I just cargo culted it from some other DBs. Probably not needed.

(sql-set-product-feature 'postgres
                         :prompt-length 0)

Regexp to match CRDB’s little continuation marker thingy:

(sql-set-product-feature 'postgres
                         :prompt-cont-regexp "^ +-> ")

Set the “end of a SQL statement” character to the semicolon. Some of the other DBs have some pretty fancy settings here, but this seems to mostly work.

(sql-set-product-feature 'postgres
                         :terminator '(";" . ";"))

Command to show all the tables in the current database.

(sql-set-product-feature 'postgres
                         :list-all "SHOW TABLES;")

And finally, this is the comint function we need to work with CRDB:

(defun sql-comint-cockroach (product options)
  "Create comint buffer and connect to CockroachDB."
  (let ((params
         (append
          (if (not (= 0 sql-port))
              (list "--port" (number-to-string sql-port)))
          (if (not (string= "" sql-user))
              (list "--user" sql-user))
          (if (not (string= "" sql-server))
              (list "--host" sql-server))
          options
          (if (not (string= "" sql-database))
              (list sql-database)))))
    (sql-comint product params)))

It’s all about the BATNA

8109693804_d735a19f95_z
 
(Image courtesy Ismael Celis under Creative Commons license.)

It seems like there is a constant stream of articles being turned out about how we’re all going to be working in Amazon fulfillment centers and holding in our pee for 12 hours while we dry-swallow bottles of Aleve and live in fear of our slave-driving lower-level warehouse managers.

You can read a lot of these types of articles on sites like the Verge for some reason. (I am beginning to think of them – at least in part – as “nominally ‘tech’ but actually ‘tech pessimism'” sites.)

Meanwhile, there is another – perhaps-less-frequent but still influential – stream of articles about how companies “can’t find” good employees, they “can’t hire”, millennials want “too much” from their employers, Americans “won’t work hard” and “don’t have the necessary skills” for “the future” ™, and so on.

You can probably read these articles in the Wall Street Journal.

The NY Times, that bourgeois rag, will happily run both types of article. (Parts of its demographic hold both views, in some cases simultaneously, and hey, the ads pay either way.)

Unfortunately there is an important concept taken from business negotiation called BATNA that is almost never even mentioned in either type of article – even though it usually explains the behaviors chronicled in the article! I could almost forgive this if the writer had studied journalism and not economics (although not really), but if they have any economics or business background at all it’s just criminal.

What is BATNA though, really? Well you can read the wiki article for more information, but it is an acronym that means “Best Alternative To Negotiated Agreement”. In other words, it’s a way of thinking during any type of negotiation about questions of the form “What’s my next best option if this deal falls through?”

For example, if you are an employer with a lot of cash on the balance sheet you can afford to wait a few quarters (or years) until employee wages come down to a level you find more appealing, maybe. If you are a wage-earning employee, you probably cannot. (Not to mention that it’s probably cheaper for companies to have their PR people push articles in the WSJ about how hard it is to hire than it is to just raise wages until hiring picks up.)

P.S. Special thanks to Andrew Kraft, who gave a great talk on BATNA and other related topics a few years back at AppNexus. Without his talk, I might never have heard of this magical acronym.)

Thoughts on Rewrites

As a user, when I hear engineers start talking about doing a rewrite of an application or an API that I depend on, I get nervous. A rewrite almost never results in something better for me.

Based on personal experience, I have some (possibly unfair) opinions:

  • Rewrites are almost always about the engineering organization
  • They are almost never about the end users
  • Inside any given organization, it’s very difficult for people to understand this because their salary depends on them not understanding it
  • Attempts at rewriting really large apps rarely get to a state of “fully done”, so the engineers may end up with a Lava Layer anyway
  • Except now users are angry because features they depended on are gone

Why am I writing this? Because I’m still mad they took away my Opera.

Until recently, I’d been using Opera for over a decade. By the time Opera 12 came out, it was amazing. It had everything I needed. It was lightweight, and could run on computers with less than a gig of RAM. With all of the keyboard shortcuts enabled, I could slice and dice my way through any website. I could browse the web for hours without removing my hands from the keyboard, popping open tabs, saving pages for later reference, downloading files. It was amazing.

Oh, and Opera also had a good email client built in. It was, like the browser part, lightweight and fast, with keyboard shortcuts for almost everything. It also read RSS feeds. Oh, and newsgroups too. It had great tagging and search, so you could really organize the information coming into your world.

Then they decided to take it all away. They didn’t want to maintain their own rendering engine anymore. They let go of most of the core rendering engine developers and decided to focus on making Yet Another Chromium Skin ™. No mail reader. Most of the keyboard shortcuts gone. Runs like shit (or not at all) in computers with 1 gig of RAM.

I realize I got exactly what I paid for. But if you are wondering why users get twitchy when engineers and PMs start talking about rewrites, wonder no longer.

After Opera stopped getting maintenance, I switched back to Firefox, and fell in love with Pentadactyl, the greatest “make my browser act like Vim” addon that ever was.

Can you guess what happened next? Yep, they decided to rewrite everything and break the addon APIs. I know they had some good reasons, but those reasons meant the end of my beloved Penta. Now I am back to using Firefox with Vimium (like an animal), and I suppose I should be grateful to have even that.

And don’t get me started on my experiences with “REST APIs”, especially in a B2B environment.

Related:

Set up Gnus on Windows

There are many “set up Gnus to read email from Emacs on Windows” posts. This one is mine. Unlike the 10,000 others on the internet, this one actually worked for me.

A nice thing is that, with a few tweaks, this setup also works on UNIX-like machines.

PREREQUISITES

OVERVIEW

At a high level, the way this all works is that:

  • A mail server is out there on the interwebs somewhere
  • stunnel runs locally, and creates an encrypted “tunnel” between a port on the mail server and a port on the local machine

  • Emacs (Gnus) connects to the local port and fetches mail from there (as far as it knows)

STEP 1. INSTALL AND CONFIGURE STUNNEL

Download and install stunnel for Windows:
https://www.stunnel.org/downloads.html

I use Fastmail, so the following configuration worked for me. I put it in the file ‘C:/Users/rml/_stunnel.conf’.

# Windows stunnel config

# 1. GLOBAL OPTIONS

debug = 7
output = C:/Users/rml/Desktop/stunnel.log

# 2. SERVICE-LEVEL OPTIONS

[IMAP (rmloveland@fastmail.fm) Incoming]
client = yes
accept = 127.0.0.1:143
connect = mail.messagingengine.com:993

[SMTP (rmloveland@fastmail.fm) Outgoing]
client = yes
accept = 127.0.0.1:465
connect = mail.messagingengine.com:465

If memory serves, you will need to do some messing around with stunnel to get it to read from a config file other than the default. Luckily it puts a little icon in the notification tray that you can right-click to get it to do things such as edit the config file or view the log file. From there, you should be able to get the config in shape as shown above.

In the particular case of Fastmail, you’ll need to set up an app password via its web UI. See your email provider’s documentation for more information.

STEP 2. CONFIGURE GNUS

On the Emacs side, we need Gnus to ask the right port on the local machine for mail. Here’s what I did:

(setq send-mail-function 'smtpmail-send-it
message-send-mail-function 'smtpmail-send-it
smtpmail-smtp-server "localhost"
smtpmail-smtp-service 465
smtpmail-stream-type nil
smtpmail-default-smtp-server "localhost")

This is the part of your Gnus config that tells it how to talk to stunnel; all of the other Gnus things are beyond the scope of this article. If you need more Gnus info, you should be able to get something going using the EmacsWiki:
https://www.emacswiki.org/emacs/CategoryGnus

A Trivial Utility: Prepend

Recently at work I needed to add a timestamp to the top of a bunch of Markdown files. There are plenty of ways to skin this particular cat. As you probably know, the semantics of how you navigate UNIX file contents mean it’s easy to add something to the end of a file, but it’s not as easy to add something to the beginning.

This is a pretty trivial task that other people have solved in lots of ways. In my case, I decided against a shell script using sh or the like because I use Windows a bunch, too, and I wanted something cross-platform. As usual for me, this meant breaking out Perl.

I decided to name the tool prepend, on the grounds that that’s what it does: it adds text content to the beginning of a file.

Since I like to design top-down, let’s look at how it’s meant to be used:

$ prepend STRING_OR_FILE FILES

There are two ways to use it:

  1. Add a string to the beginning of one or more files
  2. Add the contents of a file to the beginning of one or more files

Let’s say I wanted to add a timestamp to every Markdown file in a directory. In such a case I’d add a string like so:

$ prepend '<!-- Converted on: 1/26/2017 -->' *.md

If I had some multi-line text I wanted to add to the beginning of every Markdown file, I’d say

$ prepend /tmp/multiline-preamble.md *.md

The code is shown below. I could have written it using a more low-level function such as seek but hey, why fiddle with details when memory is cheap and I can just read the entire file into an array using Tie::File?

#!/usr/bin/env perl

use strict;
use warnings;
use experimentals;
use autodie;
use IO::All;
use Tie::File;

my $usage = <<"EOF";
Usage:
    \$ $0 MESSAGE_OR_FILE FILE(S)
e.g.,
    \$ $0 '<!-- some text for the opening line -->' *.md
OR
    \$ $0 /tmp/message.txt *.txt
EOF
die "$usage\n" unless scalar @ARGV >= 2;
my @files = @ARGV;

my $maybe_file = shift;
my $content;

if (-f $maybe_file) {
  $content = io($maybe_file)->slurp;
}
else {
  $content = $maybe_file;
}

for my $file (@files) {
  my @lines;
  tie @lines, 'Tie::File', $file;
  unshift @lines, $content;
  untie @lines;
}

Some thoughts about privacy and networked computers

According to the Merriam-Webster dictionary, privacy is:

a : the quality or state of being apart from company or observation

b : freedom from unauthorized intrusion

Historically, the most common use of this term was around one’s physical space. If you go into a room in your house and close the door, you are experiencing privacy. You are “apart from” others. They cannot see you, hear you, etc.

If you go and sit in your back yard, then depending on the visibility of your back yard to neighbors and passersby, you are experiencing some degree of privacy.

If you send a letter to a friend (the kind that is written on paper and wrapped in an envelope), you have an expectation of privacy in that you expect that your letter will not be read.

In each of these cases, there is a physical barrier that separates the space that you (or your communications) occupy from space that is available for other people to see and observe.

In each of these cases, the physical objects in question do not broadcast information. There is visual information available to any passersby or other residents of the home, but the passerby must take action to look, to seek it out.

It seems that historical notions of privacy have to do with physical presence and a third party must make an effort to transgress a boundary that you have explicitly put in place (a door, wall, envelope, etc.).

Networked computers do not have any of these characteristics!

A networked computer is, in essence, a beacon that is constantly shining in the night. Networked computers constantly transmit information to other computers, and you have to do a lot of work if you want to keep that from happening (and you will probably not succeed anyway, even if you have a lot of relevant expertise). To use the beacon metaphor, it is shining all of the time, you have to do a lot of work to keep it covered up, and any slip of the covering means you will be visible from many miles away.

This flips the notion of what we historically think of as “privacy” completely on its head. Rather than a third party being forced to transgress a boundary to see you in a private room, tear open your letter’s envelope, or jump your backyard fence, the third party running the computer network that you connect to, or the server that your browser connects to, would have to explicitly take action to drop information on the floor that they have already been given.

This is a fundamentally different thing to ask for. In the first case, you are saying, “please don’t cross this physical boundary”. In the second, you are saying “I am sending you this information, but please don’t read it. Well actually, that won’t work — you will have to read it to provide me the service I’m asking for. But after providing the service, please go back and erase the information. Definitely don’t store it anywhere.”

Whereas in the first case you were asking the third party to simply avoid a behavior, in the second case you are asking the third party to do work on your behalf. This is going to be fundamentally harder to accomplish, and you really need to understand that you are asking someone else to do something for you. Implied in asking someone else to do work on your behalf is that they are not obligated to do that work, except under certain conditions or relationships.

You almost certainly do not enjoy these conditions or relationships with network operators, computer manufacturers, the writers of web browsing software, web applications, or advertising technology. You are not in a position to demand extra work from these entities.

Another way of looking at it is: given a possibility space of all behaviors in the physical realm, traditional privacy just carves out a small area of the total space and says “don’t go here”. It looks like this:

privacy-venn-01

Given the possibility space of all of the behaviors that can be engaged in by networked computers, “privacy” carves out an area of the total space and says “you will have to go here at least once to provide me with network connectivity and other services, but I want you to then take a second pass over that area and erase/drop the information you collected during the first pass”.

privacy-venn-02

I hope the above explains why I do not really like or agree with the use of the word “privacy” in discussions about computers. It’s the wrong word. I don’t know if we even have a word for what we need going forward.

Advent of Code 2017, Day 2

This is my solution for Day 2 of this year’s Advent of Code.

You may also enjoy browsing the Day 2 solutions megathread on Reddit.

PROBLEM

The spreadsheet consists of rows of apparently-random numbers. To make sure the recovery process is on the right track, they need you to calculate the spreadsheet’s checksum. For each row, determine the difference between the largest value and the smallest value; the checksum is the sum of all of these differences.

For example, given the following spreadsheet:

5 1 9 5
7 5 3
2 4 6 8

The first row’s largest and smallest values are 9 and 1, and their difference is 8.

The second row’s largest and smallest values are 7 and 3, and their difference is 4.

The third row’s difference is 6.

In this example, the spreadsheet’s checksum would be 8 + 4 + 6 = 18.

SOLUTION

(define (line->list line)
  ;; String -> List
  (let ((read-ln (field-reader (infix-splitter (rx (+ whitespace)))))
        (in-port (make-string-input-port line)))
    (receive (record fields)
        (read-ln in-port)
      (map string->number fields))))

(define (read-spreadsheet file)
  ;; File -> List[List[Number]]
  (call-with-input-file file
    (lambda (port)
      (let loop ((line (read-line port))
                 (results '()))
        (if (eof-object? line)
            results
            (loop (read-line port) (cons line results)))))))

(define (main prog+args)
  (let ((rows (read-spreadsheet "/Users/rloveland/Code/personal/advent-of-code/2017/02/02.dat")))
    (write (apply + (map
                     (lambda (row)
                       (let* ((xs (line->list row))
                              (min (apply min xs))
                              (max (apply max xs)))
                         (- max min)))
                     rows)))
    (newline)))

Advent of Code 2017, Day 1

This is my solution for Day 1 of this year’s Advent of Code.

You may also enjoy browsing the Day 1 solutions megathread on Reddit.

PROBLEM

The captcha requires you to review a sequence of digits (your puzzle input) and find the sum of all digits that match the next digit in the list. The list is circular, so the digit after the last digit is the first digit in the list.

For example:

  • 1122 produces a sum of 3 (1 + 2) because the first digit (1) matches the second digit and the third digit (2) matches the fourth digit.

  • 1111 produces 4 because each digit (all 1) matches the next.

  • 1234 produces 0 because no digit matches the next.

  • 91212129 produces 9 because the only digit that matches the next one is the last digit, 9.

SOLUTION

(define captcha-input "5994521226795838")

'(set! captcha-input "1111")

'(set! captcha-input "1122")

'(set! captcha-input "1234")

'(set! captcha-input "91212129")

(define (gather-matches s)
  ;; String -> List
  (let ((in-port (make-string-input-port s)) (count 0) (head #f) (vals '()))
    (let loop ((cur (read-char in-port)) (next (peek-char in-port)) (count count) (vals vals))
      (if (eof-object? next)
          (if (char=? cur head)
              (cons cur vals)
              vals)
          (cond ((= count 0)
                 (begin
                   (set! head cur)
                   (loop cur next (+ 1 count) vals)))
                 ((char=? cur next)
                 (loop (read-char in-port) (peek-char in-port) (+ 1 count) (cons cur vals)))
                (else (loop (read-char in-port) (peek-char in-port) (+ 1 count) vals)))))))

(define (main prog+args)
  (let* ((matches (gather-matches captcha-input))
         (matches* (map (lambda (c) (string->number (string c))) matches))
         (sum (apply + matches*)))
    (begin
      (format #t "MATCHES*: ~A~%" matches*)
      (format #t "SUM: ~A~%" sum))))