Tag Archives: unix

A Trivial Utility: Prepend

Recently at work I needed to add a timestamp to the top of a bunch of Markdown files. There are plenty of ways to skin this particular cat. As you probably know, the semantics of how you navigate UNIX file contents mean it’s easy to add something to the end of a file, but it’s not as easy to add something to the beginning.

This is a pretty trivial task that other people have solved in lots of ways. In my case, I decided against a shell script using sh or the like because I use Windows a bunch, too, and I wanted something cross-platform. As usual for me, this meant breaking out Perl.

I decided to name the tool prepend, on the grounds that that’s what it does: it adds text content to the beginning of a file.

Since I like to design top-down, let’s look at how it’s meant to be used:

$ prepend STRING_OR_FILE FILES

There are two ways to use it:

  1. Add a string to the beginning of one or more files
  2. Add the contents of a file to the beginning of one or more files

Let’s say I wanted to add a timestamp to every Markdown file in a directory. In such a case I’d add a string like so:

$ prepend '<!-- Converted on: 1/26/2017 -->' *.md

If I had some multi-line text I wanted to add to the beginning of every Markdown file, I’d say

$ prepend /tmp/multiline-preamble.md *.md

The code is shown below. I could have written it using a more low-level function such as seek but hey, why fiddle with details when memory is cheap and I can just read the entire file into an array using Tie::File?

#!/usr/bin/env perl

use strict;
use warnings;
use experimentals;
use autodie;
use IO::All;
use Tie::File;

my $usage = <<"EOF";
Usage:
    \$ $0 MESSAGE_OR_FILE FILE(S)
e.g.,
    \$ $0 '<!-- some text for the opening line -->' *.md
OR
    \$ $0 /tmp/message.txt *.txt
EOF
die "$usage\n" unless scalar @ARGV >= 2;
my @files = @ARGV;

my $maybe_file = shift;
my $content;

if (-f $maybe_file) {
  $content = io($maybe_file)->slurp;
}
else {
  $content = $maybe_file;
}

for my $file (@files) {
  my @lines;
  tie @lines, 'Tie::File', $file;
  unshift @lines, $content;
  untie @lines;
}
Advertisements

How to install Perlbrew on Fedora 20 and use it with (m)ksh

yen-bill-crane

This post details how to go about installing perlbrew on a Fedora 20 Linux system for a user whose default shell is in the ksh family, specifically mksh, which is the shell I use.

The instructions in each section can be used separately if you are just a Fedora user or just a (m)ksh user.

Installing Perlbrew on Fedora 20

If you try to install Perlbrew using the instructions on its website, Fedora barfs. Fortunately, the workaround you need is detailed in this Github issue. (Hint: read the whole thread and then do what wumpus says.)

Using Perlbrew with ksh or mksh

The perlbrew bashrc file uses a few “bashisms” (unnecessarily, I think, but it is a “bash” rc after all). I managed to hack them out and, instead of sourcing perlbrew’s default bashrc file, I now source a kshrc built by applying the patch below.

It’s working well for me so far under mksh, but feel free to leave a comment if it doesn’t work for you and I’ll try to help.

Happy Perlbrewing!

--- bashrc	2014-12-30 12:49:14.110446784 -0500
+++ kshrc	2014-12-30 13:15:58.750454519 -0500
@@ -1,6 +1,5 @@
 export PERLBREW_BASHRC_VERSION=0.72
 
-
 __perlbrew_reinit() {
     if [[ ! -d "$PERLBREW_HOME" ]]; then
         mkdir -p "$PERLBREW_HOME"
@@ -13,7 +12,9 @@
 }
 
 __perlbrew_purify () {
-    local path patharray outsep
+    path=
+    patharray= 
+    outsep=
     if [[ -n "$BASH_VERSION" ]]; then
         IFS=: read -ra patharray <<< "$1"
     fi
@@ -30,13 +31,13 @@
 }
 
 __perlbrew_set_path () {
-    export MANPATH=$PERLBREW_MANPATH${PERLBREW_MANPATH:+:}$(__perlbrew_purify "$(manpath)")
-    export PATH=${PERLBREW_PATH:-$PERLBREW_ROOT/bin}:$(__perlbrew_purify "$PATH")
+    export MANPATH=$PERLBREW_MANPATH:$MANPATH
+    export PATH=${PERLBREW_PATH:-$PERLBREW_ROOT/bin}:$PATH
     hash -r
 }
 
 __perlbrew_set_env() {
-    local code
+    code=
     code="$($perlbrew_command env $@)" || return $?
     eval "$code"
 }
@@ -64,8 +65,8 @@
 }
 
 perlbrew () {
-    local exit_status
-    local short_option
+    exit_status=
+    short_option=
     export SHELL
 
     if [[ $1 == -* ]]; then

(Image courtesy Japanexperterna.se under Creative Commons License.)

Scsh is UNIX, Part 1: Sockets

gorge

Inspired by words from Ryan Tomayko and Jacob Kaplan-Moss, I set out to see how difficult it would be to implement a tiny UNIX echo server in scsh. It turns out to be fairly easy. In this post, we’ll cover:

  • How the UNIX socket programming model works at a high level
  • How to write a basic echo server in scsh

Unlike the echo servers of Tomayko and Kaplan-Moss, this one doesn’t use fork (yet). We will add forking to the server in a follow-up post. Hopefully this will make it easier for less experienced UNIX programmers (like me) to get started with.

The UNIX socket programming model for dummies

At a high level, the steps to open a server-side socket are:

  • create a socket “thing”
  • toggle some settings on it so the OS knows it’s for interweb use
  • bind it to an address
  • listen for connections on it
  • start accepting those connections

scsh does its part to make this easy by providing a high-level Scheme procedure, BIND-LISTEN-ACCEPT-LOOP, which handles this stuff for you, in addition to some nifty error handling and other bookkeeping chores. But we’re going to ignore that for now and write everything by hand to see how it’s done.

You should be able to enter all of the code in the next section directly in the scsh top-level. You don’t need to open any additional packages; this is just plain old scsh.

Our echo server

Our server takes two arguments: the PROC is a procedure that causes the server to actually do something; we’ll write in a minute. The PORT is the port you want to serve on, e.g., 49995:

(define (server proc port)
  (let* ((sock (create-socket protocol-family/internet socket-type/stream))
         (addr (internet-address->socket-address internet-address/any port)))
    (set-socket-option sock level/socket socket/reuse-address #t)
    (bind-socket sock addr)
    (listen-socket sock 5)
    (let loop ()
      (lambda () (accept-connection sock))
      proc
      (loop))))

The first thing you’ll notice is that it’s pretty sequential and quite simple really. We just go through the socket opening dance we described earlier: open, configure, bind, listen, and accept.

Since this is an echo server, we’ll just, like, echo stuff:

(define (echo socket address)
  (let* ((in (socket:inport socket))
         (out (socket:outport socket))
         (message (read-line in)))
      (format out "~A~%" message)
      (force-output out)))

As you can see, our ECHO procedure takes a socket and an address. (We don’t do anything with the address in this example, but our procedure needs to “implement this interface”, as they say, in order to work. You can see this for yourself in the scsh 0.6.7 tarball; it’s in scsh/network.scm.)

In our LET*-binding we create some convenient locally bound names for the socket structure’s input and output ports, and then we read a line from the socket’s input port.

Since this is echo server, we just write the same data back out with FORMAT. We call FORCE-OUTPUT to flush output to the terminal immediately. Otherwise Scheme will wait for the operating system’s output buffer to fill before writing out, and it will appear to the user that nothing is happening.

Trying it out

Let’s start it up and see if it works. Assuming you’ve loaded the procedures above in your scsh image somehow, enter this at the REPL:

> (server echo 49995)

The REPL should appear to “hang” while the server is running. Now go to your terminal and connect to the echo server. There are several ways to do it; here’s what I used:

~ $ telnet localhost 49995
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
HELLO
HELLO
who's there?
who's there?
i don't know
i don't know
could you stop that?
could you stop that?
fine then
fine then
goodbye
goodbye

Hopefully that was enough to get you started playing with sockets in scsh. I’ll write a followup post where we talk about the UNIX forking process model and update this example to make it a “preforking” server.

(Image courtesy Hope for Gorilla under a Creative Commons license.)