When is good enough, good enough?

 At work today, we were discussing centralizing our operational documentation so that everything was in one place. I stressed the fact that it didn't matter what platform we used, as long as it was applied consistently and everyone used the same system together. In our particular case, this meant using a slightly older technology because the key people involved had lots of experience with it and they would be able to simply continue using it with little change. We didn't need all of the great bells and whistles provided by the newer platforms available.

I realized that I needed to take this same advice to heart and apply it to myself. A while ago, I had switched to static blog generators to create a set of web pages to put on my GitHub Pages site. In theory, this idea meant that I would have much more control over my blog, but it also introduced several extra steps to the process of getting things written down and published to the Internet. And like most people, each extra step dragged down my productivity way out of proportion to what you might think. The old saying is that perfect is the enemy of good, or in my case good enough. While trying to get the perfect layout and look with a static site, I ended up going through several generators and I was not getting any actual writing done. This is just silly. So, here I am again, back at a basic blog engine. This way, I can get actual content out of my rusty old brain and out into the world, where other people can learn from my mistakes, and occassional flashes of brilliance.

The Amiga Workbench

The Amiga always held a bit of a mythological spot in my heart due to my exposure early in my computing experience. But, I never got a chance to own one of my own. Well, now I can play with one in my browser. The site http://taws.ch/WB.html lets you boot up one of several versions of the Amiga Workbench environment and use it from within any modern browser.

Calculator nonsense

Along with republishing (with updates) my older articles from Linux Journal, I am planning on posting items on using HP and TI calculators. There are several scientific problems that can be coded up on all of these machines, and I think it would be kind of cool to share those with other like-minded people into older tech.

Maximum Theory with maxima

First published in Linux Journal



Last month we looked at using scilab to analyze the results of an experiment. In the specific case, we were looking at a pendulum and seeing how it behaved with different lengths and weights. In science, there is usually some theory being tested with an experiment. This month, we'll look at how to use maxima to help us derive the equations describing the motion of a pendulum.

First, what is maxima? Maxima is a calculating engine. It runs on the console. However, there are several GUI front-ends for it. For example, wxMaxima. Maxima and its GUIs should be available for your distribution. If not, you can find the source code at the main web site (http://maxima.sourceforge.com).

To start maxima on the command line, it's as simple as running "maxima". You should get some licensinf information, then get dumped at a prompt.

To get a GUI, you would run "wxmaxima".

Maxima runs as a continuous session, much like you would have in Maple or Mathematica. In order to end your session and exit the program, you would execute the "quit();" command. Notice that commands within maxima end with a semi-colon. Anyone used to coding in C should be used to this. If you end a command with a "$" instead, the output from that command will not be displayed.

Sometimes, you may run a command in maxima and not realize just how long it will take to finish. If you run into something like this, you can stop execution by entering ctrl-C. Like in bash, ctrl-C stops execution of the currently running command. Because this is a forced stop, you will get an error message like

Don't worry about following up on the details of this error message unless you suspect that there is something deeper happening. Maxima also contains help files for all of its builtin functions. To see detailed information about a function, you simply need to execute "? command" to get a help page on the given command. For most commands and functions, you should get some examples as well.

When you first start maxima, the prompt is simply
   (%i1)
Every input line is labeled in a similar fashion, with the numbers incrementing. So the following input prompts would be "%i2", "%i3", "%i4", and so on. The output from commands are displayed immediately following the command, and are labeled with tags of the form "%o1", "%o2", "%o3", and so on. These behave the same as variables in maxima. This means that you can refer to earlier commands, or their results, by simply using the label. If you wanted to rerun the very first command you ran in this session, you would simply execute "%i1;" There are shortcuts for the last command and the last result. For the last command, you simply type in two single quotes (''). For the last computed result, you can type in a single percent sign (%). The percent sign also marks special, built-in values. For example, %e (natural log base), %i (square root of -1) or %pi (3.14159...).

Maxima can handle arithmetic very easily. The most common arithemtic operations are
   +         addition
   -         subtraction
   *         scalar multiplication
   /         division
   ^ or **   exponentiation
   .         matrix multiplication
   sqrt(x)   square root of x
You can apply these to both numbers and variables. Arithmetic is done exactly, whenever possible. This means that things like fractions are kept as fractions until you explicitly ask for a numeric result. So you see behaviour like
   (%i1) 1/100 + 1/101;
   (%o2)             201
                    -----
                    10100

Variables are easy to use in maxima. They are simply alphanumeric strings. Assigning a value to a vriable is done with the colon character. For example, if you wanted to find the square of a number, you could use
   (%i1) x: 2;
   (%o1)             2
   (%i2) x^2;
   (%o2)             4
You can also assign more complex functions to a variable. For example,
   (%i3) w: sin(0.5 * %pi);
   (%o3)            sin(0.5 %pi)
   (%i4) w^2;
   (%o4)            sin^2(0.5 %pi)
In certain cases, maxima will leave results in a partly computed state. If you want to force maxima to give you a number, you can add ", numer" to the end of your command. So, for the above, you would type in
   (%i5) %o4, numer;
   (%o5)             1

This use of "numer" will give you 16 significant figures, but maxima can handle arbitrarily large numbers. The function bfloat will convert your result to a numeric representation. The number of significant figures used by bfloat is set by the variable fpprec (the default is 16). As an example of really large numbers, you could try
   (%i1) %pi;
   (%o1)                                 %pi
   (%i2) %pi, numer;
   (%o2)                          3.141592653589793
   (%i3) bfloat(%pi);
   (%o3)                         3.141592653589793b0
   (%i4) fpprec: 100;
   (%o4)                                 100
   (%i5) bfloat(%pi);
   (%o5) 3.1415926535897932384626433832795028841971693993751058209749445923078164\
   06286208998628034825342117068b0

The last item we will need for figuring out what theory says we should get for a pendulum is algebra. Maxima handles this very easily. The expand function can be used to expand a polynomial. For example:
   (%i1) (x + 3*y + z)^2;
                                                2
   (%o1)                           (z + 3 y + x)
   (%i2) expand(%);
                        2                      2            2
   (%o2)               z  + 6 y z + 2 x z + 9 y  + 6 x y + x
You can do a variable replacement very easily with
   (%i3) %o2, x=5/z;
                          2           30 y   25      2
   (%o3)                 z  + 6 y z + ---- + -- + 9 y  + 10
                                       z      2
                                             z
If you want to figure out the factors for the above example, you can use
   (%i4) factor(%);
                                    2             2
                                  (z  + 3 y z + 5)
   (%o4)                          -----------------
                                          2
                                         z
The last set of algebra functions that you might need are "trigexpand" and "trigreduce". These do the same kind of function as "expand" and "factor". The function "trigexpand" uses the sum of angles formulas to make the argument of each trigonometric function as simple as possible. "trigreduce" does the opposite and tries to make the expression such that it is a sum of terms with only one sin or cos function in each term.

Now, we should have enough to use maxima and figure out what we should have found last month with the pendulum. The only force that is applied to a pendulum is gravity, constantly trying to pull the pendulum straight down. This is called a restoring force and is
   F = -mg sin(O)
where m is the mass of the pendulum bob, g is the acceleration due to gravity, and O is the angle the string makes with perpendicular pointing straight down. The sin function is a bit difficult to work with, so we should look at what we can replace it with. If we do a Taylor expansion of sin, we get
   (%i2) taylor(sin(x),[x],0,5);
                                     3    5
                                    x    x
   (%o2)/T/                     x - -- + --- + . . .
                                    6    120
From this, we can see that we can probably replace the sin function with just O, as long as this angle isn't too big. This will give us
   F = -mgO
If the angle O is measured in radians, then it is equal to x/L, where x is the actual amount of the arc that the pendulum bob travels along, and L is the length of the pendulum string. Once we do this, we've essentially minimized the nonlinear part to the point were we can treat the pendulum as a simple harmonic motion, with the force constant of (mg/L). If we plug this into the equation for harmonic motion, we get the time for an oscillation given by 2*pi*sqrt(L/g). In a maxima session, this would look like

We can see that we get a result from theory that is really close to what we actually measured.

In the upcoming months, we'll look at many of the other abilities that maxima has, including advanced algebra, calculus, and even plotting. Until then, go out and do some science of your own.

Using terminalide in another terminal emulator

It took a little experimenting, but I finally figured it out. While terminalide does come with a terminal emulator of its own, there are several others available. The one I like to use is AirTerm, as it floats over other apps on my tablet. To do this, you need to open the settings for AirTerm and go to the section for the initial command run. Here, you need to put

su 10102 -c /data/data/com.spartacusrex.spartacuside/files/termied.sh

The number "10102" is the user ID that the terminalide files are installed under. Also, this will only work if you have root access, otherwise the "su 10102" part will fail.

Mac / Linux cross-development

Introduction
Mac has always been the default alternative to Wintel systems. In the past, the Mac OS has been a fairly unique entity, without too much in common with the other OSes around, like Windows or the Unix variants. Cross-platform work has always been a bit more convoluted.
With the advent of the latest incarnation of the Mac OSX, or Darwin, we've got an alternative that is very comfortable for Linux geeks. Since Darwin is based on BSD Unix, we now have the possibility of using POSIX compliant applications on the Mac. This article will cover using Linux apps on a Mac, and some of the issues in running Mac apps on Linux. We'll also take a look at using the Mac as a development environment, developing applications that will end up on Linux.
Running Linux Code on Mac OSX
Apple provides a package called Xcode on their developer site. This packages gives you the tools you need to compile programs on the Mac. This includes a nice graphical IDE and lots of examples for developing your own applications for OSX. But that is not the only thing you can do. Xcode is based on the GNU toolset, providing tools like gcc, libtool, make, etc. This means that using Xcode, most command line applications can be compiled and run on the Mac. So our simple little hello world program

   #include <stdio.h>
   #include <stdlib.h>
   int main (int argc, char **argv) {
      printf("Hello World\n");
   }

compiles fine with gcc, giving us an executable which prints out "Hello World" on the command line. Basically, anything that is POSIX compliant should compile and run with no issues.
The process of getting graphical programs to run can be a bit more involved. Mac OSX does provide an X11 server, and all of the standard development libraries that you would need for a purely X11 application, like Xlib. However, none of the other standard libraries, like gtk or Qt, are available by default. You are responsible for getting them downloaded, compiled and installed on your system yourself. In this sense, it's a lot like the Linux From Scratch project. This works fairly well, but you do have to pick the correct configuration options, and collect all of the required dependencies. However, there's no reason why you need to go through this much pain. There are two projects in active development which provide some form of package management for GNU software, Fink and MacPorts. Using these, getting and installing GNU software is made as easy as most Linux distro.
The Fink project started in 2001. Fink is based on the Debian package management system. This means that you can use the Debian package tools like dpkg, dselect and apt-get. This makes it familiar for users of Debian-based distros, like Debian or Ubuntu. Once the base installation is done, you can start to install packages. If you like a text based manager, you can use dselect . If you like a graphical manager instead, you can use the command
sudo apt-get install synaptic
to get synaptic

 Using these applications, we can install many of the packages that we are used to in Linux. The package count, as of the writing of this article, is 10872. Not all packages are available as a binary install using these tools. For this class of packages, fink will install them directly from source, compiling and installing on your Mac. So, for example, if you wanted to install gramps and get some genealogy done, you would simply execute
sudo fink install gramps
Even installing from source, fink deals well with the dependency issues since it is still based on the Debian package management system.
The Mac Ports project started in 2002. This project decided to model itself after the BSD port packaging system. This means that you use the command port to manage the packages on your system. Once you have done the base install, you can install other software packages by simply running the command
sudo port install stellarium
There are several graphical interfaces available, as well, such as Porticus. However, they tend to be independent projects, as opposed to the Debian tools available in Fink. As such, their development cycle and behavior tend to be a bit more erratic and unstable then the older and more mature Debian tools. But still, they may be exactly what your are looking for if you prefer a graphical interface. Like the Fink project, it has both binary packages and source packages available. There are 5829 packages available in the Mac Ports project, and it is growing constantly.
Both projects provide the user with access to the full wealth of open source applications that has been available to Linux users, and the number of packages available continues to grow. This further blurs the line between what makes an OS. If the only difference between Mac OSX and, say Debian, is the kernel and maybe the GUI, are they effectively the same OS? I won't pretend to have an answer, but I think that we may have to start opening our ideas of what constitutes an OS.
Once you have one, or both, of these projects installed (they can peacefully coexist on your system) you will have all of the tools necessary to do your own code development. I have used anjuta

 on my MacBook to develop some small Gnome applications. These compile and run equally well on my MacBook, and my netbook running Ubuntu. While there isn't binary compatibility between OSX and Linux, with source compatibility we can essentially run the same code on both machines.
Running Mac OSX on Linux
This is not as easy as running Linux on Mac OSX. The real stumbling block is the graphical interface in the Mac OS. This is the area where Apple have been very secretive and proprietary. The graphical interface is based on a software technology called Quartz. While the kernel and most of the command line tools have been released as open source software, Quartz is still very closed. As of the writing of this article, I could not find any references to a reverse-engineered open source replacement for Quartz. So the only option available is running OSX inside a virtual machine. While this is not technically running Mac applications on Linux, it does provide the ability of running OSX on a Linux box.
Conclusion
Using the links and information in this article, user who straddle the Linux / Mac divide can narrow that gap a considerable amount. With projects like Fink and MacPorts, Linux users have access to the full spread of GNU and other open source software. As well, the Mac makes a good development environment, allowing a user to develop code that will run on both Linux and OSX. Hopefully, you will explore the possibilities and shrink the divide a little more.
Links

• http://developer.apple.com
• http://www.opensource.apple.com
• http://www.finkproject.org
• http://www.macports.org

What Hardware Do I Have?

There are many cases where you may not necessarily know what kind of hardware you have. You may have bought a no-name box from a smaller company. Or you may have bought a used machine. In any case you may not know what you have for hardware. This month we'll look at tools you can use to find out what you have installed.

The first tool to look at is lshw. This utility LiSts HardWare (lshw). If you run it as a regular user it will actually warn you to run it as root. So go ahead and run "sudo lshw". You should now see screens of information for your system. The first section will be general information, and should look something like

jbernard-eeepc
description: Notebook
product: 700
vendor: ASUSTeK Computer INC.
version: 0129
serial: EeePC-1234567890
width: 32 bits
capabilities: smbios-2.5 dmi-2.5 smp-1.4 smp
configuration: boot=normal chassis=notebook cpus=1 uuid=XXXXXX-XXXXX-XXXXX-XXXXX

This is what I get when I run it on my little Asus EeePC. So right away we can find out the manufacturer of this little beast (ASUSTeK), the BIOS version (0129), and the fact that it is a 32-bit machine with one CPU. There is more information broken down into the following categories

core
firmware - motherboard and BIOS information
cpu - CPU information
cache - cache information
memory - memory information
bank - specific bank memory information
pci - PCI bus information
display - PCI diplay adapter
multimedia - PCI audio adapter
pci - other PCI devices
network - PCI network adapter
usb - USB devices
ide - IDE information
disk - individual disks
volume - volumes on this disk

If we look at the section describing main memory, we get on my EeePC

*-memory
description: System Memory
physical id: 1f
slot: System board or motherboard
size: 512MiB
*-bank
description: DIMM DDR2 Synchronous 400 MHz (2.5 ns)
product: PartNum0
vendor: Manufacturer0
physical id: 0
serial: SerNum0
slot: DIMM0
size: 512MiB
width: 64 bits
clock: 400MHz (2.5ns)

we can get an idea of how much information is available.

This utility is kind of an all in one tool that spits out everything on your system in one go. But what if you want information only about specific subsystems in your machine. There is an entire suite of utilities you can use for this. These might be more useful when you want some specific piece of information, or you want to do some system querying in a script.

The first thing you may want to look at is the CPU. The utility lscpu will give you output similar to

Architecture: i686
CPU op-mode(s): 32-bit
CPU(s): 1
Thread(s) per core: 1
Core(s) per socket: 1
CPU socket(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 13
Stepping: 8
CPU MHz: 571.427

From this, you can find out the manufacturer, whether it is 32-bit or 64-bit, the exact version and model, as well as the current CPU frequency.

If you want to find out whether your video card is supported by X11, or whether you'll need to go and find a third-party driver, you can use lspci. This utility will give you a list of all of the devices plugged into you PCI bus. The output will look something like

00:02.0 VGA compatible controller: Intel Corporation Mobile 915GM/GMS/910GML Express Graphics Controller (rev 04)
00:02.1 Display controller: Intel Corporation Mobile 915GM/GMS/910GML Express Graphics Controller (rev 04)

With this information we can see that the video controller in my EeePC is an Intel controller. So, if you wanted to, you could now hit Google with this information to learn about your video card and how best to configure it. If you want to see what USB devices are on your system, you can use lsusb. On my EeePC I have an SD card installed, and it shows up as

Bus 001 Device 002: ID 0951:1606 Kingston Technology

The last piece of hardware we'll look at is the disk subsystem. You can find out what you have on your system with the utility blkid. This utility will print out all of the filesystems available to you, with an output format of

/dev/sda1: UUID="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" TYPE="ext2"
/dev/sda2: UUID="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" TYPE="swap"
/dev/sda3: UUID="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" TYPE="ext2"
/dev/sdb1: UUID="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" TYPE="ext2"

With this utility, you can find out what devices are available and what filesystems are being used on them. The associated UUID's are also available, in case you want to use them in the entries in /etc/fstab.

Now that we know what kind of hardware we have on our system, the last thig to check is to see if our kernel is actually using this available hardware. On most modern distributions, the kernel is compiled to use modules. You can check to see which modules are loaded by using the command lsmod. We get a list that looks like

agpgart 31788 2 drm,intel_agp
lp 7028 0
video 17375 1 i915
output 1871 1 video

We can see that the agpgart modules has a size of 31788 bytes, and is used by the drm and intel_agp modules.

Now that you know what you have installed for hardware, you can hopefully get it all configured and optimized so that you get the most out of it. If you find other utilites that aren't covered here, I would love to hear about them.

How To Write Unmaintainable Code

How To Write Unmaintainable Code: All programmers who want to guarantee job security really should read this.

'via Blog this'

The Web on the Console

Most people think graphical interfaces when they think of surfing the web. And under X11, there are lots of great programs, like firefox or chrome. But, the console isn't the wasteland that it might seem. There are lots of utilities available to surf the web, and also to download content from or upload content to the web.

The first thing you may want to do is surf around the web and find some content. The first utility to look at is also one of the oldest, the venerable lynx. Lynx was actually my first web browser, running on a machine that couldn't handle X11. Using it in its most basic form, you simply run it on the command line and give it a file name or a URL. So, if you wanted to hit Google, you would run
lynx http://www.google.com
The first thing you will notice is that lynx will ask you whether you want to accept a cookie that Google is trying to set. Once you either accept or reject the cookie, lynx will load the web page and render it. As you will no doubt notice, there are no images. But all of the links and the text box for entering your search query are all there. You can simply navigate around from link to link with your arrow keys. Because the layout is very simple and text-based, items will be in very different locations on the screen from what you will see using a graphical browser.

There are several options to lynx that might be handy to know. The first useful tip is that you can hand in more than one URL when you launch lynx. Lynx will add all of these URLs to the history of your session and render the last URL and display it. When you tested loading Google above, lynx asked you about whether or not to accept a cookie. Most sites these days use cookies. You may not want to hear about every cookie. You can use the option "-accept_all_cookies" to avoid all of these warning messages. You can use lynx to process web pages into a readable form by using the option "-dump". This will take the rendered output from lynx and write it out to standard out. This way you can simply process web pages to a readable format and dump them into a file for later viewing. You can choose what kind of key mapping you wish to use by using the options "-vikeys" or "-emacskeys". By using one of the options, you can use shortcut keys that match your editor of choice.

Lynx does have a few issues. It has a hard time with HTML table rendering and doesn't handle frames. Along comes the browser links. Links not only works in text mode on the command line, but can also be compiled to use a graphics display. The graphics systems supported include X11, svga, and framebuffer. You can select one of these graphics interfaces with the option "-g". Links can also write out the rendered web pages to standard output by using the option "-dump". If you need to use a proxy, you can tell links which to use with the option "-http-proxy host:port". Links is also able to deal with buggy web servers. There are several web servers out there which claim to be compliant to a particular HTTP version and aren't. To try and compensate for this, you can use the "-http-bugs.*" options. For example, "-http-bugs.http10 1" forces links to use HTTP 1.0, even when a server claims to support HTTP 1.1.

If you are looking for a strictly text replacement for the venerable lynx, there is elinks. Elinks supports colors, table rendering, frames, background downloading and tabbed browsing. One possibly useful option is "-anonymous 1". This option disables local file browsing and downloads, among other things. Another interesting option is "-lookup". When you use this, elinks will print out all of the resolved IP addresses for a given domain name.

Now that we can look at web content from the command line, how can we interact with the web? What I really mean is, how do we upload and download from the web? There are many instances where you may want an offline copy of some content from the web that you can read through at your leisure, off by the lake where you don't have any Internet access. One of the tools you can use to do this is curl. Curl can transfer data to or from a server on the Internet using HTTP, FTP, SFTP, even LDAP. It can do things like HTTP POST, SSL connections, and cookies. You can specify form name/value pairs so that the web server things you are submitting a form by using the option "-F name=value". One really interesting option is the ability to use multiple URLs through ranges. For example, you can specify multiple hosts with

curl http://site.{one,two,three}.com

which will hit all three sites. You can range through alphanumeric ranges with square brackets. The command

curl http://www.site.com/text[1-10].html

will download the files text1.html to text10.html.

But what if you want a copy of an entire site for offline browsing? The tool wget can help you out here. In this case, you will likely want to use the command

wget -k -r -p http://www.site.com

The option "-r" recurses through the links of the site starting at http://www.site.com/index.html. The option "-k" rewrites the downloaded files so that links from page to page are all relative so that you can navigate correctly through the downloaded pages. The option "-p" downloads all extra content on the page, such as images. This way you can get a mirror of a site on your desktop. Wget also handles proxies, cookies, HTTP authentication, along with many other conditions. If you're uploading content to the web, you can use wput. Wput pushing content up using ftp, with an interface like wget.

So now you should be able to interact with the Internet without ever having to use a graphical interface. Yet another reason to keep you on the command line.

Google Reader going away

Well, Google has announced that Reader is going away July 1. Argh! I was one of those who used almost religiously. Now I need to find a replacement. I think I have found it in Feedly. It seems to give me the same ability to keep multiple devices synchronized. The only think I haven't found yet is how to change the sort order so that I see the oldest articles at the top. Once I do, I think I'll be happy again.

Comparing Files

There are many times when you wish to know what has happened to a
file. You may be interested in comparing one version of a file to an
earlier one. Or you may need to check one file against a reference
file. Linux provides several tools for doing this, depending on how
deep a comparison you need to make.

The most common task involves comparing two text files. The tool of
choice for this task is diff. With diff, you can compare two files,
line by line. By default, diff will notice any differences between the
two text files, no matter how small. This could be as simple as a
space character being changed into a tab character from one file to
the next. The file will look the same to a user, but diff will find
that difference. The real power of diff comes from the options
available to ignore certain kinds of differences between files. In the
above example, you could ignore that change from a space character to
a tab character by using the option "-b" or
"--ignore-space-change". This option tells diff to ignore any
differences in the amount of whitespace from one file to the next. But
what about blank lines? The option "-B" or "--ignore-blank-lines"
tells diff to ignore any changes in the numbers of blank lines from
one file to the next. In this way, diff will effectively be only
looking at the actual characters and comparing them from one file to
the next. You have essentially narrowed the focus of diff to the
actual content.

What if that is not good enough for your situation? You may be
comparing files where one was entered with all capitals on, for some
reason. Maybe the terminal being used was misconfigured. In any case,
you may not want diff to report simple differences in case as "real"
differences. In this case, you can use the option "-i" or "--ignore-case".

What if you have files from a Windows box that you are working with?
Everyone who works on both Linux and Windows has run into the issue
with line endings on text files. Linux expects only a single newline
character while Windows uses a carriage return and a newline
character. diff can be told to ignore this with the option
"--strip-trailing-cr".

The output from diff can take a few different formats. The default
output contains the line which is different, along with a number of
lines just before and just after the line in question. These extra
lines are called context, and can be set with the option "-c", "-C" or
"--context=" and a number of lines to use for context. This default
output can be used by the program patch to change one file into the
other. In this way, you can create source code patches to upgrade code
from one version to the next. diff will also output differences
between files that can be used by ed as a script by using the option
"-e" or "--ed". diff will also output an RCS format diff by using the
option "-n" or "--rcs". The other option is to print out the
differences in two columns, side by side. The option "-y" or
"--side-by-side" will let you see each file side by side with the
differences between them highlighted.

The utility diff only compares two files. What if you need to compare
three files and see what changes exist moving from one to the others?
The utility diff3 comes to the rescue. This utility compares three
files and prints out the diff statements. Again, you can use the "-e"
option to print out a script suitable for the editor ed.

But what if you simply want to see two files and how they differ?
Another utility might be just what you are looking for, comm. With no
other options, comm takes two files and prints out three columns. The
first column contains lines unique to the first file, the second
column contains lines unique to the second file and the third column
contains lines common to both files. You can selectively suppress each
of these columns with the options "-1", "-2" and "-3". They suppress
columns 1, 2 or 3, respectively.

While this works great for text files, what if you need to compare two
binary files? You need some way to compare each and every byte in each
file. The utility that can be used for this is called cmp. It does a
byte by byte comparison of two files. The default output is a print
out of which byte and which line contains the difference. If you are
interested in seeing what the byte values are, you can use the option
"-b". The "-l" option gives even more detail, printing out the byte
count and the byte value from the two files.

Using these utilities, you can start to get a better handle on how
your files are changing. Here's hoping you keep control of you files.

SVN over SSH

There are many times when the only access to a server is over SSH. There are many reasons for this, including security. If you want to host an SVN repository on this machine, how do you access it? The first step is to create the repository. If you are creating a new one, then this easy.Just log in to the server and do the following

svnadmin create /path/to/repo

If you already have a repository that you want to move to the server, you should use svnsync to do so. First, you will want to allow a user to have svnsync access with

#!/bin/sh
USER=""
if [ "$USER" = "myusername" ]; then exit 0; fi
echo "Permission denied"
exit 1

This goes in the file /path/to/repo/hooks/pre-revprop-change. Make sure that it is set as executable

chmod +x ~/mymirror/hooks/pre-revprop-change

You can then tell svnsync to initialize the mirror.

svnsync init file:///path/to/repo
svn+ssh://username@svnmaster.domain.ext/newpath/to/repo

Once this initialization is done, you can sync the full data.

svnsync --non-interactive sync file:///path/to/repo

You can then interact with this new repository over SSH by using

ssh+svn://username@svnmaster.domain.ext/newpath/to/repo

as the connection details.

One problem that may come up when your SVN repository from one server to another is to fix up your local copy. There is a switch command that will allow you to do this.

svn switch --relocate old-url new-url

This way, you don't need to toss away your current working copy and checkout a fresh one.

Listening on the command line

On Linux, there are tons of programs you can use to listen to audio. Programs like totem, rhythmbox, mplayer, and vlc. One thing you may have noticed with each of these examples is that they all run in an X11 environment. What is a person to do if they want to listen to their favorite music on the command line? Well, let's take a look. Before we do, though, I want to stress that everything here will assume that you have a working audio subsystem on your machine. There is far too much variation in audio hardware to even begin to talk about how to setup audio on your machine.

Most distributions now use pulseaudio as the audio server. This provides a standard wrapper around the actual audio hardware. This way, software simply needs to talk to the pulseaudio server and won't need to worry about the messy details of how to talk to each sound card out there. The pulseaudio system also includes utilities that you can use on the command line.The most basic thing you will want to do is to play an audio file. To do this, you would use a command like

paplay --volume=32768 example.mp3

This will play the audio file example.mp3 at 50% volume (volume is set between 0, or silent, and 65536, or 100%). The utility paplay can play any audio format supported by the library libsndfile. So, you should be able to play most audio files you come across. If you don't have any audio files of your own, you can use the included utility parec. This program grabs raw audio data from your audio card's input, and dumps it to standard output. You can pipe this off to a file to save it for later. In order to play this raw data back, you can use the included utility pacat. It takes raw audio data and dumps it out to the audio card's output speaker.

What if you have a whole list of audio files you would like to listen to? There are several choices available to handle entire playlists of music for you from the command line. Two examples are cplay and moc. Both programs give you a file list on first starting up. From here, you can play individual files, or you can start constructing playlists to organize what audio files you are listening to. Both programs use shortcut keys to create, edit and otherwise manage your playlists. You can see the screenshots provided to get a feel for how they might look on your system.

Another audio task you might want to handle from the command line is to have your computer talk to you. Of course, once it can talk, it won't be long before it talks back. But anyway, there are several utilities available to give your computer a voice. These include recite, festival and espeak. The simplest is recite. It takes text in from standard input and outputs audio to the speakers. There aren't many options available. To speak a text file, you would simply execute

recite < example.txt

You can also simply execute recite, then start typing. Just remember that recite won't see the end of your typing until you hit ctl-D, to mark the end of input. It will then speak the entire text you just typed in. The program festival starts to provide more options. To get a basic output of text to audio, you would execute

festival --tts example.txt

You have the option of also using other languages. The default is English, but you can also use Spanish, Russian or welsh by using the command line option "--language ". Beyond this, festival also uses Scheme as a scripting language, which opens it up to a huge amount of modification. You'll need to spend some time reading the manual at http://www.cstr.ac.uk/projects/festival/manual/.

The last utility is espeak. Here you have several other command line options available. You can set the amplitude with "-a " (0 to 20, default is 10), the pitch with "-p " (0 to 99, default is 50) and speed with "-s " (in words per minute, default is 160). There are several voices available. You can find out which ones are installed on your system by using the command line option "--voices". Once you select one, use the command line option "-v " to use it. You then dump it out to a wave file by using the command line option "-w ". A full command line would look like

espeak -v en-scottish -w example.wav -f example.txt

Then you can play it using

paplay example.wav

Now your computer will speak in a Scottish accent, sort of. Speech synthesis is still not perfect.

Now that you have all of these audio files laying around, you may want to do some processing on them. One of the more useful tools to do this is sox. One utility included in the package is called soxi. It gives you file information about your audio files. In it's most basic form, you can use sox to change the file format of an audio file by simply running

sox example.wav example.au

sox is smart enough to use the file extensions to figure out that you are intending to convert the file example.wav (in wav format) to example.au (in the Sun au format). You can also do processing on the audio through command line options. The number of bits per sample to use can be set through "-b ". The number of channels to use can be set through "-c ". So, setting it to mono would be "-c 1", or stereo would be "-c 2". You can set the sample rate, in Hertz, through "-r ". There are also many options that apply filters to the audio files. Please go read the manual page to see just how much you can do with sox. One example command line you might use is

sox example.wav -b 8 -c 1 -r 8000 example.au

This example command takes an input file called example.wav and converts it to 8 bits per sample, mono (or 1 channel), with a sample rate of 8000 Hz and writes it out to a file with an AU format. The sox package also contains two other utilities, play and rec. With these, you have another way of playing audio files and recording audio to a file. You also have the full spectrum of processing and filters available from sox.

You may ask why I picked this particular example? This is so that I can show you one last interesting trick. On Linux systems, you can simply cat this output file (with this specific file format) directly to the device file /dev/audio. This dumps the output directly to the sound card. So, if you want to be sneaky, you can convert some files to the AU file format with the sox command above, copy them over to your target machine, and simply cat them to /dev/audio when you want to make a nuisance of yourself. Hopefully you now have some ideas on how to enjoy your music, and how to play with audio files, without the overhead of the GUI applications available.

Local Subversion Repositories

Everyone should be using some kind of version control system for their own files. There are tons of options available, including mercurial, git, CVS and subversion. I've recently had to move some code to subversion so that I can share code with some other groups. You don't need to have a server set up out there somewhere on the interwebs. You can set up your own private repository on your own machine and use that to track changes to your code and documents. The first step is to create space on your drive

mkdir ~/svn

After that, you need to initialize the actual repository

svnadmin create ~/svn

That's it. Now you have a repository that you can use. The very first thing is to import your current code.

svn import /home/jbernard/temprepos/my_sources file:///home/jbernard/svn/my_sources -m "Initial Import"

After this initial import, you can checkout a working copy of the code to work with.

svn checkout file:///home/jbernard/svn/my_sources

All of the other commands assume that you are actually in the subdirectory holding the checked out code. You can check the surrent status of your code with

svn status

Any edited files will be flagged. You can check in these changes with

svn commit

where it will ask you for a comment about the commit. You can grab any changes committed by someone else with

svn update

I'll be adding more commands here as I think of them.

WiFi on the Command Line

More people than ever are using wireless networks as their primary networking medium. There are great programs available under X11 that give users a graphical interface to their wireless cards. Both Gnome and KDE include network management utilities, and their is a desktop environment agnostic utility called wicd which also offers great functionality. But what do you do if you aren't running X11 and you want to manage your wireless card? I won't be covering how to get your card installed and activated. For that, you should look at projects like madwifi or ndiswrapper, in order to properly configure your system and card. In the rest of this piece, I'll assume that you have your card properly installed and configured, and that it is called wlan0. Also, most of the utilities mentioned below have to talk directly to your wireless card, or at least the card driver, and so will need to be run with root privileges. Just remember to use sudo.

The first step is to see what wireless networks are available in your area. There is a utility called iwlist which provides this information. This utility can give you all sorts of information about your wireless environment. The first thing to do is to scan your environment for available networks. You would run
sudo iwlist wlan0 scan
and get output resembling
Cell 01 - Address: 00:11:22:33:44:55
ESSID:"network-essid"
Mode:Master
Channel:11
Frequency:2.462 GHz (Channel 11)
Quality=100/100 Signal level:-47dBm Noise level=-100dBm
Encryption key:off
.
.
.
The details (address and essid) have been changed to protect the guilty. Also, the ... is extra output that may or may not be available, depending on your hardware. You will get a separate cell entry for each access point within range of your wireless card. For each access point, you can find out the hardware address, the essid and the channel it is operating on. Also, you can find out what mode the access point is operating in (whether master or ad-hoc). In most cases, you will be most interested in the essid, and what encryption is being used.

Once you have information about what is available in your immediate environment, you need to configure your wireless card to use one of these access points. You can use the utility iwconfig to set these parameters for your wireless card. The first thing you will want to set is the essid, which identifies the network access point you are interested in. You would run
sudo iwconfig wlan0 essid network-essid
Depending on your card and its driver, you may have the option to set the essid to the special value "any". In this case, your card will pick the first available access point. This is called "promiscuous mode".

You may also need to set the mode to be used by your wireless card. This will depend on your network topology. You may have a central access point that all of the other devices connect to. Or you may have an ad-hoc wireless network, where all of the devices communicate as peers. You may wish to have your computer act as an access point. If so, you can set the mode to master using iwconfig. Or, you may want to simply sniff what is happening in the air around you. You can do this by setting the mode to monitor and passively monitor all packets on the frequency your card is set to. You can set the frequency, or channel, by running
sudo iwconfig wlan0 freq 2.422G
or
sudo iwconfig wlan0 channel 3

You can also set other parameters, but you should only consider doing so if you have a really good reason. One option of interest is the sensitivity threshold. This defines how sensitive the card is to noise and signal strength. You can set the behavior of the retry mechanism for the wireless card. You may need to play with this in very noisy environments. You can set the maximum number of retries with
sudo iwconfig wlan0 retry 16
or set the maximum lifetime to keep retrying to 300 milliseconds with
sudo iwconfig wlan0 retry lifetime 300m
In a very noisy environment you may also need to play with packet fragmentation. If entire packets can't make it from point to point without corruption, your wireless card may have to break packets down into smaller chunks to try and avoid this. You can tell the card what to use as a maximum fragment size with
sudo iwconfig wlan0 frag 512
This value can be anything less than the size of a packet. The last thing you may need to run is
sudo iwconfig wlan0 commit
Some cards may not apply these settings changes immediately. In these cases, you'll need to run this command to flush all pending changes to the card and get them applied.

Two other commands that may prove useful are iwspy and iwpriv. If your card supports it, you can collect wireless statistics by using
sudo iwspy wlan0
The second command gives you access to optional parameters for your particular card. iwconfig is used for the generic options available. If you run it without any parameters
sudo iwpriv wlan0
it will list all available options for the card. If there are no extra options, then you will get an output looking like
wlan0 no private ioctls
To set one of these private options, you would run
sudo iwpriv wlan0 private-command [private parameters]

Now that you have your card configured and connected to the wireless network, you'll need to configure your networking options to actually use it. If you are using DHCP on the network, you can simply run dhclient to query the DHCP server and get your IP address and other network settings. If you wish to set these options manually, you can do so through the command ifconfig. I would suggest that you give the man page for ifconfig a read. Hopefully this article will help those on the command line to use their wireless cards and networks.

Matlab OS Checks

Matlab, at least on Linux, is pretty firm on which versions it will run on. You may see glibc warnings when you try and run on older distributions. The problem is, as long as the distro isn't too old, Matlab will probably run fine. When you run Matlab interactively, you can simply accept the warning and continue on. But this doesn't work if you want to run scripts in batch mode. To get around this, you can set an environment variable to skip the OS check. If you use bash, simply use

   export OSCHECK_ENFORCE_LIMITS=0

This should let you go ahead and run your script without pausing. Hope this helps.

Putty timeouts

Some of my users were having problems with ssh connections dropping. Don't forget to go into the connection section of the preferences and set the keepalive options.

GoFredX Convention

I'm currently attending tne GoFredX Convention in Fredericton. The talks so far have been great. I think the talks after lunch are going to be even better.

Maintaining Your System from the Command line

Many Linux distributions use some form of packaging system to to organize the applications installed on a system. Using a formal packaging system lets you install, remove and in general, maintain the software on your system in a controlled and coherent way. The three main packaging systems are the Debian deb package, the Red Hat rpm package, and the Slackware pkg package. The vast majority of the distributions today will use one of these three packaging systems. They all have graphical utilities to interact with the packaging system. But what if you want to deal with the system on the command line? Say you are running a server, or are accessing a distant machine through ssh and don't want to deal with the overhead of X11? This month we'll take a look at how to do this for Debian based systems.

The first thing you will probably want to do is install some software on your system. The preferred way to do this is through the utility apt-get. apt-get is aware of the chain of dependencies between packages. Let's say you want to do some star gazing and want to install stellarium on your system. You would run
apt-get install stellarium
This would download the relevant package file, and all of its dependencies, from a repository. What if you don't know exactly what the package is named? You can query the package management system with the utility dpkg-query. If you know that the package name has "kde" in it, you can list all of the matching packages with the command
dpkg-query -l "*kde*"
Remember to quote any search strings that have a "*" in it so that you don't inadvertently have the shell try and expand them.

This works great for software available in the given repository. But what if you want something not available there? If you have a ".deb" file available for download, you can download it and install it manually. After downloading the file, you would install it by running
dpkg -i file_to_install.deb
The utility dpkg works with the deb packaging system at a lower level than apt-get. With it, you can install, remove and maintain individual packages. If you have a whole group of packages you would like to install, you might want to add the relevant repository to your list so that apt-get will know about it. The list of repositories is stored in the configuration file /etc/apt/sources.list. Each line has the form
deb http://us.archive.ubuntu.com/ubuntu/ karmic main restricted
The first field tells apt-get what is available at this repository: deb is for binary packages, deb-src is for source packages. The second field is the URL to the repository, in this example we're looking at the Ubuntu repository. The third field is the repository name, in this case the repository for the karmic version of Ubuntu. The last fields are the sections we want to look at to install packages from. This example will look at the sections main and restricted when trying to install applications or resolve dependencies.

Now that you have some applications installed, you will probably want to maintain them and keep them updated. Every piece of software will have bugs, or security issues, that come to light over time. Software developers are always releasing new versions to fix these issues, and updating the relevant packages in the repositories. To update the list of software and their versions on your system, you would run
apt-get update
Once you've updated the list, you can tell apt-get to install these updates by running
apt-get upgrade
If you want a list of what is about to be upgraded, add the command line option "-u"
apt-get upgrade -u
Sometimes, when a new version comes out for a package (like when your distribution releases a new version), the dependencies for said package might change, too. In cases like this, a straight upgrade might be confused. In these cases you can use the command
apt-get dist-upgrade
This command tries to intelligently deal with these changes in dependencies, adding and removing packages as necessary.

What do you do if you've installed a package to try it out and don't want it anymore? You can remove a package with the command
apt-get remove stellarium
This removes all of the files that were installed as part of the package stellarium, but leaves any configuration files intact and also doesn't deal with any extra packages installed because stellarium depended on them. If you wish to completely remove a package, including all configuration files, you would run
apt-get purge stellarium

All of this software installation and removal could result in cruft accumulating on your system. You may end up with unnecessary packages wasting space on your system. To start to recover some space you can run the command
apt-get autoclean
This command will remove the package ".deb" files from the local cache for packages that can no longer be downloaded. These would be mostly useless packages. If you want to completely clean out the local cache and recover more space, you can run
apt-get clean
While "remove" and "purge" will remove a package, what can you do about any dependencies installed for this package. If you run the command
apt-get autoremove
you can uninstall all packages that were installed as dependencies for other packages and aren't needed anymore.

Another way of finding packages that aren't needed anymore is through the utility deborphan. The first thing you'll need to do is install it, using
apt-get install deborphan
since most distributions don't install it by default. Once it is installed, running it with no command line options will give you a list of all packages in the libs and oldlibs sections that have no dependents. Since no other package depends on these packages, you can safely use apt-get to remove or purge them. If you want to look in all sections, you can use the option "-a". If you're trying to save space, you can ask deborphan to print out the installed sizes for these orphan packages by using the command line option "-z". You can then sort them out by running
deborphan -z -a | sort -n
This will give you a list of packages you can safely uninstall, sorted by installed size from smallest to largest. You can then free up space on your system by getting rid of the biggest space wasters.

Hopefully this gives you a good starting point to dealing with the Debian package management system. Each of the tools discussed above have lots of other options that you should research in the relevant man pages. Also, if you use a Red Hat based system, there are equivalent commands to these to help you manage the rpm files used there.

Using sed to clean up files

You can use sed to delete particular lines, based on some unique criteria. This would look like

sed -e "/search criteria/d" -i2 file1.txt

This happens "in place", replacing the file's original contents. If you want to process a number of files in the same way, you can use

ls *.txt | xargs -i{} sed -e "/search criteria/d" -i2 {}

This takes the results from ls and hands it in to xargs. xargs then runs sed on each filename, one at a time.