GoFredX Convention

I'm currently attending tne GoFredX Convention in Fredericton. The talks so far have been great. I think the talks after lunch are going to be even better.

Maintaining Your System from the Command line

Many Linux distributions use some form of packaging system to to organize the applications installed on a system. Using a formal packaging system lets you install, remove and in general, maintain the software on your system in a controlled and coherent way. The three main packaging systems are the Debian deb package, the Red Hat rpm package, and the Slackware pkg package. The vast majority of the distributions today will use one of these three packaging systems. They all have graphical utilities to interact with the packaging system. But what if you want to deal with the system on the command line? Say you are running a server, or are accessing a distant machine through ssh and don't want to deal with the overhead of X11? This month we'll take a look at how to do this for Debian based systems.

The first thing you will probably want to do is install some software on your system. The preferred way to do this is through the utility apt-get. apt-get is aware of the chain of dependencies between packages. Let's say you want to do some star gazing and want to install stellarium on your system. You would run
apt-get install stellarium
This would download the relevant package file, and all of its dependencies, from a repository. What if you don't know exactly what the package is named? You can query the package management system with the utility dpkg-query. If you know that the package name has "kde" in it, you can list all of the matching packages with the command
dpkg-query -l "*kde*"
Remember to quote any search strings that have a "*" in it so that you don't inadvertently have the shell try and expand them.

This works great for software available in the given repository. But what if you want something not available there? If you have a ".deb" file available for download, you can download it and install it manually. After downloading the file, you would install it by running
dpkg -i file_to_install.deb
The utility dpkg works with the deb packaging system at a lower level than apt-get. With it, you can install, remove and maintain individual packages. If you have a whole group of packages you would like to install, you might want to add the relevant repository to your list so that apt-get will know about it. The list of repositories is stored in the configuration file /etc/apt/sources.list. Each line has the form
deb http://us.archive.ubuntu.com/ubuntu/ karmic main restricted
The first field tells apt-get what is available at this repository: deb is for binary packages, deb-src is for source packages. The second field is the URL to the repository, in this example we're looking at the Ubuntu repository. The third field is the repository name, in this case the repository for the karmic version of Ubuntu. The last fields are the sections we want to look at to install packages from. This example will look at the sections main and restricted when trying to install applications or resolve dependencies.

Now that you have some applications installed, you will probably want to maintain them and keep them updated. Every piece of software will have bugs, or security issues, that come to light over time. Software developers are always releasing new versions to fix these issues, and updating the relevant packages in the repositories. To update the list of software and their versions on your system, you would run
apt-get update
Once you've updated the list, you can tell apt-get to install these updates by running
apt-get upgrade
If you want a list of what is about to be upgraded, add the command line option "-u"
apt-get upgrade -u
Sometimes, when a new version comes out for a package (like when your distribution releases a new version), the dependencies for said package might change, too. In cases like this, a straight upgrade might be confused. In these cases you can use the command
apt-get dist-upgrade
This command tries to intelligently deal with these changes in dependencies, adding and removing packages as necessary.

What do you do if you've installed a package to try it out and don't want it anymore? You can remove a package with the command
apt-get remove stellarium
This removes all of the files that were installed as part of the package stellarium, but leaves any configuration files intact and also doesn't deal with any extra packages installed because stellarium depended on them. If you wish to completely remove a package, including all configuration files, you would run
apt-get purge stellarium

All of this software installation and removal could result in cruft accumulating on your system. You may end up with unnecessary packages wasting space on your system. To start to recover some space you can run the command
apt-get autoclean
This command will remove the package ".deb" files from the local cache for packages that can no longer be downloaded. These would be mostly useless packages. If you want to completely clean out the local cache and recover more space, you can run
apt-get clean
While "remove" and "purge" will remove a package, what can you do about any dependencies installed for this package. If you run the command
apt-get autoremove
you can uninstall all packages that were installed as dependencies for other packages and aren't needed anymore.

Another way of finding packages that aren't needed anymore is through the utility deborphan. The first thing you'll need to do is install it, using
apt-get install deborphan
since most distributions don't install it by default. Once it is installed, running it with no command line options will give you a list of all packages in the libs and oldlibs sections that have no dependents. Since no other package depends on these packages, you can safely use apt-get to remove or purge them. If you want to look in all sections, you can use the option "-a". If you're trying to save space, you can ask deborphan to print out the installed sizes for these orphan packages by using the command line option "-z". You can then sort them out by running
deborphan -z -a | sort -n
This will give you a list of packages you can safely uninstall, sorted by installed size from smallest to largest. You can then free up space on your system by getting rid of the biggest space wasters.

Hopefully this gives you a good starting point to dealing with the Debian package management system. Each of the tools discussed above have lots of other options that you should research in the relevant man pages. Also, if you use a Red Hat based system, there are equivalent commands to these to help you manage the rpm files used there.

Using sed to clean up files

You can use sed to delete particular lines, based on some unique criteria. This would look like

sed -e "/search criteria/d" -i2 file1.txt

This happens "in place", replacing the file's original contents. If you want to process a number of files in the same way, you can use

ls *.txt | xargs -i{} sed -e "/search criteria/d" -i2 {}

This takes the results from ls and hands it in to xargs. xargs then runs sed on each filename, one at a time.

Stupid afio Tricks

We've already looked at tar, and all of the wonderful ways that it can be used. But it is not the only tool at our disposal. Another tool that gets used quite a bit for doing backups is afio. Depending on your distribution, it may or may not be already installed. In Ubuntu, for example, you would have to run

sudo apt-get install afio

to get it installed on your system. Once you do, you have a fairly powerful tool at your disposal for archiving files and making backups.

By default, afio reads and writes the files being archived on standard input and standard output. This means that you can create your list of files to archive with another program, like find, and pipe it to afio to do the actual archive. Once you have your list of files, you can apply 5 basic commands to those files

-o create an archive
-i install (or unpack) an archive
-t test (or list) the files stored in an archive
-r verify the files stored in an archive against the file system
-p copy the files to a given directory location

If you want to create a simple archive of all of your C source code files, you would execute

find . -name *.c -print | afio -o -Z source_code

When you want to extract these files again, you would execute

afio -i -Z source_code

When you run afio as a regular user, all file paths are stored with no leading "/". This means that when you unpack an archive, it will unpack in the current directory. The idea is to hopefully avoid accidentally overwriting system files. To keep the leading "/", you need to use the command line option "-x". If you run afio as the superuser, then this behavior is reversed. Any leading "/" is maintained, and you need to use the command line option "-X" to get the usual behavior of stripping leading "/".

If space is at a premium, afio can also compress your archive, just like tar can. This is done by using the command line option "-Z". There is one very big difference, however. When you compress a tar archive, the entire archive file gets compressed. This means that if you have a corruption in one part of the file, you could potentially lose all of the files in the archive. When you compress an afio archive, the archived files are actually compressed individually. This means that if one file becomes corrupted, by whatever means, you won't actually lose any of the other files in the archive. When you do compress an archive, afio uses gzip by default. You can tell gzip what compression factor to use with the command line option "-G num", where num is the amount of compression gzip is to apply to the archived files. This is a number between 0 (for no compression) and 9 (for maximum compression), with a default of 6. You may need to balance how much CPU and how much IO time is being used during the compression phase. If so, you can put limits on when compression is to be used. The command line option "-T threshold" tells afio not to try and compress a file unless it is at least threshold bytes in size. The default setting is "-T 0k", so afio tries to compress all files, no matter how small. At the other end of the spectrum, you may want to limit how large a file can be before afio tries to compress it. You can do this with the command line option "-2 max", where max is the maximum file size. The default in this case is "-2 200m", so afio won't try and compress files larger than 200MB.

What if you don't want to use gzip as your compression method? You can change this by using the command line option "-P progname", where progname is the name of the executable to use to do the compression. If you need to hand options in to this alternate program, you can do this with the option "-Q opt". You need to use separate "-Q" options for each option you need to hand in to the alternate program. Because afio simply executes this alternate program, you can run anything at this stage. This could include an encryption program, allowing you to encrypt your archive. To encrypt your archive using PGP, you could execute

export PGPPASSFD=3
find . -name *.c -print | afio -ovz -Z -U -P pgp -Q -fc -Q +verbose=0 -3 3 archive 3
This would run PGP on each file in the archive as they are added.

The last small trick with afio is that you also have the ability to interact with archives on external systems. The way you do this is similar to how you do it with tar. The format looks like

[user@]host[%rsh][=afio]:file

The option "user@" is the user name you would use to access the external system. The default communications mechanism is rsh, but you could change that to ssh by using the option "%ssh". You can define the command to use on the external system by using the option "=afio". You can use this if the executable is named something else, or in an odd location. So, if you wanted to archive all of your source code files onto an external server over ssh, you could execute

find . -name *.c -print | afio -o -Z user@server%ssh:archive

Using afio, you can now go forth and ensure that you have proper backups of all of your important information. So now you don't have any excuses anymore.

Controlling Your Processes

I believe that it was the bard who said

All the CPU's a stage,
And all the processes and threads merely players;

or something like that. In any case, it is true. All of the processes that you want to run on your machine are like players, and you are the director. You control when they run and how they run. But, how can you do this? Well, let us look at the possibilities.

The first step is to run the executable. Normally, when you run a program, all of the input and output is connected to the console. So you see the output from the program and can type in input at the keyboard. If you add an '&' to the end of the program, this connection to the console is severed. Your program will now run in the background and you can continue working on the command line. When you run an executable the shell actually creates a child process and runs your executable in that structure. But sometimes, you don't want to do that. Let's say you have decided that no shell out there is good enough and so you have decided to write your own. When you're doing testing, you want to run it as your shell, but you probably don't want to have it as your login shell until all of the bugs have been hammered out. You can run your new shell from the command line with the 'exec' function
exec myshell
This tells the shell to actually replace itself with your new shell program. To your new shell, it will look like it is your login shell. Very cool. You can use this to also load menu programs in restricted systems. That way, if your users kill off the menu program they will get logged out, just like killing off your login shell. This might be useful in some cases.

So, now that your program is running, what can we do with it? If you need to pause your program temporarily (you may need to look up some other information, or run some other program), you can do this by typing ctrl-z (Control and z at the same time). This pauses your program and places it in the background. You can do this over and over again, collecting a list of paused and backgrounded jobs. To find out what jobs are sitting in the background, you can use the shell function 'jobs'. This will print out a list of all background jobs, with output looking like

[1]+ Stopped man bash

If you wanted to also get the process IDs for these jobs, you can use the option '-l'

[1]+ 26711 Stopped man bash

By default, jobs will give you both paused and running background processes. If you only want to see the paused jobs, use the option '-s'. If you only want to see the running background jobs, use the option '-r'. Once you finished your sidebar of work, how do you get back to your paused and backgrounded program? The shell has a function called 'fg' which lets you put a program back into the foreground. If you simply execute 'fg', the last process backgrounded is pulled back into the foreground. If you want to pick a particular job to foreground, you would use the '%' option. So if you wanted to foreground job number 1, you would execute 'fg %1'. What if you wanted your backgrounded jobs to continue working? When you use ctrl-z to put a job in the background, it is also paused. To get it to continue running in the background you can use the shell function 'bg'. This is equivalent to having run your program with a '&' at the end of it. It will stay disconnected from the console but continue running while in the background.

Once a program is backgrounded and continues running, is there any way to communicate with it? Yes there is: the signal system. You can send signals to your program with the command 'kill procid', where procid is the process ID of the program you are sending the signal to. Your program can be written to intercept these signals and do things, depending on what signals have been sent. You can send a signal by either giving the signal number or a symbolic number. Some of the signals available are

1 SIGHUP terminal line hangup
3 SIGQUIT quit program
9 SIGKILL kill program
15 SIGTERM software termination signal
30 SIGUSR1 user defined signal 1
31 SIGUSR2 user defined signal 2

If you simply execute kill, the default signal sent is a SIGTERM. This signal tells the program to shutdown, as if you had quit the program. Sometimes, your program may not want to quit. You sometimes have programs that simply will not go away. In these cases you can use 'kill -9 procid', or 'kill -s SIGKILL procid', to send a kill signal. This will usually kill the offending process with extreme prejudice.

Now that you can control when and where you program runs, what's next? You may want to control the use of resources by your program. The shell has a function called 'ulimit' which can be used to do this. This function changes the limits on certain resources available to the shell, as well as any programs started from the shell. The command 'ulimit -a' will print out all of the resources and their current limits. The resource limits that you can change will depend on your particular system. As an example, which crops up when trying to run larger Java programs, let's say you need to increase the stack size for your program to 10000KB. You would do this with the command 'ulimit -s 10000'. You can also set limits for other resources like the amount of CPU time in seconds (-t), maximum amount of virtual memory in KB (-v), or the maximum size of a core file in 512-byte blocks (-c).

The last resource that you may want to control is what proportion of the system your program uses. By default, all of your programs will be treated equivalently when it comes to deciding how often you programs get scheduled to run on the CPU. You can change this with the command 'nice'. A regular user can use nice to alter the priority of their program down from 0 to 19. So, if you are going to run some process in the background, but you don't want it to interfere with what you are running in the foreground, you can run it by executing
nice -n 10 my_program
and this will run your program with a priority of 10, rather than the default of 0. You can also change the priority of an already running process with the program 'renice'. If you have a background process that seems to be taking a lot of your CPU, you can change it with
renice -n 19 -p 27666
This will lower the priority of process 27666 all the way down to 19. Regular users can only use nice or renice to lower the priority of processes. The root user can increase the priority, all the way up to -20. This is handy when you have processes that really need as much CPU time as possible. If you look at the output from top, you can see that something like pulseaudio might have a negative niceness value. You don't want your audio skipping when you watching your movies.

The other part of the system that needs to be scheduled is access to IO, especially the hard drives. You can do this with the command 'ionice'. By default, programs are scheduled using the best effort scheduling algorithm, with a priority equal to (niceness + 20) / 5. This priority for the best effort is a value between 0 and 7. If you are running some program in the background and don't want it to interfere with your foreground programs, you can set the scheduling algorithm to 'idle' with
ionice -c 3 my_program
If you want to change the IO niceness for a program that is already running, you simply have to use the option '-p procid'. The highest possible priority is called realtime, and can be between 0 and 7. So if you have a process that needs to have first dibs on IO, you can run it with the command
ionice -c 1 -n 0 my_command
Just like the negative values for the nice command, using this realtime scheduling algorithm is only available to the root user. The best a regular user will be able to do is
ionice -c 2 -n 0 my_command
which is the best effort scheduling algorithm with a priority of 0.

Now that you know how to control how your programs use the resources on your machine, you can change how interactive your system feels.

Open Circuits

Open Circuits provides a site full of ideas and circuits put into the open source community. Using their description:

Open Circuits is a wiki for sharing open source electronics knowledge, schematics, board layouts, ports and parts libraries. This include open hardware Music Players, atomic microscopes, PC, PDA and mobile phones, and batteries.

Visit, and add what you can.

New Acer Iconia tablet

I just got a new Acer Iconia tablet, and I am in love. The only issue I had was connecting to the wireless network at UNB. I finally learned that I had to actually go through the add network option and add all of the details (including the SSID) manually. Once I did this, everything came up wonderfully. Ahhhh.

Physics Site Extraordinaire

I just saw this site this morning. I found the link in Make magazine (awesome magazine, by the way). There is far too much info here for me to go into. All I can say is that you need to go check this out if you have any interest in physics or science.

Using bc for math at the command line

This first appeared in Linux Journal, some time ago.

bc - When integers aren't enough

Most people have the need to do some kind of math when they are using a computer. In shell scripts, you can make integer calculations by using functionality in the shell itself. But what if this isn't enough? Luckily the POSIX standard includes a very useful command line utility called bc. With this you can do arbitrary precision arithmetic. Actually, it is a complete, C-like, language that can do some pretty sophisticated programming, supporting variables and functions.

In bc, numbers are all represented internally as a decimal number. They have a length which is the total number of digits, and a scale which is the number of decimal spaces. You can find these values by using the builtin functions length() and scale(). For example, the number 10.23 would have a length of 4 and a scale of 2. The variable "scale" holds the number of decimal places to keep when internal functions are executed. The default value is 0. bc supports all number bases from 2 to 16, with base-10 being the default. The input and output base of numbers can be set by using the variables ibase and obase. All of the basic mathematical operations are supported in bc. You can multiply, divide, add, subtract, do mod and exponentiation. There are all of the standard comparison operations, too. Less than, less than or equal to, greater than, greater than or equal to, equal to and not equal to all give results of 0 for false and 1 for true. This is very useful in the conditional statements available in bc.

bc can be used this in shell scripts or on the command line as a very effective calculator. It will read from a list of files given on the command line, or read from standard input. On the command line, expressions can simply echoed through a pipe to bc.

echo "1+1" | bc

will give the answer of 2. As a more complex example, the sine of 5 can be assigned to a shell variable with the following

RESULT=`echo s(5) | bc -l`

The "-l" command line option tells bc to load the math library, giving access to the trigonometric functions.

As a bit of a contrived example, say that there are two values and we need to find out which one has a larger sine. With the math library and the builtin comparison operations, this can be determined by typing

echo "s(5) < s(10)" | bc -lThe result 1 gets printed out on standard output, verifying that the sine of 5 is less then the sine of 10. bc can be made to print out a text string telling the user whether the result is true or false with the following echo 'if (s(5) < s(10)) print "true\n" else print "false\n"' | bc -lThis prints out the word true. If this string is to be stored in a variable, the newline characters would be removed from the executable line. This value can then be used later on in a shell script by saving it to a shell variable.What if we have a data file of input values and we want to apply some function to them? Say we need to calculate the logarithm base 10 of each value and dump it into another file? The following example take a list of the first 10 numbers, calculates the logarithm base 10 of each number and writes the value into the file output.lst LIST="0 1 2 3 4 5 6 7 8 9" for INPUT in $LIST do echo "l($INPUT)/l(10)" | bc -l >>output.lst
done

These examples have already done some useful work, but what if the requirements are more robust? Does this necessitate a move to a heavy weight math program, like Mathematica or Maple? Not necessarily. With bc, there is the possibility of creating and using functions to make more complicated calculations. Even recursive functions can be written, like in this example to calculate a factorial

define f (x) {
if (x <= 1) return (1);
return (f(x-1) * x);
}
print "Factorial:"; factorial = read();
print f(factorial); print "\n";
quit

This can be dumped into a file called "fact.bc" and run through bc to get the factorial of some number by executing

bc fact.bc

This script will ask the user for a number and then will find the factorial. This can be used without interaction by simply feeding the number in to standard input with a pipe

echo 10 | bc fact.bc

This will print out the factorial of 10 (3628800) to standard output. But how fast can such a program be? For a variety of values run on a generic laptop, the following times were measured:

10 0.004s
100 0.004s
1000 0.028s
10000 3.099s

These times were averaged over three runs to account for varying system load. This seems more than fast enough to be useful for a lot of heavy work.

For a more science-y example, the following bc script can be used to find how long it takes for an object to fall for a series of heights.

define t(h) {
g = 9.81;
return (sqrt(2 * h / g));
}

Now there is no excuse for abandoning a shell script simply because it can't handle some mathematical problem. With bc, a lot of really useful work can be done straight from the command line. Go forth and enumerate.

Using top on SMP machines

I've been trying to sort out some user problems where multiple users are running on an SMP machine. You can use top to figure out where processes are running. When you run top, you can hit the "1" key to get a separate usage listing for each processor. If you want to see where a particular process is running, hit "f" to select columns, then hit "j" to add the CPU column. Then the top output will now show you the CPU number for each process.

UPDATE: If you need to see all of the threads associated with a process, hit "H" in top and you'll get them all. Along with the above option, you can see just how badly abused your CPUs are.

Stargate SG1 - Season 10

Oh my god. I'm watching episode 6 from season 10, named "200". This has got to be the funniest episode I have ever seen. Everyone should have to see this.

sed tip : removing blank space from the front of a line

There are times when you want to trim text quickly, maybe in a shell script. One way to do this is to use sed. To trim off all blank spaces, including any tab characters, just use


sed -e "s/^[ \t]*//g" your_text.txt

This will go through your file and trim the front of each line.

SGE tip - selecting a host

This is the first of my tips for using SGE. Sometimes you need to tell grid engine that you want to run on a particular host. In these cases you need to  add the option

-l h=hostname

This will tell grid engine to apply whatever other rules it wants to,  but in the end only run your job if it can run on the host hostname.

Software for scientists

The site Software Carpentry is one of the better sites out there for scientists and researchers who want to learn more about computational science codes. It covers everything from Linux shells, to version control, to actual code development. I can't suggest strongly enough that people interested in this should go check the site out.

at, batch and cron

This was first published in Linux Journal, issue 184.


at,batch,cron - The ABC's of doing work when nobody's home



People have always been interested in doing more work with less effort. This drive kind of reaches its peak when work is being done, even though you aren't actually doing anything. With Linux, you can effectively do this with the trio of programs at, batch and cron. So now your computer can be busy getting productive work done, even long after you've gone home. Most people have heard of cron, fewer people have heard of at, and even fewer have heard of batch. Here you'll find out what they can do for you, and the most common options to get the most out of them.

at is actually a collection of utilities. The basic idea is that you can create queues of jobs to run on your machine at specified times. The time at runs your job is specified on the command line, and almost every time format known to man is accepted. The usual formats, like HH:MM or MM/DD/YY, are supported. The standard POSIX time format of [[CC]YY]MMDDhhmm[.SS] is also supported. You can even use words for special times, like now, noon, midnight, teatime, today or tomorrow, among others. You can also do relative dates and times. For example, you could tell at to run your job at 7PM in 3 days time by using "7PM + 3 days".

at will listen to the standard input for the commands to run, which you finish off with a ctrl-D. You can also place all of the commands to run in a text file and tell at where to find it by using the command line option "-f filename". at will use the current directory at the point of invocation as the working directory.

By default, at will dump all of your jobs into one queue named "a". But you don't need to stay in that one little bucket. You can group your jobs into a number of queues quite easily. All you need to do is add the option "-q x" to the at command, where "x" is a letter. This means that you can group your jobs into 52 queues (a-z and A-Z). This lets you use some organization in managing all of this after hours work. Queues with higher letters will run with a higher niceness. The special queue "=" is reserved for jobs currently running.

So once you've submitted a bunch of jobs, how do you manage them? The command atq will print out the list of your upcoming jobs. The output of the list is: job ID, date, hour, queue and username. If you've broken up your jobs into multiple queues, you can get the list of each queue individually by using the option "-q x" again. If you change your mind, you can delete a job from the queue by using the command "atrm x", where x is the job ID.

Now, what happens if you don't want to overload your box? Using at, your scheduled job will run at the assigned time regardless of what else may be happening. Ideally, you would want your scheduled jobs to run only when they won't interfere with other work. This is where the command batch comes in. batch behaves the same way at does, but will only run the job once the system load drops below a certain value (usually 1.5). You can change this value when atd starts up. By using the command line option "-l xx", you can tell batch not to run unless the load is below the value "xx". Also, batch defaults to putting your jobs into the queue "b".

These tools are great for single runs of jobs, but what happens if you have a recurring job which needs to run on some sort of schedule? This is where our last command, cron, comes in. As a user, you actually don't run cron. You instead run the command crontab, which lets you edit the list of jobs that cron will run for you. Your crontab entries are lines containing a time specification, and a command to execute. For example, you might have a backup program running at 1AM each evening:

0 1 * * * backup_prog

cron will accept a wide variety of time specifications. The fields available for your crontab entries include

field allowed values
minute 0-59
hour 0-23
day of month 1-31
month 1-12
day of week 0-7

Using these fields and values, we can use them directly, use groups of values separated by commas, use ranges of values, or use an asterix to represent any value. You can also use special values

string meaning
@reboot run once, at startup
@yearly run once a year (0 0 1 1 *)
@annually same as @yearly
@monthly run once a month (0 0 1 * *)
@weekly run once a week (0 0 * * 0)
@daily run once a day (0 0 * * *)
@midnight same as @daily
@hourly run once an hour (0 * * * *)

Now that you have these three utilities under your belt, you can schedule those backups to run automatically, or get a long compile to start after you've gone home, or get your machine to keep using up any idle cycles. So go out and get lots of work done, even when nobody is home.

Tunneling SSH

There are lots of times when you don't have a direct connection to a machine you want to work with. A way to make that connection is to create a tunnel through an SSH connection to a middle box. There are two ways to do this. The first is directly on the command line:

ssh -N -n -L 12345:host1:2345 host2

This command makes an ssh connection to host2. It then creates a tunnel from your current host, on port 12345, out to port 2345 on host 1.  This way you can connect to port 12345 on the current host and you get tunneled out through host2 to port 2345 on host1.

Hope this is handy.  Let me know how you use this.