My Favorite Linux Tools

Note: All these notes come without any warranty whatsoever. They might remove all your files or destroy your whole computer system, although most likely they won't.  :-)

Ask your system administrator if you are unsure about something.

-----------

Setuid - run a program with owner's rights [Ingmar]

http://en.wikipedia.org/wiki/Setuid

Access rights for directories [Ingmar]

Confusing. Surprising. Nicely summarized in the Wikipedia article.

Redirection and Pipes [Marius]

Basics: Every (text mode) program has three default "streams": stdout ('standard output'), stderr ('standard error'), and stdin ('standard input'). The first two are write-only, and the last one is read-only. By default, stdout and stderr correspond to the screen output and stdin corresponds to keyboard input. By using redirection, this can be changed.

Examples:
(The command 'foo' will be used as a placeholder for arbitrary commands.)

1) Write standard output to a file (two equivalent methods)
  foo > file.txt
  foo 1> file.txt

2) Write standard error to a file
  foo 2> errors.txt

3) Read standard input from a file (two equivalent methods)
  foo < input.txt

4) In certain cases, the redirection can also be put _before_ the command:
< input.txt foo
< input.txt > output.txt foo

5) Redirect stderr to stdout
  foo 2>&1

6) Redirect both stderr and stdout to the same file.
  foo > file.txt 2>&1
(Attention to the order. The following has a different effect:
  foo 2>&1 > file.txt
)


Piping: Connect the standard output of one process to the standard input of another process. Examples:

1) ls | grep mp3   (Filter all file names containing 'mp3')
2) ps ax | perl    (Filter all running processes with 'perl' in their name)


Tee: Pipe standard output to another program, while simultaneously copying it to a file. Example:
  ls | tee dirlist.txt | grep mp3
This first saves the output of ls to dirlist.txt, but at the same time passes it on to grep.


Unnecessary 'cat':

Use
  grep searchstring < file
instead of
  cat file | grep searchstring

More infos can be found in the bash man page under "Redirection". (Type 'man bash' to access the man

Basics of SSH [Marius]

Connect to server:
  ssh server
or, explicitly specifying the user name:
  ssh user@server
Secure copy:
  scp <source> server:/path/to/destination
or, as before
  scp <source> user@server:/path/to/destination
The following is an abbreviation for the user's home directory on the remote machine:
  scp <source> server:

Examples:
  scp file.txt lth.epfl.ch:/tmp
  scp file.txt kleiner@lth.epfl.ch:/tmp
  scp file.txt lth.epfl.ch:
   

Generate your personal public/private key pair:
  ssh-keygen

To be able to log onto a remote server with this key pair, insert the public key into the file .ssh/authorized_keys in the home directory on the remote server.

To avoid having to specify the username each time, you can add something like the following to the file .ssh/config:

  Host myserver.something.com
    User marius
   
  Host *.epfl.ch
    User kleiner

Usefulness of public keys with ssh:
- No longer need to enter password each time.
- Lets jobs that run without user input access other machines (e.g. cron job).
- Allows several users to access the same account such that they are logged separately.


Tunneling with ssh:

Example:
  ssh -L 8888:192.168.1.1:80 user@host.net
As long as the ssh connection remains established, connections to port 8888 on the local machine are forwarded to port 80 on the machine 192.168.1.1 in the remote network. This can be used e.g. to connect to a VNC server or to access another computer in the remote network that would otherwise not be publicly accessible.
 

Controling your mom's PC remotely [Ingmar]

I showed a combination of Hamachi to create a VPN and a VNC server. Tali pointed out that an easier solution is to use LogMeIn. They also offer a free version of their software, which seems to work very well.

Working remotely with a graphical interface [Ingmar]

To display graphical output you will need to start an X-server. On Linux this will generally already be running (if not, try "startx"). On Windows you'll need to download and install a (free) X-server. E.g. Xming. Some ssh clients automatically start x-forwarding (= redirect the remote display to your local display). If not you'll need the '-X' option with ssh.

ssh -X user@somemachine.epfl.ch

This is a capital X and *not* an x.

xclock

Just an arbitrary graphical application. This should now open a window on your local machine.

If you want to have your graphical application run more permanently, even if the ssh connection dies, then use VNC.

Log on to some "small" machine in your lab via ssh.

1. ssh someserver  (this needs to be accessible from the "outside" - or use VPN).

2. ssh somepc   (this is the "small" machine)

and then, on somepc, type something like:

vncserver -geometry 1111x999

Here 1111x999 is the resolution you want to have. Replace it by whatever suits your purpose.

Write down the display number you're assigned. Probably :1 if it is the first time somebody is running VNC on this machine.

Later do:

ssh -L5903:somepc:5901 user@someserver.epfl.ch

The '1' is the one from above and the '3' you can replace by your favorite prime number.

Then, locally, start a vncviewer and point it to localhost:5903

Maybe then, inside the window on somepc, do startx or startkde if you don't see a nice graphical display.

Grep, sed, uniq, .... [Ingmar]

Here's the long command I typed:

wget -O - http://www.clearwhitelight.org/hitch/hhgttg.txt |  sed 's/\W/\n/g' | grep "\w" | perl -ne 'print lc $_;' | sort | uniq -c | sort -k1,1nr | tail +10 | less

Let's break this down.

1. wget -O - http://www.clearwhitelight.org/hitch/hhgttg.txt

Download the indicated file and print its content to STDOUT. STDERR will be used to show progress but that won't interfere with the following "pipes".

2. sed 's/\W/\n/g'

Substitute all non-word characters, i.e. everything that's not a letter, a number or an underscore, by a newline character. Note the 'g' for 'global' replacing. I.e. replace *all* occurrences on one line.

3. grep "\w"

Only keep the lines which have at least one word character

4. perl -ne 'print lc $_;'

The '-e' option for Perl can be used if you feed the source code explicitly as a long argument, rather than putting it into a file.

The '-n' option enables a line-based mode which puts the current line into the $_ variable in Perl. Pretty much the same as a while(<STDIN>){}

The 'lc' casts the whole line to lower case. The 'print' then prints the result. Note that the new line character also belongs to $_.

5. sort

Sort the words alphabetically

6. uniq -c

Remove repeated lines and count how often they were repeated

7. sort -k1,1nr

Sort numerically ('n') one the first "column" ('-k1,1'). Sort in reverse = descending order ('r)

8. tail +10

Remove the *first* 10 lines and print all the other lines.

9. less

Gives you the option to scroll up and down and to search for a word by typing '/'.

Keep Applications Running Using Screen [Marius]

Start a new instance of screen:
  screen

Detach from current instance:
  CTRL-A-D

Reconnect to a running instance:
  screen -r

If multiple instances are running:
  screen -r <pid>

nohup: This command prevents applications from terminating when the the owner disconnects. However, unlike screen, it is not possible to bring the process back to the foreground, i.e., to interact with it again. Note: On most modern linux systems, applications running in the background are automatically detached and continue running if the user disconnects, making nohup obsolete.

Lightweight alternative to screen: dtach

 

Using find to search for and treat multiple files [Philippe]

find . -name '*.c'

Mind the quotes! It may or may not without them, depending on whether you have C files in the current directory (because the shell will expand unquoted *.c if you do)

find . \( -name '*.c' -o -name '*.h' \)

find . -name '*.c' -exec wc -l {} \;

The command wc -l is executed for each math. {} is substituted by the matching filename (complete with path), \; marks the end of the command

find . -name '*.c' -exec wc -l {} +

+ means the names are collected and passed altogether to the command. When using plus, you can use {} only once.

find . -name '*.c' | grep xargs wc -l

find . -name '*.c' -exec cp {} {}.bak \;

Note how you can use {} multiple times.

find . -name '.svn' -delete

find . -depth -name '.svn' -ok rm -rf {} \;

-depth does depth-first-search instead of breadth-first search. This avoids error messages which would otherwise show up when trying to traverse a directory .svn you have just deleted.

-ok is like -exec, except it needs confirmation (y or Y)

find ~ -mmin -10

Convenient to figure out where these downloads went :)

 

Using locate to search for files [Philippe]

locate *.c

Note that locate does not actually traverse the filesystem, it uses a database. Run

updatedb

to, well, update that database.