Skip to main content Skip to local navigation
Patrick Hall
Home » Research

Research

If you're not familiar with any of the following, here's where to go to get started. See also Chris Matzner's resources for the beginning astronomy researcher

Unix/Linux: an operating system and simple programming environment.

  • Start with The Official Red Hat Linux Getting Started Guide at this archive of the Redhat website. (Redhat is one version of Linux.) 
  • To edit a text file from the command line in Linux, you can use either Emacs or Vi/Vim. You can read about them both at this archived site from the Redhat website. (Prof. Hall can answer questions about Vi/Vim, but not about Emacs.)
  • See the next two sections for an illustration of some basic linux commands.

Useful Linux & IRAF commands

  • When you open a terminal window on a Mac, your working directory is your home directory: /Users/username
  • type 'cd NAME' to change the working directory to the subdirectory NAME in the current directory. You can move through more than one directory at once; for example: 'cd NAME/SUBDIRECTORY'. You can move directly to a specific directory; for example: 'cd /Users/Shared/SDSS/'.
  • If you don't have permission to move to a directory, or to open or edit a file, you can fix that if you have Administrator privileges. The 'sudo' command lets you override permissions by entering your password, and the 'chmod' command lets you change the privileges on a file or directory. To gain access to a directory or file called 'name', type: sudo chmod a+rwx name.
  • type 'man' to get help for a Linux command; e.g., 'man cd' or 'man sort'
  • type 'help' or 'phelp' to get help for an IRAF command; e.g., 'help sort'
  • Note: Linux sort & join are similar to, but DIFFERENT in detail, than IRAF sort & join!
  • Linux: cat, grep, join, sort. Try them all!
  • IRAF: average, filecalc (in the CTIO package), join, sort.
  • To view a text file without editing it, use 'more' or 'less' in Linux, or 'page' in IRAF.
  • 'tar' creates binary archives of groups of files - very useful for moving files from computer to computer. Use 'tar cvf output.tar inputfile1 ... inputfileN' to create the archive output.tar (c for create, v for verbose, f for filename); 'tar tvf output.tar' to list back the files (t for Table of Contents); and 'tar xvf output.tar' to extract all the files (x for extract) or 'tar xvf output.tar inputfilename' to extract just the named file. You can gzip (compress) a tar file, but you must gunzip (uncompress) it before extracting anything, unless you use the z flag: 'tar xvzf output.tar.gz'.
  • Redirection and Piping: You can redirect the output of a task to a file, or pipe the output of a task to another task. To redirect to a new file, use '>'. To append to an existing file, use '>>'. To pipe, use '|'. (These commands are identical in Linux and IRAF. Note: Linux will give you an error if you're trying to append to a nonexistent file, while IRAF will just treat '>>' as '>' in that case.)
  • In IRAF, if you are getting strange errors, for example if you've exited a task using ctrl-C, run the 'flpr' command ('flush processes') three times. You can run multiple commands on the same line by separating them with ';' (a semicolon): 'flpr ; flpr; flpr'.
  • 'grep' is a powerful search tool for finding strings or 'regular expressions' in files. To print out all lines which contain the string 999 with a space before it in a file, do: grep ' 999' filename. To print out all lines which do NOT contain the string 999 with a space before it, do: grep -v ' 999' filename. Use '^' for the start of a line and '$' for the end of a line, so: grep '^ 999$' filename will return all lines that consist solely of a space followed by the string 999. Do 'man grep' or search the internet to learn more about the awesome power of grep. Be aware that using 'grep' from Linux and '!grep' from IRAF sometimes give different results (for reasons I've never understood). The same applies to sed.
  • 'sed' (Stream EDitor) is a powerful utility for formatting and manipulating text & numbers. For example, let's say you need to format a Right Ascension + Declination string "123456+780900" into the format "hh:mm:ss +dd:mm:ss". You can do that by running:
    sed 's/\(^..\)\(..\)\(..\)\(.\)\(..\)\(..\)\(..\)/\1:\2:\3 \4\5:\6:\7/' < oldfile > newfile
    and "newfile" will contain the line "12:34:56 +78:09:00". Yes, the syntax takes some getting used to, but you can get help by typing 'man sed' or 'info sed' or reading this sed introduction & tutorial at least through the "Is sed recursive?" section, including this tutorial on regular expressions (at least through the "Backreferences" section) when it is referenced in the sed tutorial.

Efficiency

  • In Linux, you can use the up-arrow key to revisit earlier commands; hit return on any one of them to repeat the command (you can even edit them before running them, if you want to fix a typo for example).
  • In IRAF, typing "e" for "ehistory" will bring up the previous command, and then you can use the up-arrow key to revisit earlier commands. You can even type "e squiggle" and it will bring up the most recent command line starting with "squiggle".
  • In Linux, typing '!' before a command will repeat the last incidence of that command. Any trailing characters will be added to the command. So if you did 'ls april*' to find all files starting with 'april', you could then do '!ls may*' instead of 'ls april* may*' to find all files whose names start with 'april' or 'may'.
  • Tab completion: in Linux, whenever you hit the tab key while typing a command, it will display the longest unique string fitting what you've already typed, so you don't have to type the full filename of a long file.
  • In IRAF, you can set aliases for commonly used directories in your 'login.cl' file. If you look at that file, you'll see 'set home = ...' near the start of the file; that line lets you type 'cd home' and return to the home directory. So you can add a line that reads "set work = /some/really/long/chain/of/directory/names/" and then just type 'cd work' to get to that directory. Note that like all updates to login.cl, it won't take effect until the next time you start IRAF; however, you can type 'set work=...' on the command line to use the alias in your current IRAF session.

Communication between computers

  • Linux to Linux: ssh (secure shell) is used on one linux computer to open a text window on another linux computer. For example, from a window on ara I use 'ssh phall@fornax.phys.yorku.ca' to open a window on fornax. (I could just use 'ssh phall@fornax' since they are both in the phys.yorku.ca domain.) If I start IRAF on fornax, IRAF graphics appear on ara just fine, but image display using ds9 is trickier to configure; see the IRAF website.
  • Windows to Linux: ssh can be used on a windows machine to open a window on a linux machine, although graphics display doesn't work. If `putty' or `F-Secure Shell' doesn't already exist on your windows machine, type 'putty download' into google, and download the putty ssh executable to your Windows computer. Running the putty version of ssh is self-explanatory.
  • scp (secure copy) is used to copy files from one linux computer to another. To copy a file from my current directory on my current linux machine to my home directory on felis, I would type 'scp filename phall@felis.phys.yorku.ca:'. To copy files into a particular Shared directory on felis to the same directory on manis, I would open a terminal window on manis and type 'cd /Users/Shared/directory_name/' and 'scp phall@felis.phys.yorku.ca:/Users/Shared/directory_name/filename .' (the '.' means the current directory). You can find out your current working directory on your current machine by typing 'pwd' [print working directory]. From Windows to Linux, use `F-Secure Shell' or 'pscp' [Putty scp] instead of scp.
  • Windows to Linux: Xwin32 lets you bring up the full screen of a linux computer on your windows machine.
  • There are other communications options, but the above are the ones I'm familiar with.

LaTeX

  • Latex is a text formatting language: you write a mixture of text and latex commands in a text file and then run the program 'pdflatex' to turn that file into a document in PDF format. 
  • The following are two basic introductions to latex: you can start at the beginning of this one, or start with the Text Modes section of this one.
  • Here's an overivew of Including graphics in a LaTeX document
  • PDF files can be viewed and printed using the 'acroread' command.
  • Presentation-format PDF files can be created with the "pdfscreen.sty" style file. A very short introduction to this style file can be found in this PDF file, or you can recreate a sample presentation found on fornax as "/home/phall/sample.tar". That "tar file" is a binary archive; see tar under UNIX commands above for how to extract files from it. The 'sample.tar' archive contains pdfscreen.sty, sample.tex, and all the jpg files needed to create sample.pdf by typing 'pdflatex sample' after you've extracted all the files. Edit sample.tex to create your own presentation... you can create jpg files from other formats such as PostScript by using 'convert -density 300 image.ps image.jpg'; type 'man convert' for details.
  • NEW! Writing & Speaking Guidelines for Engineering & Science

IRAF (Image Reduction and Analysis Facility): a computer language for astronomical research.

  • The IRAF home page is iraf.noao.edu, but be warned that some parts of the website are truly ancient. By clicking on 'All Docs' under 'Documentation' in the left-hand panel, you'll go to a web page with links to documentation on a variety of subjects... as you work on different projects over time, you'll need to read more & more of these manuals. And you can always search iraf.noao.edu and IRAF.net for help if you have problems.
  • To start, click on 'Recommended Documentation for every site'. You'll want to read A Beginner's Guide to Using IRAF; if you want more after that, read A User's Introduction to the IRAF Command Language. 
  • If you click on 'A Beginner's Guide to Using IRAF', most likely it will not display because your internet browser doesn't recognize the format it's in. Here's how to view it:
    • Save the file with its default name of 'beguide.ps.Z'
    • The '.Z' and '.gz' suffixes indicate compressed files. To uncompress such a file, use 'gunzip': 'gunzip beguide.ps.Z' (To create a .gz file, use 'gzip'. Don't worry about creating .Z files.)
    • If you type 'ls' to list files in your current directory, you'll now see the uncompressed file 'beguide.ps'. The '.ps' suffix indicates a PostScript file.
    • PostScript files can be viewed using 'gv' [or 'ggv']: 'gv beguide.ps &'. The ampersand (&) tells the computer to run the gv program in the background, so you can continue to run programs from the window while 'gv' is running. [Try it without the ampersand to see the difference.]
    • If the resolution of the text in the gv window is poor, hit 'a' in the gv window. To quit the gv window, move the cursor to it and hit 'q'.
    • To print a PostScript file, just type 'lpr beguide.ps' [lpr stands for laserprinter] and it will be sent to the printer 'gutenberg', right in front of the door to room 336. You CANNOT use lpr to print a compressed file (ending in .Z or .gz or .zip); always uncompress the file first! (Also be sure to print PDF files from a PDF viewer, since 'lpr file.pdf' will just print pages of junk on many printers.)
  • Obscure but useful: IRAF usually disables the backspace key, so you have to use the delete key to erase text. If you want to use the backspace key to erase text, type (or put in your login.cl or loginuser.cl file) '!tset -e^H'. (The '!' tells IRAF that what follows it is a Linux command, not an IRAF command.)
  • Resetting some basic IRAF parameters. The default values of some common IRAF parameters can be a bit dangerous, so I always reset them. Once you've learned how to use the 'epar' command, go ahead and do the following (you have to do it separately on each computer on which you run IRAF):
    • 'epar dir' and set 'maxch=38' [so you can see long filenames in full]
    • 'epar del' and set 'verify=yes' [won't delete files without asking you]
    • 'epar imdel' and set 'verify=yes' [won't delete images without asking you]
    • 'epar imren' and set 'verbose=yes' [tells you when images are renamed]
    • 'epar movef' and set 'verbose=yes' [tells you when files are moved]
    • 'epar copy' and set 'verbose=yes' [tells you when files are copied]
    • 'epar imstat' and set 'fields=image,mean,mode,stddev,min,max' and 'binwidt=0.01'. [for technical reasons I won't go into now]
  • If you need to install IRAF on a linux machine, see see this website. Note that you should install external packages STSDAS/TABLES 3.13 instead of 3.12 (if you install them at all).

Making Plots

  • The 'xgterm' window allows you to display graphics in IRAF. You can start up a new xgterm by typing: 'xgterm -sb -bg black -fg yellow &' ... the '-sb' gives you a scrollbar, and 'bg' and 'fg' control background & foreground colors.
  • Use the 'fields' command to extract specific columns from a file, and then pipe those columns to the 'graph' task to make a simple plot. So 'fields filename 1,3 | graph' would plot column 1 of 'filename' on the x-axis, vs. column 3 on the y-axis.
  • Use the 'fields' command to extract a single column from a file and output it to another file ('fields filename 3 > filename3'). Then run the 'phistogram' task on that file ('phistogram filename3'). Alternatively, you can pipe the output of fields to phistogram, but you must tell phistogram to expect input from STDIN (standard input): 'fields filename 3 | phistogram STDIN'.
  • SM is a powerful (and therefore complex) plotting program usable under Linux. Type 'sm' to start a session. Start by reading the basic tutorial online, and then when using SM you can consult the online reference manual, which includes descriptions of all SM commands. For example, using the 'points' and 'errorbar' commands will let you create a plot showing both the X and Y uncertainties for every point, and using 'histogram' and 'shade' will allow you to overplot multiple histograms. You'll need to create macro files to save your plotting routines. Some example plots and macros can be found at the bottom of this page.
  • IGI is a plotting language that can be called from within IRAF (after loading the STSDAS and GRAPHICS packages); for details type "help igi" in IRAF.
  • NEW! To grab a postscript snapshot of a plot in an IRAF graphics window, type "=gcur" and return at the IRAF prompt. That puts you in the graphics window. Then type ":.snap eps" and return to create a postscript file. Then type "q" to quit out of the graphics window. The postscript file you created will have a name starting with sgi and ending with eps (type "dir sgi*eps" to find out its name); it should be located in the directory where you typed "=gcur". You'll probably want to rename it to an easier to remember name.
  • To trim postscript files, use the UNIX commands "eps2eps" or "ps2epsi". Although they don't always do what you want, in general they trim unwanted whitespace from around a plot.

Backups

  • These are handled by SuperDuper!.

Statistics

  • A very good reference is Data reduction and error analysis for the physical sciences (2003) by Philip R. Bevington & D. Keith Robinson. The York library has a copy, plus a copy of the 1969 version (by Bevington only). For further info, when needed, consult any version of `Numerical Recipes', or see Level 5 below.

Extragalactic Astronomy and Cosmology

  • Caltech and Carnegie run Level 5, A Knowledgebase for Extragalactic Astronomy and Cosmology. It's a good reference for introductions to many different topics.

Active Galactic Nuclei

  • Start with Brad Peterson's "An Introduction to Active Galactic Nuclei". I recommend skipping section 1.2, and all of chapter 9 except section 9.4.1. Chapters 1-5 and 7 are the most important for a basic understanding, but you're welcome to read the other chapters too (especially ones which I've mentioned as important for your specific research project!).