Scripting in Unix


Some Simple Facts About Unix

  1. who - tells you who is logged in.
  2. date - current date and time.
  3. whoami, who am i or id - currently logged in username.

The heart of an operating system is called the Kernel. The Kernel allows access through utilities to the hardware. For instance, interpretation of keys pressed on the keyboard and echoing those keys pressed onto the screen is handled by parts of the Kernel of the operating system. The Kernel is always in memory.

Commands in Unix can be added together either by executing one command after another or by executing a subsequent command based on the results of the previous command. The former separates commands with semi-colons, (date; who; ls) and the latter relies on passing the STDOUT output of one command into the STDIN of a subsequent command by use of a pipe (the | character).

The main interface to a Unix system is called a Shell. A Shell is like a DOS window in Windows. A Unix Shell is intended as a basic command line interpretive character-based interface into Unix. There are windows-type GUI tools for Unix such as X and Openwin. Unix shells allow some fairly comprehensive tools and scripting / programming languages. Shell scripts are written in a number of different forms, ie. basic shell (/bin/sh), C-shell (/bin/csh or /bin/tcsh), Korn-shell (/bin/ksh) and Bash (/bin/bash). Also different versions of Unix make for different versions of Unix scripting languages. Also Perl is widely used as a scripting language. However, Perl is more consistent with text parsing and searching utilities such as awk and sed.

Most shells allow process control, variables, regular expressions, flow control, input/ouput and functions. C-shell allows command history, aliases and job controls but no functions. C-Shell (including tcsh), Korn-shell and Bash allow file name completion allow entry of partial filenames; a key is pressed which allows finding and completion of the nearest matching filename, generally in the current directory. The basic C-Shell (/bin/csh) allows filename completion using the set filec setting with the escape key. /bin/tcsh allows filename completion using the set filec setting with the tab key. K-Shell and Bash also allow arrays and integer arithmetic. Bash allows unlimited array sizes and floating-point arithmetic.

Logging in and what Happens when Logging In

Logging in accesses the password files for the purposes of validation of and allowing a username access to the operating system. Usernames and passwords are generally stored in /etc/passwd, /etc/shadow and /etc/groups (groupings of user access rights). More specialised username and password control can be gained by using the /etc/shadow (password shadow file) by use of the passmgmt utility. When logging into a machine a user has a default profile set up from the root user level. These profiles can be over-ridden by creation of user specific profile files. Profiles contain default path settings, variable names, aliases, etc. Default profile files are contained in the /etc/skel directory. There are three types of profile files, .profile, .cshrc and .login. Note that logging into a username will always start a shell. The default shell is usually /bin/sh. The PATH variable is set to /bin:/usr/bin at a minimum. You can also set the MANPATH variable to /usr/man:/usr/share/man to allow access to Unix (man)ual pages (man command).

Files, Directories and Their Attributes

Files

List files in a directory using the ls command. Many options are available. ls -la will list filenames vertically with various attributes. awk could be used to pull out file sizes and file names as shown below. The -a is important because it shows hidden files such as login profile files.

# ls -la | awk '{ print $4,$5 }'
 
users 512
other 2048
dba 1042
dba 512
dba 512
dba 512
dba 512
dba 1024
dba 512
dba 512
dba 512

Files can be viewed on the console using one of the following methods.

The wc [-(l)ines | -(w)ords | -(c)haracters] (word-counting) utility allows for counting of lines, words or characters in a file or passed from STDIN, STDIN is the result of STDOUT passed through a pipe (|) command. The example below shows all Oracle database connections, active and inactive on an Oracle database server.

oracle> ps -ef |grep $ORACLE_SID|grep -v grep|grep -v ora_|wc -l

      5

oracle> ps -ef |grep $ORACLE_SID|grep -v grep|grep -v ora_

  oracle 21481     1  0 11:25:36 ?        0:00 oracletest (LOCAL=NO)
  oracle   845     1  0   Jul 24 ?        0:00 oracletest (LOCAL=NO)
  oracle 28398     1  0 17:15:57 ?        0:00 oracletest (LOCAL=NO)
  oracle 21479     1  0 11:25:36 ?        0:00 oracletest (LOCAL=NO)
  oracle  1365     1  0 00:42:43 ?        0:00 oracletest (LOCAL=NO)

The cp source target command allows copying of files as the mv command simply renames. Note that multiple files can be copied or moved to a single destination. The -i allows for interactivity and will request confirmation when overwriting an existing file. The rm command removes files. It is usually best to set an alias in your login profile file setting the rm command to interactive mode, ie. alias rm 'rm -i'.

Directories

A disk is divided at the lowest level into blocks. At that point different operating systems will format disks in different ways. Directory structures are tree-like structures where directories can contain other directories or files. In Unix the root node is reached with the cd / command. Files can be accessed from within a directory structure by absolute or relative pathnames or from the root node, for example /u01/oracle/product/8.1.7/bin/sqlplus or by relative paths using . to represent the current directory and .. to represent the directory above the current directory. Note that any files contained within the pathnames defined in the $PATH variable are directly accessible, this is what setting the path is for.

General directory commands are shown below.

Directories can be copied and moved using the cp and mv commands with the -r recursive option.

File Attributes and Permissions

File ownership is altered using the chown [-R] options user:group files and the chgrp options group files commands. Note the -R option in the chown command -R is directory recursive.

The example below shows some files when listed with the ls -la command. Note the first two files in the listing below. The . represents the current directory and the .. represents the parent directory. The first column represents file permissions. Note the two directories have a d in the first character of the first column, ie. directory. There are three sets of slots for read, write and executable attributes, ie, rwxrwxrwx. These three sets represent the owner of the file, the group owner belongs to and other users. These attributes can be altered using the chmod command in two ways. chmod g-x * would remove all executable group permissions from the files in the list below. All permissions for owner, group and other can be altered in a less verbose way using a sum of 4(r), 2(w) and 1(x) for each of owner, group and other. For instance, chmod 700 * on the list of files shown below will remove all access privelages from both group and other and ensure that all files are read, write and executable for the owner.

drwxr-xr-x   2 oracle   dba         3072 Jul 23 13:45 .
drwxr-xr-x  31 oracle   dba         1024 Jul 23 13:43 ..
-rwxr-xr-x   1 oracle   dba        13436 Sep  8  2000 adapters
-rwxr-x--x   1 oracle   dba        52516 Jul 23 13:45 agtctl
-rwxr-xr-x   1 oracle   dba        53488 Aug 17  2000 agtctlO
-rwxr-xr-x   1 oracle   dba      1613108 Jul 23 13:44 bulkmodify
-rwxr-xr-x   1 oracle   dba         2840 Mar 31  2000 coraenv
-rwxr-xr-x   1 oracle   dba       167920 Aug 21  2000 csscan
-rwxr-xr-x   1 oracle   dba            0 May 31  2000 csscanO
-rwxr-x--x   1 oracle   dba        29736 Jul 23 13:44 cursize
-rwxr-xr-x   1 oracle   dba        30436 Aug 17  2000 cursizeO
-rwxr-xr-x   1 oracle   dba       103216 Aug 22  2000 dbfmig

Other settings for the first character in the first column of the listing above are c, b, p, s and l. The characters c, b, p and s represent character special, block special, named pipe and socket respectively. l represents a symbolic link. A symbolic link is a file in one directory containing a reference to another file somewhere else in the file system. In the link command below -s will create the link and +s will remove the link. The link can also be removed like a file with the rm command either by deleting the link or the file itself.

ln [-+]s source target

Another first character in the first column is known as the s-bit. This setting is required for running utilities such as that changing passwords. Since the password file is owned by root, a non-root user cannot normally change the password file directly. Thus programs such as passwd and passwdmgt change the password in the /etc/shadow file for the user. This is achieved by effectively changing permissions on the password file whilst the password change utility is running thus allowing the user to change the password file.

Character and Block Special Device Files

These files are really drivers used for communicating with devices one character at a time (character special) or by way of large blocks of data at a time (block special). Block special driver files are usually for media devices such as hard-drives and cd-rom drives. Devices are located in the /dev directory.

Named Pipes and Sockets

Pipes are utilised to pass information from the STDOUT result of one command into the STDOUT of another command without having to create an unneccessary file inbetween. Named pipes are used when a file is required to be created. Named pipes are typically used for interprocess communication (IPC). A Socket is exactly like a Named Pipe except that a Socket is utilised to pass information between process on different machines over a network. Thus Named Pipes are used for interprocess communcation on one machine and Sockets are used for interprocess communication between processes on separate machines on a network.

Processes

Every Unix command starts a process in foreground or background with a unqiue process ID. Use the ps command to view currently running processes. Foreground processes take input from the keyboard and send output to the monitor. A process run in background should take input from a file and send output to another file. Execute a background process by executing <command> &. Note that if a background process requires input it does not get from a file it will stop until it receives input or until it is placed into foreground or is killed manually. The no-HangUp (nohup) command can be used to continue a process execution after the user executing the process has logged off the machine.

Moving Processes between Background and Foreground

A process must first be suspended to transfer it between foreground and background. Suspend a process by pressing Ctl-z. (Check for function performing keys by typing stty -a as shown below.

# stty -a
speed 9600 baud; 
rows = 52; columns = 113; ypixels = 0; xpixels = 0;
eucw 1:0:0:0, scrw 1:0:0:0
intr = ^c; quit = ^\; erase = ^?; kill = ^u;
eof = ^d; eol = ; eol2 = ; swtch = ;
start = ^q; stop = ^s; susp = ^z; dsusp = ^y;
rprnt = ^r; flush = ^o; werase = ^w; lnext = ^v;
-parenb -parodd cs8 -cstopb -hupcl cread -clocal -loblk -crtscts -crtsxoff -parext 
-ignbrk brkint ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl -iuclc 
ixon -ixany -ixoff imaxbel 
isig icanon -xcase echo echoe echok -echonl -noflsh 
-tostop echoctl -echoprt echoke -defecho -flusho -pendin iexten 
opost -olcuc onlcr -ocrnl -onocr -onlret -ofill -ofdel 

When typing Ctl-z you will be shown the job number as shown below. Typing bg will move the first job in the list to the background. If more than one job is suspended use the bg %n form of the bg command. Exactly the same process applies to moving a job from background to foreground where the job is suspended with the Ctl-z command and the job itself is moved into foreground using the fg %1 command.

^z[1] + Stopped (SIGTSTP) <process name>

The wait %1 command can be used to disallow entry of further commands at the command line console until all processing is completed. Be careful using this command.

Viewing Running Processes

The jobs can command be used to show all currently running and stopped jobs. The ps command can be used to show all currently running processes. The most commonly used forms of the ps command are ps -ef and ps -eaf. The addition of the -a option simply lists most frequently accessed processes as a priority. A ps -ef command is shown below, column headers will not appear when typing in the ps -ef command due the the use of the grep command. The columns represent user-ID, process-ID, parent process-ID, process CPU utilisation, process priority, process start-time and command which started the process.

# ps -ef |grep ora_|grep -v grep

      UID   PID  PPID  C    STIME TTY      TIME CMD

  oracle 17750     1  0   Jul 26 ?        0:06 ora_arc0_TEST
  oracle 17740     1  0   Jul 26 ?        0:07 ora_dbw0_TEST
  oracle 17748     1  0   Jul 26 ?        0:00 ora_reco_TEST
  oracle 17742     1  0   Jul 26 ?        0:18 ora_lgwr_TEST
  oracle 17744     1  0   Jul 26 ?        0:12 ora_ckpt_TEST
  oracle 17746     1  0   Jul 26 ?        0:00 ora_smon_TEST
  oracle 17738     1  0   Jul 26 ?        0:00 ora_pmon_TEST

Killing Processes

KIlling a foreground or background job can be done by using the kill %1 command. In general processes are killed by using the kill [-9] <PID> command. The -9 option is a last resort when the kill <PID> command does not work. Note that killing a process should kill all it's child processes aswell. Gnerally a process is forked (split into two?) when a parent process spawns a child process. A process can be overlayed with another by the exec command. For instance, when running /bin/sh the shell can be overlayed with /bin/ksh by typing the command exec ksh, effectively replacing shell with Korn-Shell.