Chris F.A. Johnson
Back-cover blurb
The shell is a programming language! A shell script is as much a program as anything written in C, Python or any other programming language. Just because shell scripts are easier to write doesn't mean they should take a back seat to compiled programs or other scripting languages.
I wrote Pro Bash Programming as a tutorial that introduces the shell, and bash specifically, as a programming language. If you need a program to accomplish a task on Linux, I want you to consider the shell before any other language. If you write a prototype for a program using the shell, I want you to realize that you don't need to translate it to another language.
This book will give you a grounding in programming techniques used in writing shell programs no matter what your past programming experience. But even if you've never written a computer program before, Pro Bash Programming will get you started and help you quickly become a proficient shell programmer. If you have written a few shell programs, this book will take you to the next level and beyond. It will enable you to do things with the shell you never thought it could do. If you are already an expert shell programmer, this book will provide insight into the aracana of shell scripting, helping you to write more, and more efficient, scripts.
This book goes beyond the standard Unix shell, and uses the many added features of bash. Bash is the shell of the Free Software Foundation's GNU project, and is the standard shell on almost all Linux distributions. It is the shell you probably use at the command line. It offers the programmer many enhancements over the standard traditional Unix shell. You will find bash on many versions of Unix. It may not be the default shell, but it is usually available for interpreting your scripts.
I hope this book will help you become a more productive programmer and that your programs will be written using the Bash shell.
Contents at a Glance
| About the Author | xvii |
| About the Technical Reviewer | xviii |
| Introduction | xix |
| Chapter 1: Hello, World! Your First Shell Program | 1 |
| Chapter 2: Input, Output, and Throughput | 7 |
| Chapter 3: Looping and Branching | 19 |
| Chapter 4: Command-Line Parsing and Expansion | 29 |
| Chapter 5: Parameters and Variables | 43 |
| Chapter 6: Shell Functions | 59 |
| Chapter 7: String Manipulation | 67 |
| Chapter 8: File Operations and Commands | 79 |
| Chapter 9: Reserved Words and Builtin Commands | 97 |
| Chapter 10: Writing Bug-Free Scripts and Debugging the Rest | 113 |
| Chapter 11: Programming for the Command Line | 125 |
| Chapter 12: Runtime Configuration | 141 |
| Chapter 13: Data Processing | 157 |
| Chapter 14: Scripting the Screen | 179 |
| Chapter 15: Entry-Level Programming | 191 |
| Appendix: Shell Variables | 205 |
| Index | 219 |
Introduction
Although most users think of the shell as an interactive command interpreter, it is really a programming language in which each statement runs a command. Because it must satisfy both the interactive and programming aspects of command execution, it is a strange language, shaped as much by history as by design. Brian Kernighan and Rob Pike, The UNIX Programming Environment, Prentice-Hall, 1984
The shell is a programming language. Don't let anyone tell you otherwise. The shell is not just glue that sticks bits together. The shell is a lot more than a tool that runs other tools. The shell is a complete programming language!
When a Linux user asked me about membership databases, I asked him what he really needed. He wanted to store names and addresses for a couple of hundred members and print mailing labels for each of them. I recommended using a text editor to store the information in a text file, and I provided a shell script to create the labels in PostScript. (The script, ps-labels, appeared in my first book, Shell Scripting Recipes: A Problem-Solution Approach.)
When the SWEN worm was dumping hundreds of megabytes of junk into my mailbox every few minutes, I wrote a shell script to filter them out on the mail server and download the remaining mail to my home computer. That script has been doing its job for several years.
I used to tell people that I did most of my programming in the shell but switched to C for anything that needed the extra speed. It has been several years since I have needed to use C, so I no longer mention it. I do everything in the shell.
A shell script is as much a program as anything written in C, Python, or any other language. Just because shell scripts are easier to write doesn't mean they should take a back seat to compiled programs or other scripting languages. I use the terms script and program interchangeably when referring to tasks written in the shell.
Why the Shell?
Some Linux users do all of their work in a GUI environment and never see a command line. Most, however, use the shell at least occasionally and know something about Unix commands. It's not a big step from there to saving oft-repeated commands in a script file. When they need to extend the capabilities of their system, the shell is the natural way to go.
The shell also has important advantages over other programming languages:
- It interfaces simply and seamlessly with the hundreds of Unix utilities.
- It automatically expands wildcards into a list of file names.
- Lists contained in a variable are automatically split into their constituent parts.
Just the Shell, Ma'am, Just the Shell
While most shell programs do call external utilities, a lot of programming can be done entirely in the shell. Many scripts call just one or two utilities for information that is used later in the script. Some scripts are little more than wrappers for other commands such as awk, grep, or sed.
This book is about programming in the shell itself. There's a sprinkling of the second type, where the script gets information (such as the current date and time) and then processes it. The third type gets barely more than a cursory nod.
A Brief History of sh
The Bourne shell was the first Unix shell in general use. It was much more limited than today's shells, so it was primarily a tool to run other tools. It had variables, loops, and conditional execution, but the real work was done almost entirely by external utilities.
The C shell, csh, added command history, arithmetic, and other features that made it popular as a command-line interpreter, but it was not suitable for more than trivial scripts.
The KornShell, developed by David Korn at AT&T Bell Labs, combined the Bourne shell syntax with features of the C shell. It was compatible with the Bourne shell, bringing important functionality into the shell itself and making script execution much faster. Until the year 2000, when it was opened up, ksh was proprietary, closed-source software.
The GNU Project, needing a free, open-source shell, introduced bash. Like all modern shells, bash is a POSIX shell. It also has many added enhancements.
Which Version of Bash?
This book is aimed at users of bash-3 and later, but much of
the book will work with bash-2.05.
Features introduced in bash-4.0, are covered, but it is noted
that they are newer additions.
Wherever possible without loss of efficiency, the scripts in
this book are written to run in any POSIX shell.
It is often noted when a statement in a script is nonstandard
or uses bash syntax.
That means that it is not POSIX compliant.
Most functions, however, will not run in a POSIX shell because
they almost always use the local builtin command.
Some Linux distributions modify their versions of bash:
Debian removes network socket capabilities.
Mandriva removes the builtin time command.
For this reason, I recommend compiling your own copy of bash.
The source code is available from
http://tiswww.case.edu/php/chet/bash/bashtop.html,
and it compiles without trouble almost anywhere.
Who Will Benefit from This Book?
If you're an experienced shell programmer, this book will provide insight into the arcana of shell scripting, helping you write more, and more efficient, scripts.
If you have dabbled in shell scripting, I hope this book will encourage you to experiment further.
If you are new to shell scripting, this book will get you started and help you quickly become a proficient shell programmer.
No matter what your level of experience, this book will enable you to program tasks that aren't already dealt with on your system.
What's in the Book?
From writing your first program to using the mouse in your scripts, this book runs the gamut from simple to complex and from the obvious to the obscure. It covers every technique you will need to write efficient shell programs.
Chapter 1, Hello, World! Your First Shell Program, presents the traditional first program in any language. It prints "Hello, World!" The chapter discusses how to write the script, what to name it, and where to put it.
Chapter 2, Input, Output, and Throughput, demonstrates output using the echo and printf commands and introduces the read command for input. It also examines redirecting both input and output and connecting commands with pipelines.
Chapter 3, Looping and Branching, explains the looping statements, for, while, and until; the branching statement if; and the conditional operators && and ||.
Chapter 4, Command-Line Parsing and Expansion, describes how the shell parses a command line, from word splitting to parameter expansion.
Chapter 5, Variables and Parameters, covers all the possibilities of parameters and variables, from scalar variables to associative arrays and from default substitution to search and replace.
Chapter 6, Shell Functions, delves into the syntax of function definitions and defines a number of useful routines.
Chapter 7, String Manipulation, contains a number of functions for dicing and splicing strings.
Chapter 8, File Operations and Commands, uses more external commands than the rest of the book put together. That's because looping through a large file with the shell is painfully slow, and the Unix utilities are very efficient. This chapter also tells you when not to use those utilities.
Chapter 9, Reserved Words and Builtin Commands, looks at a number of commands that are built into the shell itself.
Chapter 10, Writing Bug-Free Scripts and Debugging the Rest, takes a buggy script and takes you step-by-step through fixing it, as well as showing you how to prevent bugs in the first place.
Chapter 11, Programming for the Command Line, is for those people who spend a lot of time at the command prompt. These programs and functions reduce the typing to a minimum.
Chapter 12, Runtime configuration, describes seven methods of altering a program's runtime behavior.
Chapter 13, Data Processing, deals with manipulating different types of data in the shell.
Chapter 14, Scripting the Screen, shows you how to write to the screen as if it were a canvas rather than a teletypewriter.
Chapter 15, Entry-Level Programming, presents techniques for getting input from a user, both using single keypresses and entering and editing a line of text. It concludes with routines for using the mouse in shell scripts.
The appendix lists all the variables used by bash with a description of each.
Downloading the Code
The source code for this book is available to readers as a gzipped tarball file at www.apress.com in the Downloads section of this book's home page.
Please feel free to visit the Apress web site and download all the code there. You can also check for errata and find related titles from Apress.
Contacting the Author
The author maintains a web site at http://cfajohnson.com/ and can be reached via e-mail at shell@cfajohnson.com.
