Regex : Tutorial

Regular expressions, or regex, are one of the best things about Perl. So good infact, that they been copied by most other programming languages.

They are sadly let down however by Perl "man pages" and the chapter in the 'Programming Perl' camel book which in no way does them justice. It explains them, but only in a way a Computer Science PHD student would understand.

So, here's my tutorial with lots of examples.

NB 1: this is based on Perl, but you can use them in Javascript, Python etc.

NB 2: once you understand this, you should be ready for the reference material

Regex : Using a regular expression (perl specific).

There are 3 ways to call a regex:

1. Use: $x =~ //
2. Use: $x =~ m//
   or m## or m"" or m{} or anything else you care to choose
3. Use: qr()
   or qr## or qr"" or qr{}

Why so many ways. First, that's Perl. But it helps with escaping, see examples below. Just choose the delimiters that don't appear inside the rege.

The usual choices are // () ## and ""

$x =~ /andrew/;
$x =~ m/andrew/;
$x =~ m#andrew#;
$x =~ qr(andrew);

#matches speachmarks
$x =~ m/"andrew"/;

#matches HTML tags - there's no need to escape the / in </b>
$x =~ qr# <b>andrew</b> #;

Regex : Using whitepace and comments

The /x modifier

Using whitespace and comments inside your regular expression is a brilliant idea, especially as regex often look like line noise, even to yourself a week later..

If you need a space, you'll need to escape it as \ (that \ then a SPACE ) or \s (which means any whitespace character)

$x =~ / andrew /ix

# these last 2 are the same
$x =~ m# <b> \s* andrew \s* </b> #ix;

$x =~ qr"                 # " is the delimiter to start and end the regex
      <b>
        \s* andrew \s*  # allow optional whitespace (0+ chars) at start and end
      </b>
   "ix;                 # ignore case

Regex : Some simple examples

Some simple examples to start

$x = 'Andrew';

print 'match' if $x =~ /rew/;
match

print 'match' if $x =~ /And/;
match

# no match as and doesn't match And
print 'false' if $x =~ /and/;

print 'true' if $x =~ /and/i; # the 'i' is ignore case
match

Regex : How many times to match?

Place these modifiers after a character:

• + matches 1+ times
  
• * matches 0+ times
  
• {3} after a charater matches it 3 times
• {2,3} after a charater matches it 2 to 3 times

Greedy and Not Greedy

$x = "
 one 
 
 two 
";

#.* is greedy, so this matches from the first 
 to the last 

$x =~ m# 
 .* 
 #x

#.*? isn't greedy, so this matches 
 one  only

$x =~ m# 
 .*? 
 #x

Regex : Predefined Groups of characters

Groups of characters

• /s matches a whitespace character
• /S matches anything except a whitespace character
• /w a word charaters (a-z, A-Z, 0-9, Persian, Greek, Chinese alphabets, etc)
• /W anything except a word character
• /d matches 0..9
• /D matches anything EXCEPT 0..9

Regex : Groups

Use ( ) for groups.

$html =~ (andrew) 
# matches andrew

((andrew)|(john)|(peter)) 
# matches andrew OR john or peter

# the same, but with whhitespace
# m# - use # as the delimiter
# #x - allow whitespace
$html =~ m#
 (
   ( andrew   )  |
   ( peter    )  |
   ( john     )
 )#x;

Perl Complex Variables

Perl has 3 types, scalar, array and hash. These can be combined to make complex variables.

Unfortunately, the syntax isn't kind.

Defining Scalars, Arrays and Hashes.

$name = 'andrew' ;

# these 2 are the same, qw() is 'syntax sugar'
@names = qw( andrew john peter );
@names = ('andrew', 'john', 'peter');

%age = (
 'andrew' => 20,
 'peter'  => 30,
 'paul'   => 17,
 );

# note the outside hash uses (
# - but the inside hash uses {

%names = (
 'andrew' => { 
      'age'    => 20, 
      'height' => 1.76,
      },
 'peter'  => { 
      'age'    => 30, 
      'height' => 1.2, 
      'friends'=> ['andrew', 'paul'], 
      }
 );

Scalars

$x =  $names{'andrew'}{'age'};
# now $x = 20 

$names{'andrew'}{'age'} = 35 ;

Arrays

Arrays are a bit more complex - use @{ } to access them and [ ] to set them

@x = @{ $names{'peter'}{'friends'} } ;

$names{'peter'}{'friends'} = [ 'bill', 'john', 'fred' ] ;

Hashes

Hashes are similar, use %{ } to access them and { } to set them

%hash = %{ $names{'andrew'} } ;

$names{'andrew'} = { 'age' => 20, 'height' => 1.76 } ;

Important Gotcha

When using the %hash = %{} and @array = @{} syntax, you are making a reference to the data, NOT a copy of it;

%hash = %{ $names{'andrew'} } ;
# %hash is a reference, change %hash and you change %names, and visa versa

%new_hash = %hash;
# now safe

Regex : Doesn't match


# $x does not contain andrew
$x !~ /andrew/


Regex : Match on word Boundary

\b match on a word boundary

\B matches on not a word boundary

Regex : . and the /s modifier

A . matches anything except a newline

To match a newline, use the /s modifer (treat as a Single line)


$x = qq(
andrew
was
here
);

$x =~ /andrew .* here/xs;
# matches as the /s carries the search over the newlines


Regex : Anchors and the /m modifier

Anchors
^ matches at the start of a string
$ matches at the end of a string

$x = /andrew/

$x =~ /^a/ ;
# matches as andrew starts with an a

$x =~ /rew$/;
# matches as andrew ends with a rew

But what if you have a string containing several lines (e.g. a file with \n characters) ?

The /m modifier.

This treats the string as multiple lines, so ^ and $ will match the start and end of each line in the file

$x = qq(
andrew
john andrew
andrew
);
$x =~ s/^andrew/ANDREW/mg;
# will replace 1st and 3rd andrew
# without the /m, it would only replace the third
This treats the string as a single line, so

Regex : Global Replace

The /g modifier

This repeats the regex until there are no more matches

$x = /andrew andrew andrew/;

$x = s/a/A/;
# Andrew andrew andrew (1 replace)

$x = s/a/A/g;
# Andrew Andrew Andrew (all the 'a' get replaces)

Regex : Using expressions

This is the /e modifier

Example: make HTML tags upper case

# ( ) - capture the text
# .*? - match the least amount of text, i.e. an HTML tag
# uc ($1) - make the matches text uppercase
# gxe - g is global replace
# gxe - x is allow whitespace and comments
# gxe - the bit on the right is an expressions to be evaluates

$html =~ s# ( < .*? > ) # print uc($1) #gxe ;

Example : find hi byte characters

# find hi-byte characters in HTML
# - and keep a record of all the hi byte chars found in %found
my %found ;

sub find {
 my ($char) = @_ ;
 $found{ $chars } ++; ## keep a record
 return '[' . ord( $chars ) . ']';
 }

## this will find euro and £ symbols, but not €
## \x80 is hex (ascii char 128)
## \x80-\xffff is a range - if the file is utf8, it will match hi bytes chars as well
## gxe - expression, whitespace, and global replace
$html =~ s# [\x80-\xffff] # &find( $1 ) #gxe;

Regex : Substitutions

Regex : Capture Matched text

Use brackets to capture text, and $1, $2 ... to refer to it


$x = 'andrew john';

# swap the first and second word around

$x=~ s/ (\w+) \s+ (\w+*)
/ $2 $1
/ x;

# s/ - substitute
# (\w+) - 1+ word charcters, captures as $1
# \s+ - 1+ whitespace characters
# (\w+) - 1+ word chars, captured as $2
# /x - use whitesapce and comments

# $1 and $2 survive
print $2;
# prints john


Regex: Ranges

Use [ ] for ranges - match any 1 char inside the range

$x =~ /[0-9]/;
$x =~ /[a-z]/i; # so match a..z and A..Z as /i is ignore case
$x =~ /[\n\r]/;

#match 1+ chars in the range 0..9 + - and .
#\. escape the dot so it doesn't mean "any 1 char"
$x=~ /[0-9+-\.]+/


Perl Poetry

This isn't mine, but its quite cool.

$_ = reverse sort qw p ekca lre uJ reh ts
p, $/.r, map $_.$", qw e p h tona e; print

Recommended PC Shareware

On my new Windows Vista PC I have:

Google Pack (which includes)

• Firefox (web browser)
• Picasa (organises photos)
• Adobe PDF Viewer
• Spyware Doctor (anti virus)
• Google Earth

Comodo (firewall) NB: Online Armour is better for Windows XP

Ubuntu (so the PC can dual boot in Linux)

Activestate Perl (a programming language)

OpenOffice shareware version of MS Word, MS Excel etc)

Notepad++ (an editor that believes in Perl)

The National Trust in Southeast England

The National Trust is a charity that own many historic buildings and gardens in the UK. This is a review of its properties in the southeast

Many NT places are country houses set in grounds, or in rural areas that are nice for a walk, so its best to visit them on pleasent days. Very few have public transport, so you really need a car to visit most of them.

Their pricing policy makes it expensive to visit an individual property, but the annual membership is very good value. Join via the website to get 25% off on your first year. Note that they send out your next's year membership pack after 9 months or so to 'force' you into renewing.

Kent (southeast of London)

• Knowle Palace: 4/5
  Large Bishop's palace set in a massive deer park. One of the biggies. Can walk from Sevenoakes train station. Nice tea room. Outside is stunning. Interior is a dimly lit (conservation) museum of very old furniture and faded portraits. The landscaped deer park is beautiful. Recommended.
  
  
• To Do
  Chartwell, Emmetts, Igtham Mote, Quebec House, Scotney castle, Sissinghurst, Smallhythe

Sussex (south of London)

• Nymans Gardens : 5/5
  A large, beautiful, word-class garden with a ruined house set on a hill. One of the biggies (for gardeners). Hundreds of different plants, shrubs and trees. South of London, just off the M23, between Gatwick Airport and Brighton. Waymarked walks down in the nearby forest. OK tearoom. Best when the garden is (spring and summer). Go there!
  
  
• Petworth : 5/5
  Very large house, almost a palace, containing a large selection of paintings and sculpture, set in a massive landscaped park. One of the bigges. Most of the ground floor is open (and some family rooms on weekdays only). Great for a walk or picnic. Go there!



• Uppark : 2/5
  Great looking house, and a great location on a quiet part of the South Downs. However the interior is very "cold", like an unlived in museum. Visit for a walk or picnic. Little information about the people that lived in it would have made for a much more interesting visit. No public transport. A long way southwest of London, near Portsmouth. Poor tearoom would be amazing if you could sit outside in summer, so best on a nice day.

Surrey

• Clandon Park and Hatchlands : 4/5
  2 country houses close to each other in parkland. Takes about 3 hours with a walk in the parkland
  
  
• Poleseden Lacy : 4/5
  Large country house overlooking a secluded valley. Formal grassed terrace with a view. Waymarked walks in the nearby forest. South of London, just outside the M25. OK tearoom (its being relocated). Recommended.
  
  
• Winkworth Arboretum : 3/5
  Pretty trees on a hillside

Hampshire (Southwest of London)

• To Do
  Hinton Ampner, Mottisfont Abbey, The Vine

Berkshire (west of London)

• Basildon Park : 2/10
  Nice, medium sized house in a park, close to the Thames. Little about the family. Takes 1 hour.
  
  
• To Do
  Ashdown House

Oxfordshire (west of London)

• To Do
  Buscot Park, Chastleton House, Greys Court

Buckinghamshire (northwest of London)

• Claydon House : 5/5
  Beautiful, medium sized house, remote location, near Stowe. Very homely. Highly recommended. Nice tearoom. Takes 1-2 hours.
  
  
• Stowe : 3/5
  A large park with many follies. One of the biggies. I was really looking forward to this place as I like follies, but it was a bit of an anticlimax. Takes about 2-3 hours to visit as you walk around. NB the house belongs to the school, and isn't NT.
  
  
• To Do
  Hughenden Manor, Waddesdon Park, West Wycombe Manor, Cliveden, Ascott