FrontPage > YahooPipes > YahooPipesRegex

Regular Expressions in Yahoo Pipes

• official Yahoo Pipes RegEx documentation

The basics

The RegEx? module is one of the most powerful modules in Yahoo Pipes. You can do all kind of data transformations with it. This wiki page here would like to give you a short overview.

Please note: Like in the Yahoo Pipes discussions, I put RegEx? patterns within square brackets. That way, you can distinguish for example [] and [ ] easily. Please omit the square brackets unless noted otherwise.

The Modifiers

You might have noticed four checkboxes next to each RegEx? line. Those are used for modifying the way the RegEx? behaves and succeeded the so called "embedded pattern-match modifiers". But they did not completely replace them.

In fact, you can use the modifiers "I, M, S and X" in embedded notation, while the checkboxes offer the options "I, M, S and G". So there's no X for checkboxes while there's no G for the embedded notification.

The answer is taken from the Yahoo Pipes Discussions and updated with information from RegEx documentation started.

What they do

• g allow global matches. set=match every occurence; unset=match only first occurence.
• i be case insensitive. set='A' equals 'a'; unset 'A' and 'a' are treated differently
• m treat string as multiple lines. set='^' matches every start of string after a \n and/or \r . unset='^' matches only the very first character in the string.
• s allow '.' to match new lines as well. set='.' matches '\n'. unset='.' does not match '\n'.
• x allow white spaces and comments within an expression.

Embedded Notation

Hapdaniel from the Yahoo Pipes Discussions points out, the original form of specifying those flags is the "embedded notation". If you prefix your RegEx? with a (?x), you'll set the X-modificator. You cannot set a (?g) that way, though.

Checkbox Notation

To activate one of the checkbox-flags, just tick it. You can tick as many flags as you like. Except the X-flag, which apparently is not available as checkbox.

Common patterns

Matching empty

What, if you want to match "nothing"? Hapdaniel has the solution:

• [^(?!.)]

Matching not empty

And here's the opposite, again from the suggestion thread.

• [^(?=.)]

Removing whitespace

Sometimes, you'd like to remove all the linefeeds and unwanted spaces out of a field. I usually use a three- to fourfold approach to that. For each of the following replacements, use +g (the global flag)

1. replace [\n] (line feed) with [ ]
2. replace [\r] (carriage return) with [ ]
3. (as needed) replace all [<br />] (html break) with [ ]
4. replace [\s+] (all whitespace occurrences) with [ ]

With 1 and 2, you remove all hard linefeeds. With 3, you remove all "logical" linefeeds (the ones that only get rendered, when the field is interpreted as html). with 4, you make the result more compact. If for example you have 3 or more spaces in a row, those will be reduced to just one space.

Using reserved characters

In RegEx?, some characters are "reserved". That means, they are not used literally, but instead used as functions. Examples:

• [.] -- one arbitrary character. if +s flag is set, this includes the new-line character (\n). if +s flag is unset, the dot does not include the new-line character.
• [\d] -- one digit. (0..9)
• [\n] -- new line, like in C
• [\r] -- carriage return, like in C
• [\s] -- one space character. Includes ' ' and tabs (\t)
• [^] -- beginning of string. If +m flag is set, this matches every start of a line. a line is then defined as something at the very start of the string or something after a new line ('\n'). If +m flag is unset, this matches only the very first character of the string.
• [$] -- end of string. If +m flag is set, this matches every end of a line. if +m flag is unset, this matches only the very last character of the string.
• [()] -- groups. You can use the groups matched in the replacement field. For example replace [(\d)] with [0$1] results in a leading zero added.
• [[]] -- character groups. For example, [123] matches 1, 2 or 3.
• [!\d] -- combination. ! means not, \d means digit. So one character, being everything but a digit, is matched here
• [\d*] -- '*' means: 0 to n matches. This would match no or up to infinite digits.
• [\d+] -- '+' means: 1 to n matches. At least one. This would match one or more digits.

To "escape" reserved characters, that is to match them literally, you put a backslash in front. For example, matching (twitter) is possible by using \(twitter\).

Removing html tags

From a post in the Yahoo Pipes Discussion.

• [<[^>]*>] - please note that this translates to something like <[^>]*> . matches every term that's within <>.
• [<.*?>] - similar to the first statement, but "lazy match". Not as efficient.

Showing Images

From a post in the Yahoo Pipes Discussion. Sometimes, one of your field contains just an image URL. You'd like to replace that URL with an image tag, so it is rendered as an image.

• Replace [(.*)] with [<img src="$1" />]

Prefixing something

Sometimes, you'd like to add something in front of a field. For example, to add a "Yahoo: " in front of every title, you could

• Replace [(^)] with [Yahoo: $1]

$1 matches the first group used (we have only one group in this example). And ^ matches the beginning of the expression.

Source: documentation started

Postfixing something

And to suffix something, you'd use a $ instead of the ^.

• Replace [($)] with [Yahoo: $1]

Source: RegEx documentation started

Translating dates

What, if you want to change a date of format mm/dd/yy to the ISO equivalent of yyyy-mm-dd ? You could use an expression like this one:

• Replace [(\d\d)\/(\d\d)\/(\d\d)] with [20$3-$1-$2]

Here, we have three groups. In the result, I also prefix a "20" as the year was specified only with two digits.

Convert to Uppercase

Also from a Yahoo Pipes Discussion. You can use the \U flag to convert something to uppercase. For example

• replace [(.*)] with [\U$1]

Convert to Lowercase

No surprise here, you can use the \L flag to convert something to lowercase

• replace [(.*)] with [\L$1]

Last modified 2008-06-28 13:46 by rck