Encountering a situation where your regular expression pattern doesn’t match anywhere in a string can be incredibly frustrating for developers and data analysts alike. It’s a common stumbling block, transforming what seems like a straightforward task into a head-scratching debugging session. Whether you’re trying to extract specific data, validate user input, or perform complex search-and-replace operations, a non-matching regex can halt your progress. This isn’t just about a simple typo; often, the mismatch stems from subtle nuances in regex syntax, unexpected data formats, or a misunderstanding of how the regex engine interprets your pattern. Fortunately, a systematic approach to debugging can uncover the root cause and get your patterns working as intended. Let’s delve into effective strategies to diagnose and fix these elusive regex issues.
Understanding Common Regex Pitfalls and Syntax Errors
One of the primary reasons a regular expression pattern doesn’t match anywhere in a string is often a fundamental misunderstanding or a subtle error in its syntax. Regular expressions are powerful but highly sensitive to every character. A single misplaced backslash, a forgotten quantifier, or an incorrect character class can completely derail your matching efforts. For instance, the dot . character matches any character except a newline by default. If you intend to match an actual literal dot, you must escape it as \.. Failing to do so means your pattern will match any character at that position, which might not be what you intended, or it might match something you didn’t expect, leading to a “no match” when you expect a specific literal match.
Case sensitivity is another frequent culprit. By default, most regex engines are case-sensitive. If your pattern uses lowercase letters like hello but your string contains Hello, it won’t match. Many regex implementations offer a case-insensitive flag (often denoted as i) to overcome this. Similarly, understanding the difference between character sets like [abc] (matches ‘a’, ‘b’, or ‘c’) and groups like (abc) (matches the literal sequence ‘abc’) is crucial. Misusing these constructs can lead to your pattern failing to match the expected substrings. Always double-check your escapes, character classes, and flags when your regex pattern doesn’t match anywhere in a string, as these are foundational elements of effective pattern matching.
Systematic Debugging Strategies for Non-Matching Patterns
When you face a situation where your regex pattern doesn’t match anywhere in a string, a systematic debugging approach is your best friend. Instead of guessing, breaking down the problem into smaller, manageable parts can quickly pinpoint the error. Start by simplifying your regex. Remove all but the most essential parts of your pattern and test it against a known sample of your target string. If that basic pattern matches, gradually add more complexity, one component at a time, testing after each addition. This incremental approach helps you identify exactly which part of your pattern is causing the issue.
Utilizing online regex testing tools is an indispensable step in this process. Platforms like Regex101 or Rubular provide real-time feedback, explaining each component of your regex and highlighting how it interacts with your test string. They often show you exactly what matches, what doesn’t, and why, including detailed breakdowns of backtracking steps. This visual and analytical feedback is invaluable for understanding complex patterns and quickly identifying mistakes that might be invisible in your code editor. As experts in pattern matching often advise, “Never debug a complex regex in your head; always use a tool.”
Step-by-Step Regex Troubleshooting Guide:
- Isolate the Problem: Test your regex with a very simple, known-to-match string first.
- Simplify and Build: Start with the smallest possible working part of your pattern and incrementally add more components, testing after each addition.
- Use a Regex Tester: Paste your regex and test string into an online regex debugger. Pay attention to the explanation panel and match highlighting.
- Check Escapes: Ensure all special characters (
., ``,+,?,|,(,),[,{,\,^,$) are properly escaped with a backslash if you intend to match them literally. - Verify Case Sensitivity: If your pattern isn’t matching due to capitalization, consider using the case-insensitive flag (e.g.,
/pattern/iin JavaScript). - Inspect Quantifiers: Ensure quantifiers (``,
+,?,{n,m}) are applied correctly and are not making parts of your pattern optional when they should be mandatory, or vice-versa. - Review Anchors: Understand if
^and$are anchoring your match to the start/end of the string/line, which might prevent matches in the middle.
The Role of Anchors, Quantifiers, and Character Classes
A frequent reason a regular expression pattern doesn’t match anywhere in a string is the incorrect application or misunderstanding of anchors, quantifiers, and character classes. Anchors, such as ^ (start of string/line) and $ (end of string/line), constrain where a match can occur. If you use ^word$, it will only match the string “word” and nothing else. If your target string is “My word is bond”, this pattern will not match because “word” is not at the start and end of the string simultaneously. Removing the anchors ^ and $ would allow word to match within a larger string. Similarly, the word boundary anchor \b matches the position between a word character and a non-word character, which can be crucial for precise word matching.
Quantifiers (``, +, ?, {<b>Question & Answer : </b><br></br><p>I am trying to match <input> of type hidden fields using this pattern:</p> <pre>/<input type="hidden" name="([^"]*?)" value="([^"]*?)" />/ </pre> <p>This is some sample form data:</p> <pre><input type="hidden" name="SaveRequired" value="False" /><input type="hidden" name="__VIEWSTATE1" value="1H4sIAAtzrkX7QfL5VEGj6nGi+nP" /><input type="hidden" name="__VIEWSTATE2" value="0351118MK" /><input type="hidden" name="__VIEWSTATE3" value="ZVVV91yjY" /><input type="hidden" name="__VIEWSTATE0" value="3" /><input type="hidden" name="__VIEWSTATE" value="" /><input type="hidden" name="__VIEWSTATE" value="" /> </pre> <p>But I am not sure that the type, name, and value attributes will always appear in the same order. If the type attribute comes last, the match will fail because in my pattern itās at the start.</p> <p><strong>Question:</strong><br></br> How can I change my pattern so it will match regardless of the positions of the attributes in the <input> tag?</p> <p><strong>P.S.:</strong> By the way I am using the <em>Adobe Air</em> based <a href="http://gskinner.com/RegExr/desktop/" rel="nofollow noreferrer" title="RegEx Desktop tool"><em>RegEx Desktop Tool</em></a> for testing regular expressions.</p><br></br><h2><em>Oh Yes You Can</em> Use Regexes to Parse HTML!</h2> <p>For the task you are attempting, regexes are <strong><em>perfectly fine!</em></strong></p> <p>It <em>is</em> true that most people underestimate the difficulty of parsing HTML with regular expressions and therefore do so poorly.</p> <p>But this is not some fundamental flaw related to computational theory. <a href="https://stackoverflow.com/a/1732454/459233">That silliness is parroted a lot around here</a>, but donāt you believe them.</p> <p>So while it certainly can be done (this posting serves as an existence proof of this incontrovertible fact), that doesnāt mean itĀ <strong><em>should</em></strong>Ā be.</p> <p>You must decide for yourself whether youāre up to the task of writing what amounts to a dedicated, special-purpose HTML parser out of regexes. Most people are not.</p> <p>But <em>I</em> am. ā»</p> <hr></hr> <h2>General Regex-Based HTML Parsing Solutions</h2> <p>First Iāll show how easy it is to parse <strong>arbitrary</strong> HTML with regexes. The full programās at the end of this posting, but the heart of the parser is:</p> <pre class="lang-perl prettyprint-override">for (;;) { given ($html) { last when (pos || 0) >= length; printf "\@%d=", (pos || 0); print "doctype " when / \G (?&doctype) $RX_SUBS /xgc; print "cdata " when / \G (?&cdata) $RX_SUBS /xgc; print "xml " when / \G (?&xml) $RX_SUBS /xgc; print "xhook " when / \G (?&xhook) $RX_SUBS /xgc; print "script " when / \G (?&script) $RX_SUBS /xgc; print "style " when / \G (?&style) $RX_SUBS /xgc; print "comment " when / \G (?&comment) $RX_SUBS /xgc; print "tag " when / \G (?&tag) $RX_SUBS /xgc; print "untag " when / \G (?&untag) $RX_SUBS /xgc; print "nasty " when / \G (?&nasty) $RX_SUBS /xgc; print "text " when / \G (?&nontag) $RX_SUBS /xgc; default { die "UNCLASSIFIED: " . substr($_, pos || 0, (length > 65) ? 65 : length); } } } </pre> <p>See how <em>easy</em> that is to read?</p> <p>As written, it identifies each piece of HTML and tells where it found that piece. You could easily modify it to do whatever else you want with any given type of piece, or for more particular types than these.</p> <p>I have no failing test cases (left :): Iāve successfully run this code on more than 100,000 HTML files ā every single one I could quickly and easily get my hands on. Beyond those, Iāve also run it on files <em>specifically constructed</em> to break naĆÆve parsers.</p> <p>This is <em>not</em> a naĆÆve parser.</p> <p>Oh, Iām sure it isnāt perfect, but I havenāt managed to break it yet. I figure that even if something did, the fix would be easy to fit in because of the programās clear structure. Even regex-heavy programs should have stucture.</p> <p>Now that thatās out of the way, let me address the OPās question.</p> <h2>Demo of Solving the OPās Task Using Regexes</h2> <p>The little html_input_rx program I include below produces the following output, so that you can see that parsing HTML with regexes works just fine for what you wish to do:</p> <pre>% html_input_rx Amazon.com-_Online_Shopping_for_Electronics,_Apparel,_Computers,_Books,_DVDs_\&_more.htm input tag #1 at character 9955: class => "searchSelect" id => "twotabsearchtextbox" name => "field-keywords" size => "50" style => "width:100%; background-color: #FFF;" title => "Search for" type => "text" value => "" input tag #2 at character 10335: alt => "Go" src => "http://g-ecx.images-amazon.com/images/G/01/x-locale/common/transparent-pixel._V192234675_.gif" type => "image" </pre> <h2><em>Parse Input Tags, See No Evil Input</em></h2> <p>Hereās the source for the program that produced the output above.</p> <pre class="lang-perl prettyprint-override">#!/usr/bin/env perl # # html_input_rx - pull out all <input> tags from (X)HTML src # via simple regex processing # # Tom Christiansen <<a class="__cf_email__" data-cfemail="91e5f2f9e3f8e2e5d1e1f4e3fdbff2fefc" href="/cdn-cgi/l/email-protection">[emailĀ protected]</a>> # Sat Nov 20 10:17:31 MST 2010 # ################################################################ use 5.012; use strict; use autodie; use warnings FATAL => "all"; use subs qw{ see_no_evil parse_input_tags input descape dequote load_patterns }; use open ":std", IN => ":bytes", OUT => ":utf8"; use Encode qw< encode decode >; ########################################################### parse_input_tags see_no_evil input ########################################################### until eof(); sub parse_input_tags { my $_ = shift(); our($Input_Tag_Rx, $Pull_Attr_Rx); my $count = 0; while (/$Input_Tag_Rx/pig) { my $input_tag = $+{TAG}; my $place = pos() - length ${^MATCH}; printf "input tag #%d at character %d:\n", ++$count, $place; my %attr = (); while ($input_tag =~ /$Pull_Attr_Rx/g) { my ($name, $value) = @+{ qw< NAME VALUE > }; $value = dequote($value); if (exists $attr{$name}) { printf "Discarding dup attr value '%s' on %s attr\n", $attr{$name} // "<undef>", $name; } $attr{$name} = $value; } for my $name (sort keys %attr) { printf " %10s => ", $name; my $value = descape $attr{$name}; my @Q; given ($value) { @Q = qw[ " " ] when !/'/ && !/"/; @Q = qw[ " " ] when /'/ && !/"/; @Q = qw[ ' ' ] when !/'/ && /"/; @Q = qw[ q( ) ] when /'/ && /"/; default { die "NOTREACHED" } } say $Q[0], $value, $Q[1]; } print "\n"; } } sub dequote { my $_ = $_[0]; s{ (?<quote> ["'] ) (?<BODY> (?s: (?! \k<quote> ) . ) * ) \k<quote> }{$+{BODY}}six; return $_; } sub descape { my $string = $_[0]; for my $_ ($string) { s{ (?<! % ) % ( \p{Hex_Digit} {2} ) }{ chr hex $1; }gsex; s{ & \043 ( [0-9]+ ) (?: ; | (?= [^0-9] ) ) }{ chr $1; }gsex; s{ & \043 x ( \p{ASCII_HexDigit} + ) (?: ; | (?= \P{ASCII_HexDigit} ) ) }{ chr hex $1; }gsex; } return $string; } sub input { our ($RX_SUBS, $Meta_Tag_Rx); my $_ = do { local $/; <> }; my $encoding = "iso-8859-1"; # web default; wish we had the HTTP headers :( while (/$Meta_Tag_Rx/gi) { my $meta = $+{META}; next unless $meta =~ m{ $RX_SUBS (?= http-equiv ) (?&name) (?&equals) (?= (?"e)? content-type ) (?&value) }six; next unless $meta =~ m{ $RX_SUBS (?= content ) (?&name) (?&equals) (?<CONTENT> (?&value) ) }six; next unless $+{CONTENT} =~ m{ $RX_SUBS (?= charset ) (?&name) (?&equals) (?<CHARSET> (?&value) ) }six; if (lc $encoding ne lc $+{CHARSET}) { say "[RESETTING ENCODING $encoding => $+{CHARSET}]"; $encoding = $+{CHARSET}; } } return decode($encoding, $_); } sub see_no_evil { my $_ = shift(); s{ <! DOCTYPE .*? > }{}sx; s{ <! \[ CDATA \[ .*? \]\] > }{}gsx; s{ <script> .*? </script> }{}gsix; s{ \<!-- .*? --> }{}gsx; return $_; } sub load_patterns { our $RX_SUBS = qr{ (?(DEFINE) (?<nv_pair> (?&name) (?&equals) (?&value) ) (?<name> \b (?= \pL ) [\w\-] + (?<= \pL ) \b ) (?<equals> (?&might_white) = (?&might_white) ) (?<value> (?"ed_value) | (?&unquoted_value) ) (?<unwhite_chunk> (?: (?! > ) \S ) + ) (?<unquoted_value> [\w\-] * ) (?<might_white> \s * ) (?<quoted_value> (?<quote> ["'] ) (?: (?! \k<quote> ) . ) * \k<quote> ) (?<start_tag> < (?&might_white) ) (?<end_tag> (?&might_white) (?: (?&html_end_tag) | (?&xhtml_end_tag) ) ) (?<html_end_tag> > ) (?<xhtml_end_tag> / > ) ) }six; our $Meta_Tag_Rx = qr{ $RX_SUBS (?<META> (?&start_tag) meta \b (?: (?&might_white) (?&nv_pair) ) + (?&end_tag) ) }six; our $Pull_Attr_Rx = qr{ $RX_SUBS (?<NAME> (?&name) ) (?&equals) (?<VALUE> (?&value) ) }six; our $Input_Tag_Rx = qr{ $RX_SUBS (?<TAG> (?&input_tag) ) (?(DEFINE) (?<input_tag> (?&start_tag) input (?&might_white) (?&attributes) (?&might_white) (?&end_tag) ) (?<attributes> (?: (?&might_white) (?&one_attribute) ) * ) (?<one_attribute> \b (?&legal_attribute) (?&might_white) = (?&might_white) (?: (?"ed_value) | (?&unquoted_value) ) ) (?<legal_attribute> (?: (?&optional_attribute) | (?&standard_attribute) | (?&event_attribute) # for LEGAL parse only, comment out next line | (?&illegal_attribute) ) ) (?<illegal_attribute> (?&name) ) (?<required_attribute> (?#no required attributes) ) (?<optional_attribute> (?&permitted_attribute) | (?&deprecated_attribute) ) # NB: The white space in string literals # below DOES NOT COUNT! It's just # there for legibility. (?<permitted_attribute> accept | alt | bottom | check box | checked | disabled | file | hidden | image | max length | middle | name | password | radio | read only | reset | right | size | src | submit | text | top | type | value ) (?<deprecated_attribute> align ) (?<standard_attribute> access key | class | dir | ltr | id | lang | style | tab index | title | xml:lang ) (?<event_attribute> on blur | on change | on click | on dbl click | on focus | on mouse down | on mouse move | on mouse out | on mouse over | on mouse up | on key down | on key press | on key up | on select ) ) }six; } UNITCHECK { load_patterns(); } END { close(STDOUT) || die "can't close stdout: $!"; } </pre> <p>There you go! Nothing to it! :)</p> <p>Only <strong><em>you</em></strong> can judge whether your skill with regexes is up to any particular parsing task. Everyoneās level of skill is different, and every new task is different. For jobs where you have a well-defined input set, regexes are obviously the right choice, because it is trivial to put some together when you have a restricted subset of HTML to deal with. Even regex beginners should be handle those jobs with regexes. Anything else is overkill.</p> <p><strong>However</strong>, once the HTML starts becoming less nailed down, once it starts to ramify in ways you cannot predict but which are perfectly legal, once you have to match more different sorts of things or with more complex dependencies, you will eventually reach a point where you have to work harder to effect a solution that uses regexes than you would have to using a parsing class. Where that break-even point falls depends again on your own comfort level with regexes.</p> <h2>So What Should I Do?</h2> <p>Iām not going to tell you what you <em>must</em> do or what you <em>cannot</em> do. I think thatās Wrong. I just want to present you with possibilties, open your eyes a bit. You get to choose what you want to do and how you want to do it. There are no absolutes ā and nobody else knows your own situation as well as you yourself do. If something seems like itās too much work, well, maybe it is. Programming should be <strong><em>fun</em></strong>, you know. If it isnāt, you may be doing it wrong.</p> <p>One can look at my html_input_rx program in any number of valid ways. One such is that you indeed <em>can</em> parse HTML with regular expressions. But another is that it is much, much, much harder than almost anyone ever thinks it is. This can easily lead to the conclusion that my program is a testament to what you should <em>not</em> do, because it really is too hard.</p> <p>I wonāt disagree with that. Certainly if everything I do in my program doesnāt make sense to you after some study, then you should not be attempting to use regexes for this kind of task. For specific HTML, regexes are great, but for generic HTML, theyāre tantamount to madness. I use parsing classes all the time, especially if itās HTML I havenāt generated myself.</p> <h2>Regexes optimal for <em>small</em> HTML parsing problems, pessimal for large ones</h2> <p>Even if my program is taken as illustrative of why you should <strong>not</strong> use regexes for parsing general HTML ā which is OK, because I kinda meant for it to be that āŗ ā it still should be an eye-opener so more people break the terribly common and nasty, nasty habit of writing unreadable, unstructured, and unmaintainable patterns.</p> <p>Patterns do not have to be ugly, and they do not have to be hard. If you create ugly patterns, it is a reflection on you, not them.</p> <h2>Phenomenally Exquisite Regex Language</h2> <p>Iāve been asked to point out that my proferred solution to your problem has been written in Perl. Are you surprised? Did you not notice? Is this revelation a bombshell?</p> <p>It is true that not all other tools and programming languages are quite as convenient, expressive, and powerful when it comes to regexes as Perl is. Thereās a big spectrum out there, with some being more suitable than others. In general, the languages that have expressed regexes as part of the core language instead of as a library are easier to work with. Iāve done nothing with regexes that you couldnāt do in, say, PCRE, although you would structure the program differently if you were using C.</p> <p>Eventually other languages will be catch up with where Perl is now in terms of regexes. I say this because back when Perl started, nobody else had anything like Perlās regexes. Say anything you like, but this is where Perl clearly won: everybody copied Perlās regexes albeit at varying stages of their development. Perl pioneered almost (not quite all, but almost) everything that you have come to rely on in modern patterns today, no matter what tool or language you use. So eventually the others <em>will</em> catch up.</p> <p>But theyāll only catch up to where Perl was sometime in the past, just as it is now. Everything advances. In regexes if nothing else, where Perl leads, others follow. Where will Perl be once everybody else finally catches up to where Perl is now? I have no idea, but I know we too will have moved. Probably weāll be closer to <a href="http://perlcabal.org/syn/S05.html" rel="noreferrer">Perlāās style of crafting patterns</a>.</p> <p>If you like that kind of thing but would like to use it in Perlā
, you might be interested in <a href="http://search.cpan.org/search?query=regexp+grammars&mode=module" rel="noreferrer">Damian Conwayās <strong>wonderful</strong> Regexp::Grammars</a> module. Itās completely awesome, and makes what Iāve done here in my program seem just as primitive as mine makes the patterns that people cram together without whitespace or alphabetic identifiers. Check it out!</p> <hr></hr> <h2>Simple HTML Chunker</h2> <p>Here is the complete source to the parser I showed the centerpiece from at the beginning of this posting.</p> <p>I am <em>not</em> suggesting that you should use this over a rigorously tested parsing class. But I am tired of people pretending that nobody can parse HTML with regexes just because <em>they</em> canāt. You clearly can, and this program is proof of that assertion.</p> <p>Sure, it isnāt easy, but <strong>it <em>is</em> possible!</strong></p> <p>And trying to do so is a terrible waste of time, because good parsing classes exist which you <em>should</em> use for this task. The right answer to people trying to parse <em>arbitrary</em> HTML is <strong>not</strong> that it is impossible. That is a facile and disingenuous answer. The correct and honest answer is that they shouldnāt attempt it because it is too much of a bother to figure out from scratch; they should not break their back striving to reĆÆnvent a wheel that works perfectly well.</p> <p>On the other hand, HTML that falls <em>within a predicable subset</em> is ultra-easy to parse with regexes. Itās no wonder people try to use them, because for small problems, toy problems perhaps, nothing could be easier. Thatās why itās so important to distinguish the two tasks ā specific vs generic ā as these do not necessarily demand the same approach.</p> <p>I hope in the future here to see a more fair and honest treatment of questions about HTML and regexes.</p> <p>Hereās my HTML lexer. It doesnāt try to do a validating parse; it just identifies the lexical elements. You might think of it more as <strong>an HTML chunker</strong> than an HTML parser. It isnāt very forgiving of broken HTML, although it makes some very small allowances in that direction.</p> <p>Even if you never parse full HTML yourself (and why should you? itās a solved problem!), this program has lots of cool regex bits that I believe a lot of people can learn a lot from. Enjoy!</p> <pre class="lang-perl prettyprint-override">#!/usr/bin/env perl # # chunk_HTML - a regex-based HTML chunker # # Tom Christiansen <<a class="__cf_email__" data-cfemail="5420373c263d272014243126387a373b39" href="/cdn-cgi/l/email-protection">[emailĀ protected]</a> # Sun Nov 21 19:16:02 MST 2010 ######################################## use 5.012; use strict; use autodie; use warnings qw< FATAL all >; use open qw< IN :bytes OUT :utf8 :std >; MAIN: { $| = 1; lex_html(my $page = slurpy()); exit(); } ######################################################################## sub lex_html { our $RX_SUBS; ############### my $html = shift(); # Am I... # for (;;) { # forgiven? :)# given ($html) { ############### last when (pos || 0) >= length; printf "\@%d=", (pos || 0); print "doctype " when / \G (?&doctype) $RX_SUBS /xgc; print "cdata " when / \G (?&cdata) $RX_SUBS /xgc; print "xml " when / \G (?&xml) $RX_SUBS /xgc; print "xhook " when / \G (?&xhook) $RX_SUBS /xgc; print "script " when / \G (?&script) $RX_SUBS /xgc; print "style " when / \G (?&style) $RX_SUBS /xgc; print "comment " when / \G (?&comment) $RX_SUBS /xgc; print "tag " when / \G (?&tag) $RX_SUBS /xgc; print "untag " when / \G (?&untag) $RX_SUBS /xgc; print "nasty " when / \G (?&nasty) $RX_SUBS /xgc; print "text " when / \G (?&nontag) $RX_SUBS /xgc; default { die "UNCLASSIFIED: " . substr($_, pos || 0, (length > 65) ? 65 : length); } } } say "."; } ##################### # Return correctly decoded contents of next complete # file slurped in from the <ARGV> stream. # sub slurpy { our ($RX_SUBS, $Meta_Tag_Rx); my $_ = do { local $/; <ARGV> }; # read all input return unless length; use Encode qw< decode >; my $bom = ""; given ($_) { $bom = "UTF-32LE" when / ^ \xFf \xFe \0 \0 /x; # LE $bom = "UTF-32BE" when / ^ \0 \0 \xFe \xFf /x; # BE $bom = "UTF-16LE" when / ^ \xFf \xFe /x; # le $bom = "UTF-16BE" when / ^ \xFe \xFf /x; # be $bom = "UTF-8" when / ^ \xEF \xBB \xBF /x; # st00pid } if ($bom) { say "[BOM $bom]"; s/^...// if $bom eq "UTF-8"; # st00pid # Must use UTF-(16|32) w/o -[BL]E to strip BOM. $bom =~ s/-[LB]E//; return decode($bom, $_); # if BOM found, don't fall through to look # for embedded encoding spec } # Latin1 is web default if not otherwise specified. # No way to do this correctly if it was overridden # in the HTTP header, since we assume stream contains # HTML only, not also the HTTP header. my $encoding = "iso-8859-1"; while (/ (?&xml) $RX_SUBS /pgx) { my $xml = ${^MATCH}; next unless $xml =~ m{ $RX_SUBS (?= encoding ) (?&name) (?&equals) (?"e) ? (?<ENCODING> (?&value) ) }sx; if (lc $encoding ne lc $+{ENCODING}) { say "[XML ENCODING $encoding => $+{ENCODING}]"; $encoding = $+{ENCODING}; } } while (/$Meta_Tag_Rx/gi) { my $meta = $+{META}; next unless $meta =~ m{ $RX_SUBS (?= http-equiv ) (?&name) (?&equals) (?= (?"e)? content-type ) (?&value) }six; next unless $meta =~ m{ $RX_SUBS (?= content ) (?&name) (?&equals) (?<CONTENT> (?&value) ) }six; next unless $+{CONTENT} =~ m{ $RX_SUBS (?= charset ) (?&name) (?&equals) (?<CHARSET> (?&value) ) }six; if (lc $encoding ne lc $+{CHARSET}) { say "[HTTP-EQUIV ENCODING $encoding => $+{CHARSET}]"; $encoding = $+{CHARSET}; } } return decode($encoding, $_); } ######################################################################## # Make sure to this function is called # as soon as source unit has been compiled. UNITCHECK { load_rxsubs() } # useful regex subroutines for HTML parsing sub load_rxsubs { our $RX_SUBS = qr{ (?(DEFINE) (?<WS> \s * ) (?<any_nv_pair> (?&name) (?&equals) (?&value) ) (?<name> \b (?= \pL ) [\w:\-] + \b ) (?<equals> (?&WS) = (?&WS) ) (?<value> (?"ed_value) | (?&unquoted_value) ) (?<unwhite_chunk> (?: (?! > ) \S ) + ) (?<unquoted_value> [\w:\-] * ) (?<any_quote> ["'] ) (?<quoted_value> (?<quote> (?&any_quote) ) (?: (?! \k<quote> ) . ) * \k<quote> ) (?<start_tag> < (?&WS) ) (?<html_end_tag> > ) (?<xhtml_end_tag> / > ) (?<end_tag> (?&WS) (?: (?&html_end_tag) | (?&xhtml_end_tag) ) ) (?<tag> (?&start_tag) (?&name) (?: (?&WS) (?&any_nv_pair) ) * (?&end_tag) ) (?<untag> </ (?&name) > ) # starts like a tag, but has screwed up quotes inside it (?<nasty> (?&start_tag) (?&name) .*? (?&end_tag) ) (?<nontag> [^<] + ) (?<string> (?"ed_value) ) (?<word> (?&name) ) (?<doctype> ]* > ) (?<cdata> <!\[CDATA\[ .*? \]\] > ) (?<script> (?= <script ) (?&tag) .*? </script> ) (?<style> (?= <style ) (?&tag) .*? </style> ) (?<comment> <!-- .*? --> ) (?<xml> < \? xml (?: (?&WS) (?&any_nv_pair) ) * (?&WS) \? > ) (?<xhook> < \? .*? \? > ) ) }six; our $Meta_Tag_Rx = qr{ $RX_SUBS (?<META> (?&start_tag) meta \b (?: (?&WS) (?&any_nv_pair) ) + (?&end_tag) ) }six; } # nobody *ever* remembers to do this! END { close STDOUT } </pre>
