| 01:08 | <Huvet> | hi everyone! I'm playing around with the html5lib 0.11 python implementation, and is wondering if I might have hit a bug: http://dpaste.com/hold/123513/ |
| 01:08 | <Huvet> | I'm parsning the HTML of swedish newspapers, which seems to we one of the worst messes in the world :( |
| 01:09 | <Huvet> | or, I could be doing something wrong, it would not be the first time :) |
| 01:34 | <Huvet> | the same error occurs on www.unt.se, and www.uhp.se too |
| 11:40 | <Philip`> | Huvet: 0.11 is very old - you should try it with the latest source version |
| 11:47 | <Huvet> | thanks, I will |
| 12:14 | <Huvet> | gah, "hg" needed to download the latest source version? what happened to the good old svn days :( |
| 12:17 | <Philip`> | The good old svn days turned into the better new hg days |
| 12:18 | <Philip`> | It's basically the same as SVN except you use the command "hg" instead of "svn" :-) |
| 12:18 | <Philip`> | ...although I suppose it might be a bit more painful on Windows |
| 12:27 | <Huvet> | well, not really, seems to work exactly like it should |
| 12:29 | <Huvet> | hmm... strange, it checked out the whole tree, even though I requested a subdirectory |
| 12:33 | <Huvet> | hmm... "... you cannot check out only one directory of a repository" |
| 12:39 | <Huvet> | hmm... I guess I can't clone the default repository and use that? seems that is 0.11 still. Maybe the 0.2 branch? *figures out how to clone a branch* |
| 12:43 | <Huvet> | is that the latest version? or should I look into some other branch? |
| 12:46 | <Huvet> | ah, fuck it, beautifulsoup seems deprecated anyways |
| 12:47 | <Philip`> | Huvet: Yeah, Hg doesn't support partial checkouts - you just clone the entire repository |
| 12:47 | <Huvet> | yeah, I figured that out |
| 12:47 | <Philip`> | which includes all the branches and everything |
| 12:47 | <Huvet> | ah |
| 12:48 | <Huvet> | how do I know which the latest branch is? |
| 12:48 | <Philip`> | You should just use the default branch |
| 12:48 | <Huvet> | ok |
| 12:48 | <Philip`> | since the others were for temporary experiments |
| 12:49 | <Philip`> | I think the BS code is still included and should work better than the 0.11 release, though I could be wrong about that |
| 12:50 | <Huvet> | seems I still get the same error there |
| 12:50 | <Philip`> | but there are fundamental problems in BS that mean it can't work properly in html5lib, and nobody has been interested in spending a great deal of effort on it |
| 12:50 | <Huvet> | but with an extra DataLossWarning |
| 12:50 | <Huvet> | I'll just use something else then I guess |
| 12:51 | <Philip`> | Okay, so maybe it doesn't work much better than the 0.11 release :-( |
| 12:52 | <Philip`> | lxml is usually the recommended treebuilder |
| 12:53 | <Huvet> | ok, i saw the remark in the docs about lxml being an "excellent library" :) |
| 12:53 | <Huvet> | or something in those terms |
| 13:00 | <Huvet> | oh great, the lxml parser crashes on those sites too :( |
| 13:00 | <Philip`> | Hmm, seems to work okay for me with lxml |
| 13:01 | <Philip`> | (I can't test BS yet since I don't have it installed) |
| 13:01 | <Huvet> | are you parsning http://www.allehanda.se ? |
| 13:02 | <Huvet> | http://dpaste.com/123628/ |
| 13:02 | <Philip`> | No, because that timed out when I first tried downloading it |
| 13:02 | <Philip`> | but now I see the problem :-/ |
| 13:04 | <Philip`> | ihatexml.py lives up to its name |
| 13:04 | <Huvet> | heh, great name for a file, what does it do? |
| 13:07 | <Philip`> | http://code.google.com/p/html5lib/issues/detail?id=125 |
| 13:07 | <Huvet> | ah, that seems it |
| 13:07 | <Philip`> | It tries to modify the names returned by the HTML parser so they're compatible with APIs that enforce XML's name requirements |
| 13:08 | <Philip`> | (and similar things) |
| 13:12 | <Philip`> | Huvet: <a><div><div><a> seems to be the pattern the BS treebuilder dislikes |
| 13:13 | <Huvet> | heh, I can understand that |
| 13:15 | <Philip`> | Huvet: It's the same as http://code.google.com/p/html5lib/issues/detail?id=80 |
| 13:15 | <Huvet> | ah, good detective work |
| 13:16 | Philip` | should have remembered it sooner because he looked into that bug when it was new |
| 13:17 | <Philip`> | (At least that's the problem on www.unt.se, I assume the others are the same) |
| 20:50 | <Huvet> | heh, next horrendous HTML that crashes the html5 parser: http://7-harad.nu/ |
| 20:51 | <Philip`> | What error message do you get? |
| 20:51 | <Huvet> | http://dpaste.com/123783/ |
| 20:54 | <Philip`> | Hmm |
| 20:54 | <Philip`> | What treebuilder are you using? |
| 20:54 | <Huvet> | dom |
| 20:55 | <Huvet> | beautifulsoup crashed on some sites, lxml on some other ones, so I'm on dom now :) |
| 20:56 | <Huvet> | I guess it's all the advertising code on these sites that make them so badly formatted |
| 20:56 | <Philip`> | http://code.google.com/p/html5lib/issues/detail?id=123 sounds like it could be relevant |
| 20:57 | <Philip`> | but I'm not really sure |
| 20:57 | <Philip`> | It'd be good if you could produce a minimal testcase |
| 20:57 | <Huvet> | yeah, I'm not sure how to go about that... save the sourcecode locally and start stipping stuff out? |
| 20:57 | <Philip`> | by starting with the markup from the site that causes problems, then deleting half of it and seeing if the problem is still there, else delete the other half instead, and repeat until there's not much left |
| 20:58 | <Philip`> | Yeah, basically what you said :-) |
| 20:58 | <Huvet> | ok, I'll get to work right away |
| 20:58 | <Huvet> | :) |
| 21:19 | <Huvet> | oh, there's a new error |
| 21:19 | <Huvet> | http://dpaste.com/123797/ |
| 21:20 | <Huvet> | but one thing at the time |
| 21:22 | <Philip`> | Testing on real content is a good way to find bugs :-) |
| 21:23 | Philip` | wonders how many pages Huvet is running through it |
| 21:23 | <Huvet> | 351 :) |
| 21:23 | <Huvet> | I'm scaping swedish news sites for RSS urls |
| 21:24 | <Huvet> | seems that's a bit harder than I first thouht :P |
| 21:24 | <Huvet> | seems that's a bit harder than I first thouht : |
| 21:25 | <Huvet> | this is the smallest I can get it: <table><td><span><font></span><span> |
| 21:25 | <Huvet> | first one |
| 21:27 | <Huvet> | ehm... strange... the other error is if I have a file with just <table> in it :) |
| 21:31 | <Philip`> | That's quite minimal :-) |
| 21:35 | <Philip`> | Huvet: I think you could fix the processEOF easily by removing the 'token' in html5parser.py lines 1689, 1692 (the processEOF declaration/call) |
| 21:36 | <Philip`> | but it'd be good to post a new issue on the Google Code site, so someone can add a test case and fix the code and make sure it works |
| 21:37 | <Philip`> | and also for the other bug (which looks like a scary adoption agency thing) |
| 21:38 | <Huvet> | I will |
| 21:56 | <Huvet> | here's the first bug: http://code.google.com/p/html5lib/issues/detail?id=126 |
| 22:01 | <Huvet> | and here's the other one: http://code.google.com/p/html5lib/issues/detail?id=127 |
| 22:04 | <AryehGregor> | "Such a subset does not, in general, include inline script elements." |
| 22:04 | <AryehGregor> | Why can't you include inline script in polyglots? Can't you fudge things using <!CDATA[ or whatnot? |
| 23:01 | <Huvet> | yay, another bug :) |
| 23:02 | <Huvet> | I'll do the drill |
| 23:05 | <Dashiva> | Yours is the drill that will pierce bugzilla |
| 23:09 | <Huvet> | :) |
| 23:09 | <Huvet> | <form><table></form><form></table></form> |
| 23:09 | <Huvet> | wonderful code |
| 23:09 | <Huvet> | I'll post a bug for it |
| 23:09 | <Huvet> | "ValueError: list.remove(x): x not in list" |
| 23:14 | <Huvet> | Here's the bug, for those interested: http://code.google.com/p/html5lib/issues/detail?id=128 |