Module:HTMLParser
CodeDiscussionEditHistoryLinksLink count Subpages:DocumentationTestsResultsSandboxLive code All modules
LuaRock "htmlparser"
Parse HTML text into a tree of elements with selectors
Install
[edit]Htmlparser is a listed LuaRock. Install using LuaRocks: luarocks install htmlparser
At Commons, everything you need is already in place. Module:Set, for example.
Dependencies
[edit]Htmlparser depends on Lua 5.2 5.1 (backported for Commons), and on the "set" LuaRock, which is installed along automatically. To be able to run the tests, lunitx (not available at Commons) also comes along as a LuaRock
Usage
[edit]Start off with
local htmlparser = require("Module:HTMLParser")
Then, parse some html:
local root = htmlparser.parse(htmlstring)
The input to parse may be the contents of a complete html document, or any valid html snippet, as long as all tags are correctly opened and closed. Now, find sepcific contained elements by selecting:
local elements = root:select(selectorstring)
Or in shorthand:
local elements = root(selectorstring)
This wil return a Set of elements, all of which are of the same type as the root element, and thus support selecting as well, if ever needed:
for e in pairs(elements) do
mw.log(e.name)
local subs = e(subselectorstring)
for sub in pairs(subs) do
mw.log("", sub.name)
end
end
The root element is a container for the top level elements in the parsed text, i.e. the <html>
element in a parsed html document would be a child of the returned root element.
Selectors
[edit]Supported selectors are a subset of jQuery's selectors:
"*"
all contained elements"element"
elements with the given tagname"#id"
elements with the given id attribute value".class"
elements with the given classname in the class attribute"[attribute]"
elements with an attribute of the given name"[attribute='value']"
equals: elements with the given value for the given attribute"[attribute!='value']"
not equals: elements without the given attribute, or having the attribute, but with a different value"[attribute|='value']"
prefix: attribute's value is given value, or starts with given value, followed by a hyphen (-
)"[attribute*='value']"
contains: attribute's value contains given value"[attribute~='value']"
word: attribute's value is a space-separated token, where one of the tokens is the given value"[attribute^='value']"
starts with: attribute's value starts with given value"[attribute$='value']"
ends with: attribute's value ends with given value":not(selectorstring)"
elements not selected by given selector string"ancestor descendant"
elements selected by thedescendant
selector string, that are a descendant of any element selected by theancestor
selector string"parent > child"
elements selected by thechild
selector string, that are a child element of any element selected by theparent
selector string
Selectors can be combined; e.g. ".class:not([attribute]) element.class"
Element type
[edit]All tree elements provide, apart from :select
and ()
, the following accessors:
Basic
[edit].name
the element's tagname.attributes
a table with keys and values for the element's attributes;{}
if none.id
the value of the element's id attribute;nil
if not present.classes
an array with the classes listed in element's class attribute;{}
if none:getcontent()
the raw text between the opening and closing tags of the element;""
if none.nodes
an array with the element's child elements,{}
if none.parent
the elements that contains this element;root.parent
isnil
Other
[edit]:gettext()
the complete element text, starting with"<tagname"
and ending with"/>"
or"</tagname>"
.level
how deep the element is in the tree; root level is0
.root
the root element of the tree;root.root
isroot
.deepernodes
a Set containing all elements in the tree beneath this element, including this element's.nodes
;{}
if none.deeperelements
a table with a key for each distinct tagname in.deepernodes
, containing a Set of all deeper element nodes with that name;{}
in none.deeperattributes
as.deeperelements
, but keyed on attribute name.deeperids
as.deeperelements
, but keyed on id value.deeperclasses
as.deeperelements
, but keyed on class name
Limitations
[edit]- Attribute values in selector strings cannot contain any spaces, nor any of
#
,.
,[
,]
,:
,(
, or)
- The spaces before and after the
>
in aparent > child
relation are mandatory <!
elements (including doctype, comments, and CDATA) are not parsed; markup within CDATA is *not* escaped- Textnodes are no seperate tree elements; in
local root = htmlparser.parse("<p>line1<br />line2</p>")
,root.nodes[1]:getcontent()
is"line1<br />line2"
, whileroot.nodes[1].nodes[1].name
is"br"
- No start or end tags are implied when omitted. Only the void elements should not have an end tag
- No validation is done for tag or attribute names or nesting of element types. The list of void elements is in fact the only part specific to HTML
Examples
[edit]See ./doc/sample.lua
Tests
[edit]See ./tst/init.lua
License
[edit]MIT; see ./doc/LICENSE
Code
--[[
(The MIT license)
Copyright (c) 2013, Wouter Scherphof (wouter.scherphof@gmail.com)
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
--]]
-- Source: https://github.com/wscherphof/lua-htmlparser/tree/master/src
local ElementNode = require("Module:HTMLParser/ElementNode")
local voidelements = require("Module:HTMLParser/voidelements")
local HtmlParser = {}
local function parse(text)
local root = ElementNode:new(text)
local node, descend, tpos, opentags = root, true, 1, {}
while true do
local openstart, name
openstart, tpos, name = string.find(root._text,
"<" .. -- an uncaptured starting "<"
"(%w+)" .. -- name = the first word, directly following the "<"
"[^>]*>", -- include, but not capture everything up to the next ">"
tpos)
if not name then break end
local tag = ElementNode:new(name, node, descend, openstart, tpos)
node = tag
local tagst, apos = tag:gettext(), 1
while true do
local start, k, eq, quote, v
start, apos, k, eq, quote = string.find(tagst,
"%s+" .. -- some uncaptured space
"([^%s=/>]+)" .. -- k = an unspaced string up to an optional "=" or the "/" or ">"
"(=?)" .. -- eq = the optional; "=", else ""
"(['\"]?)", -- quote = an optional "'" or '"' following the "=", or ""
apos)
if not k or k == "/>" or k == ">" then break end
if eq == "=" then
local pattern = "=([^%s>]*)"
if quote ~= "" then
pattern = quote .. "([^" .. quote .. "]*)" .. quote
end
start, apos, v = string.find(tagst, pattern, apos)
end
tag:addattribute(k, v or "")
end
if voidelements[string.lower(tag.name)] then
descend = false
tag:close()
else
opentags[tag.name] = opentags[tag.name] or {}
table.insert(opentags[tag.name], tag)
end
local closeend = tpos
while true do
local closestart, closing, closename
closestart, closeend, closing, closename = string.find(root._text, "[^<]*<(/?)(%w+)", closeend)
if not closing or closing == "" then break end
tag = table.remove(opentags[closename])
closestart = string.find(root._text, "<", closestart)
tag:close(closestart, closeend + 1)
node = tag.parent
descend = true
end
end
return root
end
HtmlParser.parse = parse
return HtmlParser