OLAC Record oai:lindat.mff.cuni.cz:11858/00-097C-0000-000D-F696-9

Metadata
Title:		jusText
Bibliographic Citation:		http://hdl.handle.net/11858/00-097C-0000-000D-F696-9
Creator:		Pomikálek, Jan
Date (W3CDTF):		2013-02-05T12:04:53Z
Date Available:		2013-02-05T12:04:53Z
Description:		jusText is a heuristic based boilerplate removal tool useful for cleaning documents in large textual corpora. The tool has been implemented in Python, licensed under New BSD License and made an open source software (available for download including the source code at http://code.google.com/p/justext/). It is successfully used for cleaning large textual corpora at Natural language processing centre at Faculty of informatics, Masaryk university Brno and it's industry partners. The research leading to this piece of software was published in author's Ph.D. thesis "Removing Boilerplate and Duplicate Content from Web Corpora". The boilerplate removal algorithm is able to remove most of non-grammatical sentences from a web page like navigation, advertisements, tables, short notes and so on. It has been shown it overperforms or at least keeps up with it's competitors (according to comparison with participants of Cleaneval competition in author's Ph.D. thesis). The precise removal of unwanted content and scalability of the algorithm has been demonstrated while building corpora of American Spanish, Arabic, Czech, French, Japanese, Russian, Tajik, and six Turkic languages consisting --- over 20 TB of HTML pages were processed resulting in corpora of 70 billions tokens altogether.
		PRESEMT, Lexical Computing Ltd
Identifier (URI):		http://hdl.handle.net/11858/00-097C-0000-000D-F696-9
Language:		English
Language (ISO639):		eng
Publisher:		Masaryk University, NLP Centre
Rights:		Attribution-ShareAlike 3.0 Unported (CC BY-SA 3.0)
		http://creativecommons.org/licenses/by-sa/3.0/
Subject:		boilerplate
		web documents
		text cleaning
		boilerplate removal
		text corpora
Type:		toolService
Type (DCMI):		Software
OLAC Info
Archive:		LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University
Description:		http://www.language-archives.org/archive/lindat.mff.cuni.cz
GetRecord:		OAI-PMH request for OLAC format
GetRecord:		Pre-generated XML file
OAI Info
OaiIdentifier:		oai:lindat.mff.cuni.cz:11858/00-097C-0000-000D-F696-9
DateStamp:		2021-06-29
GetRecord:		OAI-PMH request for simple DC format
Search Info
Citation:		Pomikálek, Jan. 2013. Masaryk University, NLP Centre.
Terms:		area_Europe country_GB dcmi_Software iso639_eng