Updating gems: coderay, journey, spork, sprockets

unstable
Tod Beardsley 2012-07-06 22:03:33 -05:00
parent 3dba8273c9
commit 63e41ee6bb
153 changed files with 0 additions and 17533 deletions

View File

@ -1,504 +0,0 @@
GNU LESSER GENERAL PUBLIC LICENSE
Version 2.1, February 1999
Copyright (C) 1991, 1999 Free Software Foundation, Inc.
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
[This is the first released version of the Lesser GPL. It also counts
as the successor of the GNU Library Public License, version 2, hence
the version number 2.1.]
Preamble
The licenses for most software are designed to take away your
freedom to share and change it. By contrast, the GNU General Public
Licenses are intended to guarantee your freedom to share and change
free software--to make sure the software is free for all its users.
This license, the Lesser General Public License, applies to some
specially designated software packages--typically libraries--of the
Free Software Foundation and other authors who decide to use it. You
can use it too, but we suggest you first think carefully about whether
this license or the ordinary General Public License is the better
strategy to use in any particular case, based on the explanations below.
When we speak of free software, we are referring to freedom of use,
not price. Our General Public Licenses are designed to make sure that
you have the freedom to distribute copies of free software (and charge
for this service if you wish); that you receive source code or can get
it if you want it; that you can change the software and use pieces of
it in new free programs; and that you are informed that you can do
these things.
To protect your rights, we need to make restrictions that forbid
distributors to deny you these rights or to ask you to surrender these
rights. These restrictions translate to certain responsibilities for
you if you distribute copies of the library or if you modify it.
For example, if you distribute copies of the library, whether gratis
or for a fee, you must give the recipients all the rights that we gave
you. You must make sure that they, too, receive or can get the source
code. If you link other code with the library, you must provide
complete object files to the recipients, so that they can relink them
with the library after making changes to the library and recompiling
it. And you must show them these terms so they know their rights.
We protect your rights with a two-step method: (1) we copyright the
library, and (2) we offer you this license, which gives you legal
permission to copy, distribute and/or modify the library.
To protect each distributor, we want to make it very clear that
there is no warranty for the free library. Also, if the library is
modified by someone else and passed on, the recipients should know
that what they have is not the original version, so that the original
author's reputation will not be affected by problems that might be
introduced by others.
Finally, software patents pose a constant threat to the existence of
any free program. We wish to make sure that a company cannot
effectively restrict the users of a free program by obtaining a
restrictive license from a patent holder. Therefore, we insist that
any patent license obtained for a version of the library must be
consistent with the full freedom of use specified in this license.
Most GNU software, including some libraries, is covered by the
ordinary GNU General Public License. This license, the GNU Lesser
General Public License, applies to certain designated libraries, and
is quite different from the ordinary General Public License. We use
this license for certain libraries in order to permit linking those
libraries into non-free programs.
When a program is linked with a library, whether statically or using
a shared library, the combination of the two is legally speaking a
combined work, a derivative of the original library. The ordinary
General Public License therefore permits such linking only if the
entire combination fits its criteria of freedom. The Lesser General
Public License permits more lax criteria for linking other code with
the library.
We call this license the "Lesser" General Public License because it
does Less to protect the user's freedom than the ordinary General
Public License. It also provides other free software developers Less
of an advantage over competing non-free programs. These disadvantages
are the reason we use the ordinary General Public License for many
libraries. However, the Lesser license provides advantages in certain
special circumstances.
For example, on rare occasions, there may be a special need to
encourage the widest possible use of a certain library, so that it becomes
a de-facto standard. To achieve this, non-free programs must be
allowed to use the library. A more frequent case is that a free
library does the same job as widely used non-free libraries. In this
case, there is little to gain by limiting the free library to free
software only, so we use the Lesser General Public License.
In other cases, permission to use a particular library in non-free
programs enables a greater number of people to use a large body of
free software. For example, permission to use the GNU C Library in
non-free programs enables many more people to use the whole GNU
operating system, as well as its variant, the GNU/Linux operating
system.
Although the Lesser General Public License is Less protective of the
users' freedom, it does ensure that the user of a program that is
linked with the Library has the freedom and the wherewithal to run
that program using a modified version of the Library.
The precise terms and conditions for copying, distribution and
modification follow. Pay close attention to the difference between a
"work based on the library" and a "work that uses the library". The
former contains code derived from the library, whereas the latter must
be combined with the library in order to run.
GNU LESSER GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License Agreement applies to any software library or other
program which contains a notice placed by the copyright holder or
other authorized party saying it may be distributed under the terms of
this Lesser General Public License (also called "this License").
Each licensee is addressed as "you".
A "library" means a collection of software functions and/or data
prepared so as to be conveniently linked with application programs
(which use some of those functions and data) to form executables.
The "Library", below, refers to any such software library or work
which has been distributed under these terms. A "work based on the
Library" means either the Library or any derivative work under
copyright law: that is to say, a work containing the Library or a
portion of it, either verbatim or with modifications and/or translated
straightforwardly into another language. (Hereinafter, translation is
included without limitation in the term "modification".)
"Source code" for a work means the preferred form of the work for
making modifications to it. For a library, complete source code means
all the source code for all modules it contains, plus any associated
interface definition files, plus the scripts used to control compilation
and installation of the library.
Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope. The act of
running a program using the Library is not restricted, and output from
such a program is covered only if its contents constitute a work based
on the Library (independent of the use of the Library in a tool for
writing it). Whether that is true depends on what the Library does
and what the program that uses the Library does.
1. You may copy and distribute verbatim copies of the Library's
complete source code as you receive it, in any medium, provided that
you conspicuously and appropriately publish on each copy an
appropriate copyright notice and disclaimer of warranty; keep intact
all the notices that refer to this License and to the absence of any
warranty; and distribute a copy of this License along with the
Library.
You may charge a fee for the physical act of transferring a copy,
and you may at your option offer warranty protection in exchange for a
fee.
2. You may modify your copy or copies of the Library or any portion
of it, thus forming a work based on the Library, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:
a) The modified work must itself be a software library.
b) You must cause the files modified to carry prominent notices
stating that you changed the files and the date of any change.
c) You must cause the whole of the work to be licensed at no
charge to all third parties under the terms of this License.
d) If a facility in the modified Library refers to a function or a
table of data to be supplied by an application program that uses
the facility, other than as an argument passed when the facility
is invoked, then you must make a good faith effort to ensure that,
in the event an application does not supply such function or
table, the facility still operates, and performs whatever part of
its purpose remains meaningful.
(For example, a function in a library to compute square roots has
a purpose that is entirely well-defined independent of the
application. Therefore, Subsection 2d requires that any
application-supplied function or table used by this function must
be optional: if the application does not supply it, the square
root function must still compute square roots.)
These requirements apply to the modified work as a whole. If
identifiable sections of that work are not derived from the Library,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works. But when you
distribute the same sections as part of a whole which is a work based
on the Library, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote
it.
Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Library.
In addition, mere aggregation of another work not based on the Library
with the Library (or with a work based on the Library) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.
3. You may opt to apply the terms of the ordinary GNU General Public
License instead of this License to a given copy of the Library. To do
this, you must alter all the notices that refer to this License, so
that they refer to the ordinary GNU General Public License, version 2,
instead of to this License. (If a newer version than version 2 of the
ordinary GNU General Public License has appeared, then you can specify
that version instead if you wish.) Do not make any other change in
these notices.
Once this change is made in a given copy, it is irreversible for
that copy, so the ordinary GNU General Public License applies to all
subsequent copies and derivative works made from that copy.
This option is useful when you wish to copy part of the code of
the Library into a program that is not a library.
4. You may copy and distribute the Library (or a portion or
derivative of it, under Section 2) in object code or executable form
under the terms of Sections 1 and 2 above provided that you accompany
it with the complete corresponding machine-readable source code, which
must be distributed under the terms of Sections 1 and 2 above on a
medium customarily used for software interchange.
If distribution of object code is made by offering access to copy
from a designated place, then offering equivalent access to copy the
source code from the same place satisfies the requirement to
distribute the source code, even though third parties are not
compelled to copy the source along with the object code.
5. A program that contains no derivative of any portion of the
Library, but is designed to work with the Library by being compiled or
linked with it, is called a "work that uses the Library". Such a
work, in isolation, is not a derivative work of the Library, and
therefore falls outside the scope of this License.
However, linking a "work that uses the Library" with the Library
creates an executable that is a derivative of the Library (because it
contains portions of the Library), rather than a "work that uses the
library". The executable is therefore covered by this License.
Section 6 states terms for distribution of such executables.
When a "work that uses the Library" uses material from a header file
that is part of the Library, the object code for the work may be a
derivative work of the Library even though the source code is not.
Whether this is true is especially significant if the work can be
linked without the Library, or if the work is itself a library. The
threshold for this to be true is not precisely defined by law.
If such an object file uses only numerical parameters, data
structure layouts and accessors, and small macros and small inline
functions (ten lines or less in length), then the use of the object
file is unrestricted, regardless of whether it is legally a derivative
work. (Executables containing this object code plus portions of the
Library will still fall under Section 6.)
Otherwise, if the work is a derivative of the Library, you may
distribute the object code for the work under the terms of Section 6.
Any executables containing that work also fall under Section 6,
whether or not they are linked directly with the Library itself.
6. As an exception to the Sections above, you may also combine or
link a "work that uses the Library" with the Library to produce a
work containing portions of the Library, and distribute that work
under terms of your choice, provided that the terms permit
modification of the work for the customer's own use and reverse
engineering for debugging such modifications.
You must give prominent notice with each copy of the work that the
Library is used in it and that the Library and its use are covered by
this License. You must supply a copy of this License. If the work
during execution displays copyright notices, you must include the
copyright notice for the Library among them, as well as a reference
directing the user to the copy of this License. Also, you must do one
of these things:
a) Accompany the work with the complete corresponding
machine-readable source code for the Library including whatever
changes were used in the work (which must be distributed under
Sections 1 and 2 above); and, if the work is an executable linked
with the Library, with the complete machine-readable "work that
uses the Library", as object code and/or source code, so that the
user can modify the Library and then relink to produce a modified
executable containing the modified Library. (It is understood
that the user who changes the contents of definitions files in the
Library will not necessarily be able to recompile the application
to use the modified definitions.)
b) Use a suitable shared library mechanism for linking with the
Library. A suitable mechanism is one that (1) uses at run time a
copy of the library already present on the user's computer system,
rather than copying library functions into the executable, and (2)
will operate properly with a modified version of the library, if
the user installs one, as long as the modified version is
interface-compatible with the version that the work was made with.
c) Accompany the work with a written offer, valid for at
least three years, to give the same user the materials
specified in Subsection 6a, above, for a charge no more
than the cost of performing this distribution.
d) If distribution of the work is made by offering access to copy
from a designated place, offer equivalent access to copy the above
specified materials from the same place.
e) Verify that the user has already received a copy of these
materials or that you have already sent this user a copy.
For an executable, the required form of the "work that uses the
Library" must include any data and utility programs needed for
reproducing the executable from it. However, as a special exception,
the materials to be distributed need not include anything that is
normally distributed (in either source or binary form) with the major
components (compiler, kernel, and so on) of the operating system on
which the executable runs, unless that component itself accompanies
the executable.
It may happen that this requirement contradicts the license
restrictions of other proprietary libraries that do not normally
accompany the operating system. Such a contradiction means you cannot
use both them and the Library together in an executable that you
distribute.
7. You may place library facilities that are a work based on the
Library side-by-side in a single library together with other library
facilities not covered by this License, and distribute such a combined
library, provided that the separate distribution of the work based on
the Library and of the other library facilities is otherwise
permitted, and provided that you do these two things:
a) Accompany the combined library with a copy of the same work
based on the Library, uncombined with any other library
facilities. This must be distributed under the terms of the
Sections above.
b) Give prominent notice with the combined library of the fact
that part of it is a work based on the Library, and explaining
where to find the accompanying uncombined form of the same work.
8. You may not copy, modify, sublicense, link with, or distribute
the Library except as expressly provided under this License. Any
attempt otherwise to copy, modify, sublicense, link with, or
distribute the Library is void, and will automatically terminate your
rights under this License. However, parties who have received copies,
or rights, from you under this License will not have their licenses
terminated so long as such parties remain in full compliance.
9. You are not required to accept this License, since you have not
signed it. However, nothing else grants you permission to modify or
distribute the Library or its derivative works. These actions are
prohibited by law if you do not accept this License. Therefore, by
modifying or distributing the Library (or any work based on the
Library), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Library or works based on it.
10. Each time you redistribute the Library (or any work based on the
Library), the recipient automatically receives a license from the
original licensor to copy, distribute, link with or modify the Library
subject to these terms and conditions. You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties with
this License.
11. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Library at all. For example, if a patent
license would not permit royalty-free redistribution of the Library by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Library.
If any portion of this section is held invalid or unenforceable under any
particular circumstance, the balance of the section is intended to apply,
and the section as a whole is intended to apply in other circumstances.
It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system which is
implemented by public license practices. Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.
This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.
12. If the distribution and/or use of the Library is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Library under this License may add
an explicit geographical distribution limitation excluding those countries,
so that distribution is permitted only in or among countries not thus
excluded. In such case, this License incorporates the limitation as if
written in the body of this License.
13. The Free Software Foundation may publish revised and/or new
versions of the Lesser General Public License from time to time.
Such new versions will be similar in spirit to the present version,
but may differ in detail to address new problems or concerns.
Each version is given a distinguishing version number. If the Library
specifies a version number of this License which applies to it and
"any later version", you have the option of following the terms and
conditions either of that version or of any later version published by
the Free Software Foundation. If the Library does not specify a
license version number, you may choose any version ever published by
the Free Software Foundation.
14. If you wish to incorporate parts of the Library into other free
programs whose distribution conditions are incompatible with these,
write to the author to ask for permission. For software which is
copyrighted by the Free Software Foundation, write to the Free
Software Foundation; we sometimes make exceptions for this. Our
decision will be guided by the two goals of preserving the free status
of all derivatives of our free software and of promoting the sharing
and reuse of software generally.
NO WARRANTY
15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO
WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW.
EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR
OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY
KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE
LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME
THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN
WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY
AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU
FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR
CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE
LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING
RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A
FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF
SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH
DAMAGES.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Libraries
If you develop a new library, and you want it to be of the greatest
possible use to the public, we recommend making it free software that
everyone can redistribute and change. You can do so by permitting
redistribution under these terms (or, alternatively, under the terms of the
ordinary General Public License).
To apply these terms, attach the following notices to the library. It is
safest to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least the
"copyright" line and a pointer to where the full notice is found.
<one line to give the library's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
Also add information on how to contact you by electronic and paper mail.
You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the library, if
necessary. Here is a sample; alter the names:
Yoyodyne, Inc., hereby disclaims all copyright interest in the
library `Frob' (a library for tweaking knobs) written by James Random Hacker.
<signature of Ty Coon>, 1 April 1990
Ty Coon, President of Vice
That's all there is to it!

View File

@ -1,123 +0,0 @@
= CodeRay
Tired of blue'n'gray? Try the original version of this documentation on
coderay.rubychan.de[http://coderay.rubychan.de/doc/] :-)
== About
CodeRay is a Ruby library for syntax highlighting.
You put your code in, and you get it back colored; Keywords, strings,
floats, comments - all in different colors. And with line numbers.
*Syntax* *Highlighting*...
* makes code easier to read and maintain
* lets you detect syntax errors faster
* helps you to understand the syntax of a language
* looks nice
* is what everybody wants to have on their website
* solves all your problems and makes the girls run after you
== Installation
% gem install coderay
=== Dependencies
CodeRay needs Ruby 1.8.7+ or 1.9.2+. It also runs on Rubinius and JRuby.
== Example Usage
require 'coderay'
html = CodeRay.scan("puts 'Hello, world!'", :ruby).div(:line_numbers => :table)
== Documentation
See CodeRay.
== Credits
=== Special Thanks to
* licenser (Heinz N. Gies) for ending my QBasic career, inventing the Coder
project and the input/output plugin system.
CodeRay would not exist without him.
* bovi (Daniel Bovensiepen) for helping me out on various occasions.
=== Thanks to
* Caleb Clausen for writing RubyLexer (see
http://rubyforge.org/projects/rubylexer) and lots of very interesting mail
traffic
* birkenfeld (Georg Brandl) and mitsuhiku (Arnim Ronacher) for PyKleur, now pygments.
You guys rock!
* Jamis Buck for writing Syntax (see http://rubyforge.org/projects/syntax)
I got some useful ideas from it.
* Doug Kearns and everyone else who worked on ruby.vim - it not only helped me
coding CodeRay, but also gave me a wonderful target to reach for the Ruby
scanner.
* everyone who uses CodeBB on http://www.rubyforen.de and http://www.python-forum.de
* iGEL, magichisoka, manveru, WoNáDo and everyone I forgot from rubyforen.de
* Dethix from ruby-mine.de
* zickzackw
* Dookie (who is no longer with us...) and Leonidas from http://www.python-forum.de
* Andreas Schwarz for finding out that CaseIgnoringWordList was not case
ignoring! Such things really make you write tests.
* closure for the first version of the Scheme scanner.
* Stefan Walk for the first version of the JavaScript and PHP scanners.
* Josh Goebel for another version of the JavaScript scanner, a SQL and a Diff scanner.
* Jonathan Younger for pointing out the licence confusion caused by wrong LICENSE file.
* Jeremy Hinegardner for finding the shebang-on-empty-file bug in FileType.
* Charles Oliver Nutter and Yehuda Katz for helping me benchmark CodeRay on JRuby.
* Andreas Neuhaus for pointing out a markup bug in coderay/for_redcloth.
* 0xf30fc7 for the FileType patch concerning Delphi file extensions.
* The folks at redmine.org - thank you for using and fixing CodeRay!
* Keith Pitt for his SQL scanners
* Rob Aldred for the terminal encoder
* Trans for pointing out $DEBUG dependencies
* Flameeyes for finding that Term::ANSIColor was obsolete
* matz and all Ruby gods and gurus
* The inventors of: the computer, the internet, the true color display, HTML &
CSS, VIM, Ruby, pizza, microwaves, guitars, scouting, programming, anime,
manga, coke and green ice tea.
Where would we be without all those people?
=== Created using
* Ruby[http://ruby-lang.org/]
* Chihiro (my Sony VAIO laptop); Henrietta (my old MacBook);
Triella, born Rico (my new MacBook); as well as
Seras and Hikari (my PCs)
* RDE[http://homepage2.nifty.com/sakazuki/rde_e.html],
VIM[http://vim.org] and TextMate[http://macromates.com]
* Subversion[http://subversion.tigris.org/]
* Redmine[http://redmine.org/]
* Firefox[http://www.mozilla.org/products/firefox/],
Firebug[http://getfirebug.com/], Safari[http://www.apple.com/safari/], and
Thunderbird[http://www.mozilla.org/products/thunderbird/]
* RubyGems[http://docs.rubygems.org/] and Rake[http://rake.rubyforge.org/]
* TortoiseSVN[http://tortoisesvn.tigris.org/] using Apache via
XAMPP[http://www.apachefriends.org/en/xampp.html]
* RDoc (though I'm quite unsatisfied with it)
* Microsoft Windows (yes, I confess!) and MacOS X
* GNUWin32, MinGW and some other tools to make the shell under windows a bit
less useless
* Term::ANSIColor[http://term-ansicolor.rubyforge.org/]
* PLEAC[http://pleac.sourceforge.net/] code examples
* Github
* Travis CI (http://travis-ci.org/rubychan/github)
=== Free
* As you can see, CodeRay was created under heavy use of *free* software.
* So CodeRay is also *free*.
* If you use CodeRay to create software, think about making this software
*free*, too.
* Thanks :)

View File

@ -1,35 +0,0 @@
$:.unshift File.dirname(__FILE__) unless $:.include? '.'
ROOT = '.'
LIB_ROOT = File.join ROOT, 'lib'
task :default => :test
if File.directory? 'rake_tasks'
# load rake tasks from subfolder
for task_file in Dir['rake_tasks/*.rake'].sort
load task_file
end
else
# fallback tasks when rake_tasks folder is not present (eg. in the distribution package)
desc 'Run CodeRay tests (basic)'
task :test do
ruby './test/functional/suite.rb'
ruby './test/functional/for_redcloth.rb'
end
gem 'rdoc' if defined? gem
require 'rdoc/task'
desc 'Generate documentation for CodeRay'
Rake::RDocTask.new :doc do |rd|
rd.title = 'CodeRay Documentation'
rd.main = 'README_INDEX.rdoc'
rd.rdoc_files.add Dir['lib']
rd.rdoc_files.add rd.main
rd.rdoc_dir = 'doc'
end
end

View File

@ -1,215 +0,0 @@
#!/usr/bin/env ruby
require 'coderay'
$options, args = ARGV.partition { |arg| arg[/^-[hv]$|--\w+/] }
subcommand = args.first if /^\w/ === args.first
subcommand = nil if subcommand && File.exist?(subcommand)
args.delete subcommand
def option? *options
!($options & options).empty?
end
def tty?
$stdout.tty? || option?('--tty')
end
def version
puts <<-USAGE
CodeRay #{CodeRay::VERSION}
USAGE
end
def help
puts <<-HELP
This is CodeRay #{CodeRay::VERSION}, a syntax highlighting tool for selected languages.
usage:
coderay [-language] [input] [-format] [output]
defaults:
language detect from input file name or shebang; fall back to plain text
input STDIN
format detect from output file name or use terminal; fall back to HTML
output STDOUT
common:
coderay file.rb # highlight file to terminal
coderay file.rb > file.html # highlight file to HTML page
coderay file.rb -div > file.html # highlight file to HTML snippet
configure output:
coderay file.py output.json # output tokens as JSON
coderay file.py -loc # count lines of code in Python file
configure input:
coderay -python file # specify the input language
coderay -ruby # take input from STDIN
more:
coderay stylesheet [style] # print CSS stylesheet
HELP
end
def commands
puts <<-COMMANDS
general:
highlight code highlighting (default command, optional)
stylesheet print the CSS stylesheet with the given name (aliases: style, css)
about:
list [of] list all available plugins (or just the scanners|encoders|styles|filetypes)
commands print this list
help show some help
version print CodeRay version
COMMANDS
end
def print_list_of plugin_host
plugins = plugin_host.all_plugins.map do |plugin|
info = " #{plugin.plugin_id}: #{plugin.title}"
aliases = (plugin.aliases - [:default]).map { |key| "-#{key}" }.sort_by { |key| key.size }
if plugin.respond_to?(:file_extension) || !aliases.empty?
additional_info = []
additional_info << aliases.join(', ') unless aliases.empty?
info << " (#{additional_info.join('; ')})"
end
info << ' <-- default' if plugin.aliases.include? :default
info
end
puts plugins.sort
end
if option? '-v', '--version'
version
end
if option? '-h', '--help'
help
end
case subcommand
when 'highlight', nil
if ARGV.empty?
version
help
else
signature = args.map { |arg| arg[/^-/] ? '-' : 'f' }.join
names = args.map { |arg| arg.sub(/^-/, '') }
case signature
when /^$/
exit
when /^ff?$/
input_file, output_file, = *names
when /^f-f?$/
input_file, output_format, output_file, = *names
when /^-ff?$/
input_lang, input_file, output_file, = *names
when /^-f-f?$/
input_lang, input_file, output_format, output_file, = *names
when /^--?f?$/
input_lang, output_format, output_file, = *names
else
$stdout = $stderr
help
puts
puts "Unknown parameter order: #{args.join ' '}, expected: [-language] [input] [-format] [output]"
exit 1
end
if input_file
input_lang ||= CodeRay::FileType.fetch input_file, :text, true
end
if output_file
output_format ||= CodeRay::FileType[output_file]
else
output_format ||= :terminal
end
output_format = :page if output_format.to_s == 'html'
if input_file
input = File.read input_file
else
input = $stdin.read
end
begin
file =
if output_file
File.open output_file, 'w'
else
$stdout.sync = true
$stdout
end
CodeRay.encode(input, input_lang, output_format, :out => file)
file.puts
rescue CodeRay::PluginHost::PluginNotFound => boom
$stdout = $stderr
if boom.message[/CodeRay::(\w+)s could not load plugin :?(.*?): /]
puts "I don't know the #$1 \"#$2\"."
else
puts boom.message
end
# puts "I don't know this plugin: #{boom.message[/Could not load plugin (.*?): /, 1]}."
rescue CodeRay::Scanners::Scanner::ScanError # FIXME: rescue Errno::EPIPE
# this is sometimes raised by pagers; ignore [TODO: wtf?]
ensure
file.close if output_file
end
end
when 'li', 'list'
arg = args.first && args.first.downcase
if [nil, 's', 'sc', 'scanner', 'scanners'].include? arg
puts 'input languages (Scanners):'
print_list_of CodeRay::Scanners
end
if [nil, 'e', 'en', 'enc', 'encoder', 'encoders'].include? arg
puts 'output formats (Encoders):'
print_list_of CodeRay::Encoders
end
if [nil, 'st', 'style', 'styles'].include? arg
puts 'CSS themes for HTML output (Styles):'
print_list_of CodeRay::Styles
end
if [nil, 'f', 'ft', 'file', 'filetype', 'filetypes'].include? arg
puts 'recognized file types:'
filetypes = Hash.new { |h, k| h[k] = [] }
CodeRay::FileType::TypeFromExt.inject filetypes do |types, (ext, type)|
types[type.to_s] << ".#{ext}"
types
end
CodeRay::FileType::TypeFromName.inject filetypes do |types, (name, type)|
types[type.to_s] << name
types
end
filetypes.sort.each do |type, exts|
puts " #{type}: #{exts.sort_by { |ext| ext.size }.join(', ')}"
end
end
when 'stylesheet', 'style', 'css'
puts CodeRay::Encoders[:html]::CSS.new(args.first || :default).stylesheet
when 'commands'
commands
when 'help'
help
else
$stdout = $stderr
help
puts
if subcommand[/\A\w+\z/]
puts "Unknown command: #{subcommand}"
else
puts "File not found: #{subcommand}"
end
exit 1
end

View File

@ -1,285 +0,0 @@
# encoding: utf-8
# Encoding.default_internal = 'UTF-8'
# = CodeRay Library
#
# CodeRay is a Ruby library for syntax highlighting.
#
# I try to make CodeRay easy to use and intuitive, but at the same time fully
# featured, complete, fast and efficient.
#
# See README.
#
# It consists mainly of
# * the main engine: CodeRay (Scanners::Scanner, Tokens, Encoders::Encoder)
# * the plugin system: PluginHost, Plugin
# * the scanners in CodeRay::Scanners
# * the encoders in CodeRay::Encoders
# * the styles in CodeRay::Styles
#
# Here's a fancy graphic to light up this gray docu:
#
# http://cycnus.de/raindark/coderay/scheme.png
#
# == Documentation
#
# See CodeRay, Encoders, Scanners, Tokens.
#
# == Usage
#
# Remember you need RubyGems to use CodeRay, unless you have it in your load
# path. Run Ruby with -rubygems option if required.
#
# === Highlight Ruby code in a string as html
#
# require 'coderay'
# print CodeRay.scan('puts "Hello, world!"', :ruby).html
#
# # prints something like this:
# puts <span class="s">&quot;Hello, world!&quot;</span>
#
#
# === Highlight C code from a file in a html div
#
# require 'coderay'
# print CodeRay.scan(File.read('ruby.h'), :c).div
# print CodeRay.scan_file('ruby.h').html.div
#
# You can include this div in your page. The used CSS styles can be printed with
#
# % coderay_stylesheet
#
# === Highlight without typing too much
#
# If you are one of the hasty (or lazy, or extremely curious) people, just run this file:
#
# % ruby -rubygems /path/to/coderay/coderay.rb > example.html
#
# and look at the file it created in your browser.
#
# = CodeRay Module
#
# The CodeRay module provides convenience methods for the engine.
#
# * The +lang+ and +format+ arguments select Scanner and Encoder to use. These are
# simply lower-case symbols, like <tt>:python</tt> or <tt>:html</tt>.
# * All methods take an optional hash as last parameter, +options+, that is send to
# the Encoder / Scanner.
# * Input and language are always sorted in this order: +code+, +lang+.
# (This is in alphabetical order, if you need a mnemonic ;)
#
# You should be able to highlight everything you want just using these methods;
# so there is no need to dive into CodeRay's deep class hierarchy.
#
# The examples in the demo directory demonstrate common cases using this interface.
#
# = Basic Access Ways
#
# Read this to get a general view what CodeRay provides.
#
# == Scanning
#
# Scanning means analysing an input string, splitting it up into Tokens.
# Each Token knows about what type it is: string, comment, class name, etc.
#
# Each +lang+ (language) has its own Scanner; for example, <tt>:ruby</tt> code is
# handled by CodeRay::Scanners::Ruby.
#
# CodeRay.scan:: Scan a string in a given language into Tokens.
# This is the most common method to use.
# CodeRay.scan_file:: Scan a file and guess the language using FileType.
#
# The Tokens object you get from these methods can encode itself; see Tokens.
#
# == Encoding
#
# Encoding means compiling Tokens into an output. This can be colored HTML or
# LaTeX, a textual statistic or just the number of non-whitespace tokens.
#
# Each Encoder provides output in a specific +format+, so you select Encoders via
# formats like <tt>:html</tt> or <tt>:statistic</tt>.
#
# CodeRay.encode:: Scan and encode a string in a given language.
# CodeRay.encode_tokens:: Encode the given tokens.
# CodeRay.encode_file:: Scan a file, guess the language using FileType and encode it.
#
# == All-in-One Encoding
#
# CodeRay.encode:: Highlight a string with a given input and output format.
#
# == Instanciating
#
# You can use an Encoder instance to highlight multiple inputs. This way, the setup
# for this Encoder must only be done once.
#
# CodeRay.encoder:: Create an Encoder instance with format and options.
# CodeRay.scanner:: Create an Scanner instance for lang, with '' as default code.
#
# To make use of CodeRay.scanner, use CodeRay::Scanner::code=.
#
# The scanning methods provide more flexibility; we recommend to use these.
#
# == Reusing Scanners and Encoders
#
# If you want to re-use scanners and encoders (because that is faster), see
# CodeRay::Duo for the most convenient (and recommended) interface.
module CodeRay
$CODERAY_DEBUG ||= false
CODERAY_PATH = File.join File.dirname(__FILE__), 'coderay'
# Assuming the path is a subpath of lib/coderay/
def self.coderay_path *path
File.join CODERAY_PATH, *path
end
require coderay_path('version')
# helpers
autoload :FileType, coderay_path('helpers', 'file_type')
# Tokens
autoload :Tokens, coderay_path('tokens')
autoload :TokensProxy, coderay_path('tokens_proxy')
autoload :TokenKinds, coderay_path('token_kinds')
# Plugin system
autoload :PluginHost, coderay_path('helpers', 'plugin')
autoload :Plugin, coderay_path('helpers', 'plugin')
# Plugins
autoload :Scanners, coderay_path('scanner')
autoload :Encoders, coderay_path('encoder')
autoload :Styles, coderay_path('style')
# convenience access and reusable Encoder/Scanner pair
autoload :Duo, coderay_path('duo')
class << self
# Scans the given +code+ (a String) with the Scanner for +lang+.
#
# This is a simple way to use CodeRay. Example:
# require 'coderay'
# page = CodeRay.scan("puts 'Hello, world!'", :ruby).html
#
# See also demo/demo_simple.
def scan code, lang, options = {}, &block
# FIXME: return a proxy for direct-stream encoding
TokensProxy.new code, lang, options, block
end
# Scans +filename+ (a path to a code file) with the Scanner for +lang+.
#
# If +lang+ is :auto or omitted, the CodeRay::FileType module is used to
# determine it. If it cannot find out what type it is, it uses
# CodeRay::Scanners::Text.
#
# Calls CodeRay.scan.
#
# Example:
# require 'coderay'
# page = CodeRay.scan_file('some_c_code.c').html
def scan_file filename, lang = :auto, options = {}, &block
lang = FileType.fetch filename, :text, true if lang == :auto
code = File.read filename
scan code, lang, options, &block
end
# Encode a string.
#
# This scans +code+ with the the Scanner for +lang+ and then
# encodes it with the Encoder for +format+.
# +options+ will be passed to the Encoder.
#
# See CodeRay::Encoder.encode.
def encode code, lang, format, options = {}
encoder(format, options).encode code, lang, options
end
# Encode pre-scanned Tokens.
# Use this together with CodeRay.scan:
#
# require 'coderay'
#
# # Highlight a short Ruby code example in a HTML span
# tokens = CodeRay.scan '1 + 2', :ruby
# puts CodeRay.encode_tokens(tokens, :span)
#
def encode_tokens tokens, format, options = {}
encoder(format, options).encode_tokens tokens, options
end
# Encodes +filename+ (a path to a code file) with the Scanner for +lang+.
#
# See CodeRay.scan_file.
# Notice that the second argument is the output +format+, not the input language.
#
# Example:
# require 'coderay'
# page = CodeRay.encode_file 'some_c_code.c', :html
def encode_file filename, format, options = {}
tokens = scan_file filename, :auto, get_scanner_options(options)
encode_tokens tokens, format, options
end
# Highlight a string into a HTML <div>.
#
# CSS styles use classes, so you have to include a stylesheet
# in your output.
#
# See encode.
def highlight code, lang, options = { :css => :class }, format = :div
encode code, lang, format, options
end
# Highlight a file into a HTML <div>.
#
# CSS styles use classes, so you have to include a stylesheet
# in your output.
#
# See encode.
def highlight_file filename, options = { :css => :class }, format = :div
encode_file filename, format, options
end
# Finds the Encoder class for +format+ and creates an instance, passing
# +options+ to it.
#
# Example:
# require 'coderay'
#
# stats = CodeRay.encoder(:statistic)
# stats.encode("puts 17 + 4\n", :ruby)
#
# puts '%d out of %d tokens have the kind :integer.' % [
# stats.type_stats[:integer].count,
# stats.real_token_count
# ]
# #-> 2 out of 4 tokens have the kind :integer.
def encoder format, options = {}
Encoders[format].new options
end
# Finds the Scanner class for +lang+ and creates an instance, passing
# +options+ to it.
#
# See Scanner.new.
def scanner lang, options = {}, &block
Scanners[lang].new '', options, &block
end
# Extract the options for the scanner from the +options+ hash.
#
# Returns an empty Hash if <tt>:scanner_options</tt> is not set.
#
# This is used if a method like CodeRay.encode has to provide options
# for Encoder _and_ scanner.
def get_scanner_options options
options.fetch :scanner_options, {}
end
end
end

View File

@ -1,81 +0,0 @@
module CodeRay
# = Duo
#
# A Duo is a convenient way to use CodeRay. You just create a Duo,
# giving it a lang (language of the input code) and a format (desired
# output format), and call Duo#highlight with the code.
#
# Duo makes it easy to re-use both scanner and encoder for a repetitive
# task. It also provides a very easy interface syntax:
#
# require 'coderay'
# CodeRay::Duo[:python, :div].highlight 'import this'
#
# Until you want to do uncommon things with CodeRay, I recommend to use
# this method, since it takes care of everything.
class Duo
attr_accessor :lang, :format, :options
# Create a new Duo, holding a lang and a format to highlight code.
#
# simple:
# CodeRay::Duo[:ruby, :html].highlight 'bla 42'
#
# with options:
# CodeRay::Duo[:ruby, :html, :hint => :debug].highlight '????::??'
#
# alternative syntax without options:
# CodeRay::Duo[:ruby => :statistic].encode 'class << self; end'
#
# alternative syntax with options:
# CodeRay::Duo[{ :ruby => :statistic }, :do => :something].encode 'abc'
#
# The options are forwarded to scanner and encoder
# (see CodeRay.get_scanner_options).
def initialize lang = nil, format = nil, options = {}
if format.nil? && lang.is_a?(Hash) && lang.size == 1
@lang = lang.keys.first
@format = lang[@lang]
else
@lang = lang
@format = format
end
@options = options
end
class << self
# To allow calls like Duo[:ruby, :html].highlight.
alias [] new
end
# The scanner of the duo. Only created once.
def scanner
@scanner ||= CodeRay.scanner @lang, CodeRay.get_scanner_options(@options)
end
# The encoder of the duo. Only created once.
def encoder
@encoder ||= CodeRay.encoder @format, @options
end
# Tokenize and highlight the code using +scanner+ and +encoder+.
def encode code, options = {}
options = @options.merge options
encoder.encode(code, @lang, options)
end
alias highlight encode
# Allows to use Duo like a proc object:
#
# CodeRay::Duo[:python => :yaml].call(code)
#
# or, in Ruby 1.9 and later:
#
# CodeRay::Duo[:python => :yaml].(code)
alias call encode
end
end

View File

@ -1,201 +0,0 @@
module CodeRay
# This module holds the Encoder class and its subclasses.
# For example, the HTML encoder is named CodeRay::Encoders::HTML
# can be found in coderay/encoders/html.
#
# Encoders also provides methods and constants for the register
# mechanism and the [] method that returns the Encoder class
# belonging to the given format.
module Encoders
extend PluginHost
plugin_path File.dirname(__FILE__), 'encoders'
# = Encoder
#
# The Encoder base class. Together with Scanner and
# Tokens, it forms the highlighting triad.
#
# Encoder instances take a Tokens object and do something with it.
#
# The most common Encoder is surely the HTML encoder
# (CodeRay::Encoders::HTML). It highlights the code in a colorful
# html page.
# If you want the highlighted code in a div or a span instead,
# use its subclasses Div and Span.
class Encoder
extend Plugin
plugin_host Encoders
class << self
# If FILE_EXTENSION isn't defined, this method returns the
# downcase class name instead.
def const_missing sym
if sym == :FILE_EXTENSION
(defined?(@plugin_id) && @plugin_id || name[/\w+$/].downcase).to_s
else
super
end
end
# The default file extension for output file of this encoder class.
def file_extension
self::FILE_EXTENSION
end
end
# Subclasses are to store their default options in this constant.
DEFAULT_OPTIONS = { }
# The options you gave the Encoder at creating.
attr_accessor :options, :scanner
# Creates a new Encoder.
# +options+ is saved and used for all encode operations, as long
# as you don't overwrite it there by passing additional options.
#
# Encoder objects provide three encode methods:
# - encode simply takes a +code+ string and a +lang+
# - encode_tokens expects a +tokens+ object instead
#
# Each method has an optional +options+ parameter. These are
# added to the options you passed at creation.
def initialize options = {}
@options = self.class::DEFAULT_OPTIONS.merge options
@@CODERAY_TOKEN_INTERFACE_DEPRECATION_WARNING_GIVEN = false
end
# Encode a Tokens object.
def encode_tokens tokens, options = {}
options = @options.merge options
@scanner = tokens.scanner if tokens.respond_to? :scanner
setup options
compile tokens, options
finish options
end
# Encode the given +code+ using the Scanner for +lang+.
def encode code, lang, options = {}
options = @options.merge options
@scanner = Scanners[lang].new code, CodeRay.get_scanner_options(options).update(:tokens => self)
setup options
@scanner.tokenize
finish options
end
# You can use highlight instead of encode, if that seems
# more clear to you.
alias highlight encode
# The default file extension for this encoder.
def file_extension
self.class.file_extension
end
def << token
unless @@CODERAY_TOKEN_INTERFACE_DEPRECATION_WARNING_GIVEN
warn 'Using old Tokens#<< interface.'
@@CODERAY_TOKEN_INTERFACE_DEPRECATION_WARNING_GIVEN = true
end
self.token(*token)
end
# Called with +content+ and +kind+ of the currently scanned token.
# For simple scanners, it's enougth to implement this method.
#
# By default, it calls text_token, begin_group, end_group, begin_line,
# or end_line, depending on the +content+.
def token content, kind
case content
when String
text_token content, kind
when :begin_group
begin_group kind
when :end_group
end_group kind
when :begin_line
begin_line kind
when :end_line
end_line kind
else
raise ArgumentError, 'Unknown token content type: %p, kind = %p' % [content, kind]
end
end
# Called for each text token ([text, kind]), where text is a String.
def text_token text, kind
@out << text
end
# Starts a token group with the given +kind+.
def begin_group kind
end
# Ends a token group with the given +kind+.
def end_group kind
end
# Starts a new line token group with the given +kind+.
def begin_line kind
end
# Ends a new line token group with the given +kind+.
def end_line kind
end
protected
# Called with merged options before encoding starts.
# Sets @out to an empty string.
#
# See the HTML Encoder for an example of option caching.
def setup options
@out = get_output(options)
end
def get_output options
options[:out] || ''
end
# Append data.to_s to the output. Returns the argument.
def output data
@out << data.to_s
data
end
# Called with merged options after encoding starts.
# The return value is the result of encoding, typically @out.
def finish options
@out
end
# Do the encoding.
#
# The already created +tokens+ object must be used; it must be a
# Tokens object.
def compile tokens, options = {}
content = nil
for item in tokens
if item.is_a? Array
raise ArgumentError, 'Two-element array tokens are no longer supported.'
end
if content
token content, item
content = nil
else
content = item
end
end
raise 'odd number list for Tokens' if content
end
alias tokens compile
public :tokens
end
end
end

View File

@ -1,17 +0,0 @@
module CodeRay
module Encoders
map \
:loc => :lines_of_code,
:plain => :text,
:plaintext => :text,
:remove_comments => :comment_filter,
:stats => :statistic,
:term => :terminal,
:tty => :terminal,
:yml => :yaml
# No default because Tokens#nonsense should raise NoMethodError.
end
end

View File

@ -1,25 +0,0 @@
module CodeRay
module Encoders
load :token_kind_filter
# A simple Filter that removes all tokens of the :comment kind.
#
# Alias: +remove_comments+
#
# Usage:
# CodeRay.scan('print # foo', :ruby).comment_filter.text
# #-> "print "
#
# See also: TokenKindFilter, LinesOfCode
class CommentFilter < TokenKindFilter
register_for :comment_filter
DEFAULT_OPTIONS = superclass::DEFAULT_OPTIONS.merge \
:exclude => [:comment, :docstring]
end
end
end

View File

@ -1,39 +0,0 @@
module CodeRay
module Encoders
# Returns the number of tokens.
#
# Text and block tokens are counted.
class Count < Encoder
register_for :count
protected
def setup options
super
@count = 0
end
def finish options
output @count
end
public
def text_token text, kind
@count += 1
end
def begin_group kind
@count += 1
end
alias end_group begin_group
alias begin_line begin_group
alias end_line begin_group
end
end
end

View File

@ -1,61 +0,0 @@
module CodeRay
module Encoders
# = Debug Encoder
#
# Fast encoder producing simple debug output.
#
# It is readable and diff-able and is used for testing.
#
# You cannot fully restore the tokens information from the
# output, because consecutive :space tokens are merged.
# Use Tokens#dump for caching purposes.
#
# See also: Scanners::Debug
class Debug < Encoder
register_for :debug
FILE_EXTENSION = 'raydebug'
def initialize options = {}
super
@opened = []
end
def text_token text, kind
if kind == :space
@out << text
else
# TODO: Escape (
text = text.gsub(/[)\\]/, '\\\\\0') # escape ) and \
@out << kind.to_s << '(' << text << ')'
end
end
def begin_group kind
@opened << kind
@out << kind.to_s << '<'
end
def end_group kind
if @opened.last != kind
puts @out
raise "we are inside #{@opened.inspect}, not #{kind}"
end
@opened.pop
@out << '>'
end
def begin_line kind
@out << kind.to_s << '['
end
def end_line kind
@out << ']'
end
end
end
end

View File

@ -1,23 +0,0 @@
module CodeRay
module Encoders
load :html
# Wraps HTML output into a DIV element, using inline styles by default.
#
# See Encoders::HTML for available options.
class Div < HTML
FILE_EXTENSION = 'div.html'
register_for :div
DEFAULT_OPTIONS = HTML::DEFAULT_OPTIONS.merge \
:css => :style,
:wrap => :div,
:line_numbers => false
end
end
end

View File

@ -1,58 +0,0 @@
module CodeRay
module Encoders
# A Filter encoder has another Tokens instance as output.
# It can be subclass to select, remove, or modify tokens in the stream.
#
# Subclasses of Filter are called "Filters" and can be chained.
#
# == Options
#
# === :tokens
#
# The Tokens object which will receive the output.
#
# Default: Tokens.new
#
# See also: TokenKindFilter
class Filter < Encoder
register_for :filter
protected
def setup options
super
@tokens = options[:tokens] || Tokens.new
end
def finish options
output @tokens
end
public
def text_token text, kind # :nodoc:
@tokens.text_token text, kind
end
def begin_group kind # :nodoc:
@tokens.begin_group kind
end
def begin_line kind # :nodoc:
@tokens.begin_line kind
end
def end_group kind # :nodoc:
@tokens.end_group kind
end
def end_line kind # :nodoc:
@tokens.end_line kind
end
end
end
end

View File

@ -1,327 +0,0 @@
require 'set'
module CodeRay
module Encoders
# = HTML Encoder
#
# This is CodeRay's most important highlighter:
# It provides save, fast XHTML generation and CSS support.
#
# == Usage
#
# require 'coderay'
# puts CodeRay.scan('Some /code/', :ruby).html #-> a HTML page
# puts CodeRay.scan('Some /code/', :ruby).html(:wrap => :span)
# #-> <span class="CodeRay"><span class="co">Some</span> /code/</span>
# puts CodeRay.scan('Some /code/', :ruby).span #-> the same
#
# puts CodeRay.scan('Some code', :ruby).html(
# :wrap => nil,
# :line_numbers => :inline,
# :css => :style
# )
#
# == Options
#
# === :tab_width
# Convert \t characters to +n+ spaces (a number.)
#
# Default: 8
#
# === :css
# How to include the styles; can be :class or :style.
#
# Default: :class
#
# === :wrap
# Wrap in :page, :div, :span or nil.
#
# You can also use Encoders::Div and Encoders::Span.
#
# Default: nil
#
# === :title
#
# The title of the HTML page (works only when :wrap is set to :page.)
#
# Default: 'CodeRay output'
#
# === :break_lines
#
# Split multiline blocks at line breaks.
# Forced to true if :line_numbers option is set to :inline.
#
# Default: false
#
# === :line_numbers
# Include line numbers in :table, :inline, or nil (no line numbers)
#
# Default: nil
#
# === :line_number_anchors
# Adds anchors and links to the line numbers. Can be false (off), true (on),
# or a prefix string that will be prepended to the anchor name.
#
# The prefix must consist only of letters, digits, and underscores.
#
# Default: true, default prefix name: "line"
#
# === :line_number_start
# Where to start with line number counting.
#
# Default: 1
#
# === :bold_every
# Make every +n+-th number appear bold.
#
# Default: 10
#
# === :highlight_lines
#
# Highlights certain line numbers.
# Can be any Enumerable, typically just an Array or Range, of numbers.
#
# Bolding is deactivated when :highlight_lines is set. It only makes sense
# in combination with :line_numbers.
#
# Default: nil
#
# === :hint
# Include some information into the output using the title attribute.
# Can be :info (show token kind on mouse-over), :info_long (with full path)
# or :debug (via inspect).
#
# Default: false
class HTML < Encoder
register_for :html
FILE_EXTENSION = 'snippet.html'
DEFAULT_OPTIONS = {
:tab_width => 8,
:css => :class,
:style => :alpha,
:wrap => nil,
:title => 'CodeRay output',
:break_lines => false,
:line_numbers => nil,
:line_number_anchors => 'n',
:line_number_start => 1,
:bold_every => 10,
:highlight_lines => nil,
:hint => false,
}
autoload :Output, CodeRay.coderay_path('encoders', 'html', 'output')
autoload :CSS, CodeRay.coderay_path('encoders', 'html', 'css')
autoload :Numbering, CodeRay.coderay_path('encoders', 'html', 'numbering')
attr_reader :css
protected
HTML_ESCAPE = { #:nodoc:
'&' => '&amp;',
'"' => '&quot;',
'>' => '&gt;',
'<' => '&lt;',
}
# This was to prevent illegal HTML.
# Strange chars should still be avoided in codes.
evil_chars = Array(0x00...0x20) - [?\n, ?\t, ?\s]
evil_chars.each { |i| HTML_ESCAPE[i.chr] = ' ' }
#ansi_chars = Array(0x7f..0xff)
#ansi_chars.each { |i| HTML_ESCAPE[i.chr] = '&#%d;' % i }
# \x9 (\t) and \xA (\n) not included
#HTML_ESCAPE_PATTERN = /[\t&"><\0-\x8\xB-\x1f\x7f-\xff]/
HTML_ESCAPE_PATTERN = /[\t"&><\0-\x8\xB-\x1f]/
TOKEN_KIND_TO_INFO = Hash.new do |h, kind|
h[kind] = kind.to_s.gsub(/_/, ' ').gsub(/\b\w/) { $&.capitalize }
end
TRANSPARENT_TOKEN_KINDS = Set[
:delimiter, :modifier, :content, :escape, :inline_delimiter,
]
# Generate a hint about the given +kinds+ in a +hint+ style.
#
# +hint+ may be :info, :info_long or :debug.
def self.token_path_to_hint hint, kinds
kinds = Array kinds
title =
case hint
when :info
kinds = kinds[1..-1] if TRANSPARENT_TOKEN_KINDS.include? kinds.first
TOKEN_KIND_TO_INFO[kinds.first]
when :info_long
kinds.reverse.map { |kind| TOKEN_KIND_TO_INFO[kind] }.join('/')
when :debug
kinds.inspect
end
title ? " title=\"#{title}\"" : ''
end
def setup options
super
if options[:wrap] || options[:line_numbers]
@real_out = @out
@out = ''
end
options[:break_lines] = true if options[:line_numbers] == :inline
@break_lines = (options[:break_lines] == true)
@HTML_ESCAPE = HTML_ESCAPE.dup
@HTML_ESCAPE["\t"] = ' ' * options[:tab_width]
@opened = []
@last_opened = nil
@css = CSS.new options[:style]
hint = options[:hint]
if hint && ![:debug, :info, :info_long].include?(hint)
raise ArgumentError, "Unknown value %p for :hint; \
expected :info, :info_long, :debug, false, or nil." % hint
end
css_classes = TokenKinds
case options[:css]
when :class
@span_for_kind = Hash.new do |h, k|
if k.is_a? ::Symbol
kind = k_dup = k
else
kind = k.first
k_dup = k.dup
end
if kind != :space && (hint || css_class = css_classes[kind])
title = HTML.token_path_to_hint hint, k if hint
css_class ||= css_classes[kind]
h[k_dup] = "<span#{title}#{" class=\"#{css_class}\"" if css_class}>"
else
h[k_dup] = nil
end
end
when :style
@span_for_kind = Hash.new do |h, k|
kind = k.is_a?(Symbol) ? k : k.first
h[k.is_a?(Symbol) ? k : k.dup] =
if kind != :space && (hint || css_classes[kind])
title = HTML.token_path_to_hint hint, k if hint
style = @css.get_style Array(k).map { |c| css_classes[c] }
"<span#{title}#{" style=\"#{style}\"" if style}>"
end
end
else
raise ArgumentError, "Unknown value %p for :css." % options[:css]
end
@set_last_opened = options[:hint] || options[:css] == :style
end
def finish options
unless @opened.empty?
warn '%d tokens still open: %p' % [@opened.size, @opened] if $CODERAY_DEBUG
@out << '</span>' while @opened.pop
@last_opened = nil
end
@out.extend Output
@out.css = @css
if options[:line_numbers]
Numbering.number! @out, options[:line_numbers], options
end
@out.wrap! options[:wrap]
@out.apply_title! options[:title]
if defined?(@real_out) && @real_out
@real_out << @out
@out = @real_out
end
super
end
public
def text_token text, kind
if text =~ /#{HTML_ESCAPE_PATTERN}/o
text = text.gsub(/#{HTML_ESCAPE_PATTERN}/o) { |m| @HTML_ESCAPE[m] }
end
style = @span_for_kind[@last_opened ? [kind, *@opened] : kind]
if @break_lines && (i = text.index("\n")) && (c = @opened.size + (style ? 1 : 0)) > 0
close = '</span>' * c
reopen = ''
@opened.each_with_index do |k, index|
reopen << (@span_for_kind[index > 0 ? [k, *@opened[0 ... index ]] : k] || '<span>')
end
text[i .. -1] = text[i .. -1].gsub("\n", "#{close}\n#{reopen}#{style}")
end
if style
@out << style << text << '</span>'
else
@out << text
end
end
# token groups, eg. strings
def begin_group kind
@out << (@span_for_kind[@last_opened ? [kind, *@opened] : kind] || '<span>')
@opened << kind
@last_opened = kind if @set_last_opened
end
def end_group kind
if $CODERAY_DEBUG && (@opened.empty? || @opened.last != kind)
warn 'Malformed token stream: Trying to close a token (%p) ' \
'that is not open. Open are: %p.' % [kind, @opened[1..-1]]
end
if @opened.pop
@out << '</span>'
@last_opened = @opened.last if @last_opened
end
end
# whole lines to be highlighted, eg. a deleted line in a diff
def begin_line kind
if style = @span_for_kind[@last_opened ? [kind, *@opened] : kind]
if style['class="']
@out << style.sub('class="', 'class="line ')
else
@out << style.sub('>', ' class="line">')
end
else
@out << '<span class="line">'
end
@opened << kind
@last_opened = kind if @options[:css] == :style
end
def end_line kind
if $CODERAY_DEBUG && (@opened.empty? || @opened.last != kind)
warn 'Malformed token stream: Trying to close a line (%p) ' \
'that is not open. Open are: %p.' % [kind, @opened[1..-1]]
end
if @opened.pop
@out << '</span>'
@last_opened = @opened.last if @last_opened
end
end
end
end
end

View File

@ -1,65 +0,0 @@
module CodeRay
module Encoders
class HTML
class CSS # :nodoc:
attr :stylesheet
def CSS.load_stylesheet style = nil
CodeRay::Styles[style]
end
def initialize style = :default
@classes = Hash.new
style = CSS.load_stylesheet style
@stylesheet = [
style::CSS_MAIN_STYLES,
style::TOKEN_COLORS.gsub(/^(?!$)/, '.CodeRay ')
].join("\n")
parse style::TOKEN_COLORS
end
def get_style styles
cl = @classes[styles.first]
return '' unless cl
style = ''
1.upto styles.size do |offset|
break if style = cl[styles[offset .. -1]]
end
# warn 'Style not found: %p' % [styles] if style.empty?
return style
end
private
CSS_CLASS_PATTERN = /
( # $1 = selectors
(?:
(?: \s* \. [-\w]+ )+
\s* ,?
)+
)
\s* \{ \s*
( [^\}]+ )? # $2 = style
\s* \} \s*
|
( [^\n]+ ) # $3 = error
/mx
def parse stylesheet
stylesheet.scan CSS_CLASS_PATTERN do |selectors, style, error|
raise "CSS parse error: '#{error.inspect}' not recognized" if error
for selector in selectors.split(',')
classes = selector.scan(/[-\w]+/)
cl = classes.pop
@classes[cl] ||= Hash.new
@classes[cl][classes] = style.to_s.strip.delete(' ').chomp(';')
end
end
end
end
end
end
end

View File

@ -1,103 +0,0 @@
module CodeRay
module Encoders
class HTML
module Numbering # :nodoc:
def self.number! output, mode = :table, options = {}
return self unless mode
options = DEFAULT_OPTIONS.merge options
start = options[:line_number_start]
unless start.is_a? Integer
raise ArgumentError, "Invalid value %p for :line_number_start; Integer expected." % start
end
anchor_prefix = options[:line_number_anchors]
anchor_prefix = 'line' if anchor_prefix == true
anchor_prefix = anchor_prefix.to_s[/\w+/] if anchor_prefix
anchoring =
if anchor_prefix
proc do |line|
line = line.to_s
anchor = anchor_prefix + line
"<a href=\"##{anchor}\" name=\"#{anchor}\">#{line}</a>"
end
else
proc { |line| line.to_s } # :to_s.to_proc in Ruby 1.8.7+
end
bold_every = options[:bold_every]
highlight_lines = options[:highlight_lines]
bolding =
if bold_every == false && highlight_lines == nil
anchoring
elsif highlight_lines.is_a? Enumerable
highlight_lines = highlight_lines.to_set
proc do |line|
if highlight_lines.include? line
"<strong class=\"highlighted\">#{anchoring[line]}</strong>" # highlighted line numbers in bold
else
anchoring[line]
end
end
elsif bold_every.is_a? Integer
raise ArgumentError, ":bolding can't be 0." if bold_every == 0
proc do |line|
if line % bold_every == 0
"<strong>#{anchoring[line]}</strong>" # every bold_every-th number in bold
else
anchoring[line]
end
end
else
raise ArgumentError, 'Invalid value %p for :bolding; false or Integer expected.' % bold_every
end
line_count = output.count("\n")
position_of_last_newline = output.rindex(RUBY_VERSION >= '1.9' ? /\n/ : ?\n)
if position_of_last_newline
after_last_newline = output[position_of_last_newline + 1 .. -1]
ends_with_newline = after_last_newline[/\A(?:<\/span>)*\z/]
line_count += 1 if not ends_with_newline
end
case mode
when :inline
max_width = (start + line_count).to_s.size
line_number = start
output.gsub!(/^.*$\n?/) do |line|
line_number_text = bolding.call line_number
indent = ' ' * (max_width - line_number.to_s.size) # TODO: Optimize (10^x)
line_number += 1
"<span class=\"line-numbers\">#{indent}#{line_number_text}</span>#{line}"
end
when :table
line_numbers = (start ... start + line_count).map(&bolding).join("\n")
line_numbers << "\n"
line_numbers_table_template = Output::TABLE.apply('LINE_NUMBERS', line_numbers)
output.gsub!(/<\/div>\n/, '</div>')
output.wrap_in! line_numbers_table_template
output.wrapped_in = :div
when :list
raise NotImplementedError, 'The :list option is no longer available. Use :table.'
else
raise ArgumentError, 'Unknown value %p for mode: expected one of %p' %
[mode, [:table, :inline]]
end
output
end
end
end
end
end

View File

@ -1,166 +0,0 @@
module CodeRay
module Encoders
class HTML
# This module is included in the output String of the HTML Encoder.
#
# It provides methods like wrap, div, page etc.
#
# Remember to use #clone instead of #dup to keep the modules the object was
# extended with.
#
# TODO: Rewrite this without monkey patching.
module Output
attr_accessor :css
class << self
# Raises an exception if an object that doesn't respond to to_str is extended by Output,
# to prevent users from misuse. Use Module#remove_method to disable.
def extended o # :nodoc:
warn "The Output module is intended to extend instances of String, not #{o.class}." unless o.respond_to? :to_str
end
def make_stylesheet css, in_tag = false # :nodoc:
sheet = css.stylesheet
sheet = <<-'CSS' if in_tag
<style type="text/css">
#{sheet}
</style>
CSS
sheet
end
def page_template_for_css css # :nodoc:
sheet = make_stylesheet css
PAGE.apply 'CSS', sheet
end
end
def wrapped_in? element
wrapped_in == element
end
def wrapped_in
@wrapped_in ||= nil
end
attr_writer :wrapped_in
def wrap_in! template
Template.wrap! self, template, 'CONTENT'
self
end
def apply_title! title
self.sub!(/(<title>)(<\/title>)/) { $1 + title + $2 }
self
end
def wrap! element, *args
return self if not element or element == wrapped_in
case element
when :div
raise "Can't wrap %p in %p" % [wrapped_in, element] unless wrapped_in? nil
wrap_in! DIV
when :span
raise "Can't wrap %p in %p" % [wrapped_in, element] unless wrapped_in? nil
wrap_in! SPAN
when :page
wrap! :div if wrapped_in? nil
raise "Can't wrap %p in %p" % [wrapped_in, element] unless wrapped_in? :div
wrap_in! Output.page_template_for_css(@css)
if args.first.is_a?(Hash) && title = args.first[:title]
apply_title! title
end
self
when nil
return self
else
raise "Unknown value %p for :wrap" % element
end
@wrapped_in = element
self
end
def stylesheet in_tag = false
Output.make_stylesheet @css, in_tag
end
#-- don't include the templates in docu
class Template < String # :nodoc:
def self.wrap! str, template, target
target = Regexp.new(Regexp.escape("<%#{target}%>"))
if template =~ target
str[0,0] = $`
str << $'
else
raise "Template target <%%%p%%> not found" % target
end
end
def apply target, replacement
target = Regexp.new(Regexp.escape("<%#{target}%>"))
if self =~ target
Template.new($` + replacement + $')
else
raise "Template target <%%%p%%> not found" % target
end
end
end
SPAN = Template.new '<span class="CodeRay"><%CONTENT%></span>'
DIV = Template.new <<-DIV
<div class="CodeRay">
<div class="code"><pre><%CONTENT%></pre></div>
</div>
DIV
TABLE = Template.new <<-TABLE
<table class="CodeRay"><tr>
<td class="line-numbers" title="double click to toggle" ondblclick="with (this.firstChild.style) { display = (display == '') ? 'none' : '' }"><pre><%LINE_NUMBERS%></pre></td>
<td class="code"><pre><%CONTENT%></pre></td>
</tr></table>
TABLE
PAGE = Template.new <<-PAGE
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title></title>
<style type="text/css">
.CodeRay .line-numbers a {
text-decoration: inherit;
color: inherit;
}
body {
background-color: white;
padding: 0;
margin: 0;
}
<%CSS%>
.CodeRay {
border: none;
}
</style>
</head>
<body>
<%CONTENT%>
</body>
</html>
PAGE
end
end
end
end

View File

@ -1,83 +0,0 @@
module CodeRay
module Encoders
# A simple JSON Encoder.
#
# Example:
# CodeRay.scan('puts "Hello world!"', :ruby).json
# yields
# [
# {"type"=>"text", "text"=>"puts", "kind"=>"ident"},
# {"type"=>"text", "text"=>" ", "kind"=>"space"},
# {"type"=>"block", "action"=>"open", "kind"=>"string"},
# {"type"=>"text", "text"=>"\"", "kind"=>"delimiter"},
# {"type"=>"text", "text"=>"Hello world!", "kind"=>"content"},
# {"type"=>"text", "text"=>"\"", "kind"=>"delimiter"},
# {"type"=>"block", "action"=>"close", "kind"=>"string"},
# ]
class JSON < Encoder
begin
require 'json'
rescue LoadError
begin
require 'rubygems' unless defined? Gem
gem 'json'
require 'json'
rescue LoadError
$stderr.puts "The JSON encoder needs the JSON library.\n" \
"Please gem install json."
raise
end
end
register_for :json
FILE_EXTENSION = 'json'
protected
def setup options
super
@first = true
@out << '['
end
def finish options
@out << ']'
end
def append data
if @first
@first = false
else
@out << ','
end
@out << data.to_json
end
public
def text_token text, kind
append :type => 'text', :text => text, :kind => kind
end
def begin_group kind
append :type => 'block', :action => 'open', :kind => kind
end
def end_group kind
append :type => 'block', :action => 'close', :kind => kind
end
def begin_line kind
append :type => 'block', :action => 'begin_line', :kind => kind
end
def end_line kind
append :type => 'block', :action => 'end_line', :kind => kind
end
end
end
end

View File

@ -1,45 +0,0 @@
module CodeRay
module Encoders
# Counts the LoC (Lines of Code). Returns an Integer >= 0.
#
# Alias: +loc+
#
# Everything that is not comment, markup, doctype/shebang, or an empty line,
# is considered to be code.
#
# For example,
# * HTML files not containing JavaScript have 0 LoC
# * in a Java class without comments, LoC is the number of non-empty lines
#
# A Scanner class should define the token kinds that are not code in the
# KINDS_NOT_LOC constant, which defaults to [:comment, :doctype].
class LinesOfCode < TokenKindFilter
register_for :lines_of_code
NON_EMPTY_LINE = /^\s*\S.*$/
protected
def setup options
if scanner
kinds_not_loc = scanner.class::KINDS_NOT_LOC
else
warn "Tokens have no associated scanner, counting all nonempty lines." if $VERBOSE
kinds_not_loc = CodeRay::Scanners::Scanner::KINDS_NOT_LOC
end
options[:exclude] = kinds_not_loc
super options
end
def finish options
output @tokens.text.scan(NON_EMPTY_LINE).size
end
end
end
end

View File

@ -1,18 +0,0 @@
module CodeRay
module Encoders
# = Null Encoder
#
# Does nothing and returns an empty string.
class Null < Encoder
register_for :null
def text_token text, kind
# do nothing
end
end
end
end

View File

@ -1,24 +0,0 @@
module CodeRay
module Encoders
load :html
# Wraps the output into a HTML page, using CSS classes and
# line numbers in the table format by default.
#
# See Encoders::HTML for available options.
class Page < HTML
FILE_EXTENSION = 'html'
register_for :page
DEFAULT_OPTIONS = HTML::DEFAULT_OPTIONS.merge \
:css => :class,
:wrap => :page,
:line_numbers => :table
end
end
end

View File

@ -1,23 +0,0 @@
module CodeRay
module Encoders
load :html
# Wraps HTML output into a SPAN element, using inline styles by default.
#
# See Encoders::HTML for available options.
class Span < HTML
FILE_EXTENSION = 'span.html'
register_for :span
DEFAULT_OPTIONS = HTML::DEFAULT_OPTIONS.merge \
:css => :style,
:wrap => :span,
:line_numbers => false
end
end
end

View File

@ -1,96 +0,0 @@
module CodeRay
module Encoders
# Makes a statistic for the given tokens.
#
# Alias: +stats+
class Statistic < Encoder
register_for :statistic
attr_reader :type_stats, :real_token_count # :nodoc:
TypeStats = Struct.new :count, :size # :nodoc:
protected
def setup options
super
@type_stats = Hash.new { |h, k| h[k] = TypeStats.new 0, 0 }
@real_token_count = 0
end
STATS = <<-STATS # :nodoc:
Code Statistics
Tokens %8d
Non-Whitespace %8d
Bytes Total %8d
Token Types (%d):
type count ratio size (average)
-------------------------------------------------------------
%s
STATS
TOKEN_TYPES_ROW = <<-TKR # :nodoc:
%-20s %8d %6.2f %% %5.1f
TKR
def finish options
all = @type_stats['TOTAL']
all_count, all_size = all.count, all.size
@type_stats.each do |type, stat|
stat.size /= stat.count.to_f
end
types_stats = @type_stats.sort_by { |k, v| [-v.count, k.to_s] }.map do |k, v|
TOKEN_TYPES_ROW % [k, v.count, 100.0 * v.count / all_count, v.size]
end.join
@out << STATS % [
all_count, @real_token_count, all_size,
@type_stats.delete_if { |k, v| k.is_a? String }.size,
types_stats
]
super
end
public
def text_token text, kind
@real_token_count += 1 unless kind == :space
@type_stats[kind].count += 1
@type_stats[kind].size += text.size
@type_stats['TOTAL'].size += text.size
@type_stats['TOTAL'].count += 1
end
# TODO Hierarchy handling
def begin_group kind
block_token ':begin_group', kind
end
def end_group kind
block_token ':end_group', kind
end
def begin_line kind
block_token ':begin_line', kind
end
def end_line kind
block_token ':end_line', kind
end
def block_token action, kind
@type_stats['TOTAL'].count += 1
@type_stats[action].count += 1
@type_stats[kind].count += 1
end
end
end
end

View File

@ -1,179 +0,0 @@
module CodeRay
module Encoders
# Outputs code highlighted for a color terminal.
#
# Note: This encoder is in beta. It currently doesn't use the Styles.
#
# Alias: +term+
#
# == Authors & License
#
# By Rob Aldred (http://robaldred.co.uk)
#
# Based on idea by Nathan Weizenbaum (http://nex-3.com)
#
# MIT License (http://www.opensource.org/licenses/mit-license.php)
class Terminal < Encoder
register_for :terminal
TOKEN_COLORS = {
:annotation => '35',
:attribute_name => '33',
:attribute_value => '31',
:binary => '1;35',
:char => {
:self => '36', :delimiter => '1;34'
},
:class => '1;35',
:class_variable => '36',
:color => '32',
:comment => '37',
:complex => '1;34',
:constant => ['1;34', '4'],
:decoration => '35',
:definition => '1;32',
:directive => ['32', '4'],
:doc => '46',
:doctype => '1;30',
:doc_string => ['31', '4'],
:entity => '33',
:error => ['1;33', '41'],
:exception => '1;31',
:float => '1;35',
:function => '1;34',
:global_variable => '42',
:hex => '1;36',
:include => '33',
:integer => '1;34',
:key => '35',
:label => '1;15',
:local_variable => '33',
:octal => '1;35',
:operator_name => '1;29',
:predefined_constant => '1;36',
:predefined_type => '1;30',
:predefined => ['4', '1;34'],
:preprocessor => '36',
:pseudo_class => '1;34',
:regexp => {
:self => '31',
:content => '31',
:delimiter => '1;29',
:modifier => '35',
:function => '1;29'
},
:reserved => '1;31',
:shell => {
:self => '42',
:content => '1;29',
:delimiter => '37',
},
:string => {
:self => '32',
:modifier => '1;32',
:escape => '1;36',
:delimiter => '1;32',
},
:symbol => '1;32',
:tag => '1;34',
:type => '1;34',
:value => '36',
:variable => '1;34',
:insert => '42',
:delete => '41',
:change => '44',
:head => '45'
}
TOKEN_COLORS[:keyword] = TOKEN_COLORS[:reserved]
TOKEN_COLORS[:method] = TOKEN_COLORS[:function]
TOKEN_COLORS[:imaginary] = TOKEN_COLORS[:complex]
TOKEN_COLORS[:begin_group] = TOKEN_COLORS[:end_group] =
TOKEN_COLORS[:escape] = TOKEN_COLORS[:delimiter]
protected
def setup(options)
super
@opened = []
@subcolors = nil
end
public
def text_token text, kind
if color = (@subcolors || TOKEN_COLORS)[kind]
if Hash === color
if color[:self]
color = color[:self]
else
@out << text
return
end
end
@out << ansi_colorize(color)
@out << text.gsub("\n", ansi_clear + "\n" + ansi_colorize(color))
@out << ansi_clear
@out << ansi_colorize(@subcolors[:self]) if @subcolors && @subcolors[:self]
else
@out << text
end
end
def begin_group kind
@opened << kind
@out << open_token(kind)
end
alias begin_line begin_group
def end_group kind
if @opened.empty?
# nothing to close
else
@opened.pop
@out << ansi_clear
@out << open_token(@opened.last)
end
end
def end_line kind
if @opened.empty?
# nothing to close
else
@opened.pop
# whole lines to be highlighted,
# eg. added/modified/deleted lines in a diff
@out << "\t" * 100 + ansi_clear
@out << open_token(@opened.last)
end
end
private
def open_token kind
if color = TOKEN_COLORS[kind]
if Hash === color
@subcolors = color
ansi_colorize(color[:self]) if color[:self]
else
@subcolors = {}
ansi_colorize(color)
end
else
@subcolors = nil
''
end
end
def ansi_colorize(color)
Array(color).map { |c| "\e[#{c}m" }.join
end
def ansi_clear
ansi_colorize(0)
end
end
end
end

View File

@ -1,46 +0,0 @@
module CodeRay
module Encoders
# Concats the tokens into a single string, resulting in the original
# code string if no tokens were removed.
#
# Alias: +plain+, +plaintext+
#
# == Options
#
# === :separator
# A separator string to join the tokens.
#
# Default: empty String
class Text < Encoder
register_for :text
FILE_EXTENSION = 'txt'
DEFAULT_OPTIONS = {
:separator => nil
}
def text_token text, kind
super
if @first
@first = false
else
@out << @sep
end if @sep
end
protected
def setup options
super
@first = true
@sep = options[:separator]
end
end
end
end

View File

@ -1,111 +0,0 @@
module CodeRay
module Encoders
load :filter
# A Filter that selects tokens based on their token kind.
#
# == Options
#
# === :exclude
#
# One or many symbols (in an Array) which shall be excluded.
#
# Default: []
#
# === :include
#
# One or many symbols (in an array) which shall be included.
#
# Default: :all, which means all tokens are included.
#
# Exclusion wins over inclusion.
#
# See also: CommentFilter
class TokenKindFilter < Filter
register_for :token_kind_filter
DEFAULT_OPTIONS = {
:exclude => [],
:include => :all
}
protected
def setup options
super
@group_excluded = false
@exclude = options[:exclude]
@exclude = Array(@exclude) unless @exclude == :all
@include = options[:include]
@include = Array(@include) unless @include == :all
end
def include_text_token? text, kind
include_group? kind
end
def include_group? kind
(@include == :all || @include.include?(kind)) &&
!(@exclude == :all || @exclude.include?(kind))
end
public
# Add the token to the output stream if +kind+ matches the conditions.
def text_token text, kind
super if !@group_excluded && include_text_token?(text, kind)
end
# Add the token group to the output stream if +kind+ matches the
# conditions.
#
# If it does not, all tokens inside the group are excluded from the
# stream, even if their kinds match.
def begin_group kind
if @group_excluded
@group_excluded += 1
elsif include_group? kind
super
else
@group_excluded = 1
end
end
# See +begin_group+.
def begin_line kind
if @group_excluded
@group_excluded += 1
elsif include_group? kind
super
else
@group_excluded = 1
end
end
# Take care of re-enabling the delegation of tokens to the output stream
# if an exluded group has ended.
def end_group kind
if @group_excluded
@group_excluded -= 1
@group_excluded = false if @group_excluded.zero?
else
super
end
end
# See +end_group+.
def end_line kind
if @group_excluded
@group_excluded -= 1
@group_excluded = false if @group_excluded.zero?
else
super
end
end
end
end
end

View File

@ -1,72 +0,0 @@
module CodeRay
module Encoders
# = XML Encoder
#
# Uses REXML. Very slow.
class XML < Encoder
register_for :xml
FILE_EXTENSION = 'xml'
autoload :REXML, 'rexml/document'
DEFAULT_OPTIONS = {
:tab_width => 8,
:pretty => -1,
:transitive => false,
}
protected
def setup options
super
@doc = REXML::Document.new
@doc << REXML::XMLDecl.new
@tab_width = options[:tab_width]
@root = @node = @doc.add_element('coderay-tokens')
end
def finish options
@doc.write @out, options[:pretty], options[:transitive], true
super
end
public
def text_token text, kind
if kind == :space
token = @node
else
token = @node.add_element kind.to_s
end
text.scan(/(\x20+)|(\t+)|(\n)|[^\x20\t\n]+/) do |space, tab, nl|
case
when space
token << REXML::Text.new(space, true)
when tab
token << REXML::Text.new(tab, true)
when nl
token << REXML::Text.new(nl, true)
else
token << REXML::Text.new($&)
end
end
end
def begin_group kind
@node = @node.add_element kind.to_s
end
def end_group kind
if @node == @root
raise 'no token to close!'
end
@node = @node.parent
end
end
end
end

View File

@ -1,50 +0,0 @@
autoload :YAML, 'yaml'
module CodeRay
module Encoders
# = YAML Encoder
#
# Slow.
class YAML < Encoder
register_for :yaml
FILE_EXTENSION = 'yaml'
protected
def setup options
super
@data = []
end
def finish options
output ::YAML.dump(@data)
end
public
def text_token text, kind
@data << [text, kind]
end
def begin_group kind
@data << [:begin_group, kind]
end
def end_group kind
@data << [:end_group, kind]
end
def begin_line kind
@data << [:begin_line, kind]
end
def end_line kind
@data << [:end_line, kind]
end
end
end
end

View File

@ -1,95 +0,0 @@
module CodeRay
# A little hack to enable CodeRay highlighting in RedCloth.
#
# Usage:
# require 'coderay'
# require 'coderay/for_redcloth'
# RedCloth.new('@[ruby]puts "Hello, World!"@').to_html
#
# Make sure you have RedCloth 4.0.3 activated, for example by calling
# require 'rubygems'
# before RedCloth is loaded and before calling CodeRay.for_redcloth.
module ForRedCloth
def self.install
gem 'RedCloth', '>= 4.0.3' if defined? gem
require 'redcloth'
unless RedCloth::VERSION.to_s >= '4.0.3'
if defined? gem
raise 'CodeRay.for_redcloth needs RedCloth version 4.0.3 or later. ' +
"You have #{RedCloth::VERSION}. Please gem install RedCloth."
else
$".delete 'redcloth.rb' # sorry, but it works
require 'rubygems'
return install # retry
end
end
unless RedCloth::VERSION.to_s >= '4.2.2'
warn 'CodeRay.for_redcloth works best with RedCloth version 4.2.2 or later.'
end
RedCloth::TextileDoc.send :include, ForRedCloth::TextileDoc
RedCloth::Formatters::HTML.module_eval do
def unescape(html) # :nodoc:
replacements = {
'&amp;' => '&',
'&quot;' => '"',
'&gt;' => '>',
'&lt;' => '<',
}
html.gsub(/&(?:amp|quot|[gl]t);/) { |entity| replacements[entity] }
end
undef code, bc_open, bc_close, escape_pre
def code(opts) # :nodoc:
opts[:block] = true
if !opts[:lang] && RedCloth::VERSION.to_s >= '4.2.0'
# simulating pre-4.2 behavior
if opts[:text].sub!(/\A\[(\w+)\]/, '')
if CodeRay::Scanners[$1].lang == :text
opts[:text] = $& + opts[:text]
else
opts[:lang] = $1
end
end
end
if opts[:lang] && !filter_coderay
require 'coderay'
@in_bc ||= nil
format = @in_bc ? :div : :span
opts[:text] = unescape(opts[:text]) unless @in_bc
highlighted_code = CodeRay.encode opts[:text], opts[:lang], format
highlighted_code.sub!(/\A<(span|div)/) { |m| m + pba(@in_bc || opts) }
highlighted_code
else
"<code#{pba(opts)}>#{opts[:text]}</code>"
end
end
def bc_open(opts) # :nodoc:
opts[:block] = true
@in_bc = opts
opts[:lang] ? '' : "<pre#{pba(opts)}>"
end
def bc_close(opts) # :nodoc:
opts = @in_bc
@in_bc = nil
opts[:lang] ? '' : "</pre>\n"
end
def escape_pre(text) # :nodoc:
if @in_bc ||= nil
text
else
html_esc(text, :html_escape_preformatted)
end
end
end
end
module TextileDoc # :nodoc:
attr_accessor :filter_coderay
end
end
end
CodeRay::ForRedCloth.install

View File

@ -1,143 +0,0 @@
module CodeRay
# = FileType
#
# A simple filetype recognizer.
#
# == Usage
#
# # determine the type of the given
# lang = FileType[file_name]
#
# # return :text if the file type is unknown
# lang = FileType.fetch file_name, :text
#
# # try the shebang line, too
# lang = FileType.fetch file_name, :text, true
module FileType
UnknownFileType = Class.new Exception
class << self
# Try to determine the file type of the file.
#
# +filename+ is a relative or absolute path to a file.
#
# The file itself is only accessed when +read_shebang+ is set to true.
# That means you can get filetypes from files that don't exist.
def [] filename, read_shebang = false
name = File.basename filename
ext = File.extname(name).sub(/^\./, '') # from last dot, delete the leading dot
ext2 = filename.to_s[/\.(.*)/, 1] # from first dot
type =
TypeFromExt[ext] ||
TypeFromExt[ext.downcase] ||
(TypeFromExt[ext2] if ext2) ||
(TypeFromExt[ext2.downcase] if ext2) ||
TypeFromName[name] ||
TypeFromName[name.downcase]
type ||= shebang(filename) if read_shebang
type
end
# This works like Hash#fetch.
#
# If the filetype cannot be found, the +default+ value
# is returned.
def fetch filename, default = nil, read_shebang = false
if default && block_given?
warn 'Block supersedes default value argument; use either.'
end
if type = self[filename, read_shebang]
type
else
return yield if block_given?
return default if default
raise UnknownFileType, 'Could not determine type of %p.' % filename
end
end
protected
def shebang filename
return unless File.exist? filename
File.open filename, 'r' do |f|
if first_line = f.gets
if type = first_line[TypeFromShebang]
type.to_sym
end
end
end
end
end
TypeFromExt = {
'c' => :c,
'cfc' => :xml,
'cfm' => :xml,
'clj' => :clojure,
'css' => :css,
'diff' => :diff,
'dpr' => :delphi,
'erb' => :erb,
'gemspec' => :ruby,
'groovy' => :groovy,
'gvy' => :groovy,
'h' => :c,
'haml' => :haml,
'htm' => :html,
'html' => :html,
'html.erb' => :erb,
'java' => :java,
'js' => :java_script,
'json' => :json,
'mab' => :ruby,
'pas' => :delphi,
'patch' => :diff,
'php' => :php,
'php3' => :php,
'php4' => :php,
'php5' => :php,
'prawn' => :ruby,
'py' => :python,
'py3' => :python,
'pyw' => :python,
'rake' => :ruby,
'raydebug' => :raydebug,
'rb' => :ruby,
'rbw' => :ruby,
'rhtml' => :erb,
'rjs' => :ruby,
'rpdf' => :ruby,
'ru' => :ruby,
'rxml' => :ruby,
# 'sch' => :scheme,
'sql' => :sql,
# 'ss' => :scheme,
'tmproj' => :xml,
'xhtml' => :html,
'xml' => :xml,
'yaml' => :yaml,
'yml' => :yaml,
}
for cpp_alias in %w[cc cpp cp cxx c++ C hh hpp h++ cu]
TypeFromExt[cpp_alias] = :cpp
end
TypeFromShebang = /\b(?:ruby|perl|python|sh)\b/
TypeFromName = {
'Capfile' => :ruby,
'Rakefile' => :ruby,
'Rantfile' => :ruby,
'Gemfile' => :ruby,
}
end
end

View File

@ -1,41 +0,0 @@
module CodeRay
# A simplified interface to the gzip library +zlib+ (from the Ruby Standard Library.)
module GZip
require 'zlib'
# The default zipping level. 7 zips good and fast.
DEFAULT_GZIP_LEVEL = 7
# Unzips the given string +s+.
#
# Example:
# require 'gzip_simple'
# print GZip.gunzip(File.read('adresses.gz'))
def GZip.gunzip s
Zlib::Inflate.inflate s
end
# Zips the given string +s+.
#
# Example:
# require 'gzip_simple'
# File.open('adresses.gz', 'w') do |file
# file.write GZip.gzip('Mum: 0123 456 789', 9)
# end
#
# If you provide a +level+, you can control how strong
# the string is compressed:
# - 0: no compression, only convert to gzip format
# - 1: compress fast
# - 7: compress more, but still fast (default)
# - 8: compress more, slower
# - 9: compress best, very slow
def GZip.gzip s, level = DEFAULT_GZIP_LEVEL
Zlib::Deflate.new(level).deflate s, Zlib::FINISH
end
end
end

View File

@ -1,284 +0,0 @@
module CodeRay
# = PluginHost
#
# A simple subclass/subfolder plugin system.
#
# Example:
# class Generators
# extend PluginHost
# plugin_path 'app/generators'
# end
#
# class Generator
# extend Plugin
# PLUGIN_HOST = Generators
# end
#
# class FancyGenerator < Generator
# register_for :fancy
# end
#
# Generators[:fancy] #-> FancyGenerator
# # or
# CodeRay.require_plugin 'Generators/fancy'
# # or
# Generators::Fancy
module PluginHost
# Raised if Encoders::[] fails because:
# * a file could not be found
# * the requested Plugin is not registered
PluginNotFound = Class.new LoadError
HostNotFound = Class.new LoadError
PLUGIN_HOSTS = []
PLUGIN_HOSTS_BY_ID = {} # dummy hash
# Loads all plugins using list and load.
def load_all
for plugin in list
load plugin
end
end
# Returns the Plugin for +id+.
#
# Example:
# yaml_plugin = MyPluginHost[:yaml]
def [] id, *args, &blk
plugin = validate_id(id)
begin
plugin = plugin_hash.[] plugin, *args, &blk
end while plugin.is_a? Symbol
plugin
end
alias load []
# Tries to +load+ the missing plugin by translating +const+ to the
# underscore form (eg. LinesOfCode becomes lines_of_code).
def const_missing const
id = const.to_s.
gsub(/([A-Z]+)([A-Z][a-z])/,'\1_\2').
gsub(/([a-z\d])([A-Z])/,'\1_\2').
downcase
load id
end
class << self
# Adds the module/class to the PLUGIN_HOSTS list.
def extended mod
PLUGIN_HOSTS << mod
end
end
# The path where the plugins can be found.
def plugin_path *args
unless args.empty?
@plugin_path = File.expand_path File.join(*args)
end
@plugin_path ||= ''
end
# Map a plugin_id to another.
#
# Usage: Put this in a file plugin_path/_map.rb.
#
# class MyColorHost < PluginHost
# map :navy => :dark_blue,
# :maroon => :brown,
# :luna => :moon
# end
def map hash
for from, to in hash
from = validate_id from
to = validate_id to
plugin_hash[from] = to unless plugin_hash.has_key? from
end
end
# Define the default plugin to use when no plugin is found
# for a given id, or return the default plugin.
#
# See also map.
#
# class MyColorHost < PluginHost
# map :navy => :dark_blue
# default :gray
# end
#
# MyColorHost.default # loads and returns the Gray plugin
def default id = nil
if id
id = validate_id id
raise "The default plugin can't be named \"default\"." if id == :default
plugin_hash[:default] = id
else
load :default
end
end
# Every plugin must register itself for +id+ by calling register_for,
# which calls this method.
#
# See Plugin#register_for.
def register plugin, id
plugin_hash[validate_id(id)] = plugin
end
# A Hash of plugion_id => Plugin pairs.
def plugin_hash
@plugin_hash ||= make_plugin_hash
end
# Returns an array of all .rb files in the plugin path.
#
# The extension .rb is not included.
def list
Dir[path_to('*')].select do |file|
File.basename(file)[/^(?!_)\w+\.rb$/]
end.map do |file|
File.basename(file, '.rb').to_sym
end
end
# Returns an array of all Plugins.
#
# Note: This loads all plugins using load_all.
def all_plugins
load_all
plugin_hash.values.grep(Class)
end
# Loads the map file (see map).
#
# This is done automatically when plugin_path is called.
def load_plugin_map
mapfile = path_to '_map'
@plugin_map_loaded = true
if File.exist? mapfile
require mapfile
true
else
false
end
end
protected
# Return a plugin hash that automatically loads plugins.
def make_plugin_hash
@plugin_map_loaded ||= false
Hash.new do |h, plugin_id|
id = validate_id(plugin_id)
path = path_to id
begin
raise LoadError, "#{path} not found" unless File.exist? path
require path
rescue LoadError => boom
if @plugin_map_loaded
if h.has_key?(:default)
warn '%p could not load plugin %p; falling back to %p' % [self, id, h[:default]]
h[:default]
else
raise PluginNotFound, '%p could not load plugin %p: %s' % [self, id, boom]
end
else
load_plugin_map
h[plugin_id]
end
else
# Plugin should have registered by now
if h.has_key? id
h[id]
else
raise PluginNotFound, "No #{self.name} plugin for #{id.inspect} found in #{path}."
end
end
end
end
# Returns the expected path to the plugin file for the given id.
def path_to plugin_id
File.join plugin_path, "#{plugin_id}.rb"
end
# Converts +id+ to a Symbol if it is a String,
# or returns +id+ if it already is a Symbol.
#
# Raises +ArgumentError+ for all other objects, or if the
# given String includes non-alphanumeric characters (\W).
def validate_id id
if id.is_a? Symbol or id.nil?
id
elsif id.is_a? String
if id[/\w+/] == id
id.downcase.to_sym
else
raise ArgumentError, "Invalid id given: #{id}"
end
else
raise ArgumentError, "String or Symbol expected, but #{id.class} given."
end
end
end
# = Plugin
#
# Plugins have to include this module.
#
# IMPORTANT: Use extend for this module.
#
# See CodeRay::PluginHost for examples.
module Plugin
attr_reader :plugin_id
# Register this class for the given +id+.
#
# Example:
# class MyPlugin < PluginHost::BaseClass
# register_for :my_id
# ...
# end
#
# See PluginHost.register.
def register_for id
@plugin_id = id
plugin_host.register self, id
end
# Returns the title of the plugin, or sets it to the
# optional argument +title+.
def title title = nil
if title
@title = title.to_s
else
@title ||= name[/([^:]+)$/, 1]
end
end
# The PluginHost for this Plugin class.
def plugin_host host = nil
if host.is_a? PluginHost
const_set :PLUGIN_HOST, host
end
self::PLUGIN_HOST
end
def aliases
plugin_host.load_plugin_map
plugin_host.plugin_hash.inject [] do |aliases, (key, _)|
aliases << key if plugin_host[key] == self
aliases
end
end
end
end

View File

@ -1,77 +0,0 @@
module CodeRay
# = WordList
#
# <b>A Hash subclass designed for mapping word lists to token types.</b>
#
# Copyright (c) 2006-2011 by murphy (Kornelius Kalnbach) <murphy rubychan de>
#
# License:: LGPL / ask the author
# Version:: 2.0 (2011-05-08)
#
# A WordList is a Hash with some additional features.
# It is intended to be used for keyword recognition.
#
# WordList is optimized to be used in Scanners,
# typically to decide whether a given ident is a special token.
#
# For case insensitive words use WordList::CaseIgnoring.
#
# Example:
#
# # define word arrays
# RESERVED_WORDS = %w[
# asm break case continue default do else
# ]
#
# PREDEFINED_TYPES = %w[
# int long short char void
# ]
#
# # make a WordList
# IDENT_KIND = WordList.new(:ident).
# add(RESERVED_WORDS, :reserved).
# add(PREDEFINED_TYPES, :predefined_type)
#
# ...
#
# def scan_tokens tokens, options
# ...
#
# elsif scan(/[A-Za-z_][A-Za-z_0-9]*/)
# # use it
# kind = IDENT_KIND[match]
# ...
class WordList < Hash
# Create a new WordList with +default+ as default value.
def initialize default = false
super default
end
# Add words to the list and associate them with +value+.
#
# Returns +self+, so you can concat add calls.
def add words, value = true
words.each { |word| self[word] = value }
self
end
end
# A CaseIgnoring WordList is like a WordList, only that
# keys are compared case-insensitively (normalizing keys using +downcase+).
class WordList::CaseIgnoring < WordList
def [] key
super key.downcase
end
def []= key, value
super key.downcase, value
end
end
end

View File

@ -1,323 +0,0 @@
# encoding: utf-8
require 'strscan'
module CodeRay
autoload :WordList, coderay_path('helpers', 'word_list')
# = Scanners
#
# This module holds the Scanner class and its subclasses.
# For example, the Ruby scanner is named CodeRay::Scanners::Ruby
# can be found in coderay/scanners/ruby.
#
# Scanner also provides methods and constants for the register
# mechanism and the [] method that returns the Scanner class
# belonging to the given lang.
#
# See PluginHost.
module Scanners
extend PluginHost
plugin_path File.dirname(__FILE__), 'scanners'
# = Scanner
#
# The base class for all Scanners.
#
# It is a subclass of Ruby's great +StringScanner+, which
# makes it easy to access the scanning methods inside.
#
# It is also +Enumerable+, so you can use it like an Array of
# Tokens:
#
# require 'coderay'
#
# c_scanner = CodeRay::Scanners[:c].new "if (*p == '{') nest++;"
#
# for text, kind in c_scanner
# puts text if kind == :operator
# end
#
# # prints: (*==)++;
#
# OK, this is a very simple example :)
# You can also use +map+, +any?+, +find+ and even +sort_by+,
# if you want.
class Scanner < StringScanner
extend Plugin
plugin_host Scanners
# Raised if a Scanner fails while scanning
ScanError = Class.new StandardError
# The default options for all scanner classes.
#
# Define @default_options for subclasses.
DEFAULT_OPTIONS = { }
KINDS_NOT_LOC = [:comment, :doctype, :docstring]
attr_accessor :state
class << self
# Normalizes the given code into a string with UNIX newlines, in the
# scanner's internal encoding, with invalid and undefined charachters
# replaced by placeholders. Always returns a new object.
def normalize code
# original = code
code = code.to_s unless code.is_a? ::String
return code if code.empty?
if code.respond_to? :encoding
code = encode_with_encoding code, self.encoding
else
code = to_unix code
end
# code = code.dup if code.eql? original
code
end
# The typical filename suffix for this scanner's language.
def file_extension extension = lang
@file_extension ||= extension.to_s
end
# The encoding used internally by this scanner.
def encoding name = 'UTF-8'
@encoding ||= defined?(Encoding.find) && Encoding.find(name)
end
# The lang of this Scanner class, which is equal to its Plugin ID.
def lang
@plugin_id
end
protected
def encode_with_encoding code, target_encoding
if code.encoding == target_encoding
if code.valid_encoding?
return to_unix(code)
else
source_encoding = guess_encoding code
end
else
source_encoding = code.encoding
end
# print "encode_with_encoding from #{source_encoding} to #{target_encoding}"
code.encode target_encoding, source_encoding, :universal_newline => true, :undef => :replace, :invalid => :replace
end
def to_unix code
code.index(?\r) ? code.gsub(/\r\n?/, "\n") : code
end
def guess_encoding s
#:nocov:
IO.popen("file -b --mime -", "w+") do |file|
file.write s[0, 1024]
file.close_write
begin
Encoding.find file.gets[/charset=([-\w]+)/, 1]
rescue ArgumentError
Encoding::BINARY
end
end
#:nocov:
end
end
# Create a new Scanner.
#
# * +code+ is the input String and is handled by the superclass
# StringScanner.
# * +options+ is a Hash with Symbols as keys.
# It is merged with the default options of the class (you can
# overwrite default options here.)
#
# Else, a Tokens object is used.
def initialize code = '', options = {}
if self.class == Scanner
raise NotImplementedError, "I am only the basic Scanner class. I can't scan anything. :( Use my subclasses."
end
@options = self.class::DEFAULT_OPTIONS.merge options
super self.class.normalize(code)
@tokens = options[:tokens] || Tokens.new
@tokens.scanner = self if @tokens.respond_to? :scanner=
setup
end
# Sets back the scanner. Subclasses should redefine the reset_instance
# method instead of this one.
def reset
super
reset_instance
end
# Set a new string to be scanned.
def string= code
code = self.class.normalize(code)
super code
reset_instance
end
# the Plugin ID for this scanner
def lang
self.class.lang
end
# the default file extension for this scanner
def file_extension
self.class.file_extension
end
# Scan the code and returns all tokens in a Tokens object.
def tokenize source = nil, options = {}
options = @options.merge(options)
@tokens = options[:tokens] || @tokens || Tokens.new
@tokens.scanner = self if @tokens.respond_to? :scanner=
case source
when Array
self.string = self.class.normalize(source.join)
when nil
reset
else
self.string = self.class.normalize(source)
end
begin
scan_tokens @tokens, options
rescue => e
message = "Error in %s#scan_tokens, initial state was: %p" % [self.class, defined?(state) && state]
raise_inspect e.message, @tokens, message, 30, e.backtrace
end
@cached_tokens = @tokens
if source.is_a? Array
@tokens.split_into_parts(*source.map { |part| part.size })
else
@tokens
end
end
# Cache the result of tokenize.
def tokens
@cached_tokens ||= tokenize
end
# Traverse the tokens.
def each &block
tokens.each(&block)
end
include Enumerable
# The current line position of the scanner, starting with 1.
# See also: #column.
#
# Beware, this is implemented inefficiently. It should be used
# for debugging only.
def line pos = self.pos
return 1 if pos <= 0
binary_string[0...pos].count("\n") + 1
end
# The current column position of the scanner, starting with 1.
# See also: #line.
def column pos = self.pos
return 1 if pos <= 0
pos - (binary_string.rindex(?\n, pos - 1) || -1)
end
# The string in binary encoding.
#
# To be used with #pos, which is the index of the byte the scanner
# will scan next.
def binary_string
@binary_string ||=
if string.respond_to?(:bytesize) && string.bytesize != string.size
#:nocov:
string.dup.force_encoding('binary')
#:nocov:
else
string
end
end
protected
# Can be implemented by subclasses to do some initialization
# that has to be done once per instance.
#
# Use reset for initialization that has to be done once per
# scan.
def setup # :doc:
end
# This is the central method, and commonly the only one a
# subclass implements.
#
# Subclasses must implement this method; it must return +tokens+
# and must only use Tokens#<< for storing scanned tokens!
def scan_tokens tokens, options # :doc:
raise NotImplementedError, "#{self.class}#scan_tokens not implemented."
end
# Resets the scanner.
def reset_instance
@tokens.clear if @tokens.respond_to?(:clear) && !@options[:keep_tokens]
@cached_tokens = nil
@binary_string = nil if defined? @binary_string
end
# Scanner error with additional status information
def raise_inspect msg, tokens, state = self.state || 'No state given!', ambit = 30, backtrace = caller
raise ScanError, <<-EOE % [
***ERROR in %s: %s (after %d tokens)
tokens:
%s
current line: %d column: %d pos: %d
matched: %p state: %p
bol? = %p, eos? = %p
surrounding code:
%p ~~ %p
***ERROR***
EOE
File.basename(caller[0]),
msg,
tokens.respond_to?(:size) ? tokens.size : 0,
tokens.respond_to?(:last) ? tokens.last(10).map { |t| t.inspect }.join("\n") : '',
line, column, pos,
matched, state, bol?, eos?,
binary_string[pos - ambit, ambit],
binary_string[pos, ambit],
], backtrace
end
# Shorthand for scan_until(/\z/).
# This method also avoids a JRuby 1.9 mode bug.
def scan_rest
rest = self.rest
terminate
rest
end
end
end
end

View File

@ -1,24 +0,0 @@
module CodeRay
module Scanners
map \
:'c++' => :cpp,
:cplusplus => :cpp,
:ecmascript => :java_script,
:ecma_script => :java_script,
:rhtml => :erb,
:eruby => :erb,
:irb => :ruby,
:javascript => :java_script,
:js => :java_script,
:pascal => :delphi,
:patch => :diff,
:plain => :text,
:plaintext => :text,
:xhtml => :html,
:yml => :yaml
default :text
end
end

View File

@ -1,189 +0,0 @@
module CodeRay
module Scanners
# Scanner for C.
class C < Scanner
register_for :c
file_extension 'c'
KEYWORDS = [
'asm', 'break', 'case', 'continue', 'default', 'do',
'else', 'enum', 'for', 'goto', 'if', 'return',
'sizeof', 'struct', 'switch', 'typedef', 'union', 'while',
'restrict', # added in C99
] # :nodoc:
PREDEFINED_TYPES = [
'int', 'long', 'short', 'char',
'signed', 'unsigned', 'float', 'double',
'bool', 'complex', # added in C99
] # :nodoc:
PREDEFINED_CONSTANTS = [
'EOF', 'NULL',
'true', 'false', # added in C99
] # :nodoc:
DIRECTIVES = [
'auto', 'extern', 'register', 'static', 'void',
'const', 'volatile', # added in C89
'inline', # added in C99
] # :nodoc:
IDENT_KIND = WordList.new(:ident).
add(KEYWORDS, :keyword).
add(PREDEFINED_TYPES, :predefined_type).
add(DIRECTIVES, :directive).
add(PREDEFINED_CONSTANTS, :predefined_constant) # :nodoc:
ESCAPE = / [rbfntv\n\\'"] | x[a-fA-F0-9]{1,2} | [0-7]{1,3} /x # :nodoc:
UNICODE_ESCAPE = / u[a-fA-F0-9]{4} | U[a-fA-F0-9]{8} /x # :nodoc:
protected
def scan_tokens encoder, options
state = :initial
label_expected = true
case_expected = false
label_expected_before_preproc_line = nil
in_preproc_line = false
until eos?
case state
when :initial
if match = scan(/ \s+ | \\\n /x)
if in_preproc_line && match != "\\\n" && match.index(?\n)
in_preproc_line = false
label_expected = label_expected_before_preproc_line
end
encoder.text_token match, :space
elsif match = scan(%r! // [^\n\\]* (?: \\. [^\n\\]* )* | /\* (?: .*? \*/ | .* ) !mx)
encoder.text_token match, :comment
elsif match = scan(/ [-+*=<>?:;,!&^|()\[\]{}~%]+ | \/=? | \.(?!\d) /x)
label_expected = match =~ /[;\{\}]/
if case_expected
label_expected = true if match == ':'
case_expected = false
end
encoder.text_token match, :operator
elsif match = scan(/ [A-Za-z_][A-Za-z_0-9]* /x)
kind = IDENT_KIND[match]
if kind == :ident && label_expected && !in_preproc_line && scan(/:(?!:)/)
kind = :label
match << matched
else
label_expected = false
if kind == :keyword
case match
when 'case', 'default'
case_expected = true
end
end
end
encoder.text_token match, kind
elsif match = scan(/L?"/)
encoder.begin_group :string
if match[0] == ?L
encoder.text_token 'L', :modifier
match = '"'
end
encoder.text_token match, :delimiter
state = :string
elsif match = scan(/ \# \s* if \s* 0 /x)
match << scan_until(/ ^\# (?:elif|else|endif) .*? $ | \z /xm) unless eos?
encoder.text_token match, :comment
elsif match = scan(/#[ \t]*(\w*)/)
encoder.text_token match, :preprocessor
in_preproc_line = true
label_expected_before_preproc_line = label_expected
state = :include_expected if self[1] == 'include'
elsif match = scan(/ L?' (?: [^\'\n\\] | \\ #{ESCAPE} )? '? /ox)
label_expected = false
encoder.text_token match, :char
elsif match = scan(/\$/)
encoder.text_token match, :ident
elsif match = scan(/0[xX][0-9A-Fa-f]+/)
label_expected = false
encoder.text_token match, :hex
elsif match = scan(/(?:0[0-7]+)(?![89.eEfF])/)
label_expected = false
encoder.text_token match, :octal
elsif match = scan(/(?:\d+)(?![.eEfF])L?L?/)
label_expected = false
encoder.text_token match, :integer
elsif match = scan(/\d[fF]?|\d*\.\d+(?:[eE][+-]?\d+)?[fF]?|\d+[eE][+-]?\d+[fF]?/)
label_expected = false
encoder.text_token match, :float
else
encoder.text_token getch, :error
end
when :string
if match = scan(/[^\\\n"]+/)
encoder.text_token match, :content
elsif match = scan(/"/)
encoder.text_token match, :delimiter
encoder.end_group :string
state = :initial
label_expected = false
elsif match = scan(/ \\ (?: #{ESCAPE} | #{UNICODE_ESCAPE} ) /mox)
encoder.text_token match, :char
elsif match = scan(/ \\ | $ /x)
encoder.end_group :string
encoder.text_token match, :error
state = :initial
label_expected = false
else
raise_inspect "else case \" reached; %p not handled." % peek(1), encoder
end
when :include_expected
if match = scan(/<[^>\n]+>?|"[^"\n\\]*(?:\\.[^"\n\\]*)*"?/)
encoder.text_token match, :include
state = :initial
elsif match = scan(/\s+/)
encoder.text_token match, :space
state = :initial if match.index ?\n
else
state = :initial
end
else
raise_inspect 'Unknown state', encoder
end
end
if state == :string
encoder.end_group :string
end
encoder
end
end
end
end

View File

@ -1,217 +0,0 @@
# encoding: utf-8
module CodeRay
module Scanners
# Clojure scanner by Licenser.
class Clojure < Scanner
register_for :clojure
file_extension 'clj'
SPECIAL_FORMS = %w[
def if do let quote var fn loop recur throw try catch monitor-enter monitor-exit .
new
] # :nodoc:
CORE_FORMS = %w[
+ - -> ->> .. / * <= < = == >= > accessor aclone add-classpath add-watch
agent agent-error agent-errors aget alength alias all-ns alter alter-meta!
alter-var-root amap ancestors and apply areduce array-map aset aset-boolean
aset-byte aset-char aset-double aset-float aset-int aset-long aset-short
assert assoc assoc! assoc-in associative? atom await await-for bases bean
bigdec bigint binding bit-and bit-and-not bit-clear bit-flip bit-not bit-or
bit-set bit-shift-left bit-shift-right bit-test bit-xor boolean boolean-array
booleans bound-fn bound-fn* bound? butlast byte byte-array bytes case cast char
char-array char-escape-string char-name-string char? chars class class?
clear-agent-errors clojure-version coll? comment commute comp comparator
compare compare-and-set! compile complement concat cond condp conj conj!
cons constantly construct-proxy contains? count counted? create-ns
create-struct cycle dec decimal? declare definline defmacro defmethod defmulti
defn defn- defonce defprotocol defrecord defstruct deftype delay delay?
deliver denominator deref derive descendants disj disj! dissoc dissoc!
distinct distinct? doall doc dorun doseq dosync dotimes doto double
double-array doubles drop drop-last drop-while empty empty? ensure
enumeration-seq error-handler error-mode eval even? every? extend
extend-protocol extend-type extenders extends? false? ffirst file-seq
filter find find-doc find-ns find-var first float float-array float?
floats flush fn fn? fnext for force format future future-call future-cancel
future-cancelled? future-done? future? gen-class gen-interface gensym get
get-in get-method get-proxy-class get-thread-bindings get-validator hash
hash-map hash-set identical? identity if-let if-not ifn? import in-ns
inc init-proxy instance? int int-array integer? interleave intern
interpose into into-array ints io! isa? iterate iterator-seq juxt key
keys keyword keyword? last lazy-cat lazy-seq let letfn line-seq list list*
list? load load-file load-reader load-string loaded-libs locking long
long-array longs loop macroexpand macroexpand-1 make-array make-hierarchy
map map? mapcat max max-key memfn memoize merge merge-with meta methods
min min-key mod name namespace neg? newline next nfirst nil? nnext not
not-any? not-empty not-every? not= ns ns-aliases ns-imports ns-interns
ns-map ns-name ns-publics ns-refers ns-resolve ns-unalias ns-unmap nth
nthnext num number? numerator object-array odd? or parents partial
partition pcalls peek persistent! pmap pop pop! pop-thread-bindings
pos? pr pr-str prefer-method prefers print print-namespace-doc
print-str printf println println-str prn prn-str promise proxy
proxy-mappings proxy-super push-thread-bindings pvalues quot rand
rand-int range ratio? rationalize re-find re-groups re-matcher
re-matches re-pattern re-seq read read-line read-string reduce ref
ref-history-count ref-max-history ref-min-history ref-set refer
refer-clojure reify release-pending-sends rem remove remove-all-methods
remove-method remove-ns remove-watch repeat repeatedly replace replicate
require reset! reset-meta! resolve rest restart-agent resultset-seq
reverse reversible? rseq rsubseq satisfies? second select-keys send
send-off seq seq? seque sequence sequential? set set-error-handler!
set-error-mode! set-validator! set? short short-array shorts
shutdown-agents slurp some sort sort-by sorted-map sorted-map-by
sorted-set sorted-set-by sorted? special-form-anchor special-symbol?
split-at split-with str string? struct struct-map subs subseq subvec
supers swap! symbol symbol? sync syntax-symbol-anchor take take-last
take-nth take-while test the-ns thread-bound? time to-array to-array-2d
trampoline transient tree-seq true? type unchecked-add unchecked-dec
unchecked-divide unchecked-inc unchecked-multiply unchecked-negate
unchecked-remainder unchecked-subtract underive update-in update-proxy
use val vals var-get var-set var? vary-meta vec vector vector-of vector?
when when-first when-let when-not while with-bindings with-bindings*
with-in-str with-local-vars with-meta with-open with-out-str
with-precision xml-seq zero? zipmap
] # :nodoc:
PREDEFINED_CONSTANTS = %w[
true false nil *1 *2 *3 *agent* *clojure-version* *command-line-args*
*compile-files* *compile-path* *e *err* *file* *flush-on-newline*
*in* *ns* *out* *print-dup* *print-length* *print-level* *print-meta*
*print-readably* *read-eval* *warn-on-reflection*
] # :nodoc:
IDENT_KIND = WordList.new(:ident).
add(SPECIAL_FORMS, :keyword).
add(CORE_FORMS, :keyword).
add(PREDEFINED_CONSTANTS, :predefined_constant)
KEYWORD_NEXT_TOKEN_KIND = WordList.new(nil).
add(%w[ def defn defn- definline defmacro defmulti defmethod defstruct defonce declare ], :function).
add(%w[ ns ], :namespace).
add(%w[ defprotocol defrecord ], :class)
BASIC_IDENTIFIER = /[a-zA-Z$%*\/_+!?&<>\-=]=?[a-zA-Z0-9$&*+!\/_?<>\-\#]*/
IDENTIFIER = /(?!-\d)(?:(?:#{BASIC_IDENTIFIER}\.)*#{BASIC_IDENTIFIER}(?:\/#{BASIC_IDENTIFIER})?\.?)|\.\.?/
SYMBOL = /::?#{IDENTIFIER}/o
DIGIT = /\d/
DIGIT10 = DIGIT
DIGIT16 = /[0-9a-f]/i
DIGIT8 = /[0-7]/
DIGIT2 = /[01]/
RADIX16 = /\#x/i
RADIX8 = /\#o/i
RADIX2 = /\#b/i
RADIX10 = /\#d/i
EXACTNESS = /#i|#e/i
SIGN = /[\+-]?/
EXP_MARK = /[esfdl]/i
EXP = /#{EXP_MARK}#{SIGN}#{DIGIT}+/
SUFFIX = /#{EXP}?/
PREFIX10 = /#{RADIX10}?#{EXACTNESS}?|#{EXACTNESS}?#{RADIX10}?/
PREFIX16 = /#{RADIX16}#{EXACTNESS}?|#{EXACTNESS}?#{RADIX16}/
PREFIX8 = /#{RADIX8}#{EXACTNESS}?|#{EXACTNESS}?#{RADIX8}/
PREFIX2 = /#{RADIX2}#{EXACTNESS}?|#{EXACTNESS}?#{RADIX2}/
UINT10 = /#{DIGIT10}+#*/
UINT16 = /#{DIGIT16}+#*/
UINT8 = /#{DIGIT8}+#*/
UINT2 = /#{DIGIT2}+#*/
DECIMAL = /#{DIGIT10}+#+\.#*#{SUFFIX}|#{DIGIT10}+\.#{DIGIT10}*#*#{SUFFIX}|\.#{DIGIT10}+#*#{SUFFIX}|#{UINT10}#{EXP}/
UREAL10 = /#{UINT10}\/#{UINT10}|#{DECIMAL}|#{UINT10}/
UREAL16 = /#{UINT16}\/#{UINT16}|#{UINT16}/
UREAL8 = /#{UINT8}\/#{UINT8}|#{UINT8}/
UREAL2 = /#{UINT2}\/#{UINT2}|#{UINT2}/
REAL10 = /#{SIGN}#{UREAL10}/
REAL16 = /#{SIGN}#{UREAL16}/
REAL8 = /#{SIGN}#{UREAL8}/
REAL2 = /#{SIGN}#{UREAL2}/
IMAG10 = /i|#{UREAL10}i/
IMAG16 = /i|#{UREAL16}i/
IMAG8 = /i|#{UREAL8}i/
IMAG2 = /i|#{UREAL2}i/
COMPLEX10 = /#{REAL10}@#{REAL10}|#{REAL10}\+#{IMAG10}|#{REAL10}-#{IMAG10}|\+#{IMAG10}|-#{IMAG10}|#{REAL10}/
COMPLEX16 = /#{REAL16}@#{REAL16}|#{REAL16}\+#{IMAG16}|#{REAL16}-#{IMAG16}|\+#{IMAG16}|-#{IMAG16}|#{REAL16}/
COMPLEX8 = /#{REAL8}@#{REAL8}|#{REAL8}\+#{IMAG8}|#{REAL8}-#{IMAG8}|\+#{IMAG8}|-#{IMAG8}|#{REAL8}/
COMPLEX2 = /#{REAL2}@#{REAL2}|#{REAL2}\+#{IMAG2}|#{REAL2}-#{IMAG2}|\+#{IMAG2}|-#{IMAG2}|#{REAL2}/
NUM10 = /#{PREFIX10}?#{COMPLEX10}/
NUM16 = /#{PREFIX16}#{COMPLEX16}/
NUM8 = /#{PREFIX8}#{COMPLEX8}/
NUM2 = /#{PREFIX2}#{COMPLEX2}/
NUM = /#{NUM10}|#{NUM16}|#{NUM8}|#{NUM2}/
protected
def scan_tokens encoder, options
state = :initial
kind = nil
until eos?
case state
when :initial
if match = scan(/ \s+ | \\\n | , /x)
encoder.text_token match, :space
elsif match = scan(/['`\(\[\)\]\{\}]|\#[({]|~@?|[@\^]/)
encoder.text_token match, :operator
elsif match = scan(/;.*/)
encoder.text_token match, :comment # TODO: recognize (comment ...) too
elsif match = scan(/\#?\\(?:newline|space|.?)/)
encoder.text_token match, :char
elsif match = scan(/\#[ft]/)
encoder.text_token match, :predefined_constant
elsif match = scan(/#{IDENTIFIER}/o)
kind = IDENT_KIND[match]
encoder.text_token match, kind
if rest? && kind == :keyword
if kind = KEYWORD_NEXT_TOKEN_KIND[match]
encoder.text_token match, :space if match = scan(/\s+/o)
encoder.text_token match, kind if match = scan(/#{IDENTIFIER}/o)
end
end
elsif match = scan(/#{SYMBOL}/o)
encoder.text_token match, :symbol
elsif match = scan(/\./)
encoder.text_token match, :operator
elsif match = scan(/ \# \^ #{IDENTIFIER} /ox)
encoder.text_token match, :type
elsif match = scan(/ (\#)? " /x)
state = self[1] ? :regexp : :string
encoder.begin_group state
encoder.text_token match, :delimiter
elsif match = scan(/#{NUM}/o) and not matched.empty?
encoder.text_token match, match[/[.e\/]/i] ? :float : :integer
else
encoder.text_token getch, :error
end
when :string, :regexp
if match = scan(/[^"\\]+|\\.?/)
encoder.text_token match, :content
elsif match = scan(/"/)
encoder.text_token match, :delimiter
encoder.end_group state
state = :initial
else
raise_inspect "else case \" reached; %p not handled." % peek(1),
encoder, state
end
else
raise 'else case reached'
end
end
if [:string, :regexp].include? state
encoder.end_group state
end
encoder
end
end
end
end

View File

@ -1,215 +0,0 @@
module CodeRay
module Scanners
# Scanner for C++.
#
# Aliases: +cplusplus+, c++
class CPlusPlus < Scanner
register_for :cpp
file_extension 'cpp'
title 'C++'
#-- http://www.cppreference.com/wiki/keywords/start
KEYWORDS = [
'and', 'and_eq', 'asm', 'bitand', 'bitor', 'break',
'case', 'catch', 'class', 'compl', 'const_cast',
'continue', 'default', 'delete', 'do', 'dynamic_cast', 'else',
'enum', 'export', 'for', 'goto', 'if', 'namespace', 'new',
'not', 'not_eq', 'or', 'or_eq', 'reinterpret_cast', 'return',
'sizeof', 'static_cast', 'struct', 'switch', 'template',
'throw', 'try', 'typedef', 'typeid', 'typename', 'union',
'while', 'xor', 'xor_eq',
] # :nodoc:
PREDEFINED_TYPES = [
'bool', 'char', 'double', 'float', 'int', 'long',
'short', 'signed', 'unsigned', 'wchar_t', 'string',
] # :nodoc:
PREDEFINED_CONSTANTS = [
'false', 'true',
'EOF', 'NULL',
] # :nodoc:
PREDEFINED_VARIABLES = [
'this',
] # :nodoc:
DIRECTIVES = [
'auto', 'const', 'explicit', 'extern', 'friend', 'inline', 'mutable', 'operator',
'private', 'protected', 'public', 'register', 'static', 'using', 'virtual', 'void',
'volatile',
] # :nodoc:
IDENT_KIND = WordList.new(:ident).
add(KEYWORDS, :keyword).
add(PREDEFINED_TYPES, :predefined_type).
add(PREDEFINED_VARIABLES, :local_variable).
add(DIRECTIVES, :directive).
add(PREDEFINED_CONSTANTS, :predefined_constant) # :nodoc:
ESCAPE = / [rbfntv\n\\'"] | x[a-fA-F0-9]{1,2} | [0-7]{1,3} /x # :nodoc:
UNICODE_ESCAPE = / u[a-fA-F0-9]{4} | U[a-fA-F0-9]{8} /x # :nodoc:
protected
def scan_tokens encoder, options
state = :initial
label_expected = true
case_expected = false
label_expected_before_preproc_line = nil
in_preproc_line = false
until eos?
case state
when :initial
if match = scan(/ \s+ | \\\n /x)
if in_preproc_line && match != "\\\n" && match.index(?\n)
in_preproc_line = false
label_expected = label_expected_before_preproc_line
end
encoder.text_token match, :space
elsif match = scan(%r! // [^\n\\]* (?: \\. [^\n\\]* )* | /\* (?: .*? \*/ | .* ) !mx)
encoder.text_token match, :comment
elsif match = scan(/ \# \s* if \s* 0 /x)
match << scan_until(/ ^\# (?:elif|else|endif) .*? $ | \z /xm) unless eos?
encoder.text_token match, :comment
elsif match = scan(/ [-+*=<>?:;,!&^|()\[\]{}~%]+ | \/=? | \.(?!\d) /x)
label_expected = match =~ /[;\{\}]/
if case_expected
label_expected = true if match == ':'
case_expected = false
end
encoder.text_token match, :operator
elsif match = scan(/ [A-Za-z_][A-Za-z_0-9]* /x)
kind = IDENT_KIND[match]
if kind == :ident && label_expected && !in_preproc_line && scan(/:(?!:)/)
kind = :label
match << matched
else
label_expected = false
if kind == :keyword
case match
when 'class'
state = :class_name_expected
when 'case', 'default'
case_expected = true
end
end
end
encoder.text_token match, kind
elsif match = scan(/\$/)
encoder.text_token match, :ident
elsif match = scan(/L?"/)
encoder.begin_group :string
if match[0] == ?L
encoder.text_token match, 'L', :modifier
match = '"'
end
state = :string
encoder.text_token match, :delimiter
elsif match = scan(/#[ \t]*(\w*)/)
encoder.text_token match, :preprocessor
in_preproc_line = true
label_expected_before_preproc_line = label_expected
state = :include_expected if self[1] == 'include'
elsif match = scan(/ L?' (?: [^\'\n\\] | \\ #{ESCAPE} )? '? /ox)
label_expected = false
encoder.text_token match, :char
elsif match = scan(/0[xX][0-9A-Fa-f]+/)
label_expected = false
encoder.text_token match, :hex
elsif match = scan(/(?:0[0-7]+)(?![89.eEfF])/)
label_expected = false
encoder.text_token match, :octal
elsif match = scan(/(?:\d+)(?![.eEfF])L?L?/)
label_expected = false
encoder.text_token match, :integer
elsif match = scan(/\d[fF]?|\d*\.\d+(?:[eE][+-]?\d+)?[fF]?|\d+[eE][+-]?\d+[fF]?/)
label_expected = false
encoder.text_token match, :float
else
encoder.text_token getch, :error
end
when :string
if match = scan(/[^\\"]+/)
encoder.text_token match, :content
elsif match = scan(/"/)
encoder.text_token match, :delimiter
encoder.end_group :string
state = :initial
label_expected = false
elsif match = scan(/ \\ (?: #{ESCAPE} | #{UNICODE_ESCAPE} ) /mox)
encoder.text_token match, :char
elsif match = scan(/ \\ | $ /x)
encoder.end_group :string
encoder.text_token match, :error
state = :initial
label_expected = false
else
raise_inspect "else case \" reached; %p not handled." % peek(1), encoder
end
when :include_expected
if match = scan(/<[^>\n]+>?|"[^"\n\\]*(?:\\.[^"\n\\]*)*"?/)
encoder.text_token match, :include
state = :initial
elsif match = scan(/\s+/)
encoder.text_token match, :space
state = :initial if match.index ?\n
else
state = :initial
end
when :class_name_expected
if match = scan(/ [A-Za-z_][A-Za-z_0-9]* /x)
encoder.text_token match, :class
state = :initial
elsif match = scan(/\s+/)
encoder.text_token match, :space
else
encoder.text_token getch, :error
state = :initial
end
else
raise_inspect 'Unknown state', encoder
end
end
if state == :string
encoder.end_group :string
end
encoder
end
end
end
end

View File

@ -1,199 +0,0 @@
module CodeRay
module Scanners
class CSS < Scanner
register_for :css
KINDS_NOT_LOC = [
:comment,
:class, :pseudo_class, :type,
:constant, :directive,
:key, :value, :operator, :color, :float, :string,
:error, :important,
] # :nodoc:
module RE # :nodoc:
Hex = /[0-9a-fA-F]/
Unicode = /\\#{Hex}{1,6}(?:\r\n|\s)?/ # differs from standard because it allows uppercase hex too
Escape = /#{Unicode}|\\[^\r\n\f0-9a-fA-F]/
NMChar = /[-_a-zA-Z0-9]|#{Escape}/
NMStart = /[_a-zA-Z]|#{Escape}/
NL = /\r\n|\r|\n|\f/
String1 = /"(?:[^\n\r\f\\"]|\\#{NL}|#{Escape})*"?/ # TODO: buggy regexp
String2 = /'(?:[^\n\r\f\\']|\\#{NL}|#{Escape})*'?/ # TODO: buggy regexp
String = /#{String1}|#{String2}/
HexColor = /#(?:#{Hex}{6}|#{Hex}{3})/
Color = /#{HexColor}/
Num = /-?(?:[0-9]+|[0-9]*\.[0-9]+)/
Name = /#{NMChar}+/
Ident = /-?#{NMStart}#{NMChar}*/
AtKeyword = /@#{Ident}/
Percentage = /#{Num}%/
reldimensions = %w[em ex px]
absdimensions = %w[in cm mm pt pc]
Unit = Regexp.union(*(reldimensions + absdimensions + %w[s]))
Dimension = /#{Num}#{Unit}/
Comment = %r! /\* (?: .*? \*/ | .* ) !mx
Function = /(?:url|alpha|attr|counters?)\((?:[^)\n\r\f]|\\\))*\)?/
Id = /##{Name}/
Class = /\.#{Name}/
PseudoClass = /:#{Name}/
AttributeSelector = /\[[^\]]*\]?/
end
protected
def setup
@state = :initial
@value_expected = nil
end
def scan_tokens encoder, options
states = Array(options[:state] || @state)
value_expected = @value_expected
until eos?
if match = scan(/\s+/)
encoder.text_token match, :space
elsif case states.last
when :initial, :media
if match = scan(/(?>#{RE::Ident})(?!\()|\*/ox)
encoder.text_token match, :type
next
elsif match = scan(RE::Class)
encoder.text_token match, :class
next
elsif match = scan(RE::Id)
encoder.text_token match, :constant
next
elsif match = scan(RE::PseudoClass)
encoder.text_token match, :pseudo_class
next
elsif match = scan(RE::AttributeSelector)
# TODO: Improve highlighting inside of attribute selectors.
encoder.text_token match[0,1], :operator
encoder.text_token match[1..-2], :attribute_name if match.size > 2
encoder.text_token match[-1,1], :operator if match[-1] == ?]
next
elsif match = scan(/@media/)
encoder.text_token match, :directive
states.push :media_before_name
next
end
when :block
if match = scan(/(?>#{RE::Ident})(?!\()/ox)
if value_expected
encoder.text_token match, :value
else
encoder.text_token match, :key
end
next
end
when :media_before_name
if match = scan(RE::Ident)
encoder.text_token match, :type
states[-1] = :media_after_name
next
end
when :media_after_name
if match = scan(/\{/)
encoder.text_token match, :operator
states[-1] = :media
next
end
else
#:nocov:
raise_inspect 'Unknown state', encoder
#:nocov:
end
elsif match = scan(/\/\*(?:.*?\*\/|\z)/m)
encoder.text_token match, :comment
elsif match = scan(/\{/)
value_expected = false
encoder.text_token match, :operator
states.push :block
elsif match = scan(/\}/)
value_expected = false
encoder.text_token match, :operator
if states.last == :block || states.last == :media
states.pop
end
elsif match = scan(/#{RE::String}/o)
encoder.begin_group :string
encoder.text_token match[0, 1], :delimiter
encoder.text_token match[1..-2], :content if match.size > 2
encoder.text_token match[-1, 1], :delimiter if match.size >= 2
encoder.end_group :string
elsif match = scan(/#{RE::Function}/o)
encoder.begin_group :function
start = match[/^\w+\(/]
encoder.text_token start, :delimiter
if match[-1] == ?)
encoder.text_token match[start.size..-2], :content
encoder.text_token ')', :delimiter
else
encoder.text_token match[start.size..-1], :content
end
encoder.end_group :function
elsif match = scan(/(?: #{RE::Dimension} | #{RE::Percentage} | #{RE::Num} )/ox)
encoder.text_token match, :float
elsif match = scan(/#{RE::Color}/o)
encoder.text_token match, :color
elsif match = scan(/! *important/)
encoder.text_token match, :important
elsif match = scan(/(?:rgb|hsl)a?\([^()\n]*\)?/)
encoder.text_token match, :color
elsif match = scan(RE::AtKeyword)
encoder.text_token match, :directive
elsif match = scan(/ [+>:;,.=()\/] /x)
if match == ':'
value_expected = true
elsif match == ';'
value_expected = false
end
encoder.text_token match, :operator
else
encoder.text_token getch, :error
end
end
if options[:keep_state]
@state = states
@value_expected = value_expected
end
encoder
end
end
end
end

View File

@ -1,65 +0,0 @@
module CodeRay
module Scanners
# = Debug Scanner
#
# Interprets the output of the Encoders::Debug encoder.
class Debug < Scanner
register_for :debug
title 'CodeRay Token Dump Import'
protected
def scan_tokens encoder, options
opened_tokens = []
until eos?
if match = scan(/\s+/)
encoder.text_token match, :space
elsif match = scan(/ (\w+) \( ( [^\)\\]* ( \\. [^\)\\]* )* ) \)? /x)
kind = self[1].to_sym
match = self[2].gsub(/\\(.)/m, '\1')
unless TokenKinds.has_key? kind
kind = :error
match = matched
end
encoder.text_token match, kind
elsif match = scan(/ (\w+) ([<\[]) /x)
kind = self[1].to_sym
opened_tokens << kind
case self[2]
when '<'
encoder.begin_group kind
when '['
encoder.begin_line kind
else
raise 'CodeRay bug: This case should not be reached.'
end
elsif !opened_tokens.empty? && match = scan(/ > /x)
encoder.end_group opened_tokens.pop
elsif !opened_tokens.empty? && match = scan(/ \] /x)
encoder.end_line opened_tokens.pop
else
encoder.text_token getch, :space
end
end
encoder.end_group opened_tokens.pop until opened_tokens.empty?
encoder
end
end
end
end

View File

@ -1,144 +0,0 @@
module CodeRay
module Scanners
# Scanner for the Delphi language (Object Pascal).
#
# Alias: +pascal+
class Delphi < Scanner
register_for :delphi
file_extension 'pas'
KEYWORDS = [
'and', 'array', 'as', 'at', 'asm', 'at', 'begin', 'case', 'class',
'const', 'constructor', 'destructor', 'dispinterface', 'div', 'do',
'downto', 'else', 'end', 'except', 'exports', 'file', 'finalization',
'finally', 'for', 'function', 'goto', 'if', 'implementation', 'in',
'inherited', 'initialization', 'inline', 'interface', 'is', 'label',
'library', 'mod', 'nil', 'not', 'object', 'of', 'or', 'out', 'packed',
'procedure', 'program', 'property', 'raise', 'record', 'repeat',
'resourcestring', 'set', 'shl', 'shr', 'string', 'then', 'threadvar',
'to', 'try', 'type', 'unit', 'until', 'uses', 'var', 'while', 'with',
'xor', 'on',
] # :nodoc:
DIRECTIVES = [
'absolute', 'abstract', 'assembler', 'at', 'automated', 'cdecl',
'contains', 'deprecated', 'dispid', 'dynamic', 'export',
'external', 'far', 'forward', 'implements', 'local',
'near', 'nodefault', 'on', 'overload', 'override',
'package', 'pascal', 'platform', 'private', 'protected', 'public',
'published', 'read', 'readonly', 'register', 'reintroduce',
'requires', 'resident', 'safecall', 'stdcall', 'stored', 'varargs',
'virtual', 'write', 'writeonly',
] # :nodoc:
IDENT_KIND = WordList::CaseIgnoring.new(:ident).
add(KEYWORDS, :keyword).
add(DIRECTIVES, :directive) # :nodoc:
NAME_FOLLOWS = WordList::CaseIgnoring.new(false).
add(%w(procedure function .)) # :nodoc:
protected
def scan_tokens encoder, options
state = :initial
last_token = ''
until eos?
if state == :initial
if match = scan(/ \s+ /x)
encoder.text_token match, :space
next
elsif match = scan(%r! \{ \$ [^}]* \}? | \(\* \$ (?: .*? \*\) | .* ) !mx)
encoder.text_token match, :preprocessor
next
elsif match = scan(%r! // [^\n]* | \{ [^}]* \}? | \(\* (?: .*? \*\) | .* ) !mx)
encoder.text_token match, :comment
next
elsif match = scan(/ <[>=]? | >=? | :=? | [-+=*\/;,@\^|\(\)\[\]] | \.\. /x)
encoder.text_token match, :operator
elsif match = scan(/\./)
encoder.text_token match, :operator
next if last_token == 'end'
elsif match = scan(/ [A-Za-z_][A-Za-z_0-9]* /x)
encoder.text_token match, NAME_FOLLOWS[last_token] ? :ident : IDENT_KIND[match]
elsif match = skip(/ ' ( [^\n']|'' ) (?:'|$) /x)
encoder.begin_group :char
encoder.text_token "'", :delimiter
encoder.text_token self[1], :content
encoder.text_token "'", :delimiter
encoder.end_group :char
next
elsif match = scan(/ ' /x)
encoder.begin_group :string
encoder.text_token match, :delimiter
state = :string
elsif match = scan(/ \# (?: \d+ | \$[0-9A-Fa-f]+ ) /x)
encoder.text_token match, :char
elsif match = scan(/ \$ [0-9A-Fa-f]+ /x)
encoder.text_token match, :hex
elsif match = scan(/ (?: \d+ ) (?![eE]|\.[^.]) /x)
encoder.text_token match, :integer
elsif match = scan(/ \d+ (?: \.\d+ (?: [eE][+-]? \d+ )? | [eE][+-]? \d+ ) /x)
encoder.text_token match, :float
else
encoder.text_token getch, :error
next
end
elsif state == :string
if match = scan(/[^\n']+/)
encoder.text_token match, :content
elsif match = scan(/''/)
encoder.text_token match, :char
elsif match = scan(/'/)
encoder.text_token match, :delimiter
encoder.end_group :string
state = :initial
next
elsif match = scan(/\n/)
encoder.end_group :string
encoder.text_token match, :space
state = :initial
else
raise "else case \' reached; %p not handled." % peek(1), encoder
end
else
raise 'else-case reached', encoder
end
last_token = match
end
if state == :string
encoder.end_group state
end
encoder
end
end
end
end

View File

@ -1,199 +0,0 @@
module CodeRay
module Scanners
# Scanner for output of the diff command.
#
# Alias: +patch+
class Diff < Scanner
register_for :diff
title 'diff output'
DEFAULT_OPTIONS = {
:highlight_code => true,
:inline_diff => true,
}
protected
def scan_tokens encoder, options
line_kind = nil
state = :initial
deleted_lines = 0
scanners = Hash.new do |h, lang|
h[lang] = Scanners[lang].new '', :keep_tokens => true, :keep_state => true
end
content_scanner = scanners[:plain]
content_scanner_entry_state = nil
until eos?
if match = scan(/\n/)
deleted_lines = 0 unless line_kind == :delete
if line_kind
encoder.end_line line_kind
line_kind = nil
end
encoder.text_token match, :space
next
end
case state
when :initial
if match = scan(/--- |\+\+\+ |=+|_+/)
encoder.begin_line line_kind = :head
encoder.text_token match, :head
if match = scan(/.*?(?=$|[\t\n\x00]| \(revision)/)
encoder.text_token match, :filename
if options[:highlight_code] && match != '/dev/null'
file_type = CodeRay::FileType.fetch(match, :text)
file_type = :text if file_type == :diff
content_scanner = scanners[file_type]
content_scanner_entry_state = nil
end
end
next unless match = scan(/.+/)
encoder.text_token match, :plain
elsif match = scan(/Index: |Property changes on: /)
encoder.begin_line line_kind = :head
encoder.text_token match, :head
next unless match = scan(/.+/)
encoder.text_token match, :plain
elsif match = scan(/Added: /)
encoder.begin_line line_kind = :head
encoder.text_token match, :head
next unless match = scan(/.+/)
encoder.text_token match, :plain
state = :added
elsif match = scan(/\\ .*/)
encoder.text_token match, :comment
elsif match = scan(/@@(?>[^@\n]*)@@/)
content_scanner.state = :initial unless match?(/\n\+/)
content_scanner_entry_state = nil
if check(/\n|$/)
encoder.begin_line line_kind = :change
else
encoder.begin_group :change
end
encoder.text_token match[0,2], :change
encoder.text_token match[2...-2], :plain
encoder.text_token match[-2,2], :change
encoder.end_group :change unless line_kind
next unless match = scan(/.+/)
if options[:highlight_code]
content_scanner.tokenize match, :tokens => encoder
else
encoder.text_token match, :plain
end
next
elsif match = scan(/\+/)
encoder.begin_line line_kind = :insert
encoder.text_token match, :insert
next unless match = scan(/.+/)
if options[:highlight_code]
content_scanner.tokenize match, :tokens => encoder
else
encoder.text_token match, :plain
end
next
elsif match = scan(/-/)
deleted_lines += 1
encoder.begin_line line_kind = :delete
encoder.text_token match, :delete
if options[:inline_diff] && deleted_lines == 1 && check(/(?>.*)\n\+(?>.*)$(?!\n\+)/)
content_scanner_entry_state = content_scanner.state
skip(/(.*)\n\+(.*)$/)
head, deletion, insertion, tail = diff self[1], self[2]
pre, deleted, post = content_scanner.tokenize [head, deletion, tail], :tokens => Tokens.new
encoder.tokens pre
unless deleted.empty?
encoder.begin_group :eyecatcher
encoder.tokens deleted
encoder.end_group :eyecatcher
end
encoder.tokens post
encoder.end_line line_kind
encoder.text_token "\n", :space
encoder.begin_line line_kind = :insert
encoder.text_token '+', :insert
content_scanner.state = content_scanner_entry_state || :initial
pre, inserted, post = content_scanner.tokenize [head, insertion, tail], :tokens => Tokens.new
encoder.tokens pre
unless inserted.empty?
encoder.begin_group :eyecatcher
encoder.tokens inserted
encoder.end_group :eyecatcher
end
encoder.tokens post
elsif match = scan(/.*/)
if options[:highlight_code]
if deleted_lines == 1
content_scanner_entry_state = content_scanner.state
end
content_scanner.tokenize match, :tokens => encoder unless match.empty?
if !match?(/\n-/)
if match?(/\n\+/)
content_scanner.state = content_scanner_entry_state || :initial
end
content_scanner_entry_state = nil
end
else
encoder.text_token match, :plain
end
end
next
elsif match = scan(/ .*/)
if options[:highlight_code]
content_scanner.tokenize match, :tokens => encoder
else
encoder.text_token match, :plain
end
next
elsif match = scan(/.+/)
encoder.begin_line line_kind = :comment
encoder.text_token match, :plain
else
raise_inspect 'else case rached'
end
when :added
if match = scan(/ \+/)
encoder.begin_line line_kind = :insert
encoder.text_token match, :insert
next unless match = scan(/.+/)
encoder.text_token match, :plain
else
state = :initial
next
end
end
end
encoder.end_line line_kind if line_kind
encoder
end
private
def diff a, b
# i will be the index of the leftmost difference from the left.
i_max = [a.size, b.size].min
i = 0
i += 1 while i < i_max && a[i] == b[i]
# j_min will be the index of the leftmost difference from the right.
j_min = i - i_max
# j will be the index of the rightmost difference from the right which
# does not precede the leftmost one from the left.
j = -1
j -= 1 while j >= j_min && a[j] == b[j]
return a[0...i], a[i..j], b[i..j], (j < -1) ? a[j+1..-1] : ''
end
end
end
end

View File

@ -1,81 +0,0 @@
module CodeRay
module Scanners
load :html
load :ruby
# Scanner for HTML ERB templates.
class ERB < Scanner
register_for :erb
title 'HTML ERB Template'
KINDS_NOT_LOC = HTML::KINDS_NOT_LOC
ERB_RUBY_BLOCK = /
(<%(?!%)[-=\#]?)
((?>
[^\-%]* # normal*
(?> # special
(?: %(?!>) | -(?!%>) )
[^\-%]* # normal*
)*
))
((?: -?%> )?)
/x # :nodoc:
START_OF_ERB = /
<%(?!%)
/x # :nodoc:
protected
def setup
@ruby_scanner = CodeRay.scanner :ruby, :tokens => @tokens, :keep_tokens => true
@html_scanner = CodeRay.scanner :html, :tokens => @tokens, :keep_tokens => true, :keep_state => true
end
def reset_instance
super
@html_scanner.reset
end
def scan_tokens encoder, options
until eos?
if (match = scan_until(/(?=#{START_OF_ERB})/o) || scan_rest) and not match.empty?
@html_scanner.tokenize match, :tokens => encoder
elsif match = scan(/#{ERB_RUBY_BLOCK}/o)
start_tag = self[1]
code = self[2]
end_tag = self[3]
encoder.begin_group :inline
encoder.text_token start_tag, :inline_delimiter
if start_tag == '<%#'
encoder.text_token code, :comment
else
@ruby_scanner.tokenize code, :tokens => encoder
end unless code.empty?
encoder.text_token end_tag, :inline_delimiter unless end_tag.empty?
encoder.end_group :inline
else
raise_inspect 'else-case reached!', encoder
end
end
encoder
end
end
end
end

View File

@ -1,255 +0,0 @@
module CodeRay
module Scanners
load :java
# Scanner for Groovy.
class Groovy < Java
register_for :groovy
# TODO: check list of keywords
GROOVY_KEYWORDS = %w[
as assert def in
] # :nodoc:
KEYWORDS_EXPECTING_VALUE = WordList.new.add %w[
case instanceof new return throw typeof while as assert in
] # :nodoc:
GROOVY_MAGIC_VARIABLES = %w[ it ] # :nodoc:
IDENT_KIND = Java::IDENT_KIND.dup.
add(GROOVY_KEYWORDS, :keyword).
add(GROOVY_MAGIC_VARIABLES, :local_variable) # :nodoc:
ESCAPE = / [bfnrtv$\n\\'"] | x[a-fA-F0-9]{1,2} | [0-7]{1,3} /x # :nodoc:
UNICODE_ESCAPE = / u[a-fA-F0-9]{4} /x # :nodoc: no 4-byte unicode chars? U[a-fA-F0-9]{8}
REGEXP_ESCAPE = / [bfnrtv\n\\'"] | x[a-fA-F0-9]{1,2} | [0-7]{1,3} | \d | [bBdDsSwW\/] /x # :nodoc:
# TODO: interpretation inside ', ", /
STRING_CONTENT_PATTERN = {
"'" => /(?>\\[^\\'\n]+|[^\\'\n]+)+/,
'"' => /[^\\$"\n]+/,
"'''" => /(?>[^\\']+|'(?!''))+/,
'"""' => /(?>[^\\$"]+|"(?!""))+/,
'/' => /[^\\$\/\n]+/,
} # :nodoc:
protected
def scan_tokens encoder, options
state = :initial
inline_block_stack = []
inline_block_paren_depth = nil
string_delimiter = nil
import_clause = class_name_follows = last_token = after_def = false
value_expected = true
until eos?
case state
when :initial
if match = scan(/ \s+ | \\\n /x)
encoder.text_token match, :space
if match.index ?\n
import_clause = after_def = false
value_expected = true unless value_expected
end
next
elsif match = scan(%r! // [^\n\\]* (?: \\. [^\n\\]* )* | /\* (?: .*? \*/ | .* ) !mx)
value_expected = true
after_def = false
encoder.text_token match, :comment
elsif bol? && match = scan(/ \#!.* /x)
encoder.text_token match, :doctype
elsif import_clause && match = scan(/ (?!as) #{IDENT} (?: \. #{IDENT} )* (?: \.\* )? /ox)
after_def = value_expected = false
encoder.text_token match, :include
elsif match = scan(/ #{IDENT} | \[\] /ox)
kind = IDENT_KIND[match]
value_expected = (kind == :keyword) && KEYWORDS_EXPECTING_VALUE[match]
if last_token == '.'
kind = :ident
elsif class_name_follows
kind = :class
class_name_follows = false
elsif after_def && check(/\s*[({]/)
kind = :method
after_def = false
elsif kind == :ident && last_token != '?' && check(/:/)
kind = :key
else
class_name_follows = true if match == 'class' || (import_clause && match == 'as')
import_clause = match == 'import'
after_def = true if match == 'def'
end
encoder.text_token match, kind
elsif match = scan(/;/)
import_clause = after_def = false
value_expected = true
encoder.text_token match, :operator
elsif match = scan(/\{/)
class_name_follows = after_def = false
value_expected = true
encoder.text_token match, :operator
if !inline_block_stack.empty?
inline_block_paren_depth += 1
end
# TODO: ~'...', ~"..." and ~/.../ style regexps
elsif match = scan(/ \.\.<? | \*?\.(?!\d)@? | \.& | \?:? | [,?:(\[] | -[->] | \+\+ |
&& | \|\| | \*\*=? | ==?~ | <=?>? | [-+*%^~&|>=!]=? | <<<?=? | >>>?=? /x)
value_expected = true
value_expected = :regexp if match == '~'
after_def = false
encoder.text_token match, :operator
elsif match = scan(/ [)\]}] /x)
value_expected = after_def = false
if !inline_block_stack.empty? && match == '}'
inline_block_paren_depth -= 1
if inline_block_paren_depth == 0 # closing brace of inline block reached
encoder.text_token match, :inline_delimiter
encoder.end_group :inline
state, string_delimiter, inline_block_paren_depth = inline_block_stack.pop
next
end
end
encoder.text_token match, :operator
elsif check(/[\d.]/)
after_def = value_expected = false
if match = scan(/0[xX][0-9A-Fa-f]+/)
encoder.text_token match, :hex
elsif match = scan(/(?>0[0-7]+)(?![89.eEfF])/)
encoder.text_token match, :octal
elsif match = scan(/\d+[fFdD]|\d*\.\d+(?:[eE][+-]?\d+)?[fFdD]?|\d+[eE][+-]?\d+[fFdD]?/)
encoder.text_token match, :float
elsif match = scan(/\d+[lLgG]?/)
encoder.text_token match, :integer
end
elsif match = scan(/'''|"""/)
after_def = value_expected = false
state = :multiline_string
encoder.begin_group :string
string_delimiter = match
encoder.text_token match, :delimiter
# TODO: record.'name' syntax
elsif match = scan(/["']/)
after_def = value_expected = false
state = match == '/' ? :regexp : :string
encoder.begin_group state
string_delimiter = match
encoder.text_token match, :delimiter
elsif value_expected && match = scan(/\//)
after_def = value_expected = false
encoder.begin_group :regexp
state = :regexp
string_delimiter = '/'
encoder.text_token match, :delimiter
elsif match = scan(/ @ #{IDENT} /ox)
after_def = value_expected = false
encoder.text_token match, :annotation
elsif match = scan(/\//)
after_def = false
value_expected = true
encoder.text_token match, :operator
else
encoder.text_token getch, :error
end
when :string, :regexp, :multiline_string
if match = scan(STRING_CONTENT_PATTERN[string_delimiter])
encoder.text_token match, :content
elsif match = scan(state == :multiline_string ? /'''|"""/ : /["'\/]/)
encoder.text_token match, :delimiter
if state == :regexp
# TODO: regexp modifiers? s, m, x, i?
modifiers = scan(/[ix]+/)
encoder.text_token modifiers, :modifier if modifiers && !modifiers.empty?
end
state = :string if state == :multiline_string
encoder.end_group state
string_delimiter = nil
after_def = value_expected = false
state = :initial
next
elsif (state == :string || state == :multiline_string) &&
(match = scan(/ \\ (?: #{ESCAPE} | #{UNICODE_ESCAPE} ) /mox))
if string_delimiter[0] == ?' && !(match == "\\\\" || match == "\\'")
encoder.text_token match, :content
else
encoder.text_token match, :char
end
elsif state == :regexp && match = scan(/ \\ (?: #{REGEXP_ESCAPE} | #{UNICODE_ESCAPE} ) /mox)
encoder.text_token match, :char
elsif match = scan(/ \$ #{IDENT} /mox)
encoder.begin_group :inline
encoder.text_token '$', :inline_delimiter
match = match[1..-1]
encoder.text_token match, IDENT_KIND[match]
encoder.end_group :inline
next
elsif match = scan(/ \$ \{ /x)
encoder.begin_group :inline
encoder.text_token match, :inline_delimiter
inline_block_stack << [state, string_delimiter, inline_block_paren_depth]
inline_block_paren_depth = 1
state = :initial
next
elsif match = scan(/ \$ /mx)
encoder.text_token match, :content
elsif match = scan(/ \\. /mx)
encoder.text_token match, :content # TODO: Shouldn't this be :error?
elsif match = scan(/ \\ | \n /x)
encoder.end_group state
encoder.text_token match, :error
after_def = value_expected = false
state = :initial
else
raise_inspect "else case \" reached; %p not handled." % peek(1), encoder
end
else
raise_inspect 'Unknown state', encoder
end
last_token = match unless [:space, :comment, :doctype].include? kind
end
if [:multiline_string, :string, :regexp].include? state
encoder.end_group state
end
encoder
end
end
end
end

View File

@ -1,168 +0,0 @@
module CodeRay
module Scanners
load :ruby
load :html
load :java_script
class HAML < Scanner
register_for :haml
title 'HAML Template'
KINDS_NOT_LOC = HTML::KINDS_NOT_LOC
protected
def setup
super
@ruby_scanner = CodeRay.scanner :ruby, :tokens => @tokens, :keep_tokens => true
@embedded_ruby_scanner = CodeRay.scanner :ruby, :tokens => @tokens, :keep_tokens => true, :state => @ruby_scanner.interpreted_string_state
@html_scanner = CodeRay.scanner :html, :tokens => @tokens, :keep_tokens => true
end
def scan_tokens encoder, options
match = nil
code = ''
until eos?
if bol?
if match = scan(/!!!.*/)
encoder.text_token match, :doctype
next
end
if match = scan(/(?>( *)(\/(?!\[if)|-\#|:javascript|:ruby|:\w+) *)(?=\n)/)
encoder.text_token match, :comment
code = self[2]
if match = scan(/(?:\n+#{self[1]} .*)+/)
case code
when '/', '-#'
encoder.text_token match, :comment
when ':javascript'
# TODO: recognize #{...} snippets inside JavaScript
@java_script_scanner ||= CodeRay.scanner :java_script, :tokens => @tokens, :keep_tokens => true
@java_script_scanner.tokenize match, :tokens => encoder
when ':ruby'
@ruby_scanner.tokenize match, :tokens => encoder
when /:\w+/
encoder.text_token match, :comment
else
raise 'else-case reached: %p' % [code]
end
end
end
if match = scan(/ +/)
encoder.text_token match, :space
end
if match = scan(/\/.*/)
encoder.text_token match, :comment
next
end
if match = scan(/\\/)
encoder.text_token match, :plain
if match = scan(/.+/)
@html_scanner.tokenize match, :tokens => encoder
end
next
end
tag = false
if match = scan(/%[\w:]+\/?/)
encoder.text_token match, :tag
# if match = scan(/( +)(.+)/)
# encoder.text_token self[1], :space
# @embedded_ruby_scanner.tokenize self[2], :tokens => encoder
# end
tag = true
end
while match = scan(/([.#])[-\w]*\w/)
encoder.text_token match, self[1] == '#' ? :constant : :class
tag = true
end
if tag && match = scan(/(\()([^)]+)?(\))?/)
# TODO: recognize title=@title, class="widget_#{@widget.number}"
encoder.text_token self[1], :plain
@html_scanner.tokenize self[2], :tokens => encoder, :state => :attribute if self[2]
encoder.text_token self[3], :plain if self[3]
end
if tag && match = scan(/\{/)
encoder.text_token match, :plain
code = ''
level = 1
while true
code << scan(/([^\{\},\n]|, *\n?)*/)
case match = getch
when '{'
level += 1
code << match
when '}'
level -= 1
if level > 0
code << match
else
break
end
when "\n", ",", nil
break
end
end
@ruby_scanner.tokenize code, :tokens => encoder unless code.empty?
encoder.text_token match, :plain if match
end
if tag && match = scan(/(\[)([^\]\n]+)?(\])?/)
encoder.text_token self[1], :plain
@ruby_scanner.tokenize self[2], :tokens => encoder if self[2]
encoder.text_token self[3], :plain if self[3]
end
if tag && match = scan(/\//)
encoder.text_token match, :tag
end
if scan(/(>?<?[-=]|[&!]=|(& |!)|~)( *)([^,\n\|]+(?:(, *|\|(?=.|\n.*\|$))\n?[^,\n\|]*)*)?/)
encoder.text_token self[1] + self[3], :plain
if self[4]
if self[2]
@embedded_ruby_scanner.tokenize self[4], :tokens => encoder
else
@ruby_scanner.tokenize self[4], :tokens => encoder
end
end
elsif match = scan(/((?:<|><?)(?![!?\/\w]))?(.+)?/)
encoder.text_token self[1], :plain if self[1]
# TODO: recognize #{...} snippets
@html_scanner.tokenize self[2], :tokens => encoder if self[2]
end
elsif match = scan(/.+/)
@html_scanner.tokenize match, :tokens => encoder
end
if match = scan(/\n/)
encoder.text_token match, :space
end
end
encoder
end
end
end
end

View File

@ -1,253 +0,0 @@
module CodeRay
module Scanners
# HTML Scanner
#
# Alias: +xhtml+
#
# See also: Scanners::XML
class HTML < Scanner
register_for :html
KINDS_NOT_LOC = [
:comment, :doctype, :preprocessor,
:tag, :attribute_name, :operator,
:attribute_value, :string,
:plain, :entity, :error,
] # :nodoc:
EVENT_ATTRIBUTES = %w(
onabort onafterprint onbeforeprint onbeforeunload onblur oncanplay
oncanplaythrough onchange onclick oncontextmenu oncuechange ondblclick
ondrag ondragdrop ondragend ondragenter ondragleave ondragover
ondragstart ondrop ondurationchange onemptied onended onerror onfocus
onformchange onforminput onhashchange oninput oninvalid onkeydown
onkeypress onkeyup onload onloadeddata onloadedmetadata onloadstart
onmessage onmousedown onmousemove onmouseout onmouseover onmouseup
onmousewheel onmove onoffline ononline onpagehide onpageshow onpause
onplay onplaying onpopstate onprogress onratechange onreadystatechange
onredo onreset onresize onscroll onseeked onseeking onselect onshow
onstalled onstorage onsubmit onsuspend ontimeupdate onundo onunload
onvolumechange onwaiting
)
IN_ATTRIBUTE = WordList::CaseIgnoring.new(nil).
add(EVENT_ATTRIBUTES, :script)
ATTR_NAME = /[\w.:-]+/ # :nodoc:
TAG_END = /\/?>/ # :nodoc:
HEX = /[0-9a-fA-F]/ # :nodoc:
ENTITY = /
&
(?:
\w+
|
\#
(?:
\d+
|
x#{HEX}+
)
)
;
/ox # :nodoc:
PLAIN_STRING_CONTENT = {
"'" => /[^&'>\n]+/,
'"' => /[^&">\n]+/,
} # :nodoc:
def reset
super
@state = :initial
@plain_string_content = nil
end
protected
def setup
@state = :initial
@plain_string_content = nil
end
def scan_java_script encoder, code
if code && !code.empty?
@java_script_scanner ||= Scanners::JavaScript.new '', :keep_tokens => true
# encoder.begin_group :inline
@java_script_scanner.tokenize code, :tokens => encoder
# encoder.end_group :inline
end
end
def scan_tokens encoder, options
state = options[:state] || @state
plain_string_content = @plain_string_content
in_tag = in_attribute = nil
encoder.begin_group :string if state == :attribute_value_string
until eos?
if state != :in_special_tag && match = scan(/\s+/m)
encoder.text_token match, :space
else
case state
when :initial
if match = scan(/<!--(?:.*?-->|.*)/m)
encoder.text_token match, :comment
elsif match = scan(/<!DOCTYPE(?:.*?>|.*)/m)
encoder.text_token match, :doctype
elsif match = scan(/<\?xml(?:.*?\?>|.*)/m)
encoder.text_token match, :preprocessor
elsif match = scan(/<\?(?:.*?\?>|.*)/m)
encoder.text_token match, :comment
elsif match = scan(/<\/[-\w.:]*>?/m)
in_tag = nil
encoder.text_token match, :tag
elsif match = scan(/<(?:(script)|[-\w.:]+)(>)?/m)
encoder.text_token match, :tag
in_tag = self[1]
if self[2]
state = :in_special_tag if in_tag
else
state = :attribute
end
elsif match = scan(/[^<>&]+/)
encoder.text_token match, :plain
elsif match = scan(/#{ENTITY}/ox)
encoder.text_token match, :entity
elsif match = scan(/[<>&]/)
in_tag = nil
encoder.text_token match, :error
else
raise_inspect '[BUG] else-case reached with state %p' % [state], encoder
end
when :attribute
if match = scan(/#{TAG_END}/o)
encoder.text_token match, :tag
in_attribute = nil
if in_tag
state = :in_special_tag
else
state = :initial
end
elsif match = scan(/#{ATTR_NAME}/o)
in_attribute = IN_ATTRIBUTE[match]
encoder.text_token match, :attribute_name
state = :attribute_equal
else
in_tag = nil
encoder.text_token getch, :error
end
when :attribute_equal
if match = scan(/=/) #/
encoder.text_token match, :operator
state = :attribute_value
elsif scan(/#{ATTR_NAME}/o) || scan(/#{TAG_END}/o)
state = :attribute
next
else
encoder.text_token getch, :error
state = :attribute
end
when :attribute_value
if match = scan(/#{ATTR_NAME}/o)
encoder.text_token match, :attribute_value
state = :attribute
elsif match = scan(/["']/)
if in_attribute == :script
encoder.begin_group :inline
encoder.text_token match, :inline_delimiter
if scan(/javascript:[ \t]*/)
encoder.text_token matched, :comment
end
code = scan_until(match == '"' ? /(?="|\z)/ : /(?='|\z)/)
scan_java_script encoder, code
match = scan(/["']/)
encoder.text_token match, :inline_delimiter if match
encoder.end_group :inline
state = :attribute
in_attribute = nil
else
encoder.begin_group :string
state = :attribute_value_string
plain_string_content = PLAIN_STRING_CONTENT[match]
encoder.text_token match, :delimiter
end
elsif match = scan(/#{TAG_END}/o)
encoder.text_token match, :tag
state = :initial
else
encoder.text_token getch, :error
end
when :attribute_value_string
if match = scan(plain_string_content)
encoder.text_token match, :content
elsif match = scan(/['"]/)
encoder.text_token match, :delimiter
encoder.end_group :string
state = :attribute
elsif match = scan(/#{ENTITY}/ox)
encoder.text_token match, :entity
elsif match = scan(/&/)
encoder.text_token match, :content
elsif match = scan(/[\n>]/)
encoder.end_group :string
state = :initial
encoder.text_token match, :error
end
when :in_special_tag
case in_tag
when 'script'
encoder.text_token match, :space if match = scan(/[ \t]*\n/)
if scan(/(\s*<!--)(?:(.*?)(-->)|(.*))/m)
code = self[2] || self[4]
closing = self[3]
encoder.text_token self[1], :comment
else
code = scan_until(/(?=(?:\n\s*)?<\/script>)|\z/)
closing = false
end
unless code.empty?
encoder.begin_group :inline
scan_java_script encoder, code
encoder.end_group :inline
end
encoder.text_token closing, :comment if closing
state = :initial
else
raise 'unknown special tag: %p' % [in_tag]
end
else
raise_inspect 'Unknown state: %p' % [state], encoder
end
end
end
if options[:keep_state]
@state = state
@plain_string_content = plain_string_content
end
encoder.end_group :string if state == :attribute_value_string
encoder
end
end
end
end

View File

@ -1,174 +0,0 @@
module CodeRay
module Scanners
# Scanner for Java.
class Java < Scanner
register_for :java
autoload :BuiltinTypes, CodeRay.coderay_path('scanners', 'java', 'builtin_types')
# http://java.sun.com/docs/books/tutorial/java/nutsandbolts/_keywords.html
KEYWORDS = %w[
assert break case catch continue default do else
finally for if instanceof import new package
return switch throw try typeof while
debugger export
] # :nodoc:
RESERVED = %w[ const goto ] # :nodoc:
CONSTANTS = %w[ false null true ] # :nodoc:
MAGIC_VARIABLES = %w[ this super ] # :nodoc:
TYPES = %w[
boolean byte char class double enum float int interface long
short void
] << '[]' # :nodoc: because int[] should be highlighted as a type
DIRECTIVES = %w[
abstract extends final implements native private protected public
static strictfp synchronized throws transient volatile
] # :nodoc:
IDENT_KIND = WordList.new(:ident).
add(KEYWORDS, :keyword).
add(RESERVED, :reserved).
add(CONSTANTS, :predefined_constant).
add(MAGIC_VARIABLES, :local_variable).
add(TYPES, :type).
add(BuiltinTypes::List, :predefined_type).
add(BuiltinTypes::List.select { |builtin| builtin[/(Error|Exception)$/] }, :exception).
add(DIRECTIVES, :directive) # :nodoc:
ESCAPE = / [bfnrtv\n\\'"] | x[a-fA-F0-9]{1,2} | [0-7]{1,3} /x # :nodoc:
UNICODE_ESCAPE = / u[a-fA-F0-9]{4} | U[a-fA-F0-9]{8} /x # :nodoc:
STRING_CONTENT_PATTERN = {
"'" => /[^\\']+/,
'"' => /[^\\"]+/,
'/' => /[^\\\/]+/,
} # :nodoc:
IDENT = /[a-zA-Z_][A-Za-z_0-9]*/ # :nodoc:
protected
def scan_tokens encoder, options
state = :initial
string_delimiter = nil
package_name_expected = false
class_name_follows = false
last_token_dot = false
until eos?
case state
when :initial
if match = scan(/ \s+ | \\\n /x)
encoder.text_token match, :space
next
elsif match = scan(%r! // [^\n\\]* (?: \\. [^\n\\]* )* | /\* (?: .*? \*/ | .* ) !mx)
encoder.text_token match, :comment
next
elsif package_name_expected && match = scan(/ #{IDENT} (?: \. #{IDENT} )* /ox)
encoder.text_token match, package_name_expected
elsif match = scan(/ #{IDENT} | \[\] /ox)
kind = IDENT_KIND[match]
if last_token_dot
kind = :ident
elsif class_name_follows
kind = :class
class_name_follows = false
else
case match
when 'import'
package_name_expected = :include
when 'package'
package_name_expected = :namespace
when 'class', 'interface'
class_name_follows = true
end
end
encoder.text_token match, kind
elsif match = scan(/ \.(?!\d) | [,?:()\[\]}] | -- | \+\+ | && | \|\| | \*\*=? | [-+*\/%^~&|<>=!]=? | <<<?=? | >>>?=? /x)
encoder.text_token match, :operator
elsif match = scan(/;/)
package_name_expected = false
encoder.text_token match, :operator
elsif match = scan(/\{/)
class_name_follows = false
encoder.text_token match, :operator
elsif check(/[\d.]/)
if match = scan(/0[xX][0-9A-Fa-f]+/)
encoder.text_token match, :hex
elsif match = scan(/(?>0[0-7]+)(?![89.eEfF])/)
encoder.text_token match, :octal
elsif match = scan(/\d+[fFdD]|\d*\.\d+(?:[eE][+-]?\d+)?[fFdD]?|\d+[eE][+-]?\d+[fFdD]?/)
encoder.text_token match, :float
elsif match = scan(/\d+[lL]?/)
encoder.text_token match, :integer
end
elsif match = scan(/["']/)
state = :string
encoder.begin_group state
string_delimiter = match
encoder.text_token match, :delimiter
elsif match = scan(/ @ #{IDENT} /ox)
encoder.text_token match, :annotation
else
encoder.text_token getch, :error
end
when :string
if match = scan(STRING_CONTENT_PATTERN[string_delimiter])
encoder.text_token match, :content
elsif match = scan(/["'\/]/)
encoder.text_token match, :delimiter
encoder.end_group state
state = :initial
string_delimiter = nil
elsif state == :string && (match = scan(/ \\ (?: #{ESCAPE} | #{UNICODE_ESCAPE} ) /mox))
if string_delimiter == "'" && !(match == "\\\\" || match == "\\'")
encoder.text_token match, :content
else
encoder.text_token match, :char
end
elsif match = scan(/\\./m)
encoder.text_token match, :content
elsif match = scan(/ \\ | $ /x)
encoder.end_group state
state = :initial
encoder.text_token match, :error
else
raise_inspect "else case \" reached; %p not handled." % peek(1), encoder
end
else
raise_inspect 'Unknown state', encoder
end
last_token_dot = match == '.'
end
if state == :string
encoder.end_group state
end
encoder
end
end
end
end

View File

@ -1,421 +0,0 @@
module CodeRay
module Scanners
module Java::BuiltinTypes # :nodoc:
#:nocov:
List = %w[
AbstractAction AbstractBorder AbstractButton AbstractCellEditor AbstractCollection
AbstractColorChooserPanel AbstractDocument AbstractExecutorService AbstractInterruptibleChannel
AbstractLayoutCache AbstractList AbstractListModel AbstractMap AbstractMethodError AbstractPreferences
AbstractQueue AbstractQueuedSynchronizer AbstractSelectableChannel AbstractSelectionKey AbstractSelector
AbstractSequentialList AbstractSet AbstractSpinnerModel AbstractTableModel AbstractUndoableEdit
AbstractWriter AccessControlContext AccessControlException AccessController AccessException Accessible
AccessibleAction AccessibleAttributeSequence AccessibleBundle AccessibleComponent AccessibleContext
AccessibleEditableText AccessibleExtendedComponent AccessibleExtendedTable AccessibleExtendedText
AccessibleHyperlink AccessibleHypertext AccessibleIcon AccessibleKeyBinding AccessibleObject
AccessibleRelation AccessibleRelationSet AccessibleResourceBundle AccessibleRole AccessibleSelection
AccessibleState AccessibleStateSet AccessibleStreamable AccessibleTable AccessibleTableModelChange
AccessibleText AccessibleTextSequence AccessibleValue AccountException AccountExpiredException
AccountLockedException AccountNotFoundException Acl AclEntry AclNotFoundException Action ActionEvent
ActionListener ActionMap ActionMapUIResource Activatable ActivateFailedException ActivationDesc
ActivationException ActivationGroup ActivationGroupDesc ActivationGroupID ActivationGroup_Stub
ActivationID ActivationInstantiator ActivationMonitor ActivationSystem Activator ActiveEvent
ActivityCompletedException ActivityRequiredException Adjustable AdjustmentEvent AdjustmentListener
Adler32 AffineTransform AffineTransformOp AlgorithmParameterGenerator AlgorithmParameterGeneratorSpi
AlgorithmParameters AlgorithmParameterSpec AlgorithmParametersSpi AllPermission AlphaComposite
AlreadyBoundException AlreadyConnectedException AncestorEvent AncestorListener AnnotatedElement
Annotation AnnotationFormatError AnnotationTypeMismatchException AppConfigurationEntry Appendable Applet
AppletContext AppletInitializer AppletStub Arc2D Area AreaAveragingScaleFilter ArithmeticException Array
ArrayBlockingQueue ArrayIndexOutOfBoundsException ArrayList Arrays ArrayStoreException ArrayType
AssertionError AsyncBoxView AsynchronousCloseException AtomicBoolean AtomicInteger AtomicIntegerArray
AtomicIntegerFieldUpdater AtomicLong AtomicLongArray AtomicLongFieldUpdater AtomicMarkableReference
AtomicReference AtomicReferenceArray AtomicReferenceFieldUpdater AtomicStampedReference Attribute
AttributeChangeNotification AttributeChangeNotificationFilter AttributedCharacterIterator
AttributedString AttributeException AttributeInUseException AttributeList AttributeModificationException
AttributeNotFoundException Attributes AttributeSet AttributeSetUtilities AttributeValueExp AudioClip
AudioFileFormat AudioFileReader AudioFileWriter AudioFormat AudioInputStream AudioPermission AudioSystem
AuthenticationException AuthenticationNotSupportedException Authenticator AuthorizeCallback
AuthPermission AuthProvider Autoscroll AWTError AWTEvent AWTEventListener AWTEventListenerProxy
AWTEventMulticaster AWTException AWTKeyStroke AWTPermission BackingStoreException
BadAttributeValueExpException BadBinaryOpValueExpException BadLocationException BadPaddingException
BadStringOperationException BandCombineOp BandedSampleModel BaseRowSet BasicArrowButton BasicAttribute
BasicAttributes BasicBorders BasicButtonListener BasicButtonUI BasicCheckBoxMenuItemUI BasicCheckBoxUI
BasicColorChooserUI BasicComboBoxEditor BasicComboBoxRenderer BasicComboBoxUI BasicComboPopup
BasicControl BasicDesktopIconUI BasicDesktopPaneUI BasicDirectoryModel BasicEditorPaneUI
BasicFileChooserUI BasicFormattedTextFieldUI BasicGraphicsUtils BasicHTML BasicIconFactory
BasicInternalFrameTitlePane BasicInternalFrameUI BasicLabelUI BasicListUI BasicLookAndFeel
BasicMenuBarUI BasicMenuItemUI BasicMenuUI BasicOptionPaneUI BasicPanelUI BasicPasswordFieldUI
BasicPermission BasicPopupMenuSeparatorUI BasicPopupMenuUI BasicProgressBarUI BasicRadioButtonMenuItemUI
BasicRadioButtonUI BasicRootPaneUI BasicScrollBarUI BasicScrollPaneUI BasicSeparatorUI BasicSliderUI
BasicSpinnerUI BasicSplitPaneDivider BasicSplitPaneUI BasicStroke BasicTabbedPaneUI BasicTableHeaderUI
BasicTableUI BasicTextAreaUI BasicTextFieldUI BasicTextPaneUI BasicTextUI BasicToggleButtonUI
BasicToolBarSeparatorUI BasicToolBarUI BasicToolTipUI BasicTreeUI BasicViewportUI BatchUpdateException
BeanContext BeanContextChild BeanContextChildComponentProxy BeanContextChildSupport
BeanContextContainerProxy BeanContextEvent BeanContextMembershipEvent BeanContextMembershipListener
BeanContextProxy BeanContextServiceAvailableEvent BeanContextServiceProvider
BeanContextServiceProviderBeanInfo BeanContextServiceRevokedEvent BeanContextServiceRevokedListener
BeanContextServices BeanContextServicesListener BeanContextServicesSupport BeanContextSupport
BeanDescriptor BeanInfo Beans BevelBorder Bidi BigDecimal BigInteger BinaryRefAddr BindException Binding
BitSet Blob BlockingQueue BlockView BMPImageWriteParam Book Boolean BooleanControl Border BorderFactory
BorderLayout BorderUIResource BoundedRangeModel Box BoxLayout BoxView BreakIterator
BrokenBarrierException Buffer BufferCapabilities BufferedImage BufferedImageFilter BufferedImageOp
BufferedInputStream BufferedOutputStream BufferedReader BufferedWriter BufferOverflowException
BufferStrategy BufferUnderflowException Button ButtonGroup ButtonModel ButtonUI Byte
ByteArrayInputStream ByteArrayOutputStream ByteBuffer ByteChannel ByteLookupTable ByteOrder CachedRowSet
CacheRequest CacheResponse Calendar Callable CallableStatement Callback CallbackHandler
CancelablePrintJob CancellationException CancelledKeyException CannotProceedException
CannotRedoException CannotUndoException Canvas CardLayout Caret CaretEvent CaretListener CellEditor
CellEditorListener CellRendererPane Certificate CertificateEncodingException CertificateException
CertificateExpiredException CertificateFactory CertificateFactorySpi CertificateNotYetValidException
CertificateParsingException CertPath CertPathBuilder CertPathBuilderException CertPathBuilderResult
CertPathBuilderSpi CertPathParameters CertPathTrustManagerParameters CertPathValidator
CertPathValidatorException CertPathValidatorResult CertPathValidatorSpi CertSelector CertStore
CertStoreException CertStoreParameters CertStoreSpi ChangedCharSetException ChangeEvent ChangeListener
Channel Channels Character CharacterCodingException CharacterIterator CharArrayReader CharArrayWriter
CharBuffer CharConversionException CharSequence Charset CharsetDecoder CharsetEncoder CharsetProvider
Checkbox CheckboxGroup CheckboxMenuItem CheckedInputStream CheckedOutputStream Checksum Choice
ChoiceCallback ChoiceFormat Chromaticity Cipher CipherInputStream CipherOutputStream CipherSpi Class
ClassCastException ClassCircularityError ClassDefinition ClassDesc ClassFileTransformer ClassFormatError
ClassLoader ClassLoaderRepository ClassLoadingMXBean ClassNotFoundException Clip Clipboard
ClipboardOwner Clob Cloneable CloneNotSupportedException Closeable ClosedByInterruptException
ClosedChannelException ClosedSelectorException CMMException CoderMalfunctionError CoderResult CodeSigner
CodeSource CodingErrorAction CollationElementIterator CollationKey Collator Collection
CollectionCertStoreParameters Collections Color ColorChooserComponentFactory ColorChooserUI
ColorConvertOp ColorModel ColorSelectionModel ColorSpace ColorSupported ColorType ColorUIResource
ComboBoxEditor ComboBoxModel ComboBoxUI ComboPopup CommunicationException Comparable Comparator
CompilationMXBean Compiler CompletionService Component ComponentAdapter ComponentColorModel
ComponentEvent ComponentInputMap ComponentInputMapUIResource ComponentListener ComponentOrientation
ComponentSampleModel ComponentUI ComponentView Composite CompositeContext CompositeData
CompositeDataSupport CompositeName CompositeType CompositeView CompoundBorder CompoundControl
CompoundEdit CompoundName Compression ConcurrentHashMap ConcurrentLinkedQueue ConcurrentMap
ConcurrentModificationException Condition Configuration ConfigurationException ConfirmationCallback
ConnectException ConnectIOException Connection ConnectionEvent ConnectionEventListener
ConnectionPendingException ConnectionPoolDataSource ConsoleHandler Constructor Container
ContainerAdapter ContainerEvent ContainerListener ContainerOrderFocusTraversalPolicy ContentHandler
ContentHandlerFactory ContentModel Context ContextNotEmptyException ContextualRenderedImageFactory
Control ControlFactory ControllerEventListener ConvolveOp CookieHandler Copies CopiesSupported
CopyOnWriteArrayList CopyOnWriteArraySet CountDownLatch CounterMonitor CounterMonitorMBean CRC32
CredentialException CredentialExpiredException CredentialNotFoundException CRL CRLException CRLSelector
CropImageFilter CSS CubicCurve2D Currency Cursor Customizer CyclicBarrier DatabaseMetaData DataBuffer
DataBufferByte DataBufferDouble DataBufferFloat DataBufferInt DataBufferShort DataBufferUShort
DataFlavor DataFormatException DatagramChannel DatagramPacket DatagramSocket DatagramSocketImpl
DatagramSocketImplFactory DataInput DataInputStream DataLine DataOutput DataOutputStream DataSource
DataTruncation DatatypeConfigurationException DatatypeConstants DatatypeFactory Date DateFormat
DateFormatSymbols DateFormatter DateTimeAtCompleted DateTimeAtCreation DateTimeAtProcessing
DateTimeSyntax DebugGraphics DecimalFormat DecimalFormatSymbols DefaultBoundedRangeModel
DefaultButtonModel DefaultCaret DefaultCellEditor DefaultColorSelectionModel DefaultComboBoxModel
DefaultDesktopManager DefaultEditorKit DefaultFocusManager DefaultFocusTraversalPolicy DefaultFormatter
DefaultFormatterFactory DefaultHighlighter DefaultKeyboardFocusManager DefaultListCellRenderer
DefaultListModel DefaultListSelectionModel DefaultLoaderRepository DefaultMenuLayout DefaultMetalTheme
DefaultMutableTreeNode DefaultPersistenceDelegate DefaultSingleSelectionModel DefaultStyledDocument
DefaultTableCellRenderer DefaultTableColumnModel DefaultTableModel DefaultTextUI DefaultTreeCellEditor
DefaultTreeCellRenderer DefaultTreeModel DefaultTreeSelectionModel Deflater DeflaterOutputStream Delayed
DelayQueue DelegationPermission Deprecated Descriptor DescriptorAccess DescriptorSupport DESedeKeySpec
DesignMode DESKeySpec DesktopIconUI DesktopManager DesktopPaneUI Destination Destroyable
DestroyFailedException DGC DHGenParameterSpec DHKey DHParameterSpec DHPrivateKey DHPrivateKeySpec
DHPublicKey DHPublicKeySpec Dialog Dictionary DigestException DigestInputStream DigestOutputStream
Dimension Dimension2D DimensionUIResource DirContext DirectColorModel DirectoryManager DirObjectFactory
DirStateFactory DisplayMode DnDConstants Doc DocAttribute DocAttributeSet DocFlavor DocPrintJob Document
DocumentBuilder DocumentBuilderFactory Documented DocumentEvent DocumentFilter DocumentListener
DocumentName DocumentParser DomainCombiner DOMLocator DOMResult DOMSource Double DoubleBuffer
DragGestureEvent DragGestureListener DragGestureRecognizer DragSource DragSourceAdapter
DragSourceContext DragSourceDragEvent DragSourceDropEvent DragSourceEvent DragSourceListener
DragSourceMotionListener Driver DriverManager DriverPropertyInfo DropTarget DropTargetAdapter
DropTargetContext DropTargetDragEvent DropTargetDropEvent DropTargetEvent DropTargetListener DSAKey
DSAKeyPairGenerator DSAParameterSpec DSAParams DSAPrivateKey DSAPrivateKeySpec DSAPublicKey
DSAPublicKeySpec DTD DTDConstants DuplicateFormatFlagsException Duration DynamicMBean ECField ECFieldF2m
ECFieldFp ECGenParameterSpec ECKey ECParameterSpec ECPoint ECPrivateKey ECPrivateKeySpec ECPublicKey
ECPublicKeySpec EditorKit Element ElementIterator ElementType Ellipse2D EllipticCurve EmptyBorder
EmptyStackException EncodedKeySpec Encoder EncryptedPrivateKeyInfo Entity Enum
EnumConstantNotPresentException EnumControl Enumeration EnumMap EnumSet EnumSyntax EOFException Error
ErrorListener ErrorManager EtchedBorder Event EventContext EventDirContext EventHandler EventListener
EventListenerList EventListenerProxy EventObject EventQueue EventSetDescriptor Exception
ExceptionInInitializerError ExceptionListener Exchanger ExecutionException Executor
ExecutorCompletionService Executors ExecutorService ExemptionMechanism ExemptionMechanismException
ExemptionMechanismSpi ExpandVetoException ExportException Expression ExtendedRequest ExtendedResponse
Externalizable FactoryConfigurationError FailedLoginException FeatureDescriptor Fidelity Field
FieldPosition FieldView File FileCacheImageInputStream FileCacheImageOutputStream FileChannel
FileChooserUI FileDescriptor FileDialog FileFilter FileHandler FileImageInputStream
FileImageOutputStream FileInputStream FileLock FileLockInterruptionException FilenameFilter FileNameMap
FileNotFoundException FileOutputStream FilePermission FileReader FileSystemView FileView FileWriter
Filter FilteredImageSource FilteredRowSet FilterInputStream FilterOutputStream FilterReader FilterWriter
Finishings FixedHeightLayoutCache FlatteningPathIterator FlavorEvent FlavorException FlavorListener
FlavorMap FlavorTable Float FloatBuffer FloatControl FlowLayout FlowView Flushable FocusAdapter
FocusEvent FocusListener FocusManager FocusTraversalPolicy Font FontFormatException FontMetrics
FontRenderContext FontUIResource Format FormatConversionProvider FormatFlagsConversionMismatchException
Formattable FormattableFlags Formatter FormatterClosedException FormSubmitEvent FormView Frame Future
FutureTask GapContent GarbageCollectorMXBean GatheringByteChannel GaugeMonitor GaugeMonitorMBean
GeneralPath GeneralSecurityException GenericArrayType GenericDeclaration GenericSignatureFormatError
GlyphJustificationInfo GlyphMetrics GlyphVector GlyphView GradientPaint GraphicAttribute Graphics
Graphics2D GraphicsConfigTemplate GraphicsConfiguration GraphicsDevice GraphicsEnvironment GrayFilter
GregorianCalendar GridBagConstraints GridBagLayout GridLayout Group Guard GuardedObject GZIPInputStream
GZIPOutputStream Handler HandshakeCompletedEvent HandshakeCompletedListener HasControls HashAttributeSet
HashDocAttributeSet HashMap HashPrintJobAttributeSet HashPrintRequestAttributeSet
HashPrintServiceAttributeSet HashSet Hashtable HeadlessException HierarchyBoundsAdapter
HierarchyBoundsListener HierarchyEvent HierarchyListener Highlighter HostnameVerifier HTML HTMLDocument
HTMLEditorKit HTMLFrameHyperlinkEvent HTMLWriter HttpRetryException HttpsURLConnection HttpURLConnection
HyperlinkEvent HyperlinkListener ICC_ColorSpace ICC_Profile ICC_ProfileGray ICC_ProfileRGB Icon
IconUIResource IconView Identity IdentityHashMap IdentityScope IIOByteBuffer IIOException IIOImage
IIOInvalidTreeException IIOMetadata IIOMetadataController IIOMetadataFormat IIOMetadataFormatImpl
IIOMetadataNode IIOParam IIOParamController IIOReadProgressListener IIOReadUpdateListener
IIOReadWarningListener IIORegistry IIOServiceProvider IIOWriteProgressListener IIOWriteWarningListener
IllegalAccessError IllegalAccessException IllegalArgumentException IllegalBlockingModeException
IllegalBlockSizeException IllegalCharsetNameException IllegalClassFormatException
IllegalComponentStateException IllegalFormatCodePointException IllegalFormatConversionException
IllegalFormatException IllegalFormatFlagsException IllegalFormatPrecisionException
IllegalFormatWidthException IllegalMonitorStateException IllegalPathStateException
IllegalSelectorException IllegalStateException IllegalThreadStateException Image ImageCapabilities
ImageConsumer ImageFilter ImageGraphicAttribute ImageIcon ImageInputStream ImageInputStreamImpl
ImageInputStreamSpi ImageIO ImageObserver ImageOutputStream ImageOutputStreamImpl ImageOutputStreamSpi
ImageProducer ImageReader ImageReaderSpi ImageReaderWriterSpi ImageReadParam ImageTranscoder
ImageTranscoderSpi ImageTypeSpecifier ImageView ImageWriteParam ImageWriter ImageWriterSpi
ImagingOpException IncompatibleClassChangeError IncompleteAnnotationException IndexColorModel
IndexedPropertyChangeEvent IndexedPropertyDescriptor IndexOutOfBoundsException Inet4Address Inet6Address
InetAddress InetSocketAddress Inflater InflaterInputStream InheritableThreadLocal Inherited
InitialContext InitialContextFactory InitialContextFactoryBuilder InitialDirContext InitialLdapContext
InlineView InputContext InputEvent InputMap InputMapUIResource InputMethod InputMethodContext
InputMethodDescriptor InputMethodEvent InputMethodHighlight InputMethodListener InputMethodRequests
InputMismatchException InputStream InputStreamReader InputSubset InputVerifier Insets InsetsUIResource
InstanceAlreadyExistsException InstanceNotFoundException InstantiationError InstantiationException
Instrument Instrumentation InsufficientResourcesException IntBuffer Integer IntegerSyntax InternalError
InternalFrameAdapter InternalFrameEvent InternalFrameFocusTraversalPolicy InternalFrameListener
InternalFrameUI InternationalFormatter InterruptedException InterruptedIOException
InterruptedNamingException InterruptibleChannel IntrospectionException Introspector
InvalidActivityException InvalidAlgorithmParameterException InvalidApplicationException
InvalidAttributeIdentifierException InvalidAttributesException InvalidAttributeValueException
InvalidClassException InvalidDnDOperationException InvalidKeyException InvalidKeySpecException
InvalidMarkException InvalidMidiDataException InvalidNameException InvalidObjectException
InvalidOpenTypeException InvalidParameterException InvalidParameterSpecException
InvalidPreferencesFormatException InvalidPropertiesFormatException InvalidRelationIdException
InvalidRelationServiceException InvalidRelationTypeException InvalidRoleInfoException
InvalidRoleValueException InvalidSearchControlsException InvalidSearchFilterException
InvalidTargetObjectTypeException InvalidTransactionException InvocationEvent InvocationHandler
InvocationTargetException IOException ItemEvent ItemListener ItemSelectable Iterable Iterator
IvParameterSpec JApplet JarEntry JarException JarFile JarInputStream JarOutputStream JarURLConnection
JButton JCheckBox JCheckBoxMenuItem JColorChooser JComboBox JComponent JdbcRowSet JDesktopPane JDialog
JEditorPane JFileChooser JFormattedTextField JFrame JInternalFrame JLabel JLayeredPane JList JMenu
JMenuBar JMenuItem JMException JMRuntimeException JMXAuthenticator JMXConnectionNotification
JMXConnector JMXConnectorFactory JMXConnectorProvider JMXConnectorServer JMXConnectorServerFactory
JMXConnectorServerMBean JMXConnectorServerProvider JMXPrincipal JMXProviderException
JMXServerErrorException JMXServiceURL JobAttributes JobHoldUntil JobImpressions JobImpressionsCompleted
JobImpressionsSupported JobKOctets JobKOctetsProcessed JobKOctetsSupported JobMediaSheets
JobMediaSheetsCompleted JobMediaSheetsSupported JobMessageFromOperator JobName JobOriginatingUserName
JobPriority JobPrioritySupported JobSheets JobState JobStateReason JobStateReasons Joinable JoinRowSet
JOptionPane JPanel JPasswordField JPEGHuffmanTable JPEGImageReadParam JPEGImageWriteParam JPEGQTable
JPopupMenu JProgressBar JRadioButton JRadioButtonMenuItem JRootPane JScrollBar JScrollPane JSeparator
JSlider JSpinner JSplitPane JTabbedPane JTable JTableHeader JTextArea JTextComponent JTextField
JTextPane JToggleButton JToolBar JToolTip JTree JViewport JWindow KerberosKey KerberosPrincipal
KerberosTicket Kernel Key KeyAdapter KeyAgreement KeyAgreementSpi KeyAlreadyExistsException
KeyboardFocusManager KeyEvent KeyEventDispatcher KeyEventPostProcessor KeyException KeyFactory
KeyFactorySpi KeyGenerator KeyGeneratorSpi KeyListener KeyManagementException KeyManager
KeyManagerFactory KeyManagerFactorySpi Keymap KeyPair KeyPairGenerator KeyPairGeneratorSpi KeyRep
KeySpec KeyStore KeyStoreBuilderParameters KeyStoreException KeyStoreSpi KeyStroke Label LabelUI
LabelView LanguageCallback LastOwnerException LayeredHighlighter LayoutFocusTraversalPolicy
LayoutManager LayoutManager2 LayoutQueue LDAPCertStoreParameters LdapContext LdapName
LdapReferralException Lease Level LimitExceededException Line Line2D LineBorder LineBreakMeasurer
LineEvent LineListener LineMetrics LineNumberInputStream LineNumberReader LineUnavailableException
LinkageError LinkedBlockingQueue LinkedHashMap LinkedHashSet LinkedList LinkException LinkLoopException
LinkRef List ListCellRenderer ListDataEvent ListDataListener ListenerNotFoundException ListIterator
ListModel ListResourceBundle ListSelectionEvent ListSelectionListener ListSelectionModel ListUI ListView
LoaderHandler Locale LocateRegistry Lock LockSupport Logger LoggingMXBean LoggingPermission LoginContext
LoginException LoginModule LogManager LogRecord LogStream Long LongBuffer LookAndFeel LookupOp
LookupTable Mac MacSpi MalformedInputException MalformedLinkException MalformedObjectNameException
MalformedParameterizedTypeException MalformedURLException ManagementFactory ManagementPermission
ManageReferralControl ManagerFactoryParameters Manifest Map MappedByteBuffer MarshalException
MarshalledObject MaskFormatter Matcher MatchResult Math MathContext MatteBorder MBeanAttributeInfo
MBeanConstructorInfo MBeanException MBeanFeatureInfo MBeanInfo MBeanNotificationInfo MBeanOperationInfo
MBeanParameterInfo MBeanPermission MBeanRegistration MBeanRegistrationException MBeanServer
MBeanServerBuilder MBeanServerConnection MBeanServerDelegate MBeanServerDelegateMBean MBeanServerFactory
MBeanServerForwarder MBeanServerInvocationHandler MBeanServerNotification MBeanServerNotificationFilter
MBeanServerPermission MBeanTrustPermission Media MediaName MediaPrintableArea MediaSize MediaSizeName
MediaTracker MediaTray Member MemoryCacheImageInputStream MemoryCacheImageOutputStream MemoryHandler
MemoryImageSource MemoryManagerMXBean MemoryMXBean MemoryNotificationInfo MemoryPoolMXBean MemoryType
MemoryUsage Menu MenuBar MenuBarUI MenuComponent MenuContainer MenuDragMouseEvent MenuDragMouseListener
MenuElement MenuEvent MenuItem MenuItemUI MenuKeyEvent MenuKeyListener MenuListener MenuSelectionManager
MenuShortcut MessageDigest MessageDigestSpi MessageFormat MetaEventListener MetalBorders MetalButtonUI
MetalCheckBoxIcon MetalCheckBoxUI MetalComboBoxButton MetalComboBoxEditor MetalComboBoxIcon
MetalComboBoxUI MetalDesktopIconUI MetalFileChooserUI MetalIconFactory MetalInternalFrameTitlePane
MetalInternalFrameUI MetalLabelUI MetalLookAndFeel MetalMenuBarUI MetalPopupMenuSeparatorUI
MetalProgressBarUI MetalRadioButtonUI MetalRootPaneUI MetalScrollBarUI MetalScrollButton
MetalScrollPaneUI MetalSeparatorUI MetalSliderUI MetalSplitPaneUI MetalTabbedPaneUI MetalTextFieldUI
MetalTheme MetalToggleButtonUI MetalToolBarUI MetalToolTipUI MetalTreeUI MetaMessage Method
MethodDescriptor MGF1ParameterSpec MidiChannel MidiDevice MidiDeviceProvider MidiEvent MidiFileFormat
MidiFileReader MidiFileWriter MidiMessage MidiSystem MidiUnavailableException MimeTypeParseException
MinimalHTMLWriter MissingFormatArgumentException MissingFormatWidthException MissingResourceException
Mixer MixerProvider MLet MLetMBean ModelMBean ModelMBeanAttributeInfo ModelMBeanConstructorInfo
ModelMBeanInfo ModelMBeanInfoSupport ModelMBeanNotificationBroadcaster ModelMBeanNotificationInfo
ModelMBeanOperationInfo ModificationItem Modifier Monitor MonitorMBean MonitorNotification
MonitorSettingException MouseAdapter MouseDragGestureRecognizer MouseEvent MouseInfo MouseInputAdapter
MouseInputListener MouseListener MouseMotionAdapter MouseMotionListener MouseWheelEvent
MouseWheelListener MultiButtonUI MulticastSocket MultiColorChooserUI MultiComboBoxUI MultiDesktopIconUI
MultiDesktopPaneUI MultiDoc MultiDocPrintJob MultiDocPrintService MultiFileChooserUI
MultiInternalFrameUI MultiLabelUI MultiListUI MultiLookAndFeel MultiMenuBarUI MultiMenuItemUI
MultiOptionPaneUI MultiPanelUI MultiPixelPackedSampleModel MultipleDocumentHandling MultipleMaster
MultiPopupMenuUI MultiProgressBarUI MultiRootPaneUI MultiScrollBarUI MultiScrollPaneUI MultiSeparatorUI
MultiSliderUI MultiSpinnerUI MultiSplitPaneUI MultiTabbedPaneUI MultiTableHeaderUI MultiTableUI
MultiTextUI MultiToolBarUI MultiToolTipUI MultiTreeUI MultiViewportUI MutableAttributeSet
MutableComboBoxModel MutableTreeNode Name NameAlreadyBoundException NameCallback NameClassPair
NameNotFoundException NameParser NamespaceChangeListener NamespaceContext Naming NamingEnumeration
NamingEvent NamingException NamingExceptionEvent NamingListener NamingManager NamingSecurityException
NavigationFilter NegativeArraySizeException NetPermission NetworkInterface NoClassDefFoundError
NoConnectionPendingException NodeChangeEvent NodeChangeListener NoInitialContextException
NoninvertibleTransformException NonReadableChannelException NonWritableChannelException
NoPermissionException NoRouteToHostException NoSuchAlgorithmException NoSuchAttributeException
NoSuchElementException NoSuchFieldError NoSuchFieldException NoSuchMethodError NoSuchMethodException
NoSuchObjectException NoSuchPaddingException NoSuchProviderException NotActiveException
NotBoundException NotCompliantMBeanException NotContextException Notification NotificationBroadcaster
NotificationBroadcasterSupport NotificationEmitter NotificationFilter NotificationFilterSupport
NotificationListener NotificationResult NotOwnerException NotSerializableException NotYetBoundException
NotYetConnectedException NullCipher NullPointerException Number NumberFormat NumberFormatException
NumberFormatter NumberOfDocuments NumberOfInterveningJobs NumberUp NumberUpSupported NumericShaper
OAEPParameterSpec Object ObjectChangeListener ObjectFactory ObjectFactoryBuilder ObjectInput
ObjectInputStream ObjectInputValidation ObjectInstance ObjectName ObjectOutput ObjectOutputStream
ObjectStreamClass ObjectStreamConstants ObjectStreamException ObjectStreamField ObjectView ObjID
Observable Observer OceanTheme OpenDataException OpenMBeanAttributeInfo OpenMBeanAttributeInfoSupport
OpenMBeanConstructorInfo OpenMBeanConstructorInfoSupport OpenMBeanInfo OpenMBeanInfoSupport
OpenMBeanOperationInfo OpenMBeanOperationInfoSupport OpenMBeanParameterInfo
OpenMBeanParameterInfoSupport OpenType OperatingSystemMXBean Operation OperationNotSupportedException
OperationsException Option OptionalDataException OptionPaneUI OrientationRequested OutOfMemoryError
OutputDeviceAssigned OutputKeys OutputStream OutputStreamWriter OverlappingFileLockException
OverlayLayout Override Owner Pack200 Package PackedColorModel Pageable PageAttributes
PagedResultsControl PagedResultsResponseControl PageFormat PageRanges PagesPerMinute PagesPerMinuteColor
Paint PaintContext PaintEvent Panel PanelUI Paper ParagraphView ParameterBlock ParameterDescriptor
ParameterizedType ParameterMetaData ParseException ParsePosition Parser ParserConfigurationException
ParserDelegator PartialResultException PasswordAuthentication PasswordCallback PasswordView Patch
PathIterator Pattern PatternSyntaxException PBEKey PBEKeySpec PBEParameterSpec PDLOverrideSupported
Permission PermissionCollection Permissions PersistenceDelegate PersistentMBean PhantomReference Pipe
PipedInputStream PipedOutputStream PipedReader PipedWriter PixelGrabber PixelInterleavedSampleModel
PKCS8EncodedKeySpec PKIXBuilderParameters PKIXCertPathBuilderResult PKIXCertPathChecker
PKIXCertPathValidatorResult PKIXParameters PlainDocument PlainView Point Point2D PointerInfo Policy
PolicyNode PolicyQualifierInfo Polygon PooledConnection Popup PopupFactory PopupMenu PopupMenuEvent
PopupMenuListener PopupMenuUI Port PortableRemoteObject PortableRemoteObjectDelegate
PortUnreachableException Position Predicate PreferenceChangeEvent PreferenceChangeListener Preferences
PreferencesFactory PreparedStatement PresentationDirection Principal Printable PrinterAbortException
PrinterException PrinterGraphics PrinterInfo PrinterIOException PrinterIsAcceptingJobs PrinterJob
PrinterLocation PrinterMakeAndModel PrinterMessageFromOperator PrinterMoreInfo
PrinterMoreInfoManufacturer PrinterName PrinterResolution PrinterState PrinterStateReason
PrinterStateReasons PrinterURI PrintEvent PrintException PrintGraphics PrintJob PrintJobAdapter
PrintJobAttribute PrintJobAttributeEvent PrintJobAttributeListener PrintJobAttributeSet PrintJobEvent
PrintJobListener PrintQuality PrintRequestAttribute PrintRequestAttributeSet PrintService
PrintServiceAttribute PrintServiceAttributeEvent PrintServiceAttributeListener PrintServiceAttributeSet
PrintServiceLookup PrintStream PrintWriter PriorityBlockingQueue PriorityQueue PrivateClassLoader
PrivateCredentialPermission PrivateKey PrivateMLet PrivilegedAction PrivilegedActionException
PrivilegedExceptionAction Process ProcessBuilder ProfileDataException ProgressBarUI ProgressMonitor
ProgressMonitorInputStream Properties PropertyChangeEvent PropertyChangeListener
PropertyChangeListenerProxy PropertyChangeSupport PropertyDescriptor PropertyEditor
PropertyEditorManager PropertyEditorSupport PropertyPermission PropertyResourceBundle
PropertyVetoException ProtectionDomain ProtocolException Provider ProviderException Proxy ProxySelector
PSource PSSParameterSpec PublicKey PushbackInputStream PushbackReader QName QuadCurve2D Query QueryEval
QueryExp Queue QueuedJobCount Random RandomAccess RandomAccessFile Raster RasterFormatException RasterOp
RC2ParameterSpec RC5ParameterSpec Rdn Readable ReadableByteChannel Reader ReadOnlyBufferException
ReadWriteLock RealmCallback RealmChoiceCallback Receiver Rectangle Rectangle2D RectangularShape
ReentrantLock ReentrantReadWriteLock Ref RefAddr Reference Referenceable ReferenceQueue
ReferenceUriSchemesSupported ReferralException ReflectionException ReflectPermission Refreshable
RefreshFailedException Region RegisterableService Registry RegistryHandler RejectedExecutionException
RejectedExecutionHandler Relation RelationException RelationNotFoundException RelationNotification
RelationService RelationServiceMBean RelationServiceNotRegisteredException RelationSupport
RelationSupportMBean RelationType RelationTypeNotFoundException RelationTypeSupport Remote RemoteCall
RemoteException RemoteObject RemoteObjectInvocationHandler RemoteRef RemoteServer RemoteStub
RenderableImage RenderableImageOp RenderableImageProducer RenderContext RenderedImage
RenderedImageFactory Renderer RenderingHints RepaintManager ReplicateScaleFilter RequestingUserName
RequiredModelMBean RescaleOp ResolutionSyntax Resolver ResolveResult ResourceBundle ResponseCache Result
ResultSet ResultSetMetaData Retention RetentionPolicy ReverbType RGBImageFilter RMIClassLoader
RMIClassLoaderSpi RMIClientSocketFactory RMIConnection RMIConnectionImpl RMIConnectionImpl_Stub
RMIConnector RMIConnectorServer RMIFailureHandler RMIIIOPServerImpl RMIJRMPServerImpl
RMISecurityException RMISecurityManager RMIServer RMIServerImpl RMIServerImpl_Stub
RMIServerSocketFactory RMISocketFactory Robot Role RoleInfo RoleInfoNotFoundException RoleList
RoleNotFoundException RoleResult RoleStatus RoleUnresolved RoleUnresolvedList RootPaneContainer
RootPaneUI RoundingMode RoundRectangle2D RowMapper RowSet RowSetEvent RowSetInternal RowSetListener
RowSetMetaData RowSetMetaDataImpl RowSetReader RowSetWarning RowSetWriter RSAKey RSAKeyGenParameterSpec
RSAMultiPrimePrivateCrtKey RSAMultiPrimePrivateCrtKeySpec RSAOtherPrimeInfo RSAPrivateCrtKey
RSAPrivateCrtKeySpec RSAPrivateKey RSAPrivateKeySpec RSAPublicKey RSAPublicKeySpec RTFEditorKit
RuleBasedCollator Runnable Runtime RuntimeErrorException RuntimeException RuntimeMBeanException
RuntimeMXBean RuntimeOperationsException RuntimePermission SampleModel Sasl SaslClient SaslClientFactory
SaslException SaslServer SaslServerFactory Savepoint SAXParser SAXParserFactory SAXResult SAXSource
SAXTransformerFactory Scanner ScatteringByteChannel ScheduledExecutorService ScheduledFuture
ScheduledThreadPoolExecutor Schema SchemaFactory SchemaFactoryLoader SchemaViolationException Scrollable
Scrollbar ScrollBarUI ScrollPane ScrollPaneAdjustable ScrollPaneConstants ScrollPaneLayout ScrollPaneUI
SealedObject SearchControls SearchResult SecretKey SecretKeyFactory SecretKeyFactorySpi SecretKeySpec
SecureCacheResponse SecureClassLoader SecureRandom SecureRandomSpi Security SecurityException
SecurityManager SecurityPermission Segment SelectableChannel SelectionKey Selector SelectorProvider
Semaphore SeparatorUI Sequence SequenceInputStream Sequencer SerialArray SerialBlob SerialClob
SerialDatalink SerialException Serializable SerializablePermission SerialJavaObject SerialRef
SerialStruct ServerCloneException ServerError ServerException ServerNotActiveException ServerRef
ServerRuntimeException ServerSocket ServerSocketChannel ServerSocketFactory ServiceNotFoundException
ServicePermission ServiceRegistry ServiceUI ServiceUIFactory ServiceUnavailableException Set
SetOfIntegerSyntax Severity Shape ShapeGraphicAttribute SheetCollate Short ShortBuffer
ShortBufferException ShortLookupTable ShortMessage Sides Signature SignatureException SignatureSpi
SignedObject Signer SimpleAttributeSet SimpleBeanInfo SimpleDateFormat SimpleDoc SimpleFormatter
SimpleTimeZone SimpleType SinglePixelPackedSampleModel SingleSelectionModel Size2DSyntax
SizeLimitExceededException SizeRequirements SizeSequence Skeleton SkeletonMismatchException
SkeletonNotFoundException SliderUI Socket SocketAddress SocketChannel SocketException SocketFactory
SocketHandler SocketImpl SocketImplFactory SocketOptions SocketPermission SocketSecurityException
SocketTimeoutException SoftBevelBorder SoftReference SortControl SortedMap SortedSet
SortingFocusTraversalPolicy SortKey SortResponseControl Soundbank SoundbankReader SoundbankResource
Source SourceDataLine SourceLocator SpinnerDateModel SpinnerListModel SpinnerModel SpinnerNumberModel
SpinnerUI SplitPaneUI Spring SpringLayout SQLData SQLException SQLInput SQLInputImpl SQLOutput
SQLOutputImpl SQLPermission SQLWarning SSLContext SSLContextSpi SSLEngine SSLEngineResult SSLException
SSLHandshakeException SSLKeyException SSLPeerUnverifiedException SSLPermission SSLProtocolException
SslRMIClientSocketFactory SslRMIServerSocketFactory SSLServerSocket SSLServerSocketFactory SSLSession
SSLSessionBindingEvent SSLSessionBindingListener SSLSessionContext SSLSocket SSLSocketFactory Stack
StackOverflowError StackTraceElement StandardMBean StartTlsRequest StartTlsResponse StateEdit
StateEditable StateFactory Statement StreamCorruptedException StreamHandler StreamPrintService
StreamPrintServiceFactory StreamResult StreamSource StreamTokenizer StrictMath String StringBuffer
StringBufferInputStream StringBuilder StringCharacterIterator StringContent
StringIndexOutOfBoundsException StringMonitor StringMonitorMBean StringReader StringRefAddr
StringSelection StringTokenizer StringValueExp StringWriter Stroke Struct Stub StubDelegate
StubNotFoundException Style StyleConstants StyleContext StyledDocument StyledEditorKit StyleSheet
Subject SubjectDelegationPermission SubjectDomainCombiner SupportedValuesAttribute SuppressWarnings
SwingConstants SwingPropertyChangeSupport SwingUtilities SyncFactory SyncFactoryException
SyncFailedException SynchronousQueue SyncProvider SyncProviderException SyncResolver SynthConstants
SynthContext Synthesizer SynthGraphicsUtils SynthLookAndFeel SynthPainter SynthStyle SynthStyleFactory
SysexMessage System SystemColor SystemFlavorMap TabableView TabbedPaneUI TabExpander TableCellEditor
TableCellRenderer TableColumn TableColumnModel TableColumnModelEvent TableColumnModelListener
TableHeaderUI TableModel TableModelEvent TableModelListener TableUI TableView TabSet TabStop TabularData
TabularDataSupport TabularType TagElement Target TargetDataLine TargetedNotification Templates
TemplatesHandler TextAction TextArea TextAttribute TextComponent TextEvent TextField TextHitInfo
TextInputCallback TextLayout TextListener TextMeasurer TextOutputCallback TextSyntax TextUI TexturePaint
Thread ThreadDeath ThreadFactory ThreadGroup ThreadInfo ThreadLocal ThreadMXBean ThreadPoolExecutor
Throwable Tie TileObserver Time TimeLimitExceededException TimeoutException Timer
TimerAlarmClockNotification TimerMBean TimerNotification TimerTask Timestamp TimeUnit TimeZone
TitledBorder ToolBarUI Toolkit ToolTipManager ToolTipUI TooManyListenersException Track
TransactionalWriter TransactionRequiredException TransactionRolledbackException Transferable
TransferHandler TransformAttribute Transformer TransformerConfigurationException TransformerException
TransformerFactory TransformerFactoryConfigurationError TransformerHandler Transmitter Transparency
TreeCellEditor TreeCellRenderer TreeExpansionEvent TreeExpansionListener TreeMap TreeModel
TreeModelEvent TreeModelListener TreeNode TreePath TreeSelectionEvent TreeSelectionListener
TreeSelectionModel TreeSet TreeUI TreeWillExpandListener TrustAnchor TrustManager TrustManagerFactory
TrustManagerFactorySpi Type TypeInfoProvider TypeNotPresentException Types TypeVariable UID UIDefaults
UIManager UIResource UndeclaredThrowableException UndoableEdit UndoableEditEvent UndoableEditListener
UndoableEditSupport UndoManager UnexpectedException UnicastRemoteObject UnknownError
UnknownFormatConversionException UnknownFormatFlagsException UnknownGroupException UnknownHostException
UnknownObjectException UnknownServiceException UnmappableCharacterException UnmarshalException
UnmodifiableClassException UnmodifiableSetException UnrecoverableEntryException
UnrecoverableKeyException Unreferenced UnresolvedAddressException UnresolvedPermission
UnsatisfiedLinkError UnsolicitedNotification UnsolicitedNotificationEvent
UnsolicitedNotificationListener UnsupportedAddressTypeException UnsupportedAudioFileException
UnsupportedCallbackException UnsupportedCharsetException UnsupportedClassVersionError
UnsupportedEncodingException UnsupportedFlavorException UnsupportedLookAndFeelException
UnsupportedOperationException URI URIException URIResolver URISyntax URISyntaxException URL
URLClassLoader URLConnection URLDecoder URLEncoder URLStreamHandler URLStreamHandlerFactory
UTFDataFormatException Util UtilDelegate Utilities UUID Validator ValidatorHandler ValueExp ValueHandler
ValueHandlerMultiFormat VariableHeightLayoutCache Vector VerifyError VetoableChangeListener
VetoableChangeListenerProxy VetoableChangeSupport View ViewFactory ViewportLayout ViewportUI
VirtualMachineError Visibility VMID VoiceStatus Void VolatileImage WeakHashMap WeakReference WebRowSet
WildcardType Window WindowAdapter WindowConstants WindowEvent WindowFocusListener WindowListener
WindowStateListener WrappedPlainView WritableByteChannel WritableRaster WritableRenderedImage
WriteAbortedException Writer X500Principal X500PrivateCredential X509Certificate X509CertSelector
X509CRL X509CRLEntry X509CRLSelector X509EncodedKeySpec X509ExtendedKeyManager X509Extension
X509KeyManager X509TrustManager XAConnection XADataSource XAException XAResource Xid XMLConstants
XMLDecoder XMLEncoder XMLFormatter XMLGregorianCalendar XMLParseException XmlReader XmlWriter XPath
XPathConstants XPathException XPathExpression XPathExpressionException XPathFactory
XPathFactoryConfigurationException XPathFunction XPathFunctionException XPathFunctionResolver
XPathVariableResolver ZipEntry ZipException ZipFile ZipInputStream ZipOutputStream ZoneView
]
#:nocov:
end
end
end

View File

@ -1,213 +0,0 @@
module CodeRay
module Scanners
# Scanner for JavaScript.
#
# Aliases: +ecmascript+, +ecma_script+, +javascript+
class JavaScript < Scanner
register_for :java_script
file_extension 'js'
# The actual JavaScript keywords.
KEYWORDS = %w[
break case catch continue default delete do else
finally for function if in instanceof new
return switch throw try typeof var void while with
] # :nodoc:
PREDEFINED_CONSTANTS = %w[
false null true undefined NaN Infinity
] # :nodoc:
MAGIC_VARIABLES = %w[ this arguments ] # :nodoc: arguments was introduced in JavaScript 1.4
KEYWORDS_EXPECTING_VALUE = WordList.new.add %w[
case delete in instanceof new return throw typeof with
] # :nodoc:
# Reserved for future use.
RESERVED_WORDS = %w[
abstract boolean byte char class debugger double enum export extends
final float goto implements import int interface long native package
private protected public short static super synchronized throws transient
volatile
] # :nodoc:
IDENT_KIND = WordList.new(:ident).
add(RESERVED_WORDS, :reserved).
add(PREDEFINED_CONSTANTS, :predefined_constant).
add(MAGIC_VARIABLES, :local_variable).
add(KEYWORDS, :keyword) # :nodoc:
ESCAPE = / [bfnrtv\n\\'"] | x[a-fA-F0-9]{1,2} | [0-7]{1,3} /x # :nodoc:
UNICODE_ESCAPE = / u[a-fA-F0-9]{4} | U[a-fA-F0-9]{8} /x # :nodoc:
REGEXP_ESCAPE = / [bBdDsSwW] /x # :nodoc:
STRING_CONTENT_PATTERN = {
"'" => /[^\\']+/,
'"' => /[^\\"]+/,
'/' => /[^\\\/]+/,
} # :nodoc:
KEY_CHECK_PATTERN = {
"'" => / (?> [^\\']* (?: \\. [^\\']* )* ) ' \s* : /mx,
'"' => / (?> [^\\"]* (?: \\. [^\\"]* )* ) " \s* : /mx,
} # :nodoc:
protected
def scan_tokens encoder, options
state = :initial
string_delimiter = nil
value_expected = true
key_expected = false
function_expected = false
until eos?
case state
when :initial
if match = scan(/ \s+ | \\\n /x)
value_expected = true if !value_expected && match.index(?\n)
encoder.text_token match, :space
elsif match = scan(%r! // [^\n\\]* (?: \\. [^\n\\]* )* | /\* (?: .*? \*/ | .* ) !mx)
value_expected = true
encoder.text_token match, :comment
elsif check(/\.?\d/)
key_expected = value_expected = false
if match = scan(/0[xX][0-9A-Fa-f]+/)
encoder.text_token match, :hex
elsif match = scan(/(?>0[0-7]+)(?![89.eEfF])/)
encoder.text_token match, :octal
elsif match = scan(/\d+[fF]|\d*\.\d+(?:[eE][+-]?\d+)?[fF]?|\d+[eE][+-]?\d+[fF]?/)
encoder.text_token match, :float
elsif match = scan(/\d+/)
encoder.text_token match, :integer
end
elsif value_expected && match = scan(/<([[:alpha:]]\w*) (?: [^\/>]*\/> | .*?<\/\1>)/xim)
# TODO: scan over nested tags
xml_scanner.tokenize match, :tokens => encoder
value_expected = false
next
elsif match = scan(/ [-+*=<>?:;,!&^|(\[{~%]+ | \.(?!\d) /x)
value_expected = true
last_operator = match[-1]
key_expected = (last_operator == ?{) || (last_operator == ?,)
function_expected = false
encoder.text_token match, :operator
elsif match = scan(/ [)\]}]+ /x)
function_expected = key_expected = value_expected = false
encoder.text_token match, :operator
elsif match = scan(/ [$a-zA-Z_][A-Za-z_0-9$]* /x)
kind = IDENT_KIND[match]
value_expected = (kind == :keyword) && KEYWORDS_EXPECTING_VALUE[match]
# TODO: labels
if kind == :ident
if match.index(?$) # $ allowed inside an identifier
kind = :predefined
elsif function_expected
kind = :function
elsif check(/\s*[=:]\s*function\b/)
kind = :function
elsif key_expected && check(/\s*:/)
kind = :key
end
end
function_expected = (kind == :keyword) && (match == 'function')
key_expected = false
encoder.text_token match, kind
elsif match = scan(/["']/)
if key_expected && check(KEY_CHECK_PATTERN[match])
state = :key
else
state = :string
end
encoder.begin_group state
string_delimiter = match
encoder.text_token match, :delimiter
elsif value_expected && (match = scan(/\//))
encoder.begin_group :regexp
state = :regexp
string_delimiter = '/'
encoder.text_token match, :delimiter
elsif match = scan(/ \/ /x)
value_expected = true
key_expected = false
encoder.text_token match, :operator
else
encoder.text_token getch, :error
end
when :string, :regexp, :key
if match = scan(STRING_CONTENT_PATTERN[string_delimiter])
encoder.text_token match, :content
elsif match = scan(/["'\/]/)
encoder.text_token match, :delimiter
if state == :regexp
modifiers = scan(/[gim]+/)
encoder.text_token modifiers, :modifier if modifiers && !modifiers.empty?
end
encoder.end_group state
string_delimiter = nil
key_expected = value_expected = false
state = :initial
elsif state != :regexp && (match = scan(/ \\ (?: #{ESCAPE} | #{UNICODE_ESCAPE} ) /mox))
if string_delimiter == "'" && !(match == "\\\\" || match == "\\'")
encoder.text_token match, :content
else
encoder.text_token match, :char
end
elsif state == :regexp && match = scan(/ \\ (?: #{ESCAPE} | #{REGEXP_ESCAPE} | #{UNICODE_ESCAPE} ) /mox)
encoder.text_token match, :char
elsif match = scan(/\\./m)
encoder.text_token match, :content
elsif match = scan(/ \\ | $ /x)
encoder.end_group state
encoder.text_token match, :error
key_expected = value_expected = false
state = :initial
else
raise_inspect "else case \" reached; %p not handled." % peek(1), encoder
end
else
raise_inspect 'Unknown state', encoder
end
end
if [:string, :regexp].include? state
encoder.end_group state
end
encoder
end
protected
def reset_instance
super
@xml_scanner.reset if defined? @xml_scanner
end
def xml_scanner
@xml_scanner ||= CodeRay.scanner :xml, :tokens => @tokens, :keep_tokens => true, :keep_state => false
end
end
end
end

View File

@ -1,95 +0,0 @@
module CodeRay
module Scanners
# Scanner for JSON (JavaScript Object Notation).
class JSON < Scanner
register_for :json
file_extension 'json'
KINDS_NOT_LOC = [
:float, :char, :content, :delimiter,
:error, :integer, :operator, :value,
] # :nodoc:
ESCAPE = / [bfnrt\\"\/] /x # :nodoc:
UNICODE_ESCAPE = / u[a-fA-F0-9]{4} /x # :nodoc:
protected
# See http://json.org/ for a definition of the JSON lexic/grammar.
def scan_tokens encoder, options
state = :initial
stack = []
key_expected = false
until eos?
case state
when :initial
if match = scan(/ \s+ /x)
encoder.text_token match, :space
elsif match = scan(/"/)
state = key_expected ? :key : :string
encoder.begin_group state
encoder.text_token match, :delimiter
elsif match = scan(/ [:,\[{\]}] /x)
encoder.text_token match, :operator
case match
when ':' then key_expected = false
when ',' then key_expected = true if stack.last == :object
when '{' then stack << :object; key_expected = true
when '[' then stack << :array
when '}', ']' then stack.pop # no error recovery, but works for valid JSON
end
elsif match = scan(/ true | false | null /x)
encoder.text_token match, :value
elsif match = scan(/ -? (?: 0 | [1-9]\d* ) /x)
if scan(/ \.\d+ (?:[eE][-+]?\d+)? | [eE][-+]? \d+ /x)
match << matched
encoder.text_token match, :float
else
encoder.text_token match, :integer
end
else
encoder.text_token getch, :error
end
when :string, :key
if match = scan(/[^\\"]+/)
encoder.text_token match, :content
elsif match = scan(/"/)
encoder.text_token match, :delimiter
encoder.end_group state
state = :initial
elsif match = scan(/ \\ (?: #{ESCAPE} | #{UNICODE_ESCAPE} ) /mox)
encoder.text_token match, :char
elsif match = scan(/\\./m)
encoder.text_token match, :content
elsif match = scan(/ \\ | $ /x)
encoder.end_group state
encoder.text_token match, :error
state = :initial
else
raise_inspect "else case \" reached; %p not handled." % peek(1), encoder
end
else
raise_inspect 'Unknown state: %p' % [state], encoder
end
end
if [:string, :key].include? state
encoder.end_group state
end
encoder
end
end
end
end

View File

@ -1,510 +0,0 @@
# encoding: ASCII-8BIT
module CodeRay
module Scanners
load :html
# Scanner for PHP.
#
# Original by Stefan Walk.
class PHP < Scanner
register_for :php
file_extension 'php'
encoding 'BINARY'
KINDS_NOT_LOC = HTML::KINDS_NOT_LOC
protected
def setup
@html_scanner = CodeRay.scanner :html, :tokens => @tokens, :keep_tokens => true, :keep_state => true
end
def reset_instance
super
@html_scanner.reset
end
module Words # :nodoc:
# according to http://www.php.net/manual/en/reserved.keywords.php
KEYWORDS = %w[
abstract and array as break case catch class clone const continue declare default do else elseif
enddeclare endfor endforeach endif endswitch endwhile extends final for foreach function global
goto if implements interface instanceof namespace new or private protected public static switch
throw try use var while xor
cfunction old_function
]
TYPES = %w[ int integer float double bool boolean string array object resource ]
LANGUAGE_CONSTRUCTS = %w[
die echo empty exit eval include include_once isset list
require require_once return print unset
]
CLASSES = %w[ Directory stdClass __PHP_Incomplete_Class exception php_user_filter Closure ]
# according to http://php.net/quickref.php on 2009-04-21;
# all functions with _ excluded (module functions) and selected additional functions
BUILTIN_FUNCTIONS = %w[
abs acos acosh addcslashes addslashes aggregate array arsort ascii2ebcdic asin asinh asort assert atan atan2
atanh basename bcadd bccomp bcdiv bcmod bcmul bcpow bcpowmod bcscale bcsqrt bcsub bin2hex bindec
bindtextdomain bzclose bzcompress bzdecompress bzerrno bzerror bzerrstr bzflush bzopen bzread bzwrite
calculhmac ceil chdir checkdate checkdnsrr chgrp chmod chop chown chr chroot clearstatcache closedir closelog
compact constant copy cos cosh count crc32 crypt current date dcgettext dcngettext deaggregate decbin dechex
decoct define defined deg2rad delete dgettext die dirname diskfreespace dl dngettext doubleval each
ebcdic2ascii echo empty end ereg eregi escapeshellarg escapeshellcmd eval exec exit exp explode expm1 extract
fclose feof fflush fgetc fgetcsv fgets fgetss file fileatime filectime filegroup fileinode filemtime fileowner
fileperms filepro filesize filetype floatval flock floor flush fmod fnmatch fopen fpassthru fprintf fputcsv
fputs fread frenchtojd fscanf fseek fsockopen fstat ftell ftok ftruncate fwrite getallheaders getcwd getdate
getenv gethostbyaddr gethostbyname gethostbynamel getimagesize getlastmod getmxrr getmygid getmyinode getmypid
getmyuid getopt getprotobyname getprotobynumber getrandmax getrusage getservbyname getservbyport gettext
gettimeofday gettype glob gmdate gmmktime gmstrftime gregoriantojd gzclose gzcompress gzdecode gzdeflate
gzencode gzeof gzfile gzgetc gzgets gzgetss gzinflate gzopen gzpassthru gzputs gzread gzrewind gzseek gztell
gzuncompress gzwrite hash header hebrev hebrevc hexdec htmlentities htmlspecialchars hypot iconv idate
implode include intval ip2long iptcembed iptcparse isset
jddayofweek jdmonthname jdtofrench jdtogregorian jdtojewish jdtojulian jdtounix jewishtojd join jpeg2wbmp
juliantojd key krsort ksort lcfirst lchgrp lchown levenshtein link linkinfo list localeconv localtime log
log10 log1p long2ip lstat ltrim mail main max md5 metaphone mhash microtime min mkdir mktime msql natcasesort
natsort next ngettext nl2br nthmac octdec opendir openlog
ord overload pack passthru pathinfo pclose pfsockopen phpcredits phpinfo phpversion pi png2wbmp popen pos pow
prev print printf putenv quotemeta rad2deg rand range rawurldecode rawurlencode readdir readfile readgzfile
readline readlink realpath recode rename require reset rewind rewinddir rmdir round rsort rtrim scandir
serialize setcookie setlocale setrawcookie settype sha1 shuffle signeurlpaiement sin sinh sizeof sleep snmpget
snmpgetnext snmprealwalk snmpset snmpwalk snmpwalkoid sort soundex split spliti sprintf sqrt srand sscanf stat
strcasecmp strchr strcmp strcoll strcspn strftime stripcslashes stripos stripslashes stristr strlen
strnatcasecmp strnatcmp strncasecmp strncmp strpbrk strpos strptime strrchr strrev strripos strrpos strspn
strstr strtok strtolower strtotime strtoupper strtr strval substr symlink syslog system tan tanh tempnam
textdomain time tmpfile touch trim uasort ucfirst ucwords uksort umask uniqid unixtojd unlink unpack
unserialize unset urldecode urlencode usleep usort vfprintf virtual vprintf vsprintf wordwrap
array_change_key_case array_chunk array_combine array_count_values array_diff array_diff_assoc
array_diff_key array_diff_uassoc array_diff_ukey array_fill array_fill_keys array_filter array_flip
array_intersect array_intersect_assoc array_intersect_key array_intersect_uassoc array_intersect_ukey
array_key_exists array_keys array_map array_merge array_merge_recursive array_multisort array_pad
array_pop array_product array_push array_rand array_reduce array_reverse array_search array_shift
array_slice array_splice array_sum array_udiff array_udiff_assoc array_udiff_uassoc array_uintersect
array_uintersect_assoc array_uintersect_uassoc array_unique array_unshift array_values array_walk
array_walk_recursive
assert_options base_convert base64_decode base64_encode
chunk_split class_exists class_implements class_parents
count_chars debug_backtrace debug_print_backtrace debug_zval_dump
error_get_last error_log error_reporting extension_loaded
file_exists file_get_contents file_put_contents load_file
func_get_arg func_get_args func_num_args function_exists
get_browser get_called_class get_cfg_var get_class get_class_methods get_class_vars
get_current_user get_declared_classes get_declared_interfaces get_defined_constants
get_defined_functions get_defined_vars get_extension_funcs get_headers get_html_translation_table
get_include_path get_included_files get_loaded_extensions get_magic_quotes_gpc get_magic_quotes_runtime
get_meta_tags get_object_vars get_parent_class get_required_filesget_resource_type
gc_collect_cycles gc_disable gc_enable gc_enabled
halt_compiler headers_list headers_sent highlight_file highlight_string
html_entity_decode htmlspecialchars_decode
in_array include_once inclued_get_data
is_a is_array is_binary is_bool is_buffer is_callable is_dir is_double is_executable is_file is_finite
is_float is_infinite is_int is_integer is_link is_long is_nan is_null is_numeric is_object is_readable
is_real is_resource is_scalar is_soap_fault is_string is_subclass_of is_unicode is_uploaded_file
is_writable is_writeable
locale_get_default locale_set_default
number_format override_function parse_str parse_url
php_check_syntax php_ini_loaded_file php_ini_scanned_files php_logo_guid php_sapi_name
php_strip_whitespace php_uname
preg_filter preg_grep preg_last_error preg_match preg_match_all preg_quote preg_replace
preg_replace_callback preg_split print_r
require_once register_shutdown_function register_tick_function
set_error_handler set_exception_handler set_file_buffer set_include_path
set_magic_quotes_runtime set_time_limit shell_exec
str_getcsv str_ireplace str_pad str_repeat str_replace str_rot13 str_shuffle str_split str_word_count
strip_tags substr_compare substr_count substr_replace
time_nanosleep time_sleep_until
token_get_all token_name trigger_error
unregister_tick_function use_soap_error_handler user_error
utf8_decode utf8_encode var_dump var_export
version_compare
zend_logo_guid zend_thread_id zend_version
create_function call_user_func_array
posix_access posix_ctermid posix_get_last_error posix_getcwd posix_getegid
posix_geteuid posix_getgid posix_getgrgid posix_getgrnam posix_getgroups
posix_getlogin posix_getpgid posix_getpgrp posix_getpid posix_getppid
posix_getpwnam posix_getpwuid posix_getrlimit posix_getsid posix_getuid
posix_initgroups posix_isatty posix_kill posix_mkfifo posix_mknod
posix_setegid posix_seteuid posix_setgid posix_setpgid posix_setsid
posix_setuid posix_strerror posix_times posix_ttyname posix_uname
pcntl_alarm pcntl_exec pcntl_fork pcntl_getpriority pcntl_setpriority
pcntl_signal pcntl_signal_dispatch pcntl_sigprocmask pcntl_sigtimedwait
pcntl_sigwaitinfo pcntl_wait pcntl_waitpid pcntl_wexitstatus pcntl_wifexited
pcntl_wifsignaled pcntl_wifstopped pcntl_wstopsig pcntl_wtermsig
]
# TODO: more built-in PHP functions?
EXCEPTIONS = %w[
E_ERROR E_WARNING E_PARSE E_NOTICE E_CORE_ERROR E_CORE_WARNING E_COMPILE_ERROR E_COMPILE_WARNING
E_USER_ERROR E_USER_WARNING E_USER_NOTICE E_DEPRECATED E_USER_DEPRECATED E_ALL E_STRICT
]
CONSTANTS = %w[
null true false self parent
__LINE__ __DIR__ __FILE__ __LINE__
__CLASS__ __NAMESPACE__ __METHOD__ __FUNCTION__
PHP_VERSION PHP_MAJOR_VERSION PHP_MINOR_VERSION PHP_RELEASE_VERSION PHP_VERSION_ID PHP_EXTRA_VERSION PHP_ZTS
PHP_DEBUG PHP_MAXPATHLEN PHP_OS PHP_SAPI PHP_EOL PHP_INT_MAX PHP_INT_SIZE DEFAULT_INCLUDE_PATH
PEAR_INSTALL_DIR PEAR_EXTENSION_DIR PHP_EXTENSION_DIR PHP_PREFIX PHP_BINDIR PHP_LIBDIR PHP_DATADIR
PHP_SYSCONFDIR PHP_LOCALSTATEDIR PHP_CONFIG_FILE_PATH PHP_CONFIG_FILE_SCAN_DIR PHP_SHLIB_SUFFIX
PHP_OUTPUT_HANDLER_START PHP_OUTPUT_HANDLER_CONT PHP_OUTPUT_HANDLER_END
__COMPILER_HALT_OFFSET__
EXTR_OVERWRITE EXTR_SKIP EXTR_PREFIX_SAME EXTR_PREFIX_ALL EXTR_PREFIX_INVALID EXTR_PREFIX_IF_EXISTS
EXTR_IF_EXISTS SORT_ASC SORT_DESC SORT_REGULAR SORT_NUMERIC SORT_STRING CASE_LOWER CASE_UPPER COUNT_NORMAL
COUNT_RECURSIVE ASSERT_ACTIVE ASSERT_CALLBACK ASSERT_BAIL ASSERT_WARNING ASSERT_QUIET_EVAL CONNECTION_ABORTED
CONNECTION_NORMAL CONNECTION_TIMEOUT INI_USER INI_PERDIR INI_SYSTEM INI_ALL M_E M_LOG2E M_LOG10E M_LN2 M_LN10
M_PI M_PI_2 M_PI_4 M_1_PI M_2_PI M_2_SQRTPI M_SQRT2 M_SQRT1_2 CRYPT_SALT_LENGTH CRYPT_STD_DES CRYPT_EXT_DES
CRYPT_MD5 CRYPT_BLOWFISH DIRECTORY_SEPARATOR SEEK_SET SEEK_CUR SEEK_END LOCK_SH LOCK_EX LOCK_UN LOCK_NB
HTML_SPECIALCHARS HTML_ENTITIES ENT_COMPAT ENT_QUOTES ENT_NOQUOTES INFO_GENERAL INFO_CREDITS
INFO_CONFIGURATION INFO_MODULES INFO_ENVIRONMENT INFO_VARIABLES INFO_LICENSE INFO_ALL CREDITS_GROUP
CREDITS_GENERAL CREDITS_SAPI CREDITS_MODULES CREDITS_DOCS CREDITS_FULLPAGE CREDITS_QA CREDITS_ALL STR_PAD_LEFT
STR_PAD_RIGHT STR_PAD_BOTH PATHINFO_DIRNAME PATHINFO_BASENAME PATHINFO_EXTENSION PATH_SEPARATOR CHAR_MAX
LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_ALL LC_MESSAGES ABDAY_1 ABDAY_2 ABDAY_3 ABDAY_4 ABDAY_5
ABDAY_6 ABDAY_7 DAY_1 DAY_2 DAY_3 DAY_4 DAY_5 DAY_6 DAY_7 ABMON_1 ABMON_2 ABMON_3 ABMON_4 ABMON_5 ABMON_6
ABMON_7 ABMON_8 ABMON_9 ABMON_10 ABMON_11 ABMON_12 MON_1 MON_2 MON_3 MON_4 MON_5 MON_6 MON_7 MON_8 MON_9
MON_10 MON_11 MON_12 AM_STR PM_STR D_T_FMT D_FMT T_FMT T_FMT_AMPM ERA ERA_YEAR ERA_D_T_FMT ERA_D_FMT ERA_T_FMT
ALT_DIGITS INT_CURR_SYMBOL CURRENCY_SYMBOL CRNCYSTR MON_DECIMAL_POINT MON_THOUSANDS_SEP MON_GROUPING
POSITIVE_SIGN NEGATIVE_SIGN INT_FRAC_DIGITS FRAC_DIGITS P_CS_PRECEDES P_SEP_BY_SPACE N_CS_PRECEDES
N_SEP_BY_SPACE P_SIGN_POSN N_SIGN_POSN DECIMAL_POINT RADIXCHAR THOUSANDS_SEP THOUSEP GROUPING YESEXPR NOEXPR
YESSTR NOSTR CODESET LOG_EMERG LOG_ALERT LOG_CRIT LOG_ERR LOG_WARNING LOG_NOTICE LOG_INFO LOG_DEBUG LOG_KERN
LOG_USER LOG_MAIL LOG_DAEMON LOG_AUTH LOG_SYSLOG LOG_LPR LOG_NEWS LOG_UUCP LOG_CRON LOG_AUTHPRIV LOG_LOCAL0
LOG_LOCAL1 LOG_LOCAL2 LOG_LOCAL3 LOG_LOCAL4 LOG_LOCAL5 LOG_LOCAL6 LOG_LOCAL7 LOG_PID LOG_CONS LOG_ODELAY
LOG_NDELAY LOG_NOWAIT LOG_PERROR
]
PREDEFINED = %w[
$GLOBALS $_SERVER $_GET $_POST $_FILES $_REQUEST $_SESSION $_ENV
$_COOKIE $php_errormsg $HTTP_RAW_POST_DATA $http_response_header
$argc $argv
]
IDENT_KIND = WordList::CaseIgnoring.new(:ident).
add(KEYWORDS, :keyword).
add(TYPES, :predefined_type).
add(LANGUAGE_CONSTRUCTS, :keyword).
add(BUILTIN_FUNCTIONS, :predefined).
add(CLASSES, :predefined_constant).
add(EXCEPTIONS, :exception).
add(CONSTANTS, :predefined_constant)
VARIABLE_KIND = WordList.new(:local_variable).
add(PREDEFINED, :predefined)
end
module RE # :nodoc:
PHP_START = /
<script\s+[^>]*?language\s*=\s*"php"[^>]*?> |
<script\s+[^>]*?language\s*=\s*'php'[^>]*?> |
<\?php\d? |
<\?(?!xml)
/xi
PHP_END = %r!
</script> |
\?>
!xi
HTML_INDICATOR = /<!DOCTYPE html|<(?:html|body|div|p)[> ]/i
IDENTIFIER = /[a-z_\x7f-\xFF][a-z0-9_\x7f-\xFF]*/i
VARIABLE = /\$#{IDENTIFIER}/
OPERATOR = /
\.(?!\d)=? | # dot that is not decimal point, string concatenation
&& | \|\| | # logic
:: | -> | => | # scope, member, dictionary
\\(?!\n) | # namespace
\+\+ | -- | # increment, decrement
[,;?:()\[\]{}] | # simple delimiters
[-+*\/%&|^]=? | # ordinary math, binary logic, assignment shortcuts
[~$] | # whatever
=& | # reference assignment
[=!]=?=? | <> | # comparison and assignment
<<=? | >>=? | [<>]=? # comparison and shift
/x
end
protected
def scan_tokens encoder, options
if check(RE::PHP_START) || # starts with <?
(match?(/\s*<\S/) && check(/.{1,1000}#{RE::PHP_START}/om)) || # starts with tag and contains <?
check(/.{0,1000}#{RE::HTML_INDICATOR}/om) ||
check(/.{1,100}#{RE::PHP_START}/om) # PHP start after max 100 chars
# is HTML with embedded PHP, so start with HTML
states = [:initial]
else
# is just PHP, so start with PHP surrounded by HTML
states = [:initial, :php]
end
label_expected = true
case_expected = false
heredoc_delimiter = nil
delimiter = nil
modifier = nil
until eos?
case states.last
when :initial # HTML
if match = scan(RE::PHP_START)
encoder.text_token match, :inline_delimiter
label_expected = true
states << :php
else
match = scan_until(/(?=#{RE::PHP_START})/o) || scan_rest
@html_scanner.tokenize match unless match.empty?
end
when :php
if match = scan(/\s+/)
encoder.text_token match, :space
elsif match = scan(%r! (?m: \/\* (?: .*? \*\/ | .* ) ) | (?://|\#) .*? (?=#{RE::PHP_END}|$) !xo)
encoder.text_token match, :comment
elsif match = scan(RE::IDENTIFIER)
kind = Words::IDENT_KIND[match]
if kind == :ident && label_expected && check(/:(?!:)/)
kind = :label
label_expected = true
else
label_expected = false
if kind == :ident && match =~ /^[A-Z]/
kind = :constant
elsif kind == :keyword
case match
when 'class'
states << :class_expected
when 'function'
states << :function_expected
when 'case', 'default'
case_expected = true
end
elsif match == 'b' && check(/['"]/) # binary string literal
modifier = match
next
end
end
encoder.text_token match, kind
elsif match = scan(/(?:\d+\.\d*|\d*\.\d+)(?:e[-+]?\d+)?|\d+e[-+]?\d+/i)
label_expected = false
encoder.text_token match, :float
elsif match = scan(/0x[0-9a-fA-F]+/)
label_expected = false
encoder.text_token match, :hex
elsif match = scan(/\d+/)
label_expected = false
encoder.text_token match, :integer
elsif match = scan(/['"`]/)
encoder.begin_group :string
if modifier
encoder.text_token modifier, :modifier
modifier = nil
end
delimiter = match
encoder.text_token match, :delimiter
states.push match == "'" ? :sqstring : :dqstring
elsif match = scan(RE::VARIABLE)
label_expected = false
encoder.text_token match, Words::VARIABLE_KIND[match]
elsif match = scan(/\{/)
encoder.text_token match, :operator
label_expected = true
states.push :php
elsif match = scan(/\}/)
if states.size == 1
encoder.text_token match, :error
else
states.pop
if states.last.is_a?(::Array)
delimiter = states.last[1]
states[-1] = states.last[0]
encoder.text_token match, :delimiter
encoder.end_group :inline
else
encoder.text_token match, :operator
label_expected = true
end
end
elsif match = scan(/@/)
label_expected = false
encoder.text_token match, :exception
elsif match = scan(RE::PHP_END)
encoder.text_token match, :inline_delimiter
states = [:initial]
elsif match = scan(/<<<(?:(#{RE::IDENTIFIER})|"(#{RE::IDENTIFIER})"|'(#{RE::IDENTIFIER})')/o)
encoder.begin_group :string
# warn 'heredoc in heredoc?' if heredoc_delimiter
heredoc_delimiter = Regexp.escape(self[1] || self[2] || self[3])
encoder.text_token match, :delimiter
states.push self[3] ? :sqstring : :dqstring
heredoc_delimiter = /#{heredoc_delimiter}(?=;?$)/
elsif match = scan(/#{RE::OPERATOR}/o)
label_expected = match == ';'
if case_expected
label_expected = true if match == ':'
case_expected = false
end
encoder.text_token match, :operator
else
encoder.text_token getch, :error
end
when :sqstring
if match = scan(heredoc_delimiter ? /[^\\\n]+/ : /[^'\\]+/)
encoder.text_token match, :content
elsif !heredoc_delimiter && match = scan(/'/)
encoder.text_token match, :delimiter
encoder.end_group :string
delimiter = nil
label_expected = false
states.pop
elsif heredoc_delimiter && match = scan(/\n/)
if scan heredoc_delimiter
encoder.text_token "\n", :content
encoder.text_token matched, :delimiter
encoder.end_group :string
heredoc_delimiter = nil
label_expected = false
states.pop
else
encoder.text_token match, :content
end
elsif match = scan(heredoc_delimiter ? /\\\\/ : /\\[\\'\n]/)
encoder.text_token match, :char
elsif match = scan(/\\./m)
encoder.text_token match, :content
elsif match = scan(/\\/)
encoder.text_token match, :error
else
states.pop
end
when :dqstring
if match = scan(heredoc_delimiter ? /[^${\\\n]+/ : (delimiter == '"' ? /[^"${\\]+/ : /[^`${\\]+/))
encoder.text_token match, :content
elsif !heredoc_delimiter && match = scan(delimiter == '"' ? /"/ : /`/)
encoder.text_token match, :delimiter
encoder.end_group :string
delimiter = nil
label_expected = false
states.pop
elsif heredoc_delimiter && match = scan(/\n/)
if scan heredoc_delimiter
encoder.text_token "\n", :content
encoder.text_token matched, :delimiter
encoder.end_group :string
heredoc_delimiter = nil
label_expected = false
states.pop
else
encoder.text_token match, :content
end
elsif match = scan(/\\(?:x[0-9A-Fa-f]{1,2}|[0-7]{1,3})/)
encoder.text_token match, :char
elsif match = scan(heredoc_delimiter ? /\\[nrtvf\\$]/ : (delimiter == '"' ? /\\[nrtvf\\$"]/ : /\\[nrtvf\\$`]/))
encoder.text_token match, :char
elsif match = scan(/\\./m)
encoder.text_token match, :content
elsif match = scan(/\\/)
encoder.text_token match, :error
elsif match = scan(/#{RE::VARIABLE}/o)
if check(/\[#{RE::IDENTIFIER}\]/o)
encoder.begin_group :inline
encoder.text_token match, :local_variable
encoder.text_token scan(/\[/), :operator
encoder.text_token scan(/#{RE::IDENTIFIER}/o), :ident
encoder.text_token scan(/\]/), :operator
encoder.end_group :inline
elsif check(/\[/)
match << scan(/\[['"]?#{RE::IDENTIFIER}?['"]?\]?/o)
encoder.text_token match, :error
elsif check(/->#{RE::IDENTIFIER}/o)
encoder.begin_group :inline
encoder.text_token match, :local_variable
encoder.text_token scan(/->/), :operator
encoder.text_token scan(/#{RE::IDENTIFIER}/o), :ident
encoder.end_group :inline
elsif check(/->/)
match << scan(/->/)
encoder.text_token match, :error
else
encoder.text_token match, :local_variable
end
elsif match = scan(/\{/)
if check(/\$/)
encoder.begin_group :inline
states[-1] = [states.last, delimiter]
delimiter = nil
states.push :php
encoder.text_token match, :delimiter
else
encoder.text_token match, :content
end
elsif match = scan(/\$\{#{RE::IDENTIFIER}\}/o)
encoder.text_token match, :local_variable
elsif match = scan(/\$/)
encoder.text_token match, :content
else
states.pop
end
when :class_expected
if match = scan(/\s+/)
encoder.text_token match, :space
elsif match = scan(/#{RE::IDENTIFIER}/o)
encoder.text_token match, :class
states.pop
else
states.pop
end
when :function_expected
if match = scan(/\s+/)
encoder.text_token match, :space
elsif match = scan(/&/)
encoder.text_token match, :operator
elsif match = scan(/#{RE::IDENTIFIER}/o)
encoder.text_token match, :function
states.pop
else
states.pop
end
else
raise_inspect 'Unknown state!', encoder, states
end
end
encoder
end
end
end
end

View File

@ -1,287 +0,0 @@
module CodeRay
module Scanners
# Scanner for Python. Supports Python 3.
#
# Based on pygments' PythonLexer, see
# http://dev.pocoo.org/projects/pygments/browser/pygments/lexers/agile.py.
class Python < Scanner
register_for :python
file_extension 'py'
KEYWORDS = [
'and', 'as', 'assert', 'break', 'class', 'continue', 'def',
'del', 'elif', 'else', 'except', 'finally', 'for',
'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'not',
'or', 'pass', 'raise', 'return', 'try', 'while', 'with', 'yield',
'nonlocal', # new in Python 3
] # :nodoc:
OLD_KEYWORDS = [
'exec', 'print', # gone in Python 3
] # :nodoc:
PREDEFINED_METHODS_AND_TYPES = %w[
__import__ abs all any apply basestring bin bool buffer
bytearray bytes callable chr classmethod cmp coerce compile
complex delattr dict dir divmod enumerate eval execfile exit
file filter float frozenset getattr globals hasattr hash hex id
input int intern isinstance issubclass iter len list locals
long map max min next object oct open ord pow property range
raw_input reduce reload repr reversed round set setattr slice
sorted staticmethod str sum super tuple type unichr unicode
vars xrange zip
] # :nodoc:
PREDEFINED_EXCEPTIONS = %w[
ArithmeticError AssertionError AttributeError
BaseException DeprecationWarning EOFError EnvironmentError
Exception FloatingPointError FutureWarning GeneratorExit IOError
ImportError ImportWarning IndentationError IndexError KeyError
KeyboardInterrupt LookupError MemoryError NameError
NotImplemented NotImplementedError OSError OverflowError
OverflowWarning PendingDeprecationWarning ReferenceError
RuntimeError RuntimeWarning StandardError StopIteration
SyntaxError SyntaxWarning SystemError SystemExit TabError
TypeError UnboundLocalError UnicodeDecodeError
UnicodeEncodeError UnicodeError UnicodeTranslateError
UnicodeWarning UserWarning ValueError Warning ZeroDivisionError
] # :nodoc:
PREDEFINED_VARIABLES_AND_CONSTANTS = [
'False', 'True', 'None', # "keywords" since Python 3
'self', 'Ellipsis', 'NotImplemented',
] # :nodoc:
IDENT_KIND = WordList.new(:ident).
add(KEYWORDS, :keyword).
add(OLD_KEYWORDS, :old_keyword).
add(PREDEFINED_METHODS_AND_TYPES, :predefined).
add(PREDEFINED_VARIABLES_AND_CONSTANTS, :predefined_constant).
add(PREDEFINED_EXCEPTIONS, :exception) # :nodoc:
NAME = / [[:alpha:]_] \w* /x # :nodoc:
ESCAPE = / [abfnrtv\n\\'"] | x[a-fA-F0-9]{1,2} | [0-7]{1,3} /x # :nodoc:
UNICODE_ESCAPE = / u[a-fA-F0-9]{4} | U[a-fA-F0-9]{8} | N\{[-\w ]+\} /x # :nodoc:
OPERATOR = /
\.\.\. | # ellipsis
\.(?!\d) | # dot but not decimal point
[,;:()\[\]{}] | # simple delimiters
\/\/=? | \*\*=? | # special math
[-+*\/%&|^]=? | # ordinary math and binary logic
[~`] | # binary complement and inspection
<<=? | >>=? | [<>=]=? | != # comparison and assignment
/x # :nodoc:
STRING_DELIMITER_REGEXP = Hash.new { |h, delimiter|
h[delimiter] = Regexp.union delimiter # :nodoc:
}
STRING_CONTENT_REGEXP = Hash.new { |h, delimiter|
h[delimiter] = / [^\\\n]+? (?= \\ | $ | #{Regexp.escape(delimiter)} ) /x # :nodoc:
}
DEF_NEW_STATE = WordList.new(:initial).
add(%w(def), :def_expected).
add(%w(import from), :include_expected).
add(%w(class), :class_expected) # :nodoc:
DESCRIPTOR = /
#{NAME}
(?: \. #{NAME} )*
| \*
/x # :nodoc:
DOCSTRING_COMING = /
[ \t]* u?r? ("""|''')
/x # :nodoc:
protected
def scan_tokens encoder, options
state = :initial
string_delimiter = nil
string_raw = false
string_type = nil
docstring_coming = match?(/#{DOCSTRING_COMING}/o)
last_token_dot = false
unicode = string.respond_to?(:encoding) && string.encoding.name == 'UTF-8'
from_import_state = []
until eos?
if state == :string
if match = scan(STRING_DELIMITER_REGEXP[string_delimiter])
encoder.text_token match, :delimiter
encoder.end_group string_type
string_type = nil
state = :initial
next
elsif string_delimiter.size == 3 && match = scan(/\n/)
encoder.text_token match, :content
elsif match = scan(STRING_CONTENT_REGEXP[string_delimiter])
encoder.text_token match, :content
elsif !string_raw && match = scan(/ \\ #{ESCAPE} /ox)
encoder.text_token match, :char
elsif match = scan(/ \\ #{UNICODE_ESCAPE} /ox)
encoder.text_token match, :char
elsif match = scan(/ \\ . /x)
encoder.text_token match, :content
elsif match = scan(/ \\ | $ /x)
encoder.end_group string_type
string_type = nil
encoder.text_token match, :error
state = :initial
else
raise_inspect "else case \" reached; %p not handled." % peek(1), encoder, state
end
elsif match = scan(/ [ \t]+ | \\?\n /x)
encoder.text_token match, :space
if match == "\n"
state = :initial if state == :include_expected
docstring_coming = true if match?(/#{DOCSTRING_COMING}/o)
end
next
elsif match = scan(/ \# [^\n]* /mx)
encoder.text_token match, :comment
next
elsif state == :initial
if match = scan(/#{OPERATOR}/o)
encoder.text_token match, :operator
elsif match = scan(/(u?r?|b)?("""|"|'''|')/i)
string_delimiter = self[2]
string_type = docstring_coming ? :docstring : :string
docstring_coming = false if docstring_coming
encoder.begin_group string_type
string_raw = false
modifiers = self[1]
unless modifiers.empty?
string_raw = !!modifiers.index(?r)
encoder.text_token modifiers, :modifier
match = string_delimiter
end
state = :string
encoder.text_token match, :delimiter
# TODO: backticks
elsif match = scan(unicode ? /#{NAME}/uo : /#{NAME}/o)
kind = IDENT_KIND[match]
# TODO: keyword arguments
kind = :ident if last_token_dot
if kind == :old_keyword
kind = check(/\(/) ? :ident : :keyword
elsif kind == :predefined && check(/ *=/)
kind = :ident
elsif kind == :keyword
state = DEF_NEW_STATE[match]
from_import_state << match.to_sym if state == :include_expected
end
encoder.text_token match, kind
elsif match = scan(/@[a-zA-Z0-9_.]+[lL]?/)
encoder.text_token match, :decorator
elsif match = scan(/0[xX][0-9A-Fa-f]+[lL]?/)
encoder.text_token match, :hex
elsif match = scan(/0[bB][01]+[lL]?/)
encoder.text_token match, :binary
elsif match = scan(/(?:\d*\.\d+|\d+\.\d*)(?:[eE][+-]?\d+)?|\d+[eE][+-]?\d+/)
if scan(/[jJ]/)
match << matched
encoder.text_token match, :imaginary
else
encoder.text_token match, :float
end
elsif match = scan(/0[oO][0-7]+|0[0-7]+(?![89.eE])[lL]?/)
encoder.text_token match, :octal
elsif match = scan(/\d+([lL])?/)
if self[1] == nil && scan(/[jJ]/)
match << matched
encoder.text_token match, :imaginary
else
encoder.text_token match, :integer
end
else
encoder.text_token getch, :error
end
elsif state == :def_expected
state = :initial
if match = scan(unicode ? /#{NAME}/uo : /#{NAME}/o)
encoder.text_token match, :method
else
next
end
elsif state == :class_expected
state = :initial
if match = scan(unicode ? /#{NAME}/uo : /#{NAME}/o)
encoder.text_token match, :class
else
next
end
elsif state == :include_expected
if match = scan(unicode ? /#{DESCRIPTOR}/uo : /#{DESCRIPTOR}/o)
if match == 'as'
encoder.text_token match, :keyword
from_import_state << :as
elsif from_import_state.first == :from && match == 'import'
encoder.text_token match, :keyword
from_import_state << :import
elsif from_import_state.last == :as
# encoder.text_token match, match[0,1][unicode ? /[[:upper:]]/u : /[[:upper:]]/] ? :class : :method
encoder.text_token match, :ident
from_import_state.pop
elsif IDENT_KIND[match] == :keyword
unscan
match = nil
state = :initial
next
else
encoder.text_token match, :include
end
elsif match = scan(/,/)
from_import_state.pop if from_import_state.last == :as
encoder.text_token match, :operator
else
from_import_state = []
state = :initial
next
end
else
raise_inspect 'Unknown state', encoder, state
end
last_token_dot = match == '.'
end
if state == :string
encoder.end_group string_type
end
encoder
end
end
end
end

View File

@ -1,66 +0,0 @@
module CodeRay
module Scanners
# = Debug Scanner
#
# Parses the output of the Encoders::Debug encoder.
class Raydebug < Scanner
register_for :raydebug
file_extension 'raydebug'
title 'CodeRay Token Dump'
protected
def scan_tokens encoder, options
opened_tokens = []
until eos?
if match = scan(/\s+/)
encoder.text_token match, :space
elsif match = scan(/ (\w+) \( ( [^\)\\]* ( \\. [^\)\\]* )* ) /x)
kind = self[1]
encoder.text_token kind, :class
encoder.text_token '(', :operator
match = self[2]
encoder.text_token match, kind.to_sym
encoder.text_token match, :operator if match = scan(/\)/)
elsif match = scan(/ (\w+) ([<\[]) /x)
kind = self[1]
case self[2]
when '<'
encoder.text_token kind, :class
when '['
encoder.text_token kind, :class
else
raise 'CodeRay bug: This case should not be reached.'
end
kind = kind.to_sym
opened_tokens << kind
encoder.begin_group kind
encoder.text_token self[2], :operator
elsif !opened_tokens.empty? && match = scan(/ [>\]] /x)
encoder.text_token match, :operator
encoder.end_group opened_tokens.pop
else
encoder.text_token getch, :space
end
end
encoder.end_group opened_tokens.pop until opened_tokens.empty?
encoder
end
end
end
end

View File

@ -1,461 +0,0 @@
module CodeRay
module Scanners
# This scanner is really complex, since Ruby _is_ a complex language!
#
# It tries to highlight 100% of all common code,
# and 90% of strange codes.
#
# It is optimized for HTML highlighting, and is not very useful for
# parsing or pretty printing.
class Ruby < Scanner
register_for :ruby
file_extension 'rb'
autoload :Patterns, CodeRay.coderay_path('scanners', 'ruby', 'patterns')
autoload :StringState, CodeRay.coderay_path('scanners', 'ruby', 'string_state')
def interpreted_string_state
StringState.new :string, true, '"'
end
protected
def setup
@state = :initial
end
def scan_tokens encoder, options
state, heredocs = options[:state] || @state
heredocs = heredocs.dup if heredocs.is_a?(Array)
if state && state.instance_of?(StringState)
encoder.begin_group state.type
end
last_state = nil
method_call_expected = false
value_expected = true
inline_block_stack = nil
inline_block_curly_depth = 0
if heredocs
state = heredocs.shift
encoder.begin_group state.type
heredocs = nil if heredocs.empty?
end
# def_object_stack = nil
# def_object_paren_depth = 0
patterns = Patterns # avoid constant lookup
unicode = string.respond_to?(:encoding) && string.encoding.name == 'UTF-8'
until eos?
if state.instance_of? ::Symbol
if match = scan(/[ \t\f\v]+/)
encoder.text_token match, :space
elsif match = scan(/\n/)
if heredocs
unscan # heredoc scanning needs \n at start
state = heredocs.shift
encoder.begin_group state.type
heredocs = nil if heredocs.empty?
else
state = :initial if state == :undef_comma_expected
encoder.text_token match, :space
value_expected = true
end
elsif match = scan(bol? ? / \#(!)?.* | #{patterns::RUBYDOC_OR_DATA} /ox : /\#.*/)
encoder.text_token match, self[1] ? :doctype : :comment
elsif match = scan(/\\\n/)
if heredocs
unscan # heredoc scanning needs \n at start
encoder.text_token scan(/\\/), :space
state = heredocs.shift
encoder.begin_group state.type
heredocs = nil if heredocs.empty?
else
encoder.text_token match, :space
end
elsif state == :initial
# IDENTS #
if !method_call_expected &&
match = scan(unicode ? /#{patterns::METHOD_NAME}/uo :
/#{patterns::METHOD_NAME}/o)
value_expected = false
kind = patterns::IDENT_KIND[match]
if kind == :ident
if match[/\A[A-Z]/] && !(match[/[!?]$/] || match?(/\(/))
kind = :constant
end
elsif kind == :keyword
state = patterns::KEYWORD_NEW_STATE[match]
value_expected = true if patterns::KEYWORDS_EXPECTING_VALUE[match]
end
value_expected = true if !value_expected && check(/#{patterns::VALUE_FOLLOWS}/o)
encoder.text_token match, kind
elsif method_call_expected &&
match = scan(unicode ? /#{patterns::METHOD_AFTER_DOT}/uo :
/#{patterns::METHOD_AFTER_DOT}/o)
if method_call_expected == '::' && match[/\A[A-Z]/] && !match?(/\(/)
encoder.text_token match, :constant
else
encoder.text_token match, :ident
end
method_call_expected = false
value_expected = check(/#{patterns::VALUE_FOLLOWS}/o)
# OPERATORS #
elsif !method_call_expected && match = scan(/ (\.(?!\.)|::) | (?: \.\.\.? | ==?=? | [,\(\[\{] )() | [\)\]\}] /x)
method_call_expected = self[1]
value_expected = !method_call_expected && self[2]
if inline_block_stack
case match
when '{'
inline_block_curly_depth += 1
when '}'
inline_block_curly_depth -= 1
if inline_block_curly_depth == 0 # closing brace of inline block reached
state, inline_block_curly_depth, heredocs = inline_block_stack.pop
inline_block_stack = nil if inline_block_stack.empty?
heredocs = nil if heredocs && heredocs.empty?
encoder.text_token match, :inline_delimiter
encoder.end_group :inline
next
end
end
end
encoder.text_token match, :operator
elsif match = scan(unicode ? /#{patterns::SYMBOL}/uo :
/#{patterns::SYMBOL}/o)
case delim = match[1]
when ?', ?"
encoder.begin_group :symbol
encoder.text_token ':', :symbol
match = delim.chr
encoder.text_token match, :delimiter
state = self.class::StringState.new :symbol, delim == ?", match
else
encoder.text_token match, :symbol
value_expected = false
end
elsif match = scan(/ ' (?:(?>[^'\\]*) ')? | " (?:(?>[^"\\\#]*) ")? /mx)
encoder.begin_group :string
if match.size == 1
encoder.text_token match, :delimiter
state = self.class::StringState.new :string, match == '"', match # important for streaming
else
encoder.text_token match[0,1], :delimiter
encoder.text_token match[1..-2], :content if match.size > 2
encoder.text_token match[-1,1], :delimiter
encoder.end_group :string
value_expected = false
end
elsif match = scan(unicode ? /#{patterns::INSTANCE_VARIABLE}/uo :
/#{patterns::INSTANCE_VARIABLE}/o)
value_expected = false
encoder.text_token match, :instance_variable
elsif value_expected && match = scan(/\//)
encoder.begin_group :regexp
encoder.text_token match, :delimiter
state = self.class::StringState.new :regexp, true, '/'
elsif match = scan(value_expected ? /[-+]?#{patterns::NUMERIC}/o : /#{patterns::NUMERIC}/o)
if method_call_expected
encoder.text_token match, :error
method_call_expected = false
else
encoder.text_token match, self[1] ? :float : :integer # TODO: send :hex/:octal/:binary
end
value_expected = false
elsif match = scan(/ [-+!~^\/]=? | [:;] | [*|&]{1,2}=? | >>? /x)
value_expected = true
encoder.text_token match, :operator
elsif value_expected && match = scan(/#{patterns::HEREDOC_OPEN}/o)
quote = self[3]
delim = self[quote ? 4 : 2]
kind = patterns::QUOTE_TO_TYPE[quote]
encoder.begin_group kind
encoder.text_token match, :delimiter
encoder.end_group kind
heredocs ||= [] # create heredocs if empty
heredocs << self.class::StringState.new(kind, quote != "'", delim,
self[1] == '-' ? :indented : :linestart)
value_expected = false
elsif value_expected && match = scan(/#{patterns::FANCY_STRING_START}/o)
kind = patterns::FANCY_STRING_KIND[self[1]]
encoder.begin_group kind
state = self.class::StringState.new kind, patterns::FANCY_STRING_INTERPRETED[self[1]], self[2]
encoder.text_token match, :delimiter
elsif value_expected && match = scan(/#{patterns::CHARACTER}/o)
value_expected = false
encoder.text_token match, :integer
elsif match = scan(/ %=? | <(?:<|=>?)? | \? /x)
value_expected = true
encoder.text_token match, :operator
elsif match = scan(/`/)
encoder.begin_group :shell
encoder.text_token match, :delimiter
state = self.class::StringState.new :shell, true, match
elsif match = scan(unicode ? /#{patterns::GLOBAL_VARIABLE}/uo :
/#{patterns::GLOBAL_VARIABLE}/o)
encoder.text_token match, :global_variable
value_expected = false
elsif match = scan(unicode ? /#{patterns::CLASS_VARIABLE}/uo :
/#{patterns::CLASS_VARIABLE}/o)
encoder.text_token match, :class_variable
value_expected = false
elsif match = scan(/\\\z/)
encoder.text_token match, :space
else
if method_call_expected
method_call_expected = false
next
end
unless unicode
# check for unicode
$DEBUG_BEFORE, $DEBUG = $DEBUG, false
begin
if check(/./mu).size > 1
# seems like we should try again with unicode
unicode = true
end
rescue
# bad unicode char; use getch
ensure
$DEBUG = $DEBUG_BEFORE
end
next if unicode
end
encoder.text_token getch, :error
end
if last_state
state = last_state
last_state = nil
end
elsif state == :def_expected
if match = scan(unicode ? /(?>#{patterns::METHOD_NAME_EX})(?!\.|::)/uo :
/(?>#{patterns::METHOD_NAME_EX})(?!\.|::)/o)
encoder.text_token match, :method
state = :initial
else
last_state = :dot_expected
state = :initial
end
elsif state == :dot_expected
if match = scan(/\.|::/)
# invalid definition
state = :def_expected
encoder.text_token match, :operator
else
state = :initial
end
elsif state == :module_expected
if match = scan(/<</)
encoder.text_token match, :operator
else
state = :initial
if match = scan(unicode ? / (?:#{patterns::IDENT}::)* #{patterns::IDENT} /oux :
/ (?:#{patterns::IDENT}::)* #{patterns::IDENT} /ox)
encoder.text_token match, :class
end
end
elsif state == :undef_expected
state = :undef_comma_expected
if match = scan(unicode ? /(?>#{patterns::METHOD_NAME_EX})(?!\.|::)/uo :
/(?>#{patterns::METHOD_NAME_EX})(?!\.|::)/o)
encoder.text_token match, :method
elsif match = scan(/#{patterns::SYMBOL}/o)
case delim = match[1]
when ?', ?"
encoder.begin_group :symbol
encoder.text_token ':', :symbol
match = delim.chr
encoder.text_token match, :delimiter
state = self.class::StringState.new :symbol, delim == ?", match
state.next_state = :undef_comma_expected
else
encoder.text_token match, :symbol
end
else
state = :initial
end
elsif state == :undef_comma_expected
if match = scan(/,/)
encoder.text_token match, :operator
state = :undef_expected
else
state = :initial
end
elsif state == :alias_expected
match = scan(unicode ? /(#{patterns::METHOD_NAME_OR_SYMBOL})([ \t]+)(#{patterns::METHOD_NAME_OR_SYMBOL})/uo :
/(#{patterns::METHOD_NAME_OR_SYMBOL})([ \t]+)(#{patterns::METHOD_NAME_OR_SYMBOL})/o)
if match
encoder.text_token self[1], (self[1][0] == ?: ? :symbol : :method)
encoder.text_token self[2], :space
encoder.text_token self[3], (self[3][0] == ?: ? :symbol : :method)
end
state = :initial
else
#:nocov:
raise_inspect 'Unknown state: %p' % [state], encoder
#:nocov:
end
else # StringState
match = scan_until(state.pattern) || scan_rest
unless match.empty?
encoder.text_token match, :content
break if eos?
end
if state.heredoc && self[1] # end of heredoc
match = getch
match << scan_until(/$/) unless eos?
encoder.text_token match, :delimiter unless match.empty?
encoder.end_group state.type
state = state.next_state
next
end
case match = getch
when state.delim
if state.paren_depth
state.paren_depth -= 1
if state.paren_depth > 0
encoder.text_token match, :content
next
end
end
encoder.text_token match, :delimiter
if state.type == :regexp && !eos?
match = scan(/#{patterns::REGEXP_MODIFIERS}/o)
encoder.text_token match, :modifier unless match.empty?
end
encoder.end_group state.type
value_expected = false
state = state.next_state
when '\\'
if state.interpreted
if esc = scan(/#{patterns::ESCAPE}/o)
encoder.text_token match + esc, :char
else
encoder.text_token match, :error
end
else
case esc = getch
when nil
encoder.text_token match, :content
when state.delim, '\\'
encoder.text_token match + esc, :char
else
encoder.text_token match + esc, :content
end
end
when '#'
case peek(1)
when '{'
inline_block_stack ||= []
inline_block_stack << [state, inline_block_curly_depth, heredocs]
value_expected = true
state = :initial
inline_block_curly_depth = 1
encoder.begin_group :inline
encoder.text_token match + getch, :inline_delimiter
when '$', '@'
encoder.text_token match, :escape
last_state = state
state = :initial
else
#:nocov:
raise_inspect 'else-case # reached; #%p not handled' % [peek(1)], encoder
#:nocov:
end
when state.opening_paren
state.paren_depth += 1
encoder.text_token match, :content
else
#:nocov
raise_inspect 'else-case " reached; %p not handled, state = %p' % [match, state], encoder
#:nocov:
end
end
end
# cleaning up
if state.is_a? StringState
encoder.end_group state.type
end
if options[:keep_state]
if state.is_a?(StringState) && state.heredoc
(heredocs ||= []).unshift state
state = :initial
elsif heredocs && heredocs.empty?
heredocs = nil
end
@state = state, heredocs
end
if inline_block_stack
until inline_block_stack.empty?
state, = *inline_block_stack.pop
encoder.end_group :inline
encoder.end_group state.type
end
end
encoder
end
end
end
end

View File

@ -1,175 +0,0 @@
# encoding: utf-8
module CodeRay
module Scanners
module Ruby::Patterns # :nodoc: all
KEYWORDS = %w[
and def end in or unless begin
defined? ensure module redo super until
BEGIN break do next rescue then
when END case else for retry
while alias class elsif if not return
undef yield
]
# See http://murfy.de/ruby-constants.
PREDEFINED_CONSTANTS = %w[
nil true false self
DATA ARGV ARGF ENV
FALSE TRUE NIL
STDERR STDIN STDOUT
TOPLEVEL_BINDING
RUBY_COPYRIGHT RUBY_DESCRIPTION RUBY_ENGINE RUBY_PATCHLEVEL
RUBY_PLATFORM RUBY_RELEASE_DATE RUBY_REVISION RUBY_VERSION
__FILE__ __LINE__ __ENCODING__
]
IDENT_KIND = WordList.new(:ident).
add(KEYWORDS, :keyword).
add(PREDEFINED_CONSTANTS, :predefined_constant)
KEYWORD_NEW_STATE = WordList.new(:initial).
add(%w[ def ], :def_expected).
add(%w[ undef ], :undef_expected).
add(%w[ alias ], :alias_expected).
add(%w[ class module ], :module_expected)
IDENT = 'ä'[/[[:alpha:]]/] == 'ä' ? /[[:alpha:]_][[:alnum:]_]*/ : /[^\W\d]\w*/
METHOD_NAME = / #{IDENT} [?!]? /ox
METHOD_NAME_OPERATOR = /
\*\*? # multiplication and power
| [-+~]@? # plus, minus, tilde with and without at sign
| [\/%&|^`] # division, modulo or format strings, and, or, xor, system
| \[\]=? # array getter and setter
| << | >> # append or shift left, shift right
| <=?>? | >=? # comparison, rocket operator
| ===? | =~ # simple equality, case equality, match
| ![~=@]? # negation with and without at sign, not-equal and not-match
/ox
METHOD_SUFFIX = / (?: [?!] | = (?![~>]|=(?!>)) ) /x
METHOD_NAME_EX = / #{IDENT} #{METHOD_SUFFIX}? | #{METHOD_NAME_OPERATOR} /ox
METHOD_AFTER_DOT = / #{IDENT} [?!]? | #{METHOD_NAME_OPERATOR} /ox
INSTANCE_VARIABLE = / @ #{IDENT} /ox
CLASS_VARIABLE = / @@ #{IDENT} /ox
OBJECT_VARIABLE = / @@? #{IDENT} /ox
GLOBAL_VARIABLE = / \$ (?: #{IDENT} | [1-9]\d* | 0\w* | [~&+`'=\/,;_.<>!@$?*":\\] | -[a-zA-Z_0-9] ) /ox
PREFIX_VARIABLE = / #{GLOBAL_VARIABLE} | #{OBJECT_VARIABLE} /ox
VARIABLE = / @?@? #{IDENT} | #{GLOBAL_VARIABLE} /ox
QUOTE_TO_TYPE = {
'`' => :shell,
'/'=> :regexp,
}
QUOTE_TO_TYPE.default = :string
REGEXP_MODIFIERS = /[mousenix]*/
DECIMAL = /\d+(?:_\d+)*/
OCTAL = /0_?[0-7]+(?:_[0-7]+)*/
HEXADECIMAL = /0x[0-9A-Fa-f]+(?:_[0-9A-Fa-f]+)*/
BINARY = /0b[01]+(?:_[01]+)*/
EXPONENT = / [eE] [+-]? #{DECIMAL} /ox
FLOAT_SUFFIX = / #{EXPONENT} | \. #{DECIMAL} #{EXPONENT}? /ox
FLOAT_OR_INT = / #{DECIMAL} (?: #{FLOAT_SUFFIX} () )? /ox
NUMERIC = / (?: (?=0) (?: #{OCTAL} | #{HEXADECIMAL} | #{BINARY} ) | #{FLOAT_OR_INT} ) /ox
SYMBOL = /
:
(?:
#{METHOD_NAME_EX}
| #{PREFIX_VARIABLE}
| ['"]
)
/ox
METHOD_NAME_OR_SYMBOL = / #{METHOD_NAME_EX} | #{SYMBOL} /ox
SIMPLE_ESCAPE = /
[abefnrstv]
| [0-7]{1,3}
| x[0-9A-Fa-f]{1,2}
| .
/mx
CONTROL_META_ESCAPE = /
(?: M-|C-|c )
(?: \\ (?: M-|C-|c ) )*
(?: [^\\] | \\ #{SIMPLE_ESCAPE} )?
/mox
ESCAPE = /
#{CONTROL_META_ESCAPE} | #{SIMPLE_ESCAPE}
/mox
CHARACTER = /
\?
(?:
[^\s\\]
| \\ #{ESCAPE}
)
/mox
# NOTE: This is not completely correct, but
# nobody needs heredoc delimiters ending with \n.
HEREDOC_OPEN = /
<< (-)? # $1 = float
(?:
( [A-Za-z_0-9]+ ) # $2 = delim
|
( ["'`\/] ) # $3 = quote, type
( [^\n]*? ) \3 # $4 = delim
)
/mx
RUBYDOC = /
=begin (?!\S)
.*?
(?: \Z | ^=end (?!\S) [^\n]* )
/mx
DATA = /
__END__$
.*?
(?: \Z | (?=^\#CODE) )
/mx
RUBYDOC_OR_DATA = / #{RUBYDOC} | #{DATA} /xo
# Checks for a valid value to follow. This enables
# value_expected in method calls without parentheses.
VALUE_FOLLOWS = /
(?>[ \t\f\v]+)
(?:
[%\/][^\s=]
| <<-?\S
| [-+] \d
| #{CHARACTER}
)
/ox
KEYWORDS_EXPECTING_VALUE = WordList.new.add(%w[
and end in or unless begin
defined? ensure redo super until
break do next rescue then
when case else for retry
while elsif if not return
yield
])
FANCY_STRING_START = / % ( [QqrsWwx] | (?![a-zA-Z0-9]) ) ([^a-zA-Z0-9]) /x
FANCY_STRING_KIND = Hash.new(:string).merge({
'r' => :regexp,
's' => :symbol,
'x' => :shell,
})
FANCY_STRING_INTERPRETED = Hash.new(true).merge({
'q' => false,
's' => false,
'w' => false,
})
end
end
end

View File

@ -1,71 +0,0 @@
# encoding: utf-8
module CodeRay
module Scanners
class Ruby
class StringState < Struct.new :type, :interpreted, :delim, :heredoc,
:opening_paren, :paren_depth, :pattern, :next_state # :nodoc: all
CLOSING_PAREN = Hash[ *%w[
( )
[ ]
< >
{ }
] ].each { |k,v| k.freeze; v.freeze } # debug, if I try to change it with <<
STRING_PATTERN = Hash.new do |h, k|
delim, interpreted = *k
# delim = delim.dup # workaround for old Ruby
delim_pattern = Regexp.escape(delim)
if closing_paren = CLOSING_PAREN[delim]
delim_pattern << Regexp.escape(closing_paren)
end
delim_pattern << '\\\\' unless delim == '\\'
# special_escapes =
# case interpreted
# when :regexp_symbols
# '| [|?*+(){}\[\].^$]'
# end
h[k] =
if interpreted && delim != '#'
/ (?= [#{delim_pattern}] | \# [{$@] ) /mx
else
/ (?= [#{delim_pattern}] ) /mx
end
end
def initialize kind, interpreted, delim, heredoc = false
if heredoc
pattern = heredoc_pattern delim, interpreted, heredoc == :indented
delim = nil
else
pattern = STRING_PATTERN[ [delim, interpreted] ]
if closing_paren = CLOSING_PAREN[delim]
opening_paren = delim
delim = closing_paren
paren_depth = 1
end
end
super kind, interpreted, delim, heredoc, opening_paren, paren_depth, pattern, :initial
end
def heredoc_pattern delim, interpreted, indented
# delim = delim.dup # workaround for old Ruby
delim_pattern = Regexp.escape(delim)
delim_pattern = / (?:\A|\n) #{ '(?>[ \t]*)' if indented } #{ Regexp.new delim_pattern } $ /x
if interpreted
/ (?= #{delim_pattern}() | \\ | \# [{$@] ) /mx # $1 set == end of heredoc
else
/ (?= #{delim_pattern}() | \\ ) /mx
end
end
end
end
end
end

View File

@ -1,174 +0,0 @@
module CodeRay module Scanners
# by Josh Goebel
class SQL < Scanner
register_for :sql
KEYWORDS = %w(
all and any as before begin between by case check collate
each else end exists
for foreign from full group having if in inner is join
like not of on or order outer over references
then to union using values when where
left right distinct
)
OBJECTS = %w(
database databases table tables column columns fields index constraint
constraints transaction function procedure row key view trigger
)
COMMANDS = %w(
add alter comment create delete drop grant insert into select update set
show prompt begin commit rollback replace truncate
)
PREDEFINED_TYPES = %w(
char varchar varchar2 enum binary text tinytext mediumtext
longtext blob tinyblob mediumblob longblob timestamp
date time datetime year double decimal float int
integer tinyint mediumint bigint smallint unsigned bit
bool boolean hex bin oct
)
PREDEFINED_FUNCTIONS = %w( sum cast substring abs pi count min max avg now )
DIRECTIVES = %w(
auto_increment unique default charset initially deferred
deferrable cascade immediate read write asc desc after
primary foreign return engine
)
PREDEFINED_CONSTANTS = %w( null true false )
IDENT_KIND = WordList::CaseIgnoring.new(:ident).
add(KEYWORDS, :keyword).
add(OBJECTS, :type).
add(COMMANDS, :class).
add(PREDEFINED_TYPES, :predefined_type).
add(PREDEFINED_CONSTANTS, :predefined_constant).
add(PREDEFINED_FUNCTIONS, :predefined).
add(DIRECTIVES, :directive)
ESCAPE = / [rbfntv\n\\\/'"] | x[a-fA-F0-9]{1,2} | [0-7]{1,3} | . /mx
UNICODE_ESCAPE = / u[a-fA-F0-9]{4} | U[a-fA-F0-9]{8} /x
STRING_PREFIXES = /[xnb]|_\w+/i
def scan_tokens encoder, options
state = :initial
string_type = nil
string_content = ''
name_expected = false
until eos?
if state == :initial
if match = scan(/ \s+ | \\\n /x)
encoder.text_token match, :space
elsif match = scan(/(?:--\s?|#).*/)
encoder.text_token match, :comment
elsif match = scan(%r( /\* (!)? (?: .*? \*/ | .* ) )mx)
encoder.text_token match, self[1] ? :directive : :comment
elsif match = scan(/ [*\/=<>:;,!&^|()\[\]{}~%] | [-+\.](?!\d) /x)
name_expected = true if match == '.' && check(/[A-Za-z_]/)
encoder.text_token match, :operator
elsif match = scan(/(#{STRING_PREFIXES})?([`"'])/o)
prefix = self[1]
string_type = self[2]
encoder.begin_group :string
encoder.text_token prefix, :modifier if prefix
match = string_type
state = :string
encoder.text_token match, :delimiter
elsif match = scan(/ @? [A-Za-z_][A-Za-z_0-9]* /x)
encoder.text_token match, name_expected ? :ident : (match[0] == ?@ ? :variable : IDENT_KIND[match])
name_expected = false
elsif match = scan(/0[xX][0-9A-Fa-f]+/)
encoder.text_token match, :hex
elsif match = scan(/0[0-7]+(?![89.eEfF])/)
encoder.text_token match, :octal
elsif match = scan(/[-+]?(?>\d+)(?![.eEfF])/)
encoder.text_token match, :integer
elsif match = scan(/[-+]?(?:\d[fF]|\d*\.\d+(?:[eE][+-]?\d+)?|\d+[eE][+-]?\d+)/)
encoder.text_token match, :float
elsif match = scan(/\\N/)
encoder.text_token match, :predefined_constant
else
encoder.text_token getch, :error
end
elsif state == :string
if match = scan(/[^\\"'`]+/)
string_content << match
next
elsif match = scan(/["'`]/)
if string_type == match
if peek(1) == string_type # doubling means escape
string_content << string_type << getch
next
end
unless string_content.empty?
encoder.text_token string_content, :content
string_content = ''
end
encoder.text_token match, :delimiter
encoder.end_group :string
state = :initial
string_type = nil
else
string_content << match
end
elsif match = scan(/ \\ (?: #{ESCAPE} | #{UNICODE_ESCAPE} ) /mox)
unless string_content.empty?
encoder.text_token string_content, :content
string_content = ''
end
encoder.text_token match, :char
elsif match = scan(/ \\ . /mox)
string_content << match
next
elsif match = scan(/ \\ | $ /x)
unless string_content.empty?
encoder.text_token string_content, :content
string_content = ''
end
encoder.text_token match, :error
state = :initial
else
raise "else case \" reached; %p not handled." % peek(1), encoder
end
else
raise 'else-case reached', encoder
end
end
if state == :string
encoder.end_group state
end
encoder
end
end
end end

View File

@ -1,26 +0,0 @@
module CodeRay
module Scanners
# Scanner for plain text.
#
# Yields just one token of the kind :plain.
#
# Alias: +plaintext+, +plain+
class Text < Scanner
register_for :text
title 'Plain text'
KINDS_NOT_LOC = [:plain] # :nodoc:
protected
def scan_tokens encoder, options
encoder.text_token string, :plain
encoder
end
end
end
end

View File

@ -1,17 +0,0 @@
module CodeRay
module Scanners
load :html
# Scanner for XML.
#
# Currently this is the same scanner as Scanners::HTML.
class XML < HTML
register_for :xml
file_extension 'xml'
end
end
end

View File

@ -1,140 +0,0 @@
module CodeRay
module Scanners
# Scanner for YAML.
#
# Based on the YAML scanner from Syntax by Jamis Buck.
class YAML < Scanner
register_for :yaml
file_extension 'yml'
KINDS_NOT_LOC = :all
protected
def scan_tokens encoder, options
state = :initial
key_indent = string_indent = 0
until eos?
key_indent = nil if bol?
if match = scan(/ +[\t ]*/)
encoder.text_token match, :space
elsif match = scan(/\n+/)
encoder.text_token match, :space
state = :initial if match.index(?\n)
elsif match = scan(/#.*/)
encoder.text_token match, :comment
elsif bol? and case
when match = scan(/---|\.\.\./)
encoder.begin_group :head
encoder.text_token match, :head
encoder.end_group :head
next
when match = scan(/%.*/)
encoder.text_token match, :doctype
next
end
elsif state == :value and case
when !check(/(?:"[^"]*")(?=: |:$)/) && match = scan(/"/)
encoder.begin_group :string
encoder.text_token match, :delimiter
encoder.text_token match, :content if match = scan(/ [^"\\]* (?: \\. [^"\\]* )* /mx)
encoder.text_token match, :delimiter if match = scan(/"/)
encoder.end_group :string
next
when match = scan(/[|>][-+]?/)
encoder.begin_group :string
encoder.text_token match, :delimiter
string_indent = key_indent || column(pos - match.size) - 1
encoder.text_token matched, :content if scan(/(?:\n+ {#{string_indent + 1}}.*)+/)
encoder.end_group :string
next
when match = scan(/(?![!"*&]).+?(?=$|\s+#)/)
encoder.begin_group :string
encoder.text_token match, :content
string_indent = key_indent || column(pos - match.size) - 1
encoder.text_token matched, :content if scan(/(?:\n+ {#{string_indent + 1}}.*)+/)
encoder.end_group :string
next
end
elsif case
when match = scan(/[-:](?= |$)/)
state = :value if state == :colon && (match == ':' || match == '-')
state = :value if state == :initial && match == '-'
encoder.text_token match, :operator
next
when match = scan(/[,{}\[\]]/)
encoder.text_token match, :operator
next
when state == :initial && match = scan(/[-\w.()\/ ]*\S(?= *:(?: |$))/)
encoder.text_token match, :key
key_indent = column(pos - match.size) - 1
state = :colon
next
when match = scan(/(?:"[^"\n]*"|'[^'\n]*')(?= *:(?: |$))/)
encoder.begin_group :key
encoder.text_token match[0,1], :delimiter
encoder.text_token match[1..-2], :content
encoder.text_token match[-1,1], :delimiter
encoder.end_group :key
key_indent = column(pos - match.size) - 1
state = :colon
next
when match = scan(/(![\w\/]+)(:([\w:]+))?/)
encoder.text_token self[1], :type
if self[2]
encoder.text_token ':', :operator
encoder.text_token self[3], :class
end
next
when match = scan(/&\S+/)
encoder.text_token match, :variable
next
when match = scan(/\*\w+/)
encoder.text_token match, :global_variable
next
when match = scan(/<</)
encoder.text_token match, :class_variable
next
when match = scan(/\d\d:\d\d:\d\d/)
encoder.text_token match, :octal
next
when match = scan(/\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d(\.\d+)? [-+]\d\d:\d\d/)
encoder.text_token match, :octal
next
when match = scan(/:\w+/)
encoder.text_token match, :symbol
next
when match = scan(/[^:\s]+(:(?! |$)[^:\s]*)* .*/)
encoder.text_token match, :error
next
when match = scan(/[^:\s]+(:(?! |$)[^:\s]*)*/)
encoder.text_token match, :error
next
end
else
raise if eos?
encoder.text_token getch, :error
end
end
encoder
end
end
end
end

View File

@ -1,23 +0,0 @@
module CodeRay
# This module holds the Style class and its subclasses.
#
# See Plugin.
module Styles
extend PluginHost
plugin_path File.dirname(__FILE__), 'styles'
# Base class for styles.
#
# Styles are used by Encoders::HTML to colorize tokens.
class Style
extend Plugin
plugin_host Styles
DEFAULT_OPTIONS = { } # :nodoc:
end
end
end

View File

@ -1,7 +0,0 @@
module CodeRay
module Styles
default :alpha
end
end

View File

@ -1,142 +0,0 @@
module CodeRay
module Styles
# A colorful theme using CSS 3 colors (with alpha channel).
class Alpha < Style
register_for :alpha
code_background = 'hsl(0,0%,95%)'
numbers_background = 'hsl(180,65%,90%)'
border_color = 'silver'
normal_color = 'black'
CSS_MAIN_STYLES = <<-MAIN # :nodoc:
.CodeRay {
background-color: #{code_background};
border: 1px solid #{border_color};
color: #{normal_color};
}
.CodeRay pre {
margin: 0px;
}
span.CodeRay { white-space: pre; border: 0px; padding: 2px; }
table.CodeRay { border-collapse: collapse; width: 100%; padding: 2px; }
table.CodeRay td { padding: 2px 4px; vertical-align: top; }
.CodeRay .line-numbers {
background-color: #{numbers_background};
color: gray;
text-align: right;
-webkit-user-select: none;
-moz-user-select: none;
user-select: none;
}
.CodeRay .line-numbers a {
background-color: #{numbers_background} !important;
color: gray !important;
text-decoration: none !important;
}
.CodeRay .line-numbers a:target { color: blue !important; }
.CodeRay .line-numbers .highlighted { color: red !important; }
.CodeRay .line-numbers .highlighted a { color: red !important; }
.CodeRay span.line-numbers { padding: 0px 4px; }
.CodeRay .line { display: block; float: left; width: 100%; }
.CodeRay .code { width: 100%; }
MAIN
TOKEN_COLORS = <<-'TOKENS'
.debug { color: white !important; background: blue !important; }
.annotation { color:#007 }
.attribute-name { color:#b48 }
.attribute-value { color:#700 }
.binary { color:#509 }
.char .content { color:#D20 }
.char .delimiter { color:#710 }
.char { color:#D20 }
.class { color:#B06; font-weight:bold }
.class-variable { color:#369 }
.color { color:#0A0 }
.comment { color:#777 }
.comment .char { color:#444 }
.comment .delimiter { color:#444 }
.complex { color:#A08 }
.constant { color:#036; font-weight:bold }
.decorator { color:#B0B }
.definition { color:#099; font-weight:bold }
.delimiter { color:black }
.directive { color:#088; font-weight:bold }
.doc { color:#970 }
.doc-string { color:#D42; font-weight:bold }
.doctype { color:#34b }
.entity { color:#800; font-weight:bold }
.error { color:#F00; background-color:#FAA }
.escape { color:#666 }
.exception { color:#C00; font-weight:bold }
.float { color:#60E }
.function { color:#06B; font-weight:bold }
.global-variable { color:#d70 }
.hex { color:#02b }
.imaginary { color:#f00 }
.include { color:#B44; font-weight:bold }
.inline { background-color: hsla(0,0%,0%,0.07); color: black }
.inline-delimiter { font-weight: bold; color: #666 }
.instance-variable { color:#33B }
.integer { color:#00D }
.key .char { color: #60f }
.key .delimiter { color: #404 }
.key { color: #606 }
.keyword { color:#080; font-weight:bold }
.label { color:#970; font-weight:bold }
.local-variable { color:#963 }
.namespace { color:#707; font-weight:bold }
.octal { color:#40E }
.operator { }
.predefined { color:#369; font-weight:bold }
.predefined-constant { color:#069 }
.predefined-type { color:#0a5; font-weight:bold }
.preprocessor { color:#579 }
.pseudo-class { color:#00C; font-weight:bold }
.regexp .content { color:#808 }
.regexp .delimiter { color:#404 }
.regexp .modifier { color:#C2C }
.regexp { background-color:hsla(300,100%,50%,0.06); }
.reserved { color:#080; font-weight:bold }
.shell .content { color:#2B2 }
.shell .delimiter { color:#161 }
.shell { background-color:hsla(120,100%,50%,0.06); }
.string .char { color: #b0b }
.string .content { color: #D20 }
.string .delimiter { color: #710 }
.string .modifier { color: #E40 }
.string { background-color:hsla(0,100%,50%,0.05); }
.symbol .content { color:#A60 }
.symbol .delimiter { color:#630 }
.symbol { color:#A60 }
.tag { color:#070 }
.type { color:#339; font-weight:bold }
.value { color: #088; }
.variable { color:#037 }
.insert { background: hsla(120,100%,50%,0.12) }
.delete { background: hsla(0,100%,50%,0.12) }
.change { color: #bbf; background: #007; }
.head { color: #f8f; background: #505 }
.head .filename { color: white; }
.delete .eyecatcher { background-color: hsla(0,100%,50%,0.2); border: 1px solid hsla(0,100%,45%,0.5); margin: -1px; border-bottom: none; border-top-left-radius: 5px; border-top-right-radius: 5px; }
.insert .eyecatcher { background-color: hsla(120,100%,50%,0.2); border: 1px solid hsla(120,100%,25%,0.5); margin: -1px; border-top: none; border-bottom-left-radius: 5px; border-bottom-right-radius: 5px; }
.insert .insert { color: #0c0; background:transparent; font-weight:bold }
.delete .delete { color: #c00; background:transparent; font-weight:bold }
.change .change { color: #88f }
.head .head { color: #f4f }
TOKENS
end
end
end

View File

@ -1,90 +0,0 @@
module CodeRay
# A Hash of all known token kinds and their associated CSS classes.
TokenKinds = Hash.new do |h, k|
warn 'Undefined Token kind: %p' % [k] if $CODERAY_DEBUG
false
end
# speedup
TokenKinds.compare_by_identity if TokenKinds.respond_to? :compare_by_identity
TokenKinds.update( # :nodoc:
:annotation => 'annotation',
:attribute_name => 'attribute-name',
:attribute_value => 'attribute-value',
:binary => 'bin',
:char => 'char',
:class => 'class',
:class_variable => 'class-variable',
:color => 'color',
:comment => 'comment',
:complex => 'complex',
:constant => 'constant',
:content => 'content',
:debug => 'debug',
:decorator => 'decorator',
:definition => 'definition',
:delimiter => 'delimiter',
:directive => 'directive',
:doc => 'doc',
:doctype => 'doctype',
:doc_string => 'doc-string',
:entity => 'entity',
:error => 'error',
:escape => 'escape',
:exception => 'exception',
:filename => 'filename',
:float => 'float',
:function => 'function',
:global_variable => 'global-variable',
:hex => 'hex',
:imaginary => 'imaginary',
:important => 'important',
:include => 'include',
:inline => 'inline',
:inline_delimiter => 'inline-delimiter',
:instance_variable => 'instance-variable',
:integer => 'integer',
:key => 'key',
:keyword => 'keyword',
:label => 'label',
:local_variable => 'local-variable',
:modifier => 'modifier',
:namespace => 'namespace',
:octal => 'octal',
:predefined => 'predefined',
:predefined_constant => 'predefined-constant',
:predefined_type => 'predefined-type',
:preprocessor => 'preprocessor',
:pseudo_class => 'pseudo-class',
:regexp => 'regexp',
:reserved => 'reserved',
:shell => 'shell',
:string => 'string',
:symbol => 'symbol',
:tag => 'tag',
:type => 'type',
:value => 'value',
:variable => 'variable',
:change => 'change',
:delete => 'delete',
:head => 'head',
:insert => 'insert',
:eyecatcher => 'eyecatcher',
:ident => false,
:operator => false,
:space => false,
:plain => false
)
TokenKinds[:method] = TokenKinds[:function]
TokenKinds[:escape] = TokenKinds[:delimiter]
TokenKinds[:docstring] = TokenKinds[:comment]
TokenKinds.freeze
end

View File

@ -1,215 +0,0 @@
module CodeRay
# GZip library for writing and reading token dumps.
autoload :GZip, coderay_path('helpers', 'gzip')
# = Tokens TODO: Rewrite!
#
# The Tokens class represents a list of tokens returnd from
# a Scanner.
#
# A token is not a special object, just a two-element Array
# consisting of
# * the _token_ _text_ (the original source of the token in a String) or
# a _token_ _action_ (begin_group, end_group, begin_line, end_line)
# * the _token_ _kind_ (a Symbol representing the type of the token)
#
# A token looks like this:
#
# ['# It looks like this', :comment]
# ['3.1415926', :float]
# ['$^', :error]
#
# Some scanners also yield sub-tokens, represented by special
# token actions, namely begin_group and end_group.
#
# The Ruby scanner, for example, splits "a string" into:
#
# [
# [:begin_group, :string],
# ['"', :delimiter],
# ['a string', :content],
# ['"', :delimiter],
# [:end_group, :string]
# ]
#
# Tokens is the interface between Scanners and Encoders:
# The input is split and saved into a Tokens object. The Encoder
# then builds the output from this object.
#
# Thus, the syntax below becomes clear:
#
# CodeRay.scan('price = 2.59', :ruby).html
# # the Tokens object is here -------^
#
# See how small it is? ;)
#
# Tokens gives you the power to handle pre-scanned code very easily:
# You can convert it to a webpage, a YAML file, or dump it into a gzip'ed string
# that you put in your DB.
#
# It also allows you to generate tokens directly (without using a scanner),
# to load them from a file, and still use any Encoder that CodeRay provides.
class Tokens < Array
# The Scanner instance that created the tokens.
attr_accessor :scanner
# Encode the tokens using encoder.
#
# encoder can be
# * a symbol like :html oder :statistic
# * an Encoder class
# * an Encoder object
#
# options are passed to the encoder.
def encode encoder, options = {}
encoder = Encoders[encoder].new options if encoder.respond_to? :to_sym
encoder.encode_tokens self, options
end
# Turn tokens into a string by concatenating them.
def to_s
encode CodeRay::Encoders::Encoder.new
end
# Redirects unknown methods to encoder calls.
#
# For example, if you call +tokens.html+, the HTML encoder
# is used to highlight the tokens.
def method_missing meth, options = {}
encode meth, options
rescue PluginHost::PluginNotFound
super
end
# Split the tokens into parts of the given +sizes+.
#
# The result will be an Array of Tokens objects. The parts have
# the text size specified by the parameter. In addition, each
# part closes all opened tokens. This is useful to insert tokens
# betweem them.
#
# This method is used by @Scanner#tokenize@ when called with an Array
# of source strings. The Diff encoder uses it for inline highlighting.
def split_into_parts *sizes
parts = []
opened = []
content = nil
part = Tokens.new
part_size = 0
size = sizes.first
i = 0
for item in self
case content
when nil
content = item
when String
if size && part_size + content.size > size # token must be cut
if part_size < size # some part of the token goes into this part
content = content.dup # content may no be safe to change
part << content.slice!(0, size - part_size) << item
end
# close all open groups and lines...
closing = opened.reverse.flatten.map do |content_or_kind|
case content_or_kind
when :begin_group
:end_group
when :begin_line
:end_line
else
content_or_kind
end
end
part.concat closing
begin
parts << part
part = Tokens.new
size = sizes[i += 1]
end until size.nil? || size > 0
# ...and open them again.
part.concat opened.flatten
part_size = 0
redo unless content.empty?
else
part << content << item
part_size += content.size
end
content = nil
when Symbol
case content
when :begin_group, :begin_line
opened << [content, item]
when :end_group, :end_line
opened.pop
else
raise ArgumentError, 'Unknown token action: %p, kind = %p' % [content, item]
end
part << content << item
content = nil
else
raise ArgumentError, 'Token input junk: %p, kind = %p' % [content, item]
end
end
parts << part
parts << Tokens.new while parts.size < sizes.size
parts
end
# Dumps the object into a String that can be saved
# in files or databases.
#
# The dump is created with Marshal.dump;
# In addition, it is gzipped using GZip.gzip.
#
# The returned String object includes Undumping
# so it has an #undump method. See Tokens.load.
#
# You can configure the level of compression,
# but the default value 7 should be what you want
# in most cases as it is a good compromise between
# speed and compression rate.
#
# See GZip module.
def dump gzip_level = 7
dump = Marshal.dump self
dump = GZip.gzip dump, gzip_level
dump.extend Undumping
end
# Return the actual number of tokens.
def count
size / 2
end
# Include this module to give an object an #undump
# method.
#
# The string returned by Tokens.dump includes Undumping.
module Undumping
# Calls Tokens.load with itself.
def undump
Tokens.load self
end
end
# Undump the object using Marshal.load, then
# unzip it using GZip.gunzip.
#
# The result is commonly a Tokens object, but
# this is not guaranteed.
def Tokens.load dump
dump = GZip.gunzip dump
@dump = Marshal.load dump
end
alias text_token push
def begin_group kind; push :begin_group, kind end
def end_group kind; push :end_group, kind end
def begin_line kind; push :begin_line, kind end
def end_line kind; push :end_line, kind end
alias tokens concat
end
end

View File

@ -1,55 +0,0 @@
module CodeRay
# The result of a scan operation is a TokensProxy, but should act like Tokens.
#
# This proxy makes it possible to use the classic CodeRay.scan.encode API
# while still providing the benefits of direct streaming.
class TokensProxy
attr_accessor :input, :lang, :options, :block
# Create a new TokensProxy with the arguments of CodeRay.scan.
def initialize input, lang, options = {}, block = nil
@input = input
@lang = lang
@options = options
@block = block
end
# Call CodeRay.encode if +encoder+ is a Symbol;
# otherwise, convert the receiver to tokens and call encoder.encode_tokens.
def encode encoder, options = {}
if encoder.respond_to? :to_sym
CodeRay.encode(input, lang, encoder, options)
else
encoder.encode_tokens tokens, options
end
end
# Tries to call encode;
# delegates to tokens otherwise.
def method_missing method, *args, &blk
encode method.to_sym, *args
rescue PluginHost::PluginNotFound
tokens.send(method, *args, &blk)
end
# The (cached) result of the tokenized input; a Tokens instance.
def tokens
@tokens ||= scanner.tokenize(input)
end
# A (cached) scanner instance to use for the scan task.
def scanner
@scanner ||= CodeRay.scanner(lang, options, &block)
end
# Overwrite Struct#each.
def each *args, &blk
tokens.each(*args, &blk)
self
end
end
end

View File

@ -1,3 +0,0 @@
module CodeRay
VERSION = '1.0.6'
end

View File

@ -1,320 +0,0 @@
# encoding: utf-8
require 'test/unit'
require File.expand_path('../../lib/assert_warning', __FILE__)
$:.unshift File.expand_path('../../../lib', __FILE__)
require 'coderay'
class BasicTest < Test::Unit::TestCase
def test_version
assert_nothing_raised do
assert_match(/\A\d\.\d\.\d?\z/, CodeRay::VERSION)
end
end
def with_empty_load_path
old_load_path = $:.dup
$:.clear
yield
ensure
$:.replace old_load_path
end
def test_autoload
with_empty_load_path do
assert_nothing_raised do
CodeRay::Scanners::Java::BuiltinTypes
end
end
end
RUBY_TEST_CODE = 'puts "Hello, World!"'
RUBY_TEST_TOKENS = [
['puts', :ident],
[' ', :space],
[:begin_group, :string],
['"', :delimiter],
['Hello, World!', :content],
['"', :delimiter],
[:end_group, :string]
].flatten
def test_simple_scan
assert_nothing_raised do
assert_equal RUBY_TEST_TOKENS, CodeRay.scan(RUBY_TEST_CODE, :ruby).tokens
end
end
RUBY_TEST_HTML = 'puts <span class="string"><span class="delimiter">&quot;</span>' +
'<span class="content">Hello, World!</span><span class="delimiter">&quot;</span></span>'
def test_simple_highlight
assert_nothing_raised do
assert_equal RUBY_TEST_HTML, CodeRay.scan(RUBY_TEST_CODE, :ruby).html
end
end
def test_scan_file
CodeRay.scan_file __FILE__
end
def test_encode
assert_equal 1, CodeRay.encode('test', :python, :count)
end
def test_encode_tokens
assert_equal 1, CodeRay.encode_tokens(CodeRay::Tokens['test', :string], :count)
end
def test_encode_file
assert_equal File.read(__FILE__), CodeRay.encode_file(__FILE__, :text)
end
def test_highlight
assert_match '<pre>test</pre>', CodeRay.highlight('test', :python)
end
def test_highlight_file
assert_match "require <span class=\"string\"><span class=\"delimiter\">'</span><span class=\"content\">test/unit</span><span class=\"delimiter\">'</span></span>\n", CodeRay.highlight_file(__FILE__)
end
def test_duo
assert_equal(RUBY_TEST_CODE,
CodeRay::Duo[:plain, :text].highlight(RUBY_TEST_CODE))
assert_equal(RUBY_TEST_CODE,
CodeRay::Duo[:plain => :text].highlight(RUBY_TEST_CODE))
end
def test_duo_stream
assert_equal(RUBY_TEST_CODE,
CodeRay::Duo[:plain, :text].highlight(RUBY_TEST_CODE, :stream => true))
end
def test_comment_filter
assert_equal <<-EXPECTED, CodeRay.scan(<<-INPUT, :ruby).comment_filter.text
#!/usr/bin/env ruby
code
more code
EXPECTED
#!/usr/bin/env ruby
=begin
A multi-line comment.
=end
code
# A single-line comment.
more code # and another comment, in-line.
INPUT
end
def test_lines_of_code
assert_equal 2, CodeRay.scan(<<-INPUT, :ruby).lines_of_code
#!/usr/bin/env ruby
=begin
A multi-line comment.
=end
code
# A single-line comment.
more code # and another comment, in-line.
INPUT
rHTML = <<-RHTML
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="content-type" content="text/html;charset=UTF-8" />
<title><%= controller.controller_name.titleize %>: <%= controller.action_name %></title>
<%= stylesheet_link_tag 'scaffold' %>
</head>
<body>
<p style="color: green"><%= flash[:notice] %></p>
<div id="main">
<%= yield %>
</div>
</body>
</html>
RHTML
assert_equal 0, CodeRay.scan(rHTML, :html).lines_of_code
assert_equal 0, CodeRay.scan(rHTML, :php).lines_of_code
assert_equal 0, CodeRay.scan(rHTML, :yaml).lines_of_code
assert_equal 4, CodeRay.scan(rHTML, :erb).lines_of_code
end
def test_list_of_encoders
assert_kind_of(Array, CodeRay::Encoders.list)
assert CodeRay::Encoders.list.include?(:count)
end
def test_list_of_scanners
assert_kind_of(Array, CodeRay::Scanners.list)
assert CodeRay::Scanners.list.include?(:text)
end
def test_token_kinds
assert_kind_of Hash, CodeRay::TokenKinds
for kind, css_class in CodeRay::TokenKinds
assert_kind_of Symbol, kind
if css_class != false
assert_kind_of String, css_class, "TokenKinds[%p] == %p" % [kind, css_class]
end
end
assert_equal 'reserved', CodeRay::TokenKinds[:reserved]
assert_warning 'Undefined Token kind: :shibboleet' do
assert_equal false, CodeRay::TokenKinds[:shibboleet]
end
end
class Milk < CodeRay::Encoders::Encoder
FILE_EXTENSION = 'cocoa'
end
class HoneyBee < CodeRay::Encoders::Encoder
end
def test_encoder_file_extension
assert_nothing_raised do
assert_equal 'html', CodeRay::Encoders::Page::FILE_EXTENSION
assert_equal 'cocoa', Milk::FILE_EXTENSION
assert_equal 'cocoa', Milk.new.file_extension
assert_equal 'honeybee', HoneyBee::FILE_EXTENSION
assert_equal 'honeybee', HoneyBee.new.file_extension
end
assert_raise NameError do
HoneyBee::MISSING_CONSTANT
end
end
def test_encoder_tokens
encoder = CodeRay::Encoders::Encoder.new
encoder.send :setup, {}
assert_raise(ArgumentError) { encoder.token :strange, '' }
encoder.token 'test', :debug
end
def test_encoder_deprecated_interface
encoder = CodeRay::Encoders::Encoder.new
encoder.send :setup, {}
assert_warning 'Using old Tokens#<< interface.' do
encoder << ['test', :content]
end
assert_raise ArgumentError do
encoder << [:strange, :input]
end
assert_raise ArgumentError do
encoder.encode_tokens [['test', :token]]
end
end
def encoder_token_interface_deprecation_warning_given
CodeRay::Encoders::Encoder.send :class_variable_get, :@@CODERAY_TOKEN_INTERFACE_DEPRECATION_WARNING_GIVEN
end
def test_scanner_file_extension
assert_equal 'rb', CodeRay::Scanners::Ruby.file_extension
assert_equal 'rb', CodeRay::Scanners::Ruby.new.file_extension
assert_equal 'java', CodeRay::Scanners::Java.file_extension
assert_equal 'java', CodeRay::Scanners::Java.new.file_extension
end
def test_scanner_lang
assert_equal :ruby, CodeRay::Scanners::Ruby.lang
assert_equal :ruby, CodeRay::Scanners::Ruby.new.lang
assert_equal :java, CodeRay::Scanners::Java.lang
assert_equal :java, CodeRay::Scanners::Java.new.lang
end
def test_scanner_tokenize
assert_equal ['foo', :plain], CodeRay::Scanners::Plain.new.tokenize('foo')
assert_equal [['foo', :plain], ['bar', :plain]], CodeRay::Scanners::Plain.new.tokenize(['foo', 'bar'])
CodeRay::Scanners::Plain.new.tokenize 42
end
def test_scanner_tokens
scanner = CodeRay::Scanners::Plain.new
scanner.tokenize('foo')
assert_equal ['foo', :plain], scanner.tokens
scanner.string = ''
assert_equal ['', :plain], scanner.tokens
end
def test_scanner_line_and_column
scanner = CodeRay::Scanners::Plain.new "foo\nbär+quux"
assert_equal 0, scanner.pos
assert_equal 1, scanner.line
assert_equal 1, scanner.column
scanner.scan(/foo/)
assert_equal 3, scanner.pos
assert_equal 1, scanner.line
assert_equal 4, scanner.column
scanner.scan(/\n/)
assert_equal 4, scanner.pos
assert_equal 2, scanner.line
assert_equal 1, scanner.column
scanner.scan(/b/)
assert_equal 5, scanner.pos
assert_equal 2, scanner.line
assert_equal 2, scanner.column
scanner.scan(/a/)
assert_equal 5, scanner.pos
assert_equal 2, scanner.line
assert_equal 2, scanner.column
scanner.scan(/ä/)
assert_equal 7, scanner.pos
assert_equal 2, scanner.line
assert_equal 4, scanner.column
scanner.scan(/r/)
assert_equal 8, scanner.pos
assert_equal 2, scanner.line
assert_equal 5, scanner.column
end
def test_scanner_use_subclasses
assert_raise NotImplementedError do
CodeRay::Scanners::Scanner.new
end
end
class InvalidScanner < CodeRay::Scanners::Scanner
end
def test_scanner_scan_tokens
assert_raise NotImplementedError do
InvalidScanner.new.tokenize ''
end
end
class RaisingScanner < CodeRay::Scanners::Scanner
def scan_tokens encoder, options
raise_inspect 'message', [], :initial
end
end
def test_scanner_raise_inspect
assert_raise CodeRay::Scanners::Scanner::ScanError do
RaisingScanner.new.tokenize ''
end
end
def test_scan_a_frozen_string
assert_nothing_raised do
CodeRay.scan RUBY_VERSION, :ruby
CodeRay.scan RUBY_VERSION, :plain
end
end
def test_scan_a_non_string
assert_nothing_raised do
CodeRay.scan 42, :ruby
CodeRay.scan nil, :ruby
CodeRay.scan self, :ruby
CodeRay.encode ENV.to_hash, :ruby, :page
CodeRay.highlight CodeRay, :plain
end
end
end

View File

@ -1,129 +0,0 @@
require 'test/unit'
$:.unshift File.expand_path('../../../lib', __FILE__)
require 'coderay'
class ExamplesTest < Test::Unit::TestCase
def test_examples
# output as HTML div (using inline CSS styles)
div = CodeRay.scan('puts "Hello, world!"', :ruby).div
assert_equal <<-DIV, div
<div class="CodeRay">
<div class="code"><pre>puts <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Hello, world!</span><span style="color:#710">&quot;</span></span></pre></div>
</div>
DIV
# ...with line numbers
div = CodeRay.scan(<<-CODE.chomp, :ruby).div(:line_numbers => :table)
5.times do
puts 'Hello, world!'
end
CODE
assert_equal <<-DIV, div
<table class="CodeRay"><tr>
<td class="line-numbers" title="double click to toggle" ondblclick="with (this.firstChild.style) { display = (display == '') ? 'none' : '' }"><pre><a href="#n1" name="n1">1</a>
<a href="#n2" name="n2">2</a>
<a href="#n3" name="n3">3</a>
</pre></td>
<td class="code"><pre><span style="color:#00D">5</span>.times <span style="color:#080;font-weight:bold">do</span>
puts <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">'</span><span style="color:#D20">Hello, world!</span><span style="color:#710">'</span></span>
<span style="color:#080;font-weight:bold">end</span></pre></td>
</tr></table>
DIV
# output as standalone HTML page (using CSS classes)
page = CodeRay.scan('puts "Hello, world!"', :ruby).page
assert_match <<-PAGE, page
<body>
<table class="CodeRay"><tr>
<td class="line-numbers" title="double click to toggle" ondblclick="with (this.firstChild.style) { display = (display == '') ? 'none' : '' }"><pre>
</pre></td>
<td class="code"><pre>puts <span class="string"><span class="delimiter">&quot;</span><span class="content">Hello, world!</span><span class="delimiter">&quot;</span></span></pre></td>
</tr></table>
</body>
PAGE
# keep scanned tokens for later use
tokens = CodeRay.scan('{ "just": "an", "example": 42 }', :json)
assert_kind_of CodeRay::TokensProxy, tokens
assert_equal ["{", :operator, " ", :space, :begin_group, :key,
"\"", :delimiter, "just", :content, "\"", :delimiter,
:end_group, :key, ":", :operator, " ", :space,
:begin_group, :string, "\"", :delimiter, "an", :content,
"\"", :delimiter, :end_group, :string, ",", :operator,
" ", :space, :begin_group, :key, "\"", :delimiter,
"example", :content, "\"", :delimiter, :end_group, :key,
":", :operator, " ", :space, "42", :integer,
" ", :space, "}", :operator], tokens.tokens
# produce a token statistic
assert_equal <<-STATISTIC, tokens.statistic
Code Statistics
Tokens 26
Non-Whitespace 15
Bytes Total 31
Token Types (7):
type count ratio size (average)
-------------------------------------------------------------
TOTAL 26 100.00 % 1.2
delimiter 6 23.08 % 1.0
operator 5 19.23 % 1.0
space 5 19.23 % 1.0
key 4 15.38 % 0.0
:begin_group 3 11.54 % 0.0
:end_group 3 11.54 % 0.0
content 3 11.54 % 4.3
string 2 7.69 % 0.0
integer 1 3.85 % 2.0
STATISTIC
# count the tokens
assert_equal 26, tokens.count
# produce a HTML div, but with CSS classes
div = tokens.div(:css => :class)
assert_equal <<-DIV, div
<div class="CodeRay">
<div class="code"><pre>{ <span class="key"><span class="delimiter">&quot;</span><span class="content">just</span><span class="delimiter">&quot;</span></span>: <span class="string"><span class="delimiter">&quot;</span><span class="content">an</span><span class="delimiter">&quot;</span></span>, <span class="key"><span class="delimiter">&quot;</span><span class="content">example</span><span class="delimiter">&quot;</span></span>: <span class="integer">42</span> }</pre></div>
</div>
DIV
# highlight a file (HTML div); guess the file type base on the extension
assert_equal :ruby, CodeRay::FileType[__FILE__]
# get a new scanner for Python
python_scanner = CodeRay.scanner :python
assert_kind_of CodeRay::Scanners::Python, python_scanner
# get a new encoder for terminal
terminal_encoder = CodeRay.encoder :term
assert_kind_of CodeRay::Encoders::Terminal, terminal_encoder
# scanning into tokens
tokens = python_scanner.tokenize 'import this; # The Zen of Python'
assert_equal ["import", :keyword, " ", :space, "this", :include,
";", :operator, " ", :space, "# The Zen of Python", :comment], tokens
# format the tokens
term = terminal_encoder.encode_tokens(tokens)
assert_equal "\e[1;31mimport\e[0m \e[33mthis\e[0m; \e[37m# The Zen of Python\e[0m", term
# re-using scanner and encoder
ruby_highlighter = CodeRay::Duo[:ruby, :div]
div = ruby_highlighter.encode('puts "Hello, world!"')
assert_equal <<-DIV, div
<div class="CodeRay">
<div class="code"><pre>puts <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Hello, world!</span><span style="color:#710">&quot;</span></span></pre></div>
</div>
DIV
end
end

View File

@ -1,84 +0,0 @@
require 'test/unit'
require File.expand_path('../../lib/assert_warning', __FILE__)
$:.unshift File.expand_path('../../../lib', __FILE__)
require 'coderay'
begin
require 'rubygems' unless defined? Gem
gem 'RedCloth', '>= 4.0.3' rescue nil
require 'redcloth'
rescue LoadError
warn 'RedCloth not found - skipping for_redcloth tests.'
undef RedCloth if defined? RedCloth
end
class BasicTest < Test::Unit::TestCase
def test_for_redcloth
require 'coderay/for_redcloth'
assert_equal "<p><span lang=\"ruby\" class=\"CodeRay\">puts <span style=\"background-color:hsla(0,100%,50%,0.05)\"><span style=\"color:#710\">&quot;</span><span style=\"color:#D20\">Hello, World!</span><span style=\"color:#710\">&quot;</span></span></span></p>",
RedCloth.new('@[ruby]puts "Hello, World!"@').to_html
assert_equal <<-BLOCKCODE.chomp,
<div lang="ruby" class="CodeRay">
<div class="code"><pre>puts <span style="background-color:hsla(0,100%,50%,0.05)"><span style="color:#710">&quot;</span><span style="color:#D20">Hello, World!</span><span style="color:#710">&quot;</span></span></pre></div>
</div>
BLOCKCODE
RedCloth.new('bc[ruby]. puts "Hello, World!"').to_html
end
def test_for_redcloth_no_lang
require 'coderay/for_redcloth'
assert_equal "<p><code>puts \"Hello, World!\"</code></p>",
RedCloth.new('@puts "Hello, World!"@').to_html
assert_equal <<-BLOCKCODE.chomp,
<pre><code>puts \"Hello, World!\"</code></pre>
BLOCKCODE
RedCloth.new('bc. puts "Hello, World!"').to_html
end
def test_for_redcloth_style
require 'coderay/for_redcloth'
assert_equal <<-BLOCKCODE.chomp,
<pre style=\"color: red;\"><code style=\"color: red;\">puts \"Hello, World!\"</code></pre>
BLOCKCODE
RedCloth.new('bc{color: red}. puts "Hello, World!"').to_html
end
def test_for_redcloth_escapes
require 'coderay/for_redcloth'
assert_equal '<p><span lang="ruby" class="CodeRay">&gt;</span></p>',
RedCloth.new('@[ruby]>@').to_html
assert_equal <<-BLOCKCODE.chomp,
<div lang="ruby" class="CodeRay">
<div class="code"><pre>&amp;</pre></div>
</div>
BLOCKCODE
RedCloth.new('bc[ruby]. &').to_html
end
def test_for_redcloth_escapes2
require 'coderay/for_redcloth'
assert_equal "<p><span lang=\"c\" class=\"CodeRay\"><span style=\"color:#579\">#include</span> <span style=\"color:#B44;font-weight:bold\">&lt;test.h&gt;</span></span></p>",
RedCloth.new('@[c]#include <test.h>@').to_html
end
# See http://jgarber.lighthouseapp.com/projects/13054/tickets/124-code-markup-does-not-allow-brackets.
def test_for_redcloth_false_positive
require 'coderay/for_redcloth'
assert_warning 'CodeRay::Scanners could not load plugin :project; falling back to :text' do
assert_equal '<p><code>[project]_dff.skjd</code></p>',
RedCloth.new('@[project]_dff.skjd@').to_html
end
# false positive, but expected behavior / known issue
assert_equal "<p><span lang=\"ruby\" class=\"CodeRay\">_dff.skjd</span></p>",
RedCloth.new('@[ruby]_dff.skjd@').to_html
assert_warning 'CodeRay::Scanners could not load plugin :project; falling back to :text' do
assert_equal <<-BLOCKCODE.chomp,
<pre><code>[project]_dff.skjd</code></pre>
BLOCKCODE
RedCloth.new('bc. [project]_dff.skjd').to_html
end
end
end if defined? RedCloth

View File

@ -1,15 +0,0 @@
require 'test/unit'
$VERBOSE = $CODERAY_DEBUG = true
$:.unshift File.expand_path('../../../lib', __FILE__)
require 'coderay'
mydir = File.dirname(__FILE__)
suite = Dir[File.join(mydir, '*.rb')].
map { |tc| File.basename(tc).sub(/\.rb$/, '') } - %w'suite for_redcloth'
puts "Running basic CodeRay #{CodeRay::VERSION} tests: #{suite.join(', ')}"
for test_case in suite
load File.join(mydir, test_case + '.rb')
end

View File

@ -1,8 +0,0 @@
# -*- ruby -*-
require 'autotest/restart'
Autotest.add_hook :initialize do |at|
at.testlib = 'minitest/autorun'
at.find_directories = ARGV unless ARGV.empty?
end

View File

@ -1,7 +0,0 @@
rvm:
- 1.8.7
- 1.9.2
- 1.9.3
- jruby
- rbx
- ree

View File

@ -1,10 +0,0 @@
Wed Feb 15 11:49:41 2012 Aaron Patterson <aaron@tenderlovemaking.com>
* lib/journey/formatter.rb: reject non-truthy parameters parts.
Fixes rails ticket #4587
Mon Jan 23 17:07:53 2012 Aaron Patterson <aaron@tenderlovemaking.com>
* Added symbol? and literal? predicate methods to the ast nodes for
easier AST traversal.
* Made the dummy node a real class.

View File

@ -1,14 +0,0 @@
# -*- ruby -*-
# DO NOT EDIT THIS FILE. Instead, edit Rakefile, and run `rake bundler:gemfile`.
source :gemcutter
gem "minitest", "~>2.3", :group => [:development, :test]
gem "racc", ">=1.4.6", :group => [:development, :test]
gem "rdoc", "~>3.11", :group => [:development, :test]
gem "json", ">=0", :group => [:development, :test]
gem "hoe", "~>2.12", :group => [:development, :test]
# vim: syntax=ruby

View File

@ -1,49 +0,0 @@
.autotest
.gemtest
.travis.yml
CHANGELOG.rdoc
Gemfile
Manifest.txt
README.rdoc
Rakefile
journey.gemspec
lib/journey.rb
lib/journey/backwards.rb
lib/journey/core-ext/hash.rb
lib/journey/formatter.rb
lib/journey/gtg/builder.rb
lib/journey/gtg/simulator.rb
lib/journey/gtg/transition_table.rb
lib/journey/nfa/builder.rb
lib/journey/nfa/dot.rb
lib/journey/nfa/simulator.rb
lib/journey/nfa/transition_table.rb
lib/journey/nodes/node.rb
lib/journey/parser.rb
lib/journey/parser.y
lib/journey/parser_extras.rb
lib/journey/path/pattern.rb
lib/journey/route.rb
lib/journey/router.rb
lib/journey/router/strexp.rb
lib/journey/router/utils.rb
lib/journey/routes.rb
lib/journey/scanner.rb
lib/journey/visitors.rb
lib/journey/visualizer/fsm.css
lib/journey/visualizer/fsm.js
lib/journey/visualizer/index.html.erb
test/gtg/test_builder.rb
test/gtg/test_transition_table.rb
test/helper.rb
test/nfa/test_simulator.rb
test/nfa/test_transition_table.rb
test/nodes/test_symbol.rb
test/path/test_pattern.rb
test/route/definition/test_parser.rb
test/route/definition/test_scanner.rb
test/router/test_strexp.rb
test/router/test_utils.rb
test/test_route.rb
test/test_router.rb
test/test_routes.rb

View File

@ -1,48 +0,0 @@
= Journey
* http://github.com/rails/journey
== DESCRIPTION:
Journey is a router. It routes requests.
== FEATURES/PROBLEMS:
* Designed for rails
== SYNOPSIS:
Too complex right now. :(
== REQUIREMENTS:
* None
== INSTALL:
* gem install journey
== LICENSE:
(The MIT License)
Copyright (c) 2011 Aaron Patterson
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
'Software'), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

View File

@ -1,32 +0,0 @@
# -*- ruby -*-
require 'rubygems'
require 'hoe'
Hoe.plugins.delete :rubyforge
Hoe.plugin :minitest
Hoe.plugin :gemspec # `gem install hoe-gemspec`
Hoe.plugin :git # `gem install hoe-git`
Hoe.plugin :bundler # `gem install hoe-bundler`
Hoe.spec 'journey' do
developer('Aaron Patterson', 'aaron@tenderlovemaking.com')
self.readme_file = 'README.rdoc'
self.history_file = 'CHANGELOG.rdoc'
self.extra_rdoc_files = FileList['*.rdoc']
self.extra_dev_deps += [
["racc", ">= 1.4.6"],
["rdoc", "~> 3.11"],
["json"],
]
end
rule '.rb' => '.y' do |t|
sh "racc -l -o #{t.name} #{t.source}"
end
task :compile => "lib/journey/parser.rb"
Rake::Task[:test].prerequisites.unshift "lib/journey/parser.rb"
# vim: syntax=ruby

View File

@ -1,48 +0,0 @@
# -*- encoding: utf-8 -*-
Gem::Specification.new do |s|
s.name = "journey"
s.version = "1.0.3.20120222115248"
s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
s.authors = ["Aaron Patterson"]
s.date = "2012-02-22"
s.description = "Journey is a router. It routes requests."
s.email = ["aaron@tenderlovemaking.com"]
s.extra_rdoc_files = ["Manifest.txt", "CHANGELOG.rdoc", "README.rdoc"]
s.files = [".autotest", ".gemtest", ".travis.yml", "CHANGELOG.rdoc", "Gemfile", "Manifest.txt", "README.rdoc", "Rakefile", "journey.gemspec", "lib/journey.rb", "lib/journey/backwards.rb", "lib/journey/core-ext/hash.rb", "lib/journey/formatter.rb", "lib/journey/gtg/builder.rb", "lib/journey/gtg/simulator.rb", "lib/journey/gtg/transition_table.rb", "lib/journey/nfa/builder.rb", "lib/journey/nfa/dot.rb", "lib/journey/nfa/simulator.rb", "lib/journey/nfa/transition_table.rb", "lib/journey/nodes/node.rb", "lib/journey/parser.rb", "lib/journey/parser.y", "lib/journey/parser_extras.rb", "lib/journey/path/pattern.rb", "lib/journey/route.rb", "lib/journey/router.rb", "lib/journey/router/strexp.rb", "lib/journey/router/utils.rb", "lib/journey/routes.rb", "lib/journey/scanner.rb", "lib/journey/visitors.rb", "lib/journey/visualizer/fsm.css", "lib/journey/visualizer/fsm.js", "lib/journey/visualizer/index.html.erb", "test/gtg/test_builder.rb", "test/gtg/test_transition_table.rb", "test/helper.rb", "test/nfa/test_simulator.rb", "test/nfa/test_transition_table.rb", "test/nodes/test_symbol.rb", "test/path/test_pattern.rb", "test/route/definition/test_parser.rb", "test/route/definition/test_scanner.rb", "test/router/test_strexp.rb", "test/router/test_utils.rb", "test/test_route.rb", "test/test_router.rb", "test/test_routes.rb"]
s.homepage = "http://github.com/rails/journey"
s.rdoc_options = ["--main", "README.rdoc"]
s.require_paths = ["lib"]
s.rubyforge_project = "journey"
s.rubygems_version = "1.8.11"
s.summary = "Journey is a router"
s.test_files = ["test/gtg/test_builder.rb", "test/gtg/test_transition_table.rb", "test/nfa/test_simulator.rb", "test/nfa/test_transition_table.rb", "test/nodes/test_symbol.rb", "test/path/test_pattern.rb", "test/route/definition/test_parser.rb", "test/route/definition/test_scanner.rb", "test/router/test_strexp.rb", "test/router/test_utils.rb", "test/test_route.rb", "test/test_router.rb", "test/test_routes.rb"]
if s.respond_to? :specification_version then
s.specification_version = 3
if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
s.add_development_dependency(%q<minitest>, ["~> 2.11"])
s.add_development_dependency(%q<racc>, [">= 1.4.6"])
s.add_development_dependency(%q<rdoc>, ["~> 3.11"])
s.add_development_dependency(%q<json>, [">= 0"])
s.add_development_dependency(%q<rdoc>, ["~> 3.10"])
s.add_development_dependency(%q<hoe>, ["~> 2.13"])
else
s.add_dependency(%q<minitest>, ["~> 2.11"])
s.add_dependency(%q<racc>, [">= 1.4.6"])
s.add_dependency(%q<rdoc>, ["~> 3.11"])
s.add_dependency(%q<json>, [">= 0"])
s.add_dependency(%q<rdoc>, ["~> 3.10"])
s.add_dependency(%q<hoe>, ["~> 2.13"])
end
else
s.add_dependency(%q<minitest>, ["~> 2.11"])
s.add_dependency(%q<racc>, [">= 1.4.6"])
s.add_dependency(%q<rdoc>, ["~> 3.11"])
s.add_dependency(%q<json>, [">= 0"])
s.add_dependency(%q<rdoc>, ["~> 3.10"])
s.add_dependency(%q<hoe>, ["~> 2.13"])
end
end

View File

@ -1,5 +0,0 @@
require 'journey/router'
require 'journey/gtg/builder'
require 'journey/gtg/simulator'
require 'journey/nfa/builder'
require 'journey/nfa/simulator'

View File

@ -1,5 +0,0 @@
module Rack
Mount = Journey::Router
Mount::RouteSet = Journey::Router
Mount::RegexpWithNamedGroups = Journey::Path::Pattern
end

View File

@ -1,11 +0,0 @@
# :stopdoc:
if RUBY_VERSION < '1.9'
class Hash
def keep_if
each do |k,v|
delete(k) unless yield(k,v)
end
end
end
end
# :startdoc:

View File

@ -1,129 +0,0 @@
module Journey
###
# The Formatter class is used for formatting URLs. For example, parameters
# passed to +url_for+ in rails will eventually call Formatter#generate
class Formatter
attr_reader :routes
def initialize routes
@routes = routes
@cache = nil
end
def generate key, name, options, recall = {}, parameterize = nil
constraints = recall.merge options
match_route(name, constraints) do |route|
data = constraints.dup
keys_to_keep = route.parts.reverse.drop_while { |part|
!options.key?(part) || (options[part] || recall[part]).nil?
} | route.required_parts
(data.keys - keys_to_keep).each do |bad_key|
data.delete bad_key
end
parameterized_parts = data.dup
if parameterize
parameterized_parts.each do |k,v|
parameterized_parts[k] = parameterize.call(k, v)
end
end
parameterized_parts.keep_if { |_,v| v }
next unless verify_required_parts!(route, parameterized_parts)
z = Hash[options.to_a - data.to_a - route.defaults.to_a]
return [route.format(parameterized_parts), z]
end
raise Router::RoutingError
end
def clear
@cache = nil
end
private
def named_routes
routes.named_routes
end
def match_route name, options
if named_routes.key? name
yield named_routes[name]
else
#routes = possibles(@cache, options.to_a)
routes = non_recursive(cache, options.to_a)
hash = routes.group_by { |_, r|
r.score options
}
hash.keys.sort.reverse_each do |score|
next if score < 0
hash[score].sort_by { |i,_| i }.each do |_,route|
yield route
end
end
end
end
def non_recursive cache, options
routes = []
stack = [cache]
while stack.any?
c = stack.shift
routes.concat c[:___routes] if c.key? :___routes
options.each do |pair|
stack << c[pair] if c.key? pair
end
end
routes
end
def possibles cache, options, depth = 0
cache.fetch(:___routes) { [] } + options.find_all { |pair|
cache.key? pair
}.map { |pair|
possibles(cache[pair], options, depth + 1)
}.flatten(1)
end
def verify_required_parts! route, parts
tests = route.path.requirements
route.required_parts.all? { |key|
if tests.key? key
/\A#{tests[key]}\Z/ === parts[key]
else
parts.fetch(key) { false }
end
}
end
def build_cache
kash = {}
routes.each_with_index do |route, i|
money = kash
route.required_defaults.each do |tuple|
hash = (money[tuple] ||= {})
money = hash
end
(money[:___routes] ||= []) << [i, route]
end
kash
end
def cache
@cache ||= build_cache
end
end
end

View File

@ -1,159 +0,0 @@
require 'journey/gtg/transition_table'
module Journey
module GTG
class Builder
DUMMY = Nodes::Dummy.new # :nodoc:
attr_reader :root, :ast, :endpoints
def initialize root
@root = root
@ast = Nodes::Cat.new root, DUMMY
@followpos = nil
end
def transition_table
dtrans = TransitionTable.new
marked = {}
state_id = Hash.new { |h,k| h[k] = h.length }
start = firstpos(root)
dstates = [start]
until dstates.empty?
s = dstates.shift
next if marked[s]
marked[s] = true # mark s
s.group_by { |state| symbol(state) }.each do |sym, ps|
u = ps.map { |l| followpos(l) }.flatten
next if u.empty?
if u.uniq == [DUMMY]
from = state_id[s]
to = state_id[Object.new]
dtrans[from, to] = sym
dtrans.add_accepting to
ps.each { |state| dtrans.add_memo to, state.memo }
else
dtrans[state_id[s], state_id[u]] = sym
if u.include? DUMMY
to = state_id[u]
accepting = ps.find_all { |l| followpos(l).include? DUMMY }
accepting.each { |accepting_state|
dtrans.add_memo to, accepting_state.memo
}
dtrans.add_accepting state_id[u]
end
end
dstates << u
end
end
dtrans
end
def nullable? node
case node
when Nodes::Group
true
when Nodes::Star
true
when Nodes::Or
node.children.any? { |c| nullable?(c) }
when Nodes::Cat
nullable?(node.left) && nullable?(node.right)
when Nodes::Terminal
!node.left
when Nodes::Unary
nullable? node.left
else
raise ArgumentError, 'unknown nullable: %s' % node.class.name
end
end
def firstpos node
case node
when Nodes::Star
firstpos(node.left)
when Nodes::Cat
if nullable? node.left
firstpos(node.left) | firstpos(node.right)
else
firstpos(node.left)
end
when Nodes::Or
node.children.map { |c| firstpos(c) }.flatten.uniq
when Nodes::Unary
firstpos(node.left)
when Nodes::Terminal
nullable?(node) ? [] : [node]
else
raise ArgumentError, 'unknown firstpos: %s' % node.class.name
end
end
def lastpos node
case node
when Nodes::Star
firstpos(node.left)
when Nodes::Or
node.children.map { |c| lastpos(c) }.flatten.uniq
when Nodes::Cat
if nullable? node.right
lastpos(node.left) | lastpos(node.right)
else
lastpos(node.right)
end
when Nodes::Terminal
nullable?(node) ? [] : [node]
when Nodes::Unary
lastpos(node.left)
else
raise ArgumentError, 'unknown lastpos: %s' % node.class.name
end
end
def followpos node
followpos_table[node]
end
private
def followpos_table
@followpos ||= build_followpos
end
def build_followpos
table = Hash.new { |h,k| h[k] = [] }
@ast.each do |n|
case n
when Nodes::Cat
lastpos(n.left).each do |i|
table[i] += firstpos(n.right)
end
when Nodes::Star
lastpos(n).each do |i|
table[i] += firstpos(n)
end
end
end
table
end
def symbol edge
case edge
when Journey::Nodes::Symbol
edge.regexp
else
edge.left
end
end
end
end
end

View File

@ -1,43 +0,0 @@
require 'strscan'
module Journey
module GTG
class MatchData
attr_reader :memos
def initialize memos
@memos = memos
end
end
class Simulator
attr_reader :tt
def initialize transition_table
@tt = transition_table
end
def simulate string
input = StringScanner.new string
state = [0]
while sym = input.scan(/[\/\.\?]|[^\/\.\?]+/)
state = tt.move(state, sym)
end
acceptance_states = state.find_all { |s|
tt.accepting? s
}
return if acceptance_states.empty?
memos = acceptance_states.map { |x| tt.memo x }.flatten.compact
MatchData.new memos
end
alias :=~ :simulate
alias :match :simulate
end
end
end

View File

@ -1,153 +0,0 @@
require 'journey/nfa/dot'
module Journey
module GTG
class TransitionTable
include Journey::NFA::Dot
attr_reader :memos
def initialize
@regexp_states = Hash.new { |h,k| h[k] = {} }
@string_states = Hash.new { |h,k| h[k] = {} }
@accepting = {}
@memos = Hash.new { |h,k| h[k] = [] }
end
def add_accepting state
@accepting[state] = true
end
def accepting_states
@accepting.keys
end
def accepting? state
@accepting[state]
end
def add_memo idx, memo
@memos[idx] << memo
end
def memo idx
@memos[idx]
end
def eclosure t
Array(t)
end
def move t, a
move_string(t, a).concat move_regexp(t, a)
end
def to_json
require 'json'
simple_regexp = Hash.new { |h,k| h[k] = {} }
@regexp_states.each do |from, hash|
hash.each do |re, to|
simple_regexp[from][re.source] = to
end
end
JSON.dump({
:regexp_states => simple_regexp,
:string_states => @string_states,
:accepting => @accepting
})
end
def to_svg
svg = IO.popen("dot -Tsvg", 'w+') { |f|
f.write to_dot
f.close_write
f.readlines
}
3.times { svg.shift }
svg.join.sub(/width="[^"]*"/, '').sub(/height="[^"]*"/, '')
end
def visualizer paths, title = 'FSM'
viz_dir = File.join File.dirname(__FILE__), '..', 'visualizer'
fsm_js = File.read File.join(viz_dir, 'fsm.js')
fsm_css = File.read File.join(viz_dir, 'fsm.css')
erb = File.read File.join(viz_dir, 'index.html.erb')
states = "function tt() { return #{to_json}; }"
fun_routes = paths.shuffle.first(3).map do |ast|
ast.map { |n|
case n
when Nodes::Symbol
case n.left
when ':id' then rand(100).to_s
when ':format' then %w{ xml json }.shuffle.first
else
'omg'
end
when Nodes::Terminal then n.symbol
else
nil
end
}.compact.join
end
stylesheets = [fsm_css]
svg = to_svg
javascripts = [states, fsm_js]
# Annoying hack for 1.9 warnings
fun_routes = fun_routes
stylesheets = stylesheets
svg = svg
javascripts = javascripts
require 'erb'
template = ERB.new erb
template.result(binding)
end
def []= from, to, sym
case sym
when String
@string_states[from][sym] = to
when Regexp
@regexp_states[from][sym] = to
else
raise ArgumentError, 'unknown symbol: %s' % sym.class
end
end
def states
ss = @string_states.keys + @string_states.values.map(&:values).flatten
rs = @regexp_states.keys + @regexp_states.values.map(&:values).flatten
(ss + rs).uniq
end
def transitions
@string_states.map { |from, hash|
hash.map { |s, to| [from, s, to] }
}.flatten(1) + @regexp_states.map { |from, hash|
hash.map { |s, to| [from, s, to] }
}.flatten(1)
end
private
def move_regexp t, a
return [] if t.empty?
t.map { |s|
@regexp_states[s].map { |re,v| re === a ? v : nil }
}.flatten.compact.uniq
end
def move_string t, a
return [] if t.empty?
t.map { |s| @string_states[s][a] }.compact
end
end
end
end

View File

@ -1,74 +0,0 @@
require 'journey/nfa/transition_table'
require 'journey/gtg/transition_table'
module Journey
module NFA
class Visitor < Visitors::Visitor
def initialize tt
@tt = tt
@i = -1
end
def visit_CAT node
left = visit node.left
right = visit node.right
@tt.merge left.last, right.first
[left.first, right.last]
end
def visit_GROUP node
from = @i += 1
left = visit node.left
to = @i += 1
@tt.accepting = to
@tt[from, left.first] = nil
@tt[left.last, to] = nil
@tt[from, to] = nil
[from, to]
end
def visit_OR node
from = @i += 1
children = node.children.map { |c| visit c }
to = @i += 1
children.each do |child|
@tt[from, child.first] = nil
@tt[child.last, to] = nil
end
@tt.accepting = to
[from, to]
end
def terminal node
from_i = @i += 1 # new state
to_i = @i += 1 # new state
@tt[from_i, to_i] = node
@tt.accepting = to_i
@tt.add_memo to_i, node.memo
[from_i, to_i]
end
end
class Builder
def initialize ast
@ast = ast
end
def transition_table
tt = TransitionTable.new
Visitor.new(tt).accept @ast
tt
end
end
end
end

View File

@ -1,34 +0,0 @@
# encoding: utf-8
module Journey
module NFA
module Dot
def to_dot
edges = transitions.map { |from, sym, to|
" #{from} -> #{to} [label=\"#{sym || 'ε'}\"];"
}
#memo_nodes = memos.values.flatten.map { |n|
# label = n
# if Journey::Route === n
# label = "#{n.verb.source} #{n.path.spec}"
# end
# " #{n.object_id} [label=\"#{label}\", shape=box];"
#}
#memo_edges = memos.map { |k, memos|
# (memos || []).map { |v| " #{k} -> #{v.object_id};" }
#}.flatten.uniq
<<-eodot
digraph nfa {
rankdir=LR;
node [shape = doublecircle];
#{accepting_states.join ' '};
node [shape = circle];
#{edges.join "\n"}
}
eodot
end
end
end
end

View File

@ -1,45 +0,0 @@
require 'strscan'
module Journey
module NFA
class MatchData
attr_reader :memos
def initialize memos
@memos = memos
end
end
class Simulator
attr_reader :tt
def initialize transition_table
@tt = transition_table
end
def simulate string
input = StringScanner.new string
state = tt.eclosure 0
until input.eos?
sym = input.scan(/[\/\.\?]|[^\/\.\?]+/)
# FIXME: tt.eclosure is not needed for the GTG
state = tt.eclosure tt.move(state, sym)
end
acceptance_states = state.find_all { |s|
tt.accepting? tt.eclosure(s).sort.last
}
return if acceptance_states.empty?
memos = acceptance_states.map { |x| tt.memo x }.flatten.compact
MatchData.new memos
end
alias :=~ :simulate
alias :match :simulate
end
end
end

View File

@ -1,164 +0,0 @@
require 'journey/nfa/dot'
module Journey
module NFA
class TransitionTable
include Journey::NFA::Dot
attr_accessor :accepting
attr_reader :memos
def initialize
@table = Hash.new { |h,f| h[f] = {} }
@memos = {}
@accepting = nil
@inverted = nil
end
def accepting? state
accepting == state
end
def accepting_states
[accepting]
end
def add_memo idx, memo
@memos[idx] = memo
end
def memo idx
@memos[idx]
end
def []= i, f, s
@table[f][i] = s
end
def merge left, right
@memos[right] = @memos.delete left
@table[right] = @table.delete(left)
end
def states
(@table.keys + @table.values.map(&:keys).flatten).uniq
end
###
# Returns a generalized transition graph with reduced states. The states
# are reduced like a DFA, but the table must be simulated like an NFA.
#
# Edges of the GTG are regular expressions
def generalized_table
gt = GTG::TransitionTable.new
marked = {}
state_id = Hash.new { |h,k| h[k] = h.length }
alphabet = self.alphabet
stack = [eclosure(0)]
until stack.empty?
state = stack.pop
next if marked[state] || state.empty?
marked[state] = true
alphabet.each do |alpha|
next_state = eclosure(following_states(state, alpha))
next if next_state.empty?
gt[state_id[state], state_id[next_state]] = alpha
stack << next_state
end
end
final_groups = state_id.keys.find_all { |s|
s.sort.last == accepting
}
final_groups.each do |states|
id = state_id[states]
gt.add_accepting id
save = states.find { |s|
@memos.key?(s) && eclosure(s).sort.last == accepting
}
gt.add_memo id, memo(save)
end
gt
end
###
# Returns set of NFA states to which there is a transition on ast symbol
# +a+ from some state +s+ in +t+.
def following_states t, a
Array(t).map { |s| inverted[s][a] }.flatten.uniq
end
###
# Returns set of NFA states to which there is a transition on ast symbol
# +a+ from some state +s+ in +t+.
def move t, a
Array(t).map { |s|
inverted[s].keys.compact.find_all { |sym|
sym === a
}.map { |sym| inverted[s][sym] }
}.flatten.uniq
end
def alphabet
inverted.values.map(&:keys).flatten.compact.uniq.sort_by { |x| x.to_s }
end
###
# Returns a set of NFA states reachable from some NFA state +s+ in set
# +t+ on nil-transitions alone.
def eclosure t
stack = Array(t)
seen = {}
children = []
until stack.empty?
s = stack.pop
next if seen[s]
seen[s] = true
children << s
stack.concat inverted[s][nil]
end
children.uniq
end
def transitions
@table.map { |to, hash|
hash.map { |from, sym| [from, sym, to] }
}.flatten(1)
end
private
def inverted
return @inverted if @inverted
@inverted = Hash.new { |h,from|
h[from] = Hash.new { |j,s| j[s] = [] }
}
@table.each { |to, hash|
hash.each { |from, sym|
if sym
sym = Nodes::Symbol === sym ? sym.regexp : sym.left
end
@inverted[from][sym] << to
}
}
@inverted
end
end
end
end

View File

@ -1,122 +0,0 @@
require 'journey/visitors'
module Journey
module Nodes
class Node # :nodoc:
include Enumerable
attr_accessor :left, :memo
def initialize left
@left = left
@memo = nil
end
def each(&block)
Visitors::Each.new(block).accept(self)
end
def to_s
Visitors::String.new.accept(self)
end
def to_dot
Visitors::Dot.new.accept(self)
end
def to_sym
name.to_sym
end
def name
left.tr ':', ''
end
def type
raise NotImplementedError
end
def symbol?; false; end
def literal?; false; end
end
class Terminal < Node
alias :symbol :left
end
class Literal < Terminal
def literal?; true; end
def type; :LITERAL; end
end
class Dummy < Literal
def initialize x = Object.new
super
end
def literal?; false; end
end
%w{ Symbol Slash Dot }.each do |t|
class_eval %{
class #{t} < Terminal
def type; :#{t.upcase}; end
end
}
end
class Symbol < Terminal
attr_accessor :regexp
alias :symbol :regexp
DEFAULT_EXP = /[^\.\/\?]+/
def initialize left
super
@regexp = DEFAULT_EXP
end
def default_regexp?
regexp == DEFAULT_EXP
end
def symbol?; true; end
end
class Unary < Node
def children; [value] end
end
class Group < Unary
def type; :GROUP; end
end
class Star < Unary
def type; :STAR; end
end
class Binary < Node
attr_accessor :right
def initialize left, right
super(left)
@right = right
end
def children; [left, right] end
end
class Cat < Binary
def type; :CAT; end
end
class Or < Node
attr_reader :children
def initialize children
@children = children
end
def type; :OR; end
end
end
end

View File

@ -1,204 +0,0 @@
#
# DO NOT MODIFY!!!!
# This file is automatically generated by Racc 1.4.6
# from Racc grammer file "".
#
require 'racc/parser.rb'
require 'journey/parser_extras'
module Journey
class Parser < Racc::Parser
##### State transition tables begin ###
racc_action_table = [
17, 22, 13, 15, 14, 7, 15, 16, 8, 19,
13, 15, 14, 7, 24, 16, 8, 19, 13, 15,
14, 7, nil, 16, 8, 13, 15, 14, 7, nil,
16, 8, 13, 15, 14, 7, nil, 16, 8 ]
racc_action_check = [
1, 17, 1, 1, 1, 1, 8, 1, 1, 1,
20, 20, 20, 20, 20, 20, 20, 20, 7, 7,
7, 7, nil, 7, 7, 19, 19, 19, 19, nil,
19, 19, 0, 0, 0, 0, nil, 0, 0 ]
racc_action_pointer = [
30, 0, nil, nil, nil, nil, nil, 16, 3, nil,
nil, nil, nil, nil, nil, nil, nil, 1, nil, 23,
8, nil, nil, nil, nil ]
racc_action_default = [
-18, -18, -2, -3, -4, -5, -6, -18, -18, -10,
-11, -12, -13, -14, -15, -16, -17, -18, -1, -18,
-18, -9, 25, -8, -7 ]
racc_goto_table = [
18, 1, 21, nil, nil, nil, nil, nil, 20, nil,
nil, nil, nil, nil, nil, nil, nil, nil, 23, 18 ]
racc_goto_check = [
2, 1, 7, nil, nil, nil, nil, nil, 1, nil,
nil, nil, nil, nil, nil, nil, nil, nil, 2, 2 ]
racc_goto_pointer = [
nil, 1, -1, nil, nil, nil, nil, -6, nil, nil,
nil ]
racc_goto_default = [
nil, nil, 2, 3, 4, 5, 6, 10, 9, 11,
12 ]
racc_reduce_table = [
0, 0, :racc_error,
2, 11, :_reduce_1,
1, 11, :_reduce_2,
1, 11, :_reduce_none,
1, 12, :_reduce_none,
1, 12, :_reduce_none,
1, 12, :_reduce_none,
3, 15, :_reduce_7,
3, 13, :_reduce_8,
2, 16, :_reduce_9,
1, 14, :_reduce_none,
1, 14, :_reduce_none,
1, 14, :_reduce_none,
1, 14, :_reduce_none,
1, 19, :_reduce_14,
1, 18, :_reduce_15,
1, 17, :_reduce_16,
1, 20, :_reduce_17 ]
racc_reduce_n = 18
racc_shift_n = 25
racc_token_table = {
false => 0,
:error => 1,
:SLASH => 2,
:LITERAL => 3,
:SYMBOL => 4,
:LPAREN => 5,
:RPAREN => 6,
:DOT => 7,
:STAR => 8,
:OR => 9 }
racc_nt_base = 10
racc_use_result_var = true
Racc_arg = [
racc_action_table,
racc_action_check,
racc_action_default,
racc_action_pointer,
racc_goto_table,
racc_goto_check,
racc_goto_default,
racc_goto_pointer,
racc_nt_base,
racc_reduce_table,
racc_token_table,
racc_shift_n,
racc_reduce_n,
racc_use_result_var ]
Racc_token_to_s_table = [
"$end",
"error",
"SLASH",
"LITERAL",
"SYMBOL",
"LPAREN",
"RPAREN",
"DOT",
"STAR",
"OR",
"$start",
"expressions",
"expression",
"or",
"terminal",
"group",
"star",
"literal",
"symbol",
"slash",
"dot" ]
Racc_debug_parser = false
##### State transition tables end #####
# reduce 0 omitted
def _reduce_1(val, _values, result)
result = Cat.new(val.first, val.last)
result
end
def _reduce_2(val, _values, result)
result = val.first
result
end
# reduce 3 omitted
# reduce 4 omitted
# reduce 5 omitted
# reduce 6 omitted
def _reduce_7(val, _values, result)
result = Group.new(val[1])
result
end
def _reduce_8(val, _values, result)
result = Or.new([val.first, val.last])
result
end
def _reduce_9(val, _values, result)
result = Star.new(Symbol.new(val.last.left))
result
end
# reduce 10 omitted
# reduce 11 omitted
# reduce 12 omitted
# reduce 13 omitted
def _reduce_14(val, _values, result)
result = Slash.new('/')
result
end
def _reduce_15(val, _values, result)
result = Symbol.new(val.first)
result
end
def _reduce_16(val, _values, result)
result = Literal.new(val.first)
result
end
def _reduce_17(val, _values, result)
result = Dot.new(val.first)
result
end
def _reduce_none(val, _values, result)
val[0]
end
end # class Parser
end # module Journey

View File

@ -1,47 +0,0 @@
class Journey::Parser
token SLASH LITERAL SYMBOL LPAREN RPAREN DOT STAR OR
rule
expressions
: expressions expression { result = Cat.new(val.first, val.last) }
| expression { result = val.first }
| or
;
expression
: terminal
| group
| star
;
group
: LPAREN expressions RPAREN { result = Group.new(val[1]) }
;
or
: expressions OR expression { result = Or.new([val.first, val.last]) }
;
star
: STAR literal { result = Star.new(Symbol.new(val.last.left)) }
;
terminal
: symbol
| literal
| slash
| dot
;
slash
: SLASH { result = Slash.new('/') }
;
symbol
: SYMBOL { result = Symbol.new(val.first) }
;
literal
: LITERAL { result = Literal.new(val.first) }
dot
: DOT { result = Dot.new(val.first) }
;
end
---- header
require 'journey/parser_extras'

View File

@ -1,21 +0,0 @@
require 'journey/scanner'
require 'journey/nodes/node'
module Journey
class Parser < Racc::Parser
include Journey::Nodes
def initialize
@scanner = Scanner.new
end
def parse string
@scanner.scan_setup string
do_parse
end
def next_token
@scanner.next_token
end
end
end

View File

@ -1,193 +0,0 @@
module Journey
module Path
class Pattern
attr_reader :spec, :requirements, :anchored
def initialize strexp
parser = Journey::Parser.new
@anchored = true
case strexp
when String
@spec = parser.parse strexp
@requirements = {}
@separators = "/.?"
when Router::Strexp
@spec = parser.parse strexp.path
@requirements = strexp.requirements
@separators = strexp.separators.join
@anchored = strexp.anchor
else
raise "wtf bro: #{strexp}"
end
@names = nil
@optional_names = nil
@required_names = nil
@re = nil
@offsets = nil
end
def ast
@spec.grep(Nodes::Symbol).each do |node|
re = @requirements[node.to_sym]
node.regexp = re if re
end
@spec.grep(Nodes::Star).each do |node|
node = node.left
node.regexp = @requirements[node.to_sym] || /(.+)/
end
@spec
end
def names
@names ||= spec.grep(Nodes::Symbol).map { |n| n.name }
end
def required_names
@required_names ||= names - optional_names
end
def optional_names
@optional_names ||= spec.grep(Nodes::Group).map { |group|
group.grep(Nodes::Symbol)
}.flatten.map { |n| n.name }.uniq
end
class RegexpOffsets < Journey::Visitors::Visitor # :nodoc:
attr_reader :offsets
def initialize matchers
@matchers = matchers
@capture_count = [0]
end
def visit node
super
@capture_count
end
def visit_SYMBOL node
node = node.to_sym
if @matchers.key? node
re = /#{@matchers[node]}|/
@capture_count.push((re.match('').length - 1) + (@capture_count.last || 0))
else
@capture_count << (@capture_count.last || 0)
end
end
end
class AnchoredRegexp < Journey::Visitors::Visitor # :nodoc:
def initialize separator, matchers
@separator = separator
@matchers = matchers
@separator_re = "([^#{separator}]+)"
super()
end
def accept node
%r{\A#{visit node}\Z}
end
def visit_CAT node
[visit(node.left), visit(node.right)].join
end
def visit_SYMBOL node
node = node.to_sym
return @separator_re unless @matchers.key? node
re = @matchers[node]
"(#{re})"
end
def visit_GROUP node
"(?:#{visit node.left})?"
end
def visit_LITERAL node
Regexp.escape node.left
end
alias :visit_DOT :visit_LITERAL
def visit_SLASH node
node.left
end
def visit_STAR node
re = @matchers[node.left.to_sym] || '.+'
"(#{re})"
end
end
class UnanchoredRegexp < AnchoredRegexp # :nodoc:
def accept node
%r{\A#{visit node}}
end
end
class MatchData
attr_reader :names
def initialize names, offsets, match
@names = names
@offsets = offsets
@match = match
end
def captures
(length - 1).times.map { |i| self[i + 1] }
end
def [] x
idx = @offsets[x - 1] + x
@match[idx]
end
def length
@offsets.length
end
def post_match
@match.post_match
end
def to_s
@match.to_s
end
end
def match other
return unless match = to_regexp.match(other)
MatchData.new names, offsets, match
end
alias :=~ :match
def source
to_regexp.source
end
def to_regexp
@re ||= regexp_visitor.new(@separators, @requirements).accept spec
end
private
def regexp_visitor
@anchored ? AnchoredRegexp : UnanchoredRegexp
end
def offsets
return @offsets if @offsets
viz = RegexpOffsets.new @requirements
@offsets = viz.accept spec
end
end
end
end

View File

@ -1,95 +0,0 @@
module Journey
class Route
attr_reader :app, :path, :verb, :defaults, :ip, :name
attr_reader :constraints
alias :conditions :constraints
attr_accessor :precedence
##
# +path+ is a path constraint.
# +constraints+ is a hash of constraints to be applied to this route.
def initialize name, app, path, constraints, defaults = {}
constraints = constraints.dup
@name = name
@app = app
@path = path
@verb = constraints[:request_method] || //
@ip = constraints.delete(:ip) || //
@constraints = constraints
@constraints.keep_if { |_,v| Regexp === v || String === v }
@defaults = defaults
@required_defaults = nil
@required_parts = nil
@parts = nil
@decorated_ast = nil
@precedence = 0
end
def ast
return @decorated_ast if @decorated_ast
@decorated_ast = path.ast
@decorated_ast.grep(Nodes::Terminal).each { |n| n.memo = self }
@decorated_ast
end
def requirements # :nodoc:
# needed for rails `rake routes`
path.requirements.merge(@defaults).delete_if { |_,v|
/.+?/ == v
}
end
def segments
@path.names
end
def required_keys
path.required_names.map { |x| x.to_sym } + required_defaults.keys
end
def score constraints
required_keys = path.required_names
supplied_keys = constraints.map { |k,v| v && k.to_s }.compact
return -1 unless (required_keys - supplied_keys).empty?
score = (supplied_keys & path.names).length
score + (required_defaults.length * 2)
end
def parts
@parts ||= segments.map { |n| n.to_sym }
end
alias :segment_keys :parts
def format path_options
(defaults.keys - required_parts).each do |key|
path_options.delete key if defaults[key].to_s == path_options[key].to_s
end
formatter = Visitors::Formatter.new(path_options)
formatted_path = formatter.accept(path.spec)
formatted_path.gsub(/\/\x00/, '')
end
def optional_parts
path.optional_names.map { |n| n.to_sym }
end
def required_parts
@required_parts ||= path.required_names.map { |n| n.to_sym }
end
def required_defaults
@required_defaults ||= begin
matches = parts
@defaults.dup.delete_if { |k,_| matches.include? k }
end
end
end
end

View File

@ -1,145 +0,0 @@
require 'journey/core-ext/hash'
require 'journey/router/utils'
require 'journey/router/strexp'
require 'journey/routes'
require 'journey/formatter'
before = $-w
$-w = false
require 'journey/parser'
$-w = before
require 'journey/route'
require 'journey/path/pattern'
module Journey
class Router
class RoutingError < ::StandardError
end
VERSION = '1.0.3'
class NullReq # :nodoc:
attr_reader :env
def initialize env
@env = env
end
def request_method
env['REQUEST_METHOD']
end
def path_info
env['PATH_INFO']
end
def ip
env['REMOTE_ADDR']
end
def [](k); env[k]; end
end
attr_reader :request_class, :formatter
attr_accessor :routes
def initialize routes, options
@options = options
@params_key = options[:parameters_key]
@request_class = options[:request_class] || NullReq
@routes = routes
end
def call env
env['PATH_INFO'] = Utils.normalize_path env['PATH_INFO']
find_routes(env).each do |match, parameters, route|
script_name, path_info, set_params = env.values_at('SCRIPT_NAME',
'PATH_INFO',
@params_key)
unless route.path.anchored
env['SCRIPT_NAME'] = script_name.to_s + match.to_s
env['PATH_INFO'] = match.post_match
end
env[@params_key] = (set_params || {}).merge parameters
status, headers, body = route.app.call(env)
if 'pass' == headers['X-Cascade']
env['SCRIPT_NAME'] = script_name
env['PATH_INFO'] = path_info
env[@params_key] = set_params
next
end
return [status, headers, body]
end
return [404, {'X-Cascade' => 'pass'}, ['Not Found']]
end
def recognize req
find_routes(req.env).each do |match, parameters, route|
unless route.path.anchored
req.env['SCRIPT_NAME'] = match.to_s
req.env['PATH_INFO'] = match.post_match.sub(/^([^\/])/, '/\1')
end
yield(route, nil, parameters)
end
end
def visualizer
tt = GTG::Builder.new(ast).transition_table
groups = partitioned_routes.first.map(&:ast).group_by { |a| a.to_s }
asts = groups.values.map { |v| v.first }
tt.visualizer asts
end
private
def partitioned_routes
routes.partitioned_routes
end
def ast
routes.ast
end
def simulator
routes.simulator
end
def custom_routes
partitioned_routes.last
end
def filter_routes path
return [] unless ast
data = simulator.match(path)
data ? data.memos : []
end
def find_routes env
req = request_class.new env
routes = filter_routes(req.path_info) + custom_routes.find_all { |r|
r.path.match(req.path_info)
}
routes.sort_by(&:precedence).find_all { |r|
r.constraints.all? { |k,v| v === req.send(k) } &&
r.verb === req.request_method
}.reject { |r| req.ip && !(r.ip === req.ip) }.map { |r|
match_data = r.path.match(req.path_info)
match_names = match_data.names.map { |n| n.to_sym }
match_values = match_data.captures.map { |v| v && Utils.unescape_uri(v) }
info = Hash[match_names.zip(match_values).find_all { |_,y| y }]
[match_data, r.defaults.merge(info), r]
}
end
end
end

View File

@ -1,22 +0,0 @@
module Journey
class Router
class Strexp
class << self
alias :compile :new
end
attr_reader :path, :requirements, :separators, :anchor
def initialize path, requirements, separators, anchor = true
@path = path
@requirements = requirements
@separators = separators
@anchor = anchor
end
def names
@path.scan(/:\w+/).map { |s| s.tr(':', '') }
end
end
end
end

View File

@ -1,57 +0,0 @@
require 'uri'
module Journey
class Router
class Utils
# Normalizes URI path.
#
# Strips off trailing slash and ensures there is a leading slash.
#
# normalize_path("/foo") # => "/foo"
# normalize_path("/foo/") # => "/foo"
# normalize_path("foo") # => "/foo"
# normalize_path("") # => "/"
def self.normalize_path(path)
path = "/#{path}"
path.squeeze!('/')
path.sub!(%r{/+\Z}, '')
path = '/' if path == ''
path
end
# URI path and fragment escaping
# http://tools.ietf.org/html/rfc3986
module UriEscape
# Symbol captures can generate multiple path segments, so include /.
reserved_segment = '/'
reserved_fragment = '/?'
reserved_pchar = ':@&=+$,;%'
safe_pchar = "#{URI::REGEXP::PATTERN::UNRESERVED}#{reserved_pchar}"
safe_segment = "#{safe_pchar}#{reserved_segment}"
safe_fragment = "#{safe_pchar}#{reserved_fragment}"
if RUBY_VERSION >= '1.9'
UNSAFE_SEGMENT = Regexp.new("[^#{safe_segment}]", false).freeze
UNSAFE_FRAGMENT = Regexp.new("[^#{safe_fragment}]", false).freeze
else
UNSAFE_SEGMENT = Regexp.new("[^#{safe_segment}]", false, 'N').freeze
UNSAFE_FRAGMENT = Regexp.new("[^#{safe_fragment}]", false, 'N').freeze
end
end
Parser = URI.const_defined?(:Parser) ? URI::Parser.new : URI
def self.escape_path(path)
Parser.escape(path.to_s, UriEscape::UNSAFE_SEGMENT)
end
def self.escape_fragment(fragment)
Parser.escape(fragment.to_s, UriEscape::UNSAFE_FRAGMENT)
end
def self.unescape_uri(uri)
Parser.unescape(uri)
end
end
end
end

Some files were not shown because too many files have changed in this diff Show More