metasploit-framework/data/john/doc/CHANGES

The following changes have been made between John 1.7.7 and 1.7.8:

* The bitslice DES S-box expressions have been replaced with those generated
by Roman Rusakov specifically for John the Ripper.  The corresponding assembly
code for x86 with MMX, SSE2, and for x86-64 with SSE2 has been re-generated.
For other CPUs and for AVX/XOP, C compilers do a reasonably good job of
generating the code from the supplied C source files (with intrinsics where
relevant).  The S-box expressions that we were using before had a 21% larger
gate count, so theoretically this could provide a 21% speedup.  In practice,
though, a 12% to 14% speedup at DES-based crypt(3) hashes is typical.
This effort has been sponsored by Rapid7: http://www.rapid7.com
* Corrected support for bcrypt (OpenBSD Blowfish) hashes of passwords
containing non-ASCII characters (that is, characters with the 8th bit set).
Added support for such hashes produced by crypt_blowfish up to 1.0.4, which
contained a sign extension bug (inherited from older versions of John).
The old buggy behavior may be enabled per-hash, using the "$2x$" prefix.
* The external mode virtual machine's performance has been improved through
additional multi-op instructions matching common instruction sequences
(assign-pop and some triple- and quad-push VM instructions were added).
* A few minor bug fixes and enhancements were made.

The following changes have been made between John 1.7.6.1 and 1.7.7:

* Added Intel AVX and AMD XOP instruction sets support for bitslice DES
(with C compiler intrinsics).  New make targets: linux-x86-64-avx,
linux-x86-64-xop, linux-x86-avx, and linux-x86-xop (these require recent
versions of GCC and GNU binutils).
* A "dummy" "format" is now supported (plaintext passwords encoded in
hexadecimal and prefixed with "$dummy$") - for faster testing and tuning of
custom wordlists, rule sets, .chr files, and external modes on already known or
artificial passwords, as well as for testing of future and modified versions of
John itself.
* Apache "$apr1$" MD5-based password hashes are now supported along with the
FreeBSD-style MD5-based crypt(3) hashes that were supported previously.  Hashes
of both of these types may be loaded for cracking simultaneously.
* The "--salts" option threshold is now applied before removal of previously
cracked hashes for consistent behavior with interrupted and continued sessions.
* The "Idle = Y" setting (which is the default) is now ignored for
OpenMP-enabled hash types when the actual number of threads is greater than 1.
(Unfortunately, it did not work right at least with GNU libgomp on Linux.)
* When a cracking session terminates or is interrupted, John will now warn the
user if the cracked passwords printed to the terminal while cracking are
potentially incomplete.  It will advise the user to use the "--show" option to
see the complete set of cracked passwords with proper post-processing.
* When loading hashes specified on a line on their own (feature introduced in
1.7.6), the loader will now ignore leading and trailing whitespace.
* Unless a hash type is forced from the command line, the loader will now print
warnings about additional hash types seen in the input files (beyond the hash
type autodetected initially).
* For use primarily by the jumbo patch (and later by future enhancements to the
official versions as well), the loader now includes logic to warn the user of
ambiguous hash encodings (e.g. LM vs. NTLM vs. raw-MD5, all of which may be
represented as 32 hexadecimal characters) and of excessive partial hash
collisions, which it works around (these are typically caused by an incomplete
implementation of a new hash type).
* The "unique" and "unshadow" programs have been made significantly faster.
* "DateTime", "Repeats", "Subsets", "AtLeast1-Simple", "AtLeast1-Generic", and
"Policy" external mode samples have been added to the default john.conf.
* The self-tests have been enhanced to detect more kinds of program bugs.
* A few minor bug fixes and enhancements were made.

The following changes have been made between John 1.7.6 and 1.7.6.1:

* Corrected a logic error introduced in JtR 1.7.4.2: in "single crack" mode,
we need a salt's key buffer even when we have no words corresponding to that
salt's hashes to base candidate passwords on.  We need this buffer to hold
other salts' successful guesses for testing against this salt's hashes.

The following changes have been made between John 1.7.5.1 and 1.7.6:

* Generic crypt(3) support (enabled with "--format=crypt") has been added for
auditing password hash types supported by the system but not yet supported by
John's own optimized cryptographic routines (such as SHA-crypt and SunMD5).
* Optional parallelization of the above has been implemented by means of OpenMP
along with glibc's crypt_r(3) or Solaris' MT-safe crypt(3C).
* Optional parallelization of John's own optimized code for the OpenBSD-style
Blowfish-based crypt(3) (bcrypt) hashes with OpenMP has been added.
* A more suitable version of 32-bit x86 assembly code for Blowfish is now
chosen on Core i7 and similar CPUs (when they happen to run a 32-bit build).
* More optimal DES S-box expressions for PowerPC with AltiVec (making use of
the conditional select operation) contributed by Dumplinger Boy (Dango-Chu)
have been integrated.
* The bitslice DES C source code has been reworked to allow for the use of
arbitrary SIMD intrinsics, which was previously only implemented for AltiVec
as a special case.
* Support for SSE2 and MMX intrinsics with bitslice DES (as an alternative to
the supplied assembly code) has been added (currently only enabled for SSE2 on
x86-64 when compiling with GCC 4.4+).
* Support for mixed-type longer virtual vectors (such as SSE2+MMX, SSE2+ALU,
AltiVec+ALU, and other combinations) with bitslice DES has been added (not
enabled by default yet, primarily intended for easy benchmarks on future CPUs,
with future compiler versions, with even more SIMD instruction sets, and with
different DES S-box expressions that might be available in the future).
* The obsolete 32-bit SPARC assembly implementation of DES has been dropped.
* The loader will now detect password hashes specified on a line on their own,
not only as part of an /etc/passwd or PWDUMP format file.
* When run in "--stdin" mode and reading candidate passwords from a terminal
(to be typed by the user), John will no longer mess with the terminal settings.
* John will now restore terminal settings not only on normal termination or
interrupt, but also when forcibly interrupted with two Ctrl-C keypresses.

The following changes have been made between John 1.7.5 and 1.7.5.1:

* A new numeric variable has been added to the word mangling rules engine:
"p" for position of the character last found with the "/" or "%" commands.

The following changes have been made between John 1.7.4.2 and 1.7.5:

* Support for the use of "--format" along with "--show" or "--make-charset" has
been added.
* The choice of .rec and .log filenames for custom session names has been made
more intuitive.
* Support for "\r" (character lists with repeats) and "\p0" (reference to the
immediately preceding character list/range) has been added to the word mangling
rules preprocessor.
* The undefined and undocumented behavior of some subtle word mangling rules
preprocessor constructs has been changed to arguably be more sensible.
* Some bugs were fixed, most notably JtR crashing on no password hashes loaded
(bug introduced in 1.7.4.2).

The following changes have been made between John 1.7.4 and 1.7.4.2:

* Major performance improvements for processing of very large password files
or sets of files, especially with salt-less or same-salt hashes, achieved
primarily through introduction of two additional hash table sizes (64K and 1M
entries), changes to the loader, and smarter processing of successful guesses
(to accommodate getting thousands of hashes successfully cracked per second).
* Many default buffer and hash table sizes have been increased and thresholds
for the use of hash tables lowered, meaning that John will now tend to use
more memory to achieve better speed (unless it is told not to with the
"--save-memory" option).
* Some previously missed common website passwords found on public lists of
"top N passwords" have been added to the bundled common passwords list.
* Some bugs introduced in 1.7.4 and affecting wordlist mode's elimination of
consecutive duplicate candidate passwords have been fixed.

The following changes have been made between John 1.7.3.4 and 1.7.4:

* Support for back-references and "parallel" ranges has been added to the
word mangling rules preprocessor.
* The notion of numeric variables (to be used for character positions
and substring lengths along with numeric constants supported previously)
has been introduced into the rules engine.  Two pre-defined variables
("l" for initial or updated word's length and "m" for initial or
memorized word's last character position) and 11 user-defined variables
("a" through "k") have been added.  Additionally, there's a new numeric
constant: "z" for "infinite" position or length.
* New rule commands have been added: "A" (append, insert, or prefix with a
string), "X" (extract a substring from memory and insert), "v" (subtract
and assign to a numeric variable).
* New rule reject flags have been added: ":" (no-op, for use along with the
"parallel" ranges feature of the preprocessor) and "p" (reject unless word
pair commands are allowed, for sharing of the same ruleset between "single
crack" and wordlist modes).
* Processing of word mangling rules has been made significantly faster in
multiple ways (caching of the current length, less copying of data, code
and data placement changes for better branch prediction and L1 cache usage,
compiler-friendly use of local variables, code micro-optimizations,
removal of no-op rule commands in an initial pass).
* The default rulesets for "single crack" and wordlist modes have been
revised to make use of the new features, for speed, to produce fewer
duplicates, and to attempt additional kinds of candidate passwords (such
as for years 2010 through 2019 with "year-based" rules).
* The idle priority emulation code has been optimized for lower overhead when
there appears to be no other demand for CPU time.
* The default for the Idle setting has been changed from N to Y.

The following changes have been made between John 1.7.3.1 and 1.7.3.4:

* "make check" has been implemented (for Unix-like systems only).
* The "--test" option will now take an optional argument - the duration of each
benchmark in seconds.
* Section .note.GNU-stack has been added to all assembly files to avoid the
stack area unnecessarily being made executable on Linux systems that use this
mechanism.
* Some very minor bugs that did not affect normal operation have been fixed.
* Some unimportant compiler warnings have been fixed, a source code comment has
been made more verbose and more complete.

The following changes have been made between John 1.7.3 and 1.7.3.1:

* Corrected the x86 assembly files for building on Mac OS X.
* Merged in some generic changes from JtR Pro.

The following changes have been made between John 1.7.2 and 1.7.3:

* Two Blowfish-based crypt(3) hashes may now be computed in parallel for much
better performance on modern multi-issue CPUs with a sufficient number of
registers (e.g., x86-64).
* Bitslice DES assembly code for x86-64 has been converted to use
instruction pointer relative addressing (needed for Mac OS X support).
* New make targets: macosx-universal, macosx-x86-64, solaris-x86-64-cc,
solaris-x86-64-gcc, solaris-x86-sse2-cc, solaris-x86-sse2-gcc,
solaris-x86-mmx-cc, solaris-x86-mmx-gcc, solaris-x86-any-cc, linux-ia64;
other changes to the Makefile.
* Minor bug fixes.
* "DumbForce" and "KnownForce" external mode samples have been added to the
default john.conf.

The following changes have been made between John 1.7.1 and 1.7.2:

* Bitslice DES assembly code for x86-64 making use of the 64-bit mode
extended SSE2 with 16 XMM registers has been added for better performance
at DES-based crypt(3) hashes with x86-64 builds on AMD processors.
* New make target for FreeBSD/x86-64.

The following changes have been made between John 1.7.0.2 and 1.7.1:

* Bitslice DES code for x86 with SSE2 has been added for better performance
at DES-based crypt(3) hashes on Pentium 4 and SSE2-capable AMD processors.
* Assorted high-level changes have been applied to improve performance
on current x86-64 processors.
* New make target for NetBSD/SPARC64.
* Minor source code cleanups.

The following changes have been made between John 1.7 and 1.7.0.2:

* Minor bug and portability fixes.
* Better handling of certain uncommon scenarios and improper uses of John.
* Bonus: "Keyboard" cracker included in the default john.conf (john.ini)
that will try sequences of adjacent keys on a keyboard as passwords.

The following changes have been made between John 1.6 and 1.7:

* Bitslice DES code for x86 with MMX: more than twice faster than older
non-bitslice MMX code.
* Bitsliced the LM hash code as well: now several times faster.
* Significant improvements to the generic bitslice DES code: +20% on RISC.
* PowerPC G4+ AltiVec support (Mac OS X and Linux): effective 128-bitness
for bitslice DES, resulting in huge speedups.
* First attempt at generic vectorization support for bitslice DES.
* Two MD5 hashes at a time for extra ILP on RISC: up to +80% on Alpha EV5+.
* Generic Blowfish x86 assembly code in addition to the original Pentium
version: +15% on the Pentium Pro family (up to and including Pentium III),
+20% on AMD K6 (Pentium 4 and newer AMD CPUs are more happy running the
original Pentium code for Blowfish).
* Verbose logging of events to the global or a session-specific log file.
* Better idle priority emulation with POSIX.1b (POSIX.4) scheduling calls.
* System-wide installation support for *BSD ports and Linux distributions.
* AIX, DU/Tru64 C2, HP-UX tcb files support in unshadow.
* New make targets for Linux/x86-64, Linux/PowerPC, FreeBSD/Alpha,
OpenBSD/x86-64, OpenBSD/Alpha, OpenBSD/SPARC, OpenBSD/SPARC64,
OpenBSD/PowerPC, OpenBSD/PA-RISC, OpenBSD/VAX, NetBSD/VAX, Solaris/SPARC64,
Mac OS X (PowerPC and x86), SCO, BeOS.
* Bug and portability fixes, and new bugs.
* Bonus: "Strip" cracker included in the default john.conf (john.ini).

$Owl: Owl/packages/john/john/doc/CHANGES,v 1.58 2011/06/22 13:39:46 solar Exp $