Add bin and edit workflow
Gitea Actions Demo / Explore-Gitea-Actions (push) Failing after 9s

This commit is contained in:
2026-09-16 13:11:16 -06:00
parent c8ac4fcae5
commit 4cee170d66
17576 changed files with 895740 additions and 2 deletions
+26
View File
@@ -0,0 +1,26 @@
# frozen_string_literal: true
require_relative 'lib/Ascii85/version'
Gem::Specification.new do |s|
s.name = 'Ascii85'
s.version = Ascii85::VERSION
s.platform = Gem::Platform::RUBY
s.author = 'Johannes Holzfuß'
s.email = 'johannes@holzfuss.name'
s.license = 'MIT'
s.homepage = 'https://github.com/DataWraith/ascii85gem/'
s.summary = 'Ascii85 encoder/decoder'
s.description = "Ascii85 provides methods to encode/decode Adobe's binary-to-text encoding of the same name."
s.required_ruby_version = '>= 2.7.0'
s.add_development_dependency 'minitest', '~> 5', '>= 5.12.0'
s.add_development_dependency 'rake', '~> 13'
s.files = `git ls-files`.split("\n") - ['.gitignore', '.github/workflows/ruby.yml']
s.test_files = `git ls-files -- spec/*`.split("\n")
s.executables = `git ls-files -- bin/*`.split("\n").map { |f| File.basename(f) }
s.require_paths = ['lib']
s.extra_rdoc_files = ['README.md', 'LICENSE']
end
+70
View File
@@ -0,0 +1,70 @@
# Ascii85 Changelog
## [2.0.1] - 2024-09-15
### Fixed
- Decoding binary data could lead to Encoding errors (Issue #8)
## [2.0.0] - 2024-08-20
### BREAKING CHANGES
- The minimum required Ruby version has been raised to 2.7.0.
### Added
- `Ascii85.decode_raw` method that doesn't expect the input to be wrapped in `<~` and `~>` delimiters.
- `Ascii85.extract` method to extract encoded text from between `<~` and `~>` for feeding to `#decode_raw`.
- Option to pass an IO-object as input to `#encode` and `#decode_raw` instead of a String.
- Option to pass an IO-object to `#encode` and `#decode_raw` for output. Output is written to the object instead of being returned as a String.
- Streaming capability for `#encode` and `#decode_raw` when both input and output are IO objects, using constant memory.
## [1.1.1] - 2024-05-09
### Fixed
- Make `bin/ascii85` Ruby 3.2-compatible (thanks @tylerwillingham)
- Slightly improved error handling of `bin/ascii85`
## [1.1.0] - 2020-11-11
### Added
- Make use of frozen_string_literal (thanks @aliismayilov)
### Changed
- Updated tests to use newer minitest syntax
## [1.0.3] - 2018-01-25
### Changed
- Updated the gem's metadata
## [1.0.2] - 2012-09-16
### Changed
- Changed test runner from RSpec to MiniSpec
- Added support for rubygems-test
- Minor changes to make packaging easier
## [1.0.1] - 2011-05-05
### Changed
- Removed `hoe` dependency in favor of `bundler`
- Minor corrections in the documentation
## [1.0.0] - 2009-12-25
### Added
- Ruby 1.9 compatibility
- Command-line en- and decoder
## [0.9.0] - 2009-02-17
- Initial release
+6
View File
@@ -0,0 +1,6 @@
# frozen_string_literal: true
source 'http://rubygems.org'
# Specify your gem's dependencies in Ascii85.gemspec
gemspec
+19
View File
@@ -0,0 +1,19 @@
Copyright (c) 2009 Johannes Holzfuß
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
of the Software, and to permit persons to whom the Software is furnished to do
so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+78
View File
@@ -0,0 +1,78 @@
**Status**: This project is feature-complete. With the exception of fixes to reported bugs, no further development will take place.
# Ascii85
## Description
Ascii85 is a Ruby gem that provides methods for encoding/decoding Adobe's
binary-to-text encoding of the same name.
See the Adobe PostScript Language Reference ([archived version][PLRM]) page 131
and [Wikipedia](https://en.wikipedia.org/wiki/Ascii85) for more information
about the format.
[PLRM]: https://web.archive.org/web/20161222092741/https://www.adobe.com/products/postscript/pdfs/PLRM.pdf
## Installation
`$ gem install Ascii85`
> [!IMPORTANT]
> Note that the gem name is capitalized.
## Usage
```ruby
require 'ascii85'
Ascii85.encode("Ruby")
=> "<~;KZGo~>"
Ascii85.decode("<~;KZGo~>")
=> "Ruby"
Ascii85.extract("Foo<~;KZGo~>Bar")
=> ";KZGo"
Ascii85.decode_raw(";KZGo")
=> "Ruby"
```
In addition, `Ascii85.encode` can take a second parameter that specifies the
length of the returned lines. The default is 80; use `false` for unlimited.
`Ascii85.decode` expects the input to be enclosed in `<~` and `~>` — it
ignores everything outside of these, while `Ascii85.decode_raw` assumes that
the entire String passed in is encoded in Ascii85. If you need to, you can use
`Ascii85.extract` to find and extract the first substring of the input that is
enclosed by the `<~` and `~>` delimiters.
The output of `Ascii85.decode` and `Ascii85.decode_raw` will be a String that
has the `ASCII-8BIT` encoding, so you may have to use `String#force_encoding` to
convert it to the desired encoding.
For further options, see the [Documentation](https://www.rubydoc.info/gems/Ascii85/).
## Command-line utility
This gem includes `ascii85`, a command-line utility modeled after `base64` from
the GNU coreutils. It can be used to encode/decode Ascii85 directly from the
command-line:
```
Usage: ascii85 [OPTIONS] [FILE]
Encodes or decodes FILE or STDIN using Ascii85 and writes to STDOUT.
-w, --wrap COLUMN Wrap lines at COLUMN. Default is 80, use 0 for no wrapping
-d, --decode Decode the input
-h, --help Display this help and exit
-V, --version Output version information
```
## License
Ascii85 is distributed under the MIT License. See the accompanying LICENSE file
for details.
+14
View File
@@ -0,0 +1,14 @@
# frozen_string_literal: true
require 'bundler'
Bundler::GemHelper.install_tasks
require 'rake/testtask'
Rake::TestTask.new do |t|
t.test_files = FileList['spec/**/*_spec.rb']
end
task specs: :test
task tests: :test
task default: :test
+112
View File
@@ -0,0 +1,112 @@
#!/usr/bin/env ruby
# frozen_string_literal: true
#
# A simple command-line tool to de- and encode Ascii85, modeled after `base64`
# from the GNU Coreutils.
#
require 'optparse'
require File.join(File.dirname(__FILE__), '..', 'lib', 'ascii85')
require File.join(File.dirname(__FILE__), '..', 'lib', 'Ascii85', 'version')
class CLI
attr_reader :options
def initialize(argv, stdin: $stdin, stdout: $stdout)
@in = stdin
@out = stdout
@options = {
wrap: 80,
action: :encode
}
parse_options(argv)
end
def parse_options(argv)
@parser = OptionParser.new do |opts|
opts.banner = "Usage: #{File.basename($PROGRAM_NAME)} [OPTIONS] [FILE]\n" \
'Encodes or decodes FILE or STDIN using Ascii85 and writes to STDOUT.'
opts.on('-w', '--wrap COLUMN', Integer,
'Wrap lines at COLUMN. Default is 80, use 0 for no wrapping') do |opt|
@options[:wrap] = opt.abs
@options[:wrap] = false if opt.zero?
end
opts.on('-d', '--decode', 'Decode the input') do
@options[:action] = :decode
end
opts.on('-h', '--help', 'Display this help and exit') do
@options[:action] = :help
end
opts.on('-V', '--version', 'Output version information') do |_opt|
@options[:action] = :version
end
end
remaining_args = @parser.parse!(argv)
case remaining_args.size
when 0
@options[:file] = '-'
when 1
@options[:file] = remaining_args.first
else
raise(OptionParser::ParseError, "Superfluous operand(s): \"#{remaining_args[1..].join('", "')}\"")
end
end
def input
fn = @options[:file]
return @in.binmode if fn == '-'
raise(StandardError, "File not found: \"#{fn}\"") unless File.exist?(fn)
raise(StandardError, "File is not readable: \"#{fn}\"") unless File.readable_real?(fn)
File.new(fn, 'rb')
end
def decode
Ascii85.decode(input.read, out: @out)
end
def encode
Ascii85.encode(input, @options[:wrap], out: @out)
end
def version
"Ascii85 v#{Ascii85::VERSION},\nwritten by Johannes Holzfuß"
end
def help
@parser
end
def call
case @options[:action]
when :help then @out.puts help
when :version then @out.puts version
when :encode then encode
when :decode then decode
end
end
end
if File.basename($PROGRAM_NAME) == "ascii85"
begin
CLI.new(ARGV).call
rescue OptionParser::ParseError => e
abort e.message
rescue Ascii85::DecodingError => e
abort "Decoding Error: #{e.message}"
rescue StandardError => e
abort "Error: #{e.message}"
end
end
@@ -0,0 +1,5 @@
# frozen_string_literal: true
module Ascii85
VERSION = '2.0.1'
end
+469
View File
@@ -0,0 +1,469 @@
# frozen_string_literal: true
require 'stringio'
#
# Ascii85 is an implementation of Adobe's binary-to-text encoding of the
# same name in pure Ruby.
#
# See http://en.wikipedia.org/wiki/Ascii85 for more information about the
# format.
#
# Author:: Johannes Holzfuß (johannes@holzfuss.name)
# License:: Distributed under the MIT License (see LICENSE file)
#
module Ascii85
class << self
EMPTY_STRING = ''.dup.force_encoding(Encoding::ASCII_8BIT)
START_MARKER = '<~'.dup.force_encoding(Encoding::ASCII_8BIT)
ENDING_MARKER = '~>'.dup.force_encoding(Encoding::ASCII_8BIT)
LINE_BREAK = "\n".dup.force_encoding(Encoding::ASCII_8BIT)
#
# Encodes the bytes of the given String or IO-like object as Ascii85.
#
# @param str_or_io [String, IO] The input to encode
# @param wrap_lines [Integer, false] The line length for wrapping, or +false+ for no wrapping
# @param out [IO, nil] An optional IO-like object to write the output to
#
# @return [String, IO] The encoded String or the output IO object that was passed in
#
# @example Encoding a simple String
# Ascii85.encode("Ruby")
# # => <~;KZGo~>
#
# @example Encoding with line wrapping
# Ascii85.encode("Supercalifragilisticexpialidocious", 15)
# # => <~;g!%jEarNoBkD
# # BoB5)0rF*),+AU&
# # 0.@;KXgDe!L"F`R
# # ~>
#
# @example Encoding without line wrapping
# Ascii85.encode("Supercalifragilisticexpialidocious", false)
# # => <~;g!%jEarNoBkDBoB5)0rF*),+AU&0.@;KXgDe!L"F`R~>
#
# @example Encoding from an IO-like object
# input = StringIO.new("Ruby")
# Ascii85.encode(input)
# # => "<~;KZGo~>"
#
# @example Encoding to an IO object
# output = StringIO.new
# Ascii85.encode("Ruby", out: output)
# # => output (with "<~;KZGo~>" written to it)
#
def encode(str_or_io, wrap_lines = 80, out: nil)
reader = if io_like?(str_or_io)
str_or_io
else
StringIO.new(str_or_io.to_s, 'rb')
end
return EMPTY_STRING.dup if reader.eof?
# Setup buffered Reader and Writers
bufreader = BufferedReader.new(reader, unencoded_chunk_size)
bufwriter = BufferedWriter.new(out || StringIO.new(String.new, 'wb'), encoded_chunk_size)
writer = wrap_lines ? Wrapper.new(bufwriter, wrap_lines) : DummyWrapper.new(bufwriter)
padding = unfrozen_binary_copy("\0\0\0\0")
tuplebuf = unfrozen_binary_copy('!!!!!')
exclamations = unfrozen_binary_copy('!!!!!')
z = unfrozen_binary_copy('z')
bufreader.each_chunk do |chunk|
chunk.unpack('N*').each do |word|
# Encode each big-endian 32-bit word into a 5-character tuple (except
# for 0, which encodes to 'z')
if word.zero?
writer.write(z)
else
word, b0 = word.divmod(85)
word, b1 = word.divmod(85)
word, b2 = word.divmod(85)
word, b3 = word.divmod(85)
b4 = word
tuplebuf.setbyte(0, b4 + 33)
tuplebuf.setbyte(1, b3 + 33)
tuplebuf.setbyte(2, b2 + 33)
tuplebuf.setbyte(3, b1 + 33)
tuplebuf.setbyte(4, b0 + 33)
writer.write(tuplebuf)
end
end
next if (chunk.bytesize & 0b11).zero?
# If we have leftover bytes, we need to zero-pad to a multiple of four
# before converting to a 32-bit word.
padding_length = (-chunk.bytesize) % 4
trailing = chunk[-(4 - padding_length)..]
word = (trailing + padding[0...padding_length]).unpack1('N')
# Encode the last word and cut off any padding
if word.zero?
writer.write(exclamations[0..(4 - padding_length)])
else
word, b0 = word.divmod(85)
word, b1 = word.divmod(85)
word, b2 = word.divmod(85)
word, b3 = word.divmod(85)
b4 = word
tuplebuf.setbyte(0, b4 + 33)
tuplebuf.setbyte(1, b3 + 33)
tuplebuf.setbyte(2, b2 + 33)
tuplebuf.setbyte(3, b1 + 33)
tuplebuf.setbyte(4, b0 + 33)
writer.write(tuplebuf[0..(4 - padding_length)])
end
end
# If no output IO-object was provided, extract the encoded String from the
# default StringIO writer. We force the encoding to 'ASCII-8BIT' to work
# around a TruffleRuby bug.
return writer.finish.io.string.force_encoding(Encoding::ASCII_8BIT) if out.nil?
# Otherwise we make sure to flush the output writer, and then return it.
writer.finish.io
end
# Searches through a String and extracts the first substring enclosed by '<~' and '~>'.
#
# @param str [String] The String to search through
#
# @return [String] The extracted substring, or an empty String if no valid delimiters are found
#
# @example Extracting Ascii85 content
# Ascii85.extract("Foo<~;KZGo~>Bar<~z~>Baz")
# # => ";KZGo"
#
# @example When no delimiters are found
# Ascii85.extract("No delimiters")
# # => ""
#
# @note This method only accepts a String, not an IO-like object, as the entire input
# needs to be available to ensure validity.
#
def extract(str)
input = str.to_s
# Make sure the delimiter Strings have the correct encoding.
opening_delim = '<~'.encode(input.encoding)
closing_delim = '~>'.encode(input.encoding)
# Get the positions of the opening/closing delimiters. If there is no pair
# of opening/closing delimiters, return an unfrozen empty String.
(start_pos = input.index(opening_delim)) or return EMPTY_STRING.dup
(end_pos = input.index(closing_delim, start_pos + 2)) or return EMPTY_STRING.dup
# Get the String inside the delimiter-pair
input[(start_pos + 2)...end_pos]
end
#
# Searches through a String and decodes the first substring enclosed by '<~' and '~>'.
#
# @param str [String] The String containing Ascii85-encoded content
# @param out [IO, nil] An optional IO-like object to write the output to
#
# @return [String, IO] The decoded String (in ASCII-8BIT encoding) or the output IO object (if it was provided)
#
# @raise [Ascii85::DecodingError] When malformed input is encountered
#
# @example Decoding Ascii85 content
# Ascii85.decode("<~;KZGo~>")
# # => "Ruby"
#
# @example Decoding with multiple Ascii85 blocks present (ignores all but the first)
# Ascii85.decode("Foo<~;KZGo~>Bar<~87cURDZ~>Baz")
# # => "Ruby"
#
# @example When no delimiters are found
# Ascii85.decode("No delimiters")
# # => ""
#
# @example Decoding to an IO object
# output = StringIO.new
# Ascii85.decode("<~;KZGo~>", out: output)
# # => output (with "Ruby" written to it)
#
# @note This method only accepts a String, not an IO-like object, as the entire input
# needs to be available to ensure validity.
#
def decode(str, out: nil)
decode_raw(extract(str), out: out)
end
#
# Decodes the given raw Ascii85-encoded String or IO-like object.
#
# @param str_or_io [String, IO] The Ascii85-encoded input to decode
# @param out [IO, nil] An optional IO-like object to write the output to
#
# @return [String, IO] The decoded String (in ASCII-8BIT encoding) or the output IO object (if it was provided)
#
# @raise [Ascii85::DecodingError] When malformed input is encountered
#
# @example Decoding a raw Ascii85 String
# Ascii85.decode_raw(";KZGo")
# # => "Ruby"
#
# @example Decoding from an IO-like object
# input = StringIO.new(";KZGo")
# Ascii85.decode_raw(input)
# # => "Ruby"
#
# @example Decoding to an IO object
# output = StringIO.new
# Ascii85.decode_raw(";KZGo", out: output)
# # => output (with "Ruby" written to it)
#
# @note The input must not be enclosed in '<~' and '~>' delimiters.
#
def decode_raw(str_or_io, out: nil)
reader = if io_like?(str_or_io)
str_or_io
else
StringIO.new(str_or_io.to_s, 'rb')
end
# Return an unfrozen String on empty input
return EMPTY_STRING.dup if reader.eof?
# Setup buffered Reader and Writers
bufreader = BufferedReader.new(reader, encoded_chunk_size)
bufwriter = BufferedWriter.new(out || StringIO.new(String.new, 'wb'), unencoded_chunk_size)
# Populate the lookup table (caches the exponentiation)
lut = (0..4).map { |count| 85**(4 - count) }
# Decode
word = 0
count = 0
zeroes = unfrozen_binary_copy("\0\0\0\0")
wordbuf = zeroes.dup
bufreader.each_chunk do |chunk|
chunk.each_byte do |c|
case c.chr
when ' ', "\t", "\r", "\n", "\f", "\0"
# Ignore whitespace
next
when 'z'
raise(Ascii85::DecodingError, "Found 'z' inside Ascii85 5-tuple") unless count.zero?
# Expand z to 0-word
bufwriter.write(zeroes)
when '!'..'u'
# Decode 5 characters into a 4-byte word
word += (c - 33) * lut[count]
count += 1
if count == 5 && word > 0xffffffff
raise(Ascii85::DecodingError, "Invalid Ascii85 5-tuple (#{word} >= 2**32)")
elsif count == 5
b3 = word & 0xff; word >>= 8
b2 = word & 0xff; word >>= 8
b1 = word & 0xff; word >>= 8
b0 = word
wordbuf.setbyte(0, b0)
wordbuf.setbyte(1, b1)
wordbuf.setbyte(2, b2)
wordbuf.setbyte(3, b3)
bufwriter.write(wordbuf)
word = 0
count = 0
end
else
raise(Ascii85::DecodingError, "Illegal character inside Ascii85: #{c.chr.dump}")
end
end
end
# We're done if all 5-tuples have been consumed
if count.zero?
bufwriter.flush
return out || bufwriter.io.string.force_encoding(Encoding::ASCII_8BIT)
end
raise(Ascii85::DecodingError, 'Last 5-tuple consists of single character') if count == 1
# Finish last, partially decoded 32-bit word
count -= 1
word += lut[count]
bufwriter.write((word >> 24).chr) if count >= 1
bufwriter.write(((word >> 16) & 0xff).chr) if count >= 2
bufwriter.write(((word >> 8) & 0xff).chr) if count == 3
bufwriter.flush
out || bufwriter.io.string.force_encoding(Encoding::ASCII_8BIT)
end
private
# Copies the given String and forces the encoding of the returned copy to
# be Encoding::ASCII_8BIT.
def unfrozen_binary_copy(str)
str.dup.force_encoding(Encoding::ASCII_8BIT)
end
# Buffers an underlying IO object to increase efficiency. You do not need
# to use this directly.
#
# @private
#
class BufferedReader
def initialize(io, buffer_size)
@io = io
@buffer_size = buffer_size
end
def each_chunk
return enum_for(:each_chunk) unless block_given?
until @io.eof?
chunk = @io.read(@buffer_size)
yield chunk if chunk
end
end
end
# Buffers an underlying IO object to increase efficiency. You do not need
# to use this directly.
#
# @private
#
class BufferedWriter
attr_accessor :io
def initialize(io, buffer_size)
@io = io
@buffer_size = buffer_size
@buffer = String.new(capacity: buffer_size, encoding: Encoding::ASCII_8BIT)
end
def write(tuple)
flush if @buffer.bytesize + tuple.bytesize > @buffer_size
@buffer << tuple
end
def flush
@io.write(@buffer)
@buffer.clear
end
end
# Wraps the input in '<~' and '~>' delimiters and passes it through
# unmodified to the underlying IO object otherwise. You do not need to
# use this directly.
#
# @private
#
class DummyWrapper
def initialize(out)
@out = out
@out.write(START_MARKER)
end
def write(buffer)
@out.write(buffer)
end
def finish
@out.write(ENDING_MARKER)
@out.flush
@out
end
end
# Wraps the input in '<~' and '~>' delimiters and ensures that no line is
# longer than the specified length. You do not need to use this directly.
#
# @private
#
class Wrapper
def initialize(out, wrap_lines)
@line_length = [2, wrap_lines.to_i].max
@out = out
@out.write(START_MARKER)
@cur_len = 2
end
def write(buffer)
loop do
s = buffer.bytesize
if @cur_len + s < @line_length
@out.write(buffer)
@cur_len += s
return
end
remaining = @line_length - @cur_len
@out.write(buffer[0...remaining])
@out.write(LINE_BREAK)
@cur_len = 0
buffer = buffer[remaining..]
return if buffer.empty?
end
end
def finish
# Add the closing delimiter (may need to be pushed to the next line)
@out.write(LINE_BREAK) if @cur_len + 2 > @line_length
@out.write(ENDING_MARKER)
@out.flush
@out
end
end
# Check if an object is IO-like
#
# @private
#
def io_like?(obj)
obj.respond_to?(:read) &&
obj.respond_to?(:eof?)
end
# @return [Integer] Buffer size for to-be-encoded input
#
def unencoded_chunk_size
4 * 2048
end
# @return [Integer] Buffer size for encoded output
#
def encoded_chunk_size
5 * 2048
end
end
#
# Error raised when Ascii85 encounters problems while decoding the input.
#
# This error is raised for the following issues:
# * An invalid character (valid characters are '!'..'u' and 'z')
# * A 'z' character inside a 5-tuple ('z' is only valid on its own)
# * An invalid 5-tuple that decodes to >= 2**32
# * The last tuple consisting of a single character. Valid tuples always have
# at least two characters.
#
class DecodingError < StandardError; end
end
+211
View File
@@ -0,0 +1,211 @@
# frozen_string_literal: true
require 'stringio'
require 'tempfile'
require 'minitest/autorun'
# We can't require the executable file because it doesn't
# have the '.rb' extension, so we have to load it.
unless defined?(CLI)
load File.join(__dir__, '..','..', 'bin', 'ascii85')
end
describe 'CLI' do
it 'should recognize the -h and --help options' do
[%w[-h], %w[--help]].each do |args|
cli = CLI.new(args)
assert_equal :help, cli.options[:action]
end
end
it 'should recognize the -V and --version options' do
[%w[-V], %w[--version]].each do |args|
cli = CLI.new(args)
assert_equal :version, cli.options[:action]
end
end
it 'should complain about superfluous arguments' do
assert_raises(OptionParser::ParseError) do
CLI.new(%w[foo bar])
end
end
describe 'wrap' do
it 'should default to wrapping at 80 characters' do
cli = CLI.new([])
assert_equal 80, cli.options[:wrap]
end
it 'should recognize the -w and --wrap options' do
[%w[-w 17], %w[--wrap 17]].each do |args|
cli = CLI.new(args)
assert_equal 17, cli.options[:wrap]
end
end
it 'should recognize the no-wrapping setting' do
cli = CLI.new(%w[-w 0])
assert_equal false, cli.options[:wrap]
end
it 'should raise an error if the wrap option is not an integer' do
assert_raises(OptionParser::ParseError) do
CLI.new(%w[-w foo])
end
end
end
describe 'encoding' do
it 'should encode from STDIN' do
stdin = StringIO.new('Ruby')
stdout = StringIO.new
CLI.new([], stdin: stdin, stdout: stdout).call
assert_equal '<~;KZGo~>', stdout.string
end
it 'should accept "-" as a file name' do
stdin = StringIO.new('Ruby')
stdout = StringIO.new
CLI.new(['-'], stdin: stdin, stdout: stdout).call
assert_equal '<~;KZGo~>', stdout.string
end
it 'should encode a file' do
begin
f = Tempfile.create('ascii85_encode')
f.write('Ruby')
f.close
stdout = StringIO.new
CLI.new([f.path], stdout: stdout).call
assert_equal '<~;KZGo~>', stdout.string
ensure
File.unlink(f.path)
end
end
it 'should wrap lines' do
begin
f = Tempfile.create('ascii85_wrap')
f.write('a' * 20)
f.close
stdout = StringIO.new
CLI.new([f.path, '-w2'], stdout: stdout).call
assert stdout.string.lines.all? { |l| l.chomp.length <= 2 }
ensure
File.unlink(f.path)
end
end
it 'should fail when the input file is not found' do
assert_raises(StandardError) do
CLI.new(['./foo/bar/baz']).call
end
end
it 'should fail when the input file is not readable' do
begin
f = Tempfile.create('ascii85_encode')
f.chmod(0o000)
assert_raises(StandardError) do
CLI.new([f.path]).call
end
ensure
File.unlink(f.path)
end
end
end
describe 'decoding' do
it 'should decode from STDIN' do
stdin = StringIO.new('<~;KZGo~>')
stdout = StringIO.new
CLI.new(['-d'], stdin: stdin, stdout: stdout).call
assert_equal 'Ruby', stdout.string
end
it 'should accept "-" as a file name' do
stdin = StringIO.new('<~;KZGo~>')
stdout = StringIO.new
CLI.new(['-d','-'], stdin: stdin, stdout: stdout).call
assert_equal 'Ruby', stdout.string
end
it 'should decode a file' do
begin
f = Tempfile.create('ascii85_decode')
f.write('<~;KZGo~>')
f.close
stdout = StringIO.new
CLI.new(['-d', f.path], stdout: stdout).call
assert_equal 'Ruby', stdout.string
ensure
File.unlink(f.path)
end
end
it 'should fail when the input file is not found' do
assert_raises(StandardError) do
CLI.new(['-d', './foo/bar/baz']).call
end
end
it 'should fail when the input file is not readable' do
begin
f = Tempfile.create('ascii85_decode')
f.chmod(0o000)
assert_raises(StandardError) do
CLI.new(['-d', f.path]).call
end
ensure
File.unlink(f.path)
end
end
describe 'invalid input' do
it 'should return the empty string when the input does not have delimiters' do
stdin = StringIO.new('No delimiters')
stdout = StringIO.new
CLI.new(['-d'], stdin: stdin, stdout: stdout).call
assert_equal '', stdout.string
end
ERROR_CASES = [
'<~!!y!!~>',
'<~!!z!!~>',
'<~s8W-#~>',
'<~!~>',
]
it 'should raise an error when invalid input is encountered' do
ERROR_CASES.each do |input|
stdin = StringIO.new(input)
stdout = StringIO.new
assert_raises(Ascii85::DecodingError) do
CLI.new(['-d'], stdin: stdin, stdout: stdout).call
end
end
end
end
end
end
@@ -0,0 +1,263 @@
# frozen_string_literal: true
require 'minitest/autorun'
require 'stringio'
# Require implementation
require File.expand_path('../../lib/ascii85', __dir__)
TEST_CASES = {
'' => '',
' ' => '<~+9~>',
"\0" * 1 => '<~!!~>',
"\0" * 2 => '<~!!!~>',
"\0" * 3 => '<~!!!!~>',
"\0" * 4 => '<~z~>',
"\0" * 5 => '<~z!!~>',
"A\0\0\0\0" => '<~5l^lb!!~>', # No z-abbreviation!
'A' => '<~5l~>',
'AB' => '<~5sb~>',
'ABC' => '<~5sdp~>',
'ABCD' => '<~5sdq,~>',
'ABCDE' => '<~5sdq,70~>',
'ABCDEF' => '<~5sdq,77I~>',
'ABCDEFG' => '<~5sdq,77Kc~>',
'ABCDEFGH' => '<~5sdq,77Kd<~>',
'ABCDEFGHI' => '<~5sdq,77Kd<8H~>',
'Ascii85' => '<~6$$OMBfIs~>',
'Antidisestablishmentarianism' => '<~6#LdYA8-*rF*(i"Ch[s(D.RU,@<-\'jDJ=0/~>',
# Dōmo arigatō, Mr. Roboto (according to Wikipedia)
'どうもありがとうミスターロボット' =>
'<~j+42iJVN3:K&_E6j+<0KJW/W?W8iG`j+EuaK"9on^Z0sZj+FJoK:LtSKB%T?~>',
[Math::PI].pack('G') => '<~5RAV2<(&;T~>',
[Math::E].pack('G') => '<~5R"n0M\\K6,~>',
# Minified example from Github issue 8.
# Note that OT and OU as the trailing characters are equivalent.
"\x9B\xB6\xB9+\x91" => '<~S$ojXOT~>'
}.freeze
describe Ascii85 do
it '#decode should be the inverse of #encode' do
# Generate a test string that contains all possible bytes
test_str = String.new
(0..255).each do |c|
test_str << c.chr
end
encoded = Ascii85.encode(test_str)
decoded = Ascii85.decode(encoded)
assert_equal test_str, decoded
end
describe '#encode' do
it 'should encode all specified test-cases correctly' do
TEST_CASES.each_pair do |input, encoded|
assert_equal encoded, Ascii85.encode(input)
end
end
it 'should always return unfrozen Strings' do
TEST_CASES.each_pair do |input, encoded|
assert_equal false, Ascii85.encode(input).frozen?
end
end
it 'should encode Strings in different encodings correctly' do
input_euc_jp = 'どうもありがとうミスターロボット'.encode('EUC-JP')
input_binary = input_euc_jp.force_encoding('ASCII-8BIT')
assert_equal Ascii85.encode(input_binary), Ascii85.encode(input_euc_jp)
end
it 'should produce output lines no longer than specified' do
test_str = '0123456789' * 30
#
# No wrap
#
assert_equal 0, Ascii85.encode(test_str, false).count("\n")
#
# x characters per line, except for the last one
#
(2..12).each do |x|
encoded = Ascii85.encode(test_str, x)
# Determine the length of all lines
count_arr = []
encoded.each_line do |line|
count_arr << line.chomp.length
end
# The last line is allowed to be shorter than x, so remove it
count_arr.pop if count_arr.last <= x
# If the end-marker is on a line of its own, the next-to-last line is
# allowed to be shorter than specified by exactly one character
count_arr.pop if (encoded[-3].chr =~ /[\r\n]/) && (count_arr.last == x - 1)
# Remove all line-lengths that are of length x from count_arr
count_arr.delete_if { |len| len == x }
# Now count_arr should be empty
assert_empty count_arr
end
end
it 'should not split the end-marker to achieve correct line length' do
assert_equal "<~z\n~>", Ascii85.encode("\0" * 4, 4)
end
it 'should encode to an IO object when provided' do
output = StringIO.new
result = Ascii85.encode('Ruby', out: output)
assert_equal output, result
assert_equal '<~;KZGo~>', output.string
end
it 'should encode from an IO object' do
input = StringIO.new('Ruby')
result = Ascii85.encode(input)
assert_equal '<~;KZGo~>', result
end
end
describe '#extract' do
it 'should extract data within delimiters only' do
assert_empty Ascii85.extract('<~~>')
assert_empty Ascii85.extract("Doesn't contain delimiters")
assert_empty Ascii85.extract('Mismatched ~> delimiters 1')
assert_empty Ascii85.extract('Mismatched <~ delimiters 2')
assert_empty Ascii85.extract('Mismatched ~><~ delimiters 3')
assert_equal ';KZGo', Ascii85.extract('<~;KZGo~><~z~>')
assert_equal 'z', Ascii85.extract('FooBar<~z~>BazQux')
end
end
describe '#decode' do
it 'should decode all specified test-cases correctly' do
TEST_CASES.each_pair do |decoded, input|
assert_equal decoded.dup.force_encoding('ASCII-8BIT'), Ascii85.decode(input)
end
end
it 'should always return unfrozen Strings' do
TEST_CASES.each_pair do |input, encoded|
assert_equal false, Ascii85.decode(encoded).frozen?
end
end
it 'should accept valid input in encodings other than the default' do
input = 'Ragnarök τέχνη русский язык I ♥ Ruby'
input_ascii85 = Ascii85.encode(input)
# Try to encode input_ascii85 in all possible encodings and see if we
# do the right thing in #decode.
Encoding.list.each do |encoding|
next if encoding.dummy?
next unless encoding.ascii_compatible?
# CP949 is a Microsoft Codepage for Korean, which apparently does not
# include a backslash, even though #ascii_compatible? returns true. This
# leads to an Ascii85::DecodingError, so we simply skip the encoding.
next if encoding.name == 'CP949'
begin
to_test = input_ascii85.encode(encoding)
assert_equal input, Ascii85.decode(to_test).force_encoding('UTF-8')
rescue Encoding::ConverterNotFoundError
# Ignore this encoding
end
end
end
it 'should only process data within delimiters' do
assert_empty Ascii85.decode('<~~>')
assert_empty Ascii85.decode("Doesn't contain delimiters")
assert_empty Ascii85.decode('Mismatched ~> delimiters 1')
assert_empty Ascii85.decode('Mismatched <~ delimiters 2')
assert_empty Ascii85.decode('Mismatched ~><~ delimiters 3')
assert_equal 'Ruby', Ascii85.decode('<~;KZGo~><~z~>')
assert_equal "\0\0\0\0", Ascii85.decode('FooBar<~z~>BazQux')
end
it 'should ignore whitespace' do
decoded = Ascii85.decode("<~6 #LdYA\r\08\n \n\n- *rF*(i\"Ch[s \t(D.RU,@ <-\'jDJ=0\f/~>")
assert_equal 'Antidisestablishmentarianism', decoded
end
it 'should return ASCII-8BIT encoded strings' do
assert_equal 'ASCII-8BIT', Ascii85.decode('<~;KZGo~>').encoding.name
end
it 'should decode to an IO object when provided' do
output = StringIO.new
result = Ascii85.decode('<~;KZGo~>', out: output)
assert_equal output, result
assert_equal 'Ruby', output.string
end
describe 'Error conditions' do
it 'should raise DecodingError if it encounters a word >= 2**32' do
assert_raises(Ascii85::DecodingError) { Ascii85.decode('<~s8W-#~>') }
end
it 'should raise DecodingError if it encounters an invalid character' do
assert_raises(Ascii85::DecodingError) { Ascii85.decode('<~!!y!!~>') }
end
it 'should raise DecodingError if the last tuple consists of a single character' do
assert_raises(Ascii85::DecodingError) { Ascii85.decode('<~!~>') }
end
it 'should raise DecodingError if a z is found inside a 5-tuple' do
assert_raises(Ascii85::DecodingError) { Ascii85.decode('<~!!z!!~>') }
end
end
end
describe '#decode_raw' do
it 'should decode raw Ascii85 without delimiters' do
TEST_CASES.each_pair do |decoded, input|
raw_input = input[2...-2] # Remove '<~' and '~>'
assert_equal decoded.dup.force_encoding('ASCII-8BIT'), Ascii85.decode_raw(raw_input)
end
end
it 'should always return unfrozen Strings' do
TEST_CASES.each_pair do |decoded, input|
raw_input = input[2...-2] # Remove '<~' and '~>'
assert_equal false, Ascii85.decode_raw(raw_input).frozen?
end
end
it 'should decode from an IO object' do
input = StringIO.new(';KZGo')
result = Ascii85.decode_raw(input)
assert_equal 'Ruby', result
end
it 'should decode to an IO object when provided' do
output = StringIO.new
result = Ascii85.decode_raw(';KZGo', out: output)
assert_equal output, result
assert_equal 'Ruby', output.string
end
it 'should raise DecodingError for invalid input' do
assert_raises(Ascii85::DecodingError) { Ascii85.decode_raw('s8W-#') }
assert_raises(Ascii85::DecodingError) { Ascii85.decode_raw('!!y!!') }
assert_raises(Ascii85::DecodingError) { Ascii85.decode_raw('!') }
assert_raises(Ascii85::DecodingError) { Ascii85.decode_raw('!!z!!') }
end
end
end
+301
View File
@@ -0,0 +1,301 @@
# Addressable 2.8.7 <a name="v2.8.7">
- Allow `public_suffix` 6 ([#535])
[#535]: https://github.com/sporkmonger/addressable/pull/535
# Addressable 2.8.6 <a name="v2.8.6">
- Memoize regexps for common character classes ([#524])
[#524]: https://github.com/sporkmonger/addressable/pull/524
# Addressable 2.8.5 <a name="v2.8.5">
- Fix thread safety issue with encoding tables ([#515])
- Define URI::NONE as a module to avoid serialization issues ([#509])
- Fix YAML serialization ([#508])
[#508]: https://github.com/sporkmonger/addressable/pull/508
[#509]: https://github.com/sporkmonger/addressable/pull/509
[#515]: https://github.com/sporkmonger/addressable/pull/515
# Addressable 2.8.4 <a name="v2.8.4">
- Restore `Addressable::IDNA.unicode_normalize_kc` as a deprecated method ([#504])
[#504]: https://github.com/sporkmonger/addressable/pull/504
# Addressable 2.8.3 <a name="v2.8.3">
- Fix template expand level 2 hash support for non-string objects ([#499], [#498])
[#499]: https://github.com/sporkmonger/addressable/pull/499
[#498]: https://github.com/sporkmonger/addressable/pull/498
# Addressable 2.8.2 <a name="v2.8.2">
- Improve cache hits and JIT friendliness ([#486](https://github.com/sporkmonger/addressable/pull/486))
- Improve code style and test coverage ([#482](https://github.com/sporkmonger/addressable/pull/482))
- Ensure reset of deferred validation ([#481](https://github.com/sporkmonger/addressable/pull/481))
- Resolve normalization differences between `IDNA::Native` and `IDNA::Pure` ([#408](https://github.com/sporkmonger/addressable/issues/408), [#492])
- Remove redundant colon in `Addressable::URI::CharacterClasses::AUTHORITY` regex ([#438](https://github.com/sporkmonger/addressable/pull/438)) (accidentally reverted by [#449] merge but [added back](https://github.com/sporkmonger/addressable/pull/492#discussion_r1105125280) in [#492])
[#492]: https://github.com/sporkmonger/addressable/pull/492
# Addressable 2.8.1 <a name="v2.8.1">
- refactor `Addressable::URI.normalize_path` to address linter offenses ([#430](https://github.com/sporkmonger/addressable/pull/430))
- update gemspec to reflect supported Ruby versions ([#466], [#464], [#463])
- compatibility w/ public_suffix 5.x ([#466], [#465], [#460])
- fixes "invalid byte sequence in UTF-8" exception when unencoding URLs containing non UTF-8 characters ([#459](https://github.com/sporkmonger/addressable/pull/459))
- `Ractor` compatibility ([#449])
- use the whole string instead of a single line for template match ([#431](https://github.com/sporkmonger/addressable/pull/431))
- force UTF-8 encoding only if needed ([#341](https://github.com/sporkmonger/addressable/pull/341))
[#449]: https://github.com/sporkmonger/addressable/pull/449
[#460]: https://github.com/sporkmonger/addressable/pull/460
[#463]: https://github.com/sporkmonger/addressable/pull/463
[#464]: https://github.com/sporkmonger/addressable/pull/464
[#465]: https://github.com/sporkmonger/addressable/pull/465
[#466]: https://github.com/sporkmonger/addressable/pull/466
# Addressable 2.8.0 <a name="v2.8.0">
- fixes ReDoS vulnerability in Addressable::Template#match
- no longer replaces `+` with spaces in queries for non-http(s) schemes
- fixed encoding ipv6 literals
- the `:compacted` flag for `normalized_query` now dedupes parameters
- fix broken `escape_component` alias
- dropping support for Ruby 2.0 and 2.1
- adding Ruby 3.0 compatibility for development tasks
- drop support for `rack-mount` and remove Addressable::Template#generate
- performance improvements
- switch CI/CD to GitHub Actions
# Addressable 2.7.0 <a name="v2.7.0">
- added `:compacted` flag to `normalized_query`
- `heuristic_parse` handles `mailto:` more intuitively
- dropped explicit support for JRuby 9.0.5.0
- compatibility w/ public_suffix 4.x
- performance improvements
# Addressable 2.6.0 <a name="v2.6.0">
- added `tld=` method to allow assignment to the public suffix
- most `heuristic_parse` patterns are now case-insensitive
- `heuristic_parse` handles more `file://` URI variations
- fixes bug in `heuristic_parse` when uri starts with digit
- fixes bug in `request_uri=` with query strings
- fixes template issues with `nil` and `?` operator
- `frozen_string_literal` pragmas added
- minor performance improvements in regexps
- fixes to eliminate warnings
# Addressable 2.5.2 <a name="v2.5.2">
- better support for frozen string literals
- fixed bug w/ uppercase characters in scheme
- IDNA errors w/ emoji URLs
- compatibility w/ public_suffix 3.x
# Addressable 2.5.1 <a name="v2.5.1">
- allow unicode normalization to be disabled for URI Template expansion
- removed duplicate test
# Addressable 2.5.0 <a name="v2.5.0">
- dropping support for Ruby 1.9
- adding support for Ruby 2.4 preview
- add support for public suffixes and tld; first runtime dependency
- hostname escaping should match RFC; underscores in hostnames no longer escaped
- paths beginning with // and missing an authority are now considered invalid
- validation now also takes place after setting a path
- handle backslashes in authority more like a browser for `heuristic_parse`
- unescaped backslashes in host now raise an `InvalidURIError`
- `merge!`, `join!`, `omit!` and `normalize!` don't disable deferred validation
- `heuristic_parse` now trims whitespace before parsing
- host parts longer than 63 bytes will be ignored and not passed to libidn
- normalized values always encoded as UTF-8
# Addressable 2.4.0 <a name="v2.4.0">
- support for 1.8.x dropped
- double quotes in a host now raises an error
- newlines in host will no longer get unescaped during normalization
- stricter handling of bogus scheme values
- stricter handling of encoded port values
- calling `require 'addressable'` will now load both the URI and Template files
- assigning to the `hostname` component with an `IPAddr` object is now supported
- assigning to the `origin` component is now supported
- fixed minor bug where an exception would be thrown for a missing ACE suffix
- better partial expansion of URI templates
# Addressable 2.3.8 <a name="v2.3.8">
- fix warnings
- update dependency gems
- support for 1.8.x officially deprecated
# Addressable 2.3.7 <a name="v2.3.7">
- fix scenario in which invalid URIs don't get an exception until inspected
- handle hostnames with two adjacent periods correctly
- upgrade of RSpec
# Addressable 2.3.6 <a name="v2.3.6">
- normalization drops empty query string
- better handling in template extract for missing values
- template modifier for `'?'` now treated as optional
- fixed issue where character class parameters were modified
- templates can now be tested for equality
- added `:sorted` option to normalization of query strings
- fixed issue with normalization of hosts given in `'example.com.'` form
# Addressable 2.3.5 <a name="v2.3.5">
- added Addressable::URI#empty? method
- Addressable::URI#hostname methods now strip square brackets from IPv6 hosts
- compatibility with Net::HTTP in Ruby 2.0.0
- Addressable::URI#route_from should always give relative URIs
# Addressable 2.3.4 <a name="v2.3.4">
- fixed issue with encoding altering its inputs
- query string normalization now leaves ';' characters alone
- FakeFS is detected before attempting to load unicode tables
- additional testing to ensure frozen objects don't cause problems
# Addressable 2.3.3 <a name="v2.3.3">
- fixed issue with converting common primitives during template expansion
- fixed port encoding issue
- removed a few warnings
- normalize should now ignore %2B in query strings
- the IDNA logic should now be handled by libidn in Ruby 1.9
- no template match should now result in nil instead of an empty MatchData
- added license information to gemspec
# Addressable 2.3.2 <a name="v2.3.2">
- added Addressable::URI#default_port method
- fixed issue with Marshalling Unicode data on Windows
- improved heuristic parsing to better handle IPv4 addresses
# Addressable 2.3.1 <a name="v2.3.1">
- fixed missing unicode data file
# Addressable 2.3.0 <a name="v2.3.0">
- updated Addressable::Template to use RFC 6570, level 4
- fixed compatibility problems with some versions of Ruby
- moved unicode tables into a data file for performance reasons
- removing support for multiple query value notations
# Addressable 2.2.8 <a name="v2.2.8">
- fixed issues with dot segment removal code
- form encoding can now handle multiple values per key
- updated development environment
# Addressable 2.2.7 <a name="v2.2.7">
- fixed issues related to Addressable::URI#query_values=
- the Addressable::URI.parse method is now polymorphic
# Addressable 2.2.6 <a name="v2.2.6">
- changed the way ambiguous paths are handled
- fixed bug with frozen URIs
- https supported in heuristic parsing
# Addressable 2.2.5 <a name="v2.2.5">
- 'parsing' a pre-parsed URI object is now a dup operation
- introduced conditional support for libidn
- fixed normalization issue on ampersands in query strings
- added additional tests around handling of query strings
# Addressable 2.2.4 <a name="v2.2.4">
- added origin support from draft-ietf-websec-origin-00
- resolved issue with attempting to navigate below root
- fixed bug with string splitting in query strings
# Addressable 2.2.3 <a name="v2.2.3">
- added :flat_array notation for query strings
# Addressable 2.2.2 <a name="v2.2.2">
- fixed issue with percent escaping of '+' character in query strings
# Addressable 2.2.1 <a name="v2.2.1">
- added support for application/x-www-form-urlencoded.
# Addressable 2.2.0 <a name="v2.2.0">
- added site methods
- improved documentation
# Addressable 2.1.2 <a name="v2.1.2">
- added HTTP request URI methods
- better handling of Windows file paths
- validation_deferred boolean replaced with defer_validation block
- normalization of percent-encoded paths should now be correct
- fixed issue with constructing URIs with relative paths
- fixed warnings
# Addressable 2.1.1 <a name="v2.1.1">
- more type checking changes
- fixed issue with unicode normalization
- added method to find template defaults
- symbolic keys are now allowed in template mappings
- numeric values and symbolic values are now allowed in template mappings
# Addressable 2.1.0 <a name="v2.1.0">
- refactored URI template support out into its own class
- removed extract method due to being useless and unreliable
- removed Addressable::URI.expand_template
- removed Addressable::URI#extract_mapping
- added partial template expansion
- fixed minor bugs in the parse and heuristic_parse methods
- fixed incompatibility with Ruby 1.9.1
- fixed bottleneck in Addressable::URI#hash and Addressable::URI#to_s
- fixed unicode normalization exception
- updated query_values methods to better handle subscript notation
- worked around issue with freezing URIs
- improved specs
# Addressable 2.0.2 <a name="v2.0.2">
- fixed issue with URI template expansion
- fixed issue with percent escaping characters 0-15
# Addressable 2.0.1 <a name="v2.0.1">
- fixed issue with query string assignment
- fixed issue with improperly encoded components
# Addressable 2.0.0 <a name="v2.0.0">
- the initialize method now takes an options hash as its only parameter
- added query_values method to URI class
- completely replaced IDNA implementation with pure Ruby
- renamed Addressable::ADDRESSABLE_VERSION to Addressable::VERSION
- completely reworked the Rakefile
- changed the behavior of the port method significantly
- Addressable::URI.encode_segment, Addressable::URI.unencode_segment renamed
- documentation is now in YARD format
- more rigorous type checking
- to_str method implemented, implicit conversion to Strings now allowed
- Addressable::URI#omit method added, Addressable::URI#merge method replaced
- updated URI Template code to match v 03 of the draft spec
- added a bunch of new specifications
# Addressable 1.0.4 <a name="v1.0.4">
- switched to using RSpec's pending system for specs that rely on IDN
- fixed issue with creating URIs with paths that are not prefixed with '/'
# Addressable 1.0.3 <a name="v1.0.3">
- implemented a hash method
# Addressable 1.0.2 <a name="v1.0.2">
- fixed minor bug with the extract_mapping method
# Addressable 1.0.1 <a name="v1.0.1">
- fixed minor bug with the extract_mapping method
# Addressable 1.0.0 <a name="v1.0.0">
- heuristic parse method added
- parsing is slightly more strict
- replaced to_h with to_hash
- fixed routing methods
- improved specifications
- improved heckle rake task
- no surviving heckle mutations
# Addressable 0.1.2 <a name="v0.1.2">
- improved normalization
- fixed bug in joining algorithm
- updated specifications
# Addressable 0.1.1 <a name="v0.1.1">
- updated documentation
- added URI Template variable extraction
# Addressable 0.1.0 <a name="v0.1.0">
- initial release
- implementation based on RFC 3986, 3987
- support for IRIs via libidn
- support for the URI Template draft spec
+31
View File
@@ -0,0 +1,31 @@
# frozen_string_literal: true
source 'https://rubygems.org'
gemspec
group :test do
gem 'bigdecimal' if RUBY_VERSION > '2.4'
gem 'rspec', '~> 3.8'
gem 'rspec-its', '~> 1.3'
end
group :coverage do
gem "coveralls", "> 0.7", require: false, platforms: :mri
gem "simplecov", require: false
end
group :development do
gem 'launchy', '~> 2.4', '>= 2.4.3'
gem 'redcarpet', :platform => :mri_19
gem 'yard'
end
group :test, :development do
gem 'memory_profiler'
gem "rake", ">= 12.3.3"
end
unless ENV["IDNA_MODE"] == "pure"
gem "idn-ruby", platform: :mri
end
+202
View File
@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
+121
View File
@@ -0,0 +1,121 @@
# Addressable
<dl>
<dt>Homepage</dt><dd><a href="https://github.com/sporkmonger/addressable">github.com/sporkmonger/addressable</a></dd>
<dt>Author</dt><dd><a href="mailto:bob@sporkmonger.com">Bob Aman</a></dd>
<dt>Copyright</dt><dd>Copyright © Bob Aman</dd>
<dt>License</dt><dd>Apache 2.0</dd>
</dl>
[![Gem Version](https://img.shields.io/gem/dt/addressable.svg)][gem]
[![Build Status](https://github.com/sporkmonger/addressable/workflows/CI/badge.svg)][actions]
[![Test Coverage Status](https://img.shields.io/coveralls/sporkmonger/addressable.svg)][coveralls]
[![Documentation Coverage Status](https://inch-ci.org/github/sporkmonger/addressable.svg?branch=master)][inch]
[gem]: https://rubygems.org/gems/addressable
[actions]: https://github.com/sporkmonger/addressable/actions
[coveralls]: https://coveralls.io/r/sporkmonger/addressable
[inch]: https://inch-ci.org/github/sporkmonger/addressable
# Description
Addressable is an alternative implementation to the URI implementation
that is part of Ruby's standard library. It is flexible, offers heuristic
parsing, and additionally provides extensive support for IRIs and URI templates.
Addressable closely conforms to RFC 3986, RFC 3987, and RFC 6570 (level 4).
# Reference
- {Addressable::URI}
- {Addressable::Template}
# Example usage
```ruby
require "addressable/uri"
uri = Addressable::URI.parse("http://example.com/path/to/resource/")
uri.scheme
#=> "http"
uri.host
#=> "example.com"
uri.path
#=> "/path/to/resource/"
uri = Addressable::URI.parse("http://www.詹姆斯.com/")
uri.normalize
#=> #<Addressable::URI:0xc9a4c8 URI:http://www.xn--8ws00zhy3a.com/>
```
# URI Templates
For more details, see [RFC 6570](https://www.rfc-editor.org/rfc/rfc6570.txt).
```ruby
require "addressable/template"
template = Addressable::Template.new("http://example.com/{?query*}")
template.expand({
"query" => {
'foo' => 'bar',
'color' => 'red'
}
})
#=> #<Addressable::URI:0xc9d95c URI:http://example.com/?foo=bar&color=red>
template = Addressable::Template.new("http://example.com/{?one,two,three}")
template.partial_expand({"one" => "1", "three" => 3}).pattern
#=> "http://example.com/?one=1{&two}&three=3"
template = Addressable::Template.new(
"http://{host}{/segments*}/{?one,two,bogus}{#fragment}"
)
uri = Addressable::URI.parse(
"http://example.com/a/b/c/?one=1&two=2#foo"
)
template.extract(uri)
#=>
# {
# "host" => "example.com",
# "segments" => ["a", "b", "c"],
# "one" => "1",
# "two" => "2",
# "fragment" => "foo"
# }
```
# Install
```console
$ gem install addressable
```
You may optionally turn on native IDN support by installing libidn and the
idn gem:
```console
$ sudo apt-get install libidn11-dev # Debian/Ubuntu
$ brew install libidn # OS X
$ gem install idn-ruby
```
# Semantic Versioning
This project uses [Semantic Versioning](https://semver.org/). You can (and should) specify your
dependency using a pessimistic version constraint covering the major and minor
values:
```ruby
spec.add_dependency 'addressable', '~> 2.7'
```
If you need a specific bug fix, you can also specify minimum tiny versions
without preventing updates to the latest minor release:
```ruby
spec.add_dependency 'addressable', '~> 2.3', '>= 2.3.7'
```
+37
View File
@@ -0,0 +1,37 @@
# frozen_string_literal: true
require 'rubygems'
require 'rake'
require File.join(File.dirname(__FILE__), 'lib', 'addressable', 'version')
PKG_DISPLAY_NAME = 'Addressable'
PKG_NAME = PKG_DISPLAY_NAME.downcase
PKG_VERSION = Addressable::VERSION::STRING
PKG_FILE_NAME = "#{PKG_NAME}-#{PKG_VERSION}"
RELEASE_NAME = "REL #{PKG_VERSION}"
PKG_SUMMARY = "URI Implementation"
PKG_DESCRIPTION = <<-TEXT
Addressable is an alternative implementation to the URI implementation that is
part of Ruby's standard library. It is flexible, offers heuristic parsing, and
additionally provides extensive support for IRIs and URI templates.
TEXT
PKG_FILES = FileList[
"data/**/*",
"lib/**/*.rb",
"spec/**/*.rb",
"tasks/**/*.rake",
"addressable.gemspec",
"CHANGELOG.md",
"Gemfile",
"LICENSE.txt",
"README.md",
"Rakefile",
]
task :default => "spec"
Dir['tasks/**/*.rake'].each { |rake| load rake }
@@ -0,0 +1,28 @@
# -*- encoding: utf-8 -*-
# stub: addressable 2.8.7 ruby lib
Gem::Specification.new do |s|
s.name = "addressable".freeze
s.version = "2.8.7".freeze
s.required_rubygems_version = Gem::Requirement.new(">= 0".freeze) if s.respond_to? :required_rubygems_version=
s.metadata = { "changelog_uri" => "https://github.com/sporkmonger/addressable/blob/main/CHANGELOG.md#v2.8.7" } if s.respond_to? :metadata=
s.require_paths = ["lib".freeze]
s.authors = ["Bob Aman".freeze]
s.date = "2024-06-21"
s.description = "Addressable is an alternative implementation to the URI implementation that is\npart of Ruby's standard library. It is flexible, offers heuristic parsing, and\nadditionally provides extensive support for IRIs and URI templates.\n".freeze
s.email = "bob@sporkmonger.com".freeze
s.extra_rdoc_files = ["README.md".freeze]
s.files = ["CHANGELOG.md".freeze, "Gemfile".freeze, "LICENSE.txt".freeze, "README.md".freeze, "Rakefile".freeze, "addressable.gemspec".freeze, "data/unicode.data".freeze, "lib/addressable.rb".freeze, "lib/addressable/idna.rb".freeze, "lib/addressable/idna/native.rb".freeze, "lib/addressable/idna/pure.rb".freeze, "lib/addressable/template.rb".freeze, "lib/addressable/uri.rb".freeze, "lib/addressable/version.rb".freeze, "spec/addressable/idna_spec.rb".freeze, "spec/addressable/net_http_compat_spec.rb".freeze, "spec/addressable/security_spec.rb".freeze, "spec/addressable/template_spec.rb".freeze, "spec/addressable/uri_spec.rb".freeze, "spec/spec_helper.rb".freeze, "tasks/clobber.rake".freeze, "tasks/gem.rake".freeze, "tasks/git.rake".freeze, "tasks/metrics.rake".freeze, "tasks/profile.rake".freeze, "tasks/rspec.rake".freeze, "tasks/yard.rake".freeze]
s.homepage = "https://github.com/sporkmonger/addressable".freeze
s.licenses = ["Apache-2.0".freeze]
s.rdoc_options = ["--main".freeze, "README.md".freeze]
s.required_ruby_version = Gem::Requirement.new(">= 2.2".freeze)
s.rubygems_version = "3.5.11".freeze
s.summary = "URI Implementation".freeze
s.specification_version = 4
s.add_runtime_dependency(%q<public_suffix>.freeze, [">= 2.0.2".freeze, "< 7.0".freeze])
s.add_development_dependency(%q<bundler>.freeze, [">= 1.0".freeze, "< 3.0".freeze])
end
Binary file not shown.
@@ -0,0 +1,4 @@
# frozen_string_literal: true
require 'addressable/uri'
require 'addressable/template'
@@ -0,0 +1,26 @@
# frozen_string_literal: true
#--
# Copyright (C) Bob Aman
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#++
begin
require "addressable/idna/native"
rescue LoadError
# libidn or the idn gem was not available, fall back on a pure-Ruby
# implementation...
require "addressable/idna/pure"
end
@@ -0,0 +1,66 @@
# frozen_string_literal: true
#--
# Copyright (C) Bob Aman
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#++
require "idn"
module Addressable
module IDNA
def self.punycode_encode(value)
IDN::Punycode.encode(value.to_s)
end
def self.punycode_decode(value)
IDN::Punycode.decode(value.to_s)
end
class << self
# @deprecated Use {String#unicode_normalize(:nfkc)} instead
def unicode_normalize_kc(value)
value.to_s.unicode_normalize(:nfkc)
end
extend Gem::Deprecate
deprecate :unicode_normalize_kc, "String#unicode_normalize(:nfkc)", 2023, 4
end
def self.to_ascii(value)
value.to_s.split('.', -1).map do |segment|
if segment.size > 0 && segment.size < 64
IDN::Idna.toASCII(segment, IDN::Idna::ALLOW_UNASSIGNED)
elsif segment.size >= 64
segment
else
''
end
end.join('.')
end
def self.to_unicode(value)
value.to_s.split('.', -1).map do |segment|
if segment.size > 0 && segment.size < 64
IDN::Idna.toUnicode(segment, IDN::Idna::ALLOW_UNASSIGNED)
elsif segment.size >= 64
segment
else
''
end
end.join('.')
end
end
end
@@ -0,0 +1,505 @@
# frozen_string_literal: true
#--
# Copyright (C) Bob Aman
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#++
module Addressable
module IDNA
# This module is loosely based on idn_actionmailer by Mick Staugaard,
# the unicode library by Yoshida Masato, and the punycode implementation
# by Kazuhiro Nishiyama. Most of the code was copied verbatim, but
# some reformatting was done, and some translation from C was done.
#
# Without their code to work from as a base, we'd all still be relying
# on the presence of libidn. Which nobody ever seems to have installed.
#
# Original sources:
# http://github.com/staugaard/idn_actionmailer
# http://www.yoshidam.net/Ruby.html#unicode
# http://rubyforge.org/frs/?group_id=2550
UNICODE_TABLE = File.expand_path(
File.join(File.dirname(__FILE__), '../../..', 'data/unicode.data')
)
ACE_PREFIX = "xn--"
UTF8_REGEX = /\A(?:
[\x09\x0A\x0D\x20-\x7E] # ASCII
| [\xC2-\xDF][\x80-\xBF] # non-overlong 2-byte
| \xE0[\xA0-\xBF][\x80-\xBF] # excluding overlongs
| [\xE1-\xEC\xEE\xEF][\x80-\xBF]{2} # straight 3-byte
| \xED[\x80-\x9F][\x80-\xBF] # excluding surrogates
| \xF0[\x90-\xBF][\x80-\xBF]{2} # planes 1-3
| [\xF1-\xF3][\x80-\xBF]{3} # planes 4nil5
| \xF4[\x80-\x8F][\x80-\xBF]{2} # plane 16
)*\z/mnx
UTF8_REGEX_MULTIBYTE = /(?:
[\xC2-\xDF][\x80-\xBF] # non-overlong 2-byte
| \xE0[\xA0-\xBF][\x80-\xBF] # excluding overlongs
| [\xE1-\xEC\xEE\xEF][\x80-\xBF]{2} # straight 3-byte
| \xED[\x80-\x9F][\x80-\xBF] # excluding surrogates
| \xF0[\x90-\xBF][\x80-\xBF]{2} # planes 1-3
| [\xF1-\xF3][\x80-\xBF]{3} # planes 4nil5
| \xF4[\x80-\x8F][\x80-\xBF]{2} # plane 16
)/mnx
# :startdoc:
# Converts from a Unicode internationalized domain name to an ASCII
# domain name as described in RFC 3490.
def self.to_ascii(input)
input = input.to_s unless input.is_a?(String)
input = input.dup.force_encoding(Encoding::UTF_8).unicode_normalize(:nfkc)
if input.respond_to?(:force_encoding)
input.force_encoding(Encoding::ASCII_8BIT)
end
if input =~ UTF8_REGEX && input =~ UTF8_REGEX_MULTIBYTE
parts = unicode_downcase(input).split('.')
parts.map! do |part|
if part.respond_to?(:force_encoding)
part.force_encoding(Encoding::ASCII_8BIT)
end
if part =~ UTF8_REGEX && part =~ UTF8_REGEX_MULTIBYTE
ACE_PREFIX + punycode_encode(part)
else
part
end
end
parts.join('.')
else
input
end
end
# Converts from an ASCII domain name to a Unicode internationalized
# domain name as described in RFC 3490.
def self.to_unicode(input)
input = input.to_s unless input.is_a?(String)
parts = input.split('.')
parts.map! do |part|
if part =~ /^#{ACE_PREFIX}(.+)/
begin
punycode_decode(part[/^#{ACE_PREFIX}(.+)/, 1])
rescue Addressable::IDNA::PunycodeBadInput
# toUnicode is explicitly defined as never-fails by the spec
part
end
else
part
end
end
output = parts.join('.')
if output.respond_to?(:force_encoding)
output.force_encoding(Encoding::UTF_8)
end
output
end
class << self
# @deprecated Use {String#unicode_normalize(:nfkc)} instead
def unicode_normalize_kc(value)
value.to_s.unicode_normalize(:nfkc)
end
extend Gem::Deprecate
deprecate :unicode_normalize_kc, "String#unicode_normalize(:nfkc)", 2023, 4
end
##
# Unicode aware downcase method.
#
# @api private
# @param [String] input
# The input string.
# @return [String] The downcased result.
def self.unicode_downcase(input)
input = input.to_s unless input.is_a?(String)
unpacked = input.unpack("U*")
unpacked.map! { |codepoint| lookup_unicode_lowercase(codepoint) }
return unpacked.pack("U*")
end
private_class_method :unicode_downcase
def self.lookup_unicode_lowercase(codepoint)
codepoint_data = UNICODE_DATA[codepoint]
(codepoint_data ?
(codepoint_data[UNICODE_DATA_LOWERCASE] || codepoint) :
codepoint)
end
private_class_method :lookup_unicode_lowercase
UNICODE_DATA_COMBINING_CLASS = 0
UNICODE_DATA_EXCLUSION = 1
UNICODE_DATA_CANONICAL = 2
UNICODE_DATA_COMPATIBILITY = 3
UNICODE_DATA_UPPERCASE = 4
UNICODE_DATA_LOWERCASE = 5
UNICODE_DATA_TITLECASE = 6
begin
if defined?(FakeFS)
fakefs_state = FakeFS.activated?
FakeFS.deactivate!
end
# This is a sparse Unicode table. Codepoints without entries are
# assumed to have the value: [0, 0, nil, nil, nil, nil, nil]
UNICODE_DATA = File.open(UNICODE_TABLE, "rb") do |file|
Marshal.load(file.read)
end
ensure
if defined?(FakeFS)
FakeFS.activate! if fakefs_state
end
end
COMPOSITION_TABLE = {}
UNICODE_DATA.each do |codepoint, data|
canonical = data[UNICODE_DATA_CANONICAL]
exclusion = data[UNICODE_DATA_EXCLUSION]
if canonical && exclusion == 0
COMPOSITION_TABLE[canonical.unpack("C*")] = codepoint
end
end
UNICODE_MAX_LENGTH = 256
ACE_MAX_LENGTH = 256
PUNYCODE_BASE = 36
PUNYCODE_TMIN = 1
PUNYCODE_TMAX = 26
PUNYCODE_SKEW = 38
PUNYCODE_DAMP = 700
PUNYCODE_INITIAL_BIAS = 72
PUNYCODE_INITIAL_N = 0x80
PUNYCODE_DELIMITER = 0x2D
PUNYCODE_MAXINT = 1 << 64
PUNYCODE_PRINT_ASCII =
"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" +
"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n" +
" !\"\#$%&'()*+,-./" +
"0123456789:;<=>?" +
"@ABCDEFGHIJKLMNO" +
"PQRSTUVWXYZ[\\]^_" +
"`abcdefghijklmno" +
"pqrstuvwxyz{|}~\n"
# Input is invalid.
class PunycodeBadInput < StandardError; end
# Output would exceed the space provided.
class PunycodeBigOutput < StandardError; end
# Input needs wider integers to process.
class PunycodeOverflow < StandardError; end
def self.punycode_encode(unicode)
unicode = unicode.to_s unless unicode.is_a?(String)
input = unicode.unpack("U*")
output = [0] * (ACE_MAX_LENGTH + 1)
input_length = input.size
output_length = [ACE_MAX_LENGTH]
# Initialize the state
n = PUNYCODE_INITIAL_N
delta = out = 0
max_out = output_length[0]
bias = PUNYCODE_INITIAL_BIAS
# Handle the basic code points:
input_length.times do |j|
if punycode_basic?(input[j])
if max_out - out < 2
raise PunycodeBigOutput,
"Output would exceed the space provided."
end
output[out] = input[j]
out += 1
end
end
h = b = out
# h is the number of code points that have been handled, b is the
# number of basic code points, and out is the number of characters
# that have been output.
if b > 0
output[out] = PUNYCODE_DELIMITER
out += 1
end
# Main encoding loop:
while h < input_length
# All non-basic code points < n have been
# handled already. Find the next larger one:
m = PUNYCODE_MAXINT
input_length.times do |j|
m = input[j] if (n...m) === input[j]
end
# Increase delta enough to advance the decoder's
# <n,i> state to <m,0>, but guard against overflow:
if m - n > (PUNYCODE_MAXINT - delta) / (h + 1)
raise PunycodeOverflow, "Input needs wider integers to process."
end
delta += (m - n) * (h + 1)
n = m
input_length.times do |j|
# Punycode does not need to check whether input[j] is basic:
if input[j] < n
delta += 1
if delta == 0
raise PunycodeOverflow,
"Input needs wider integers to process."
end
end
if input[j] == n
# Represent delta as a generalized variable-length integer:
q = delta; k = PUNYCODE_BASE
while true
if out >= max_out
raise PunycodeBigOutput,
"Output would exceed the space provided."
end
t = (
if k <= bias
PUNYCODE_TMIN
elsif k >= bias + PUNYCODE_TMAX
PUNYCODE_TMAX
else
k - bias
end
)
break if q < t
output[out] =
punycode_encode_digit(t + (q - t) % (PUNYCODE_BASE - t))
out += 1
q = (q - t) / (PUNYCODE_BASE - t)
k += PUNYCODE_BASE
end
output[out] = punycode_encode_digit(q)
out += 1
bias = punycode_adapt(delta, h + 1, h == b)
delta = 0
h += 1
end
end
delta += 1
n += 1
end
output_length[0] = out
outlen = out
outlen.times do |j|
c = output[j]
unless c >= 0 && c <= 127
raise StandardError, "Invalid output char."
end
unless PUNYCODE_PRINT_ASCII[c]
raise PunycodeBadInput, "Input is invalid."
end
end
output[0..outlen].map { |x| x.chr }.join("").sub(/\0+\z/, "")
end
private_class_method :punycode_encode
def self.punycode_decode(punycode)
input = []
output = []
if ACE_MAX_LENGTH * 2 < punycode.size
raise PunycodeBigOutput, "Output would exceed the space provided."
end
punycode.each_byte do |c|
unless c >= 0 && c <= 127
raise PunycodeBadInput, "Input is invalid."
end
input.push(c)
end
input_length = input.length
output_length = [UNICODE_MAX_LENGTH]
# Initialize the state
n = PUNYCODE_INITIAL_N
out = i = 0
max_out = output_length[0]
bias = PUNYCODE_INITIAL_BIAS
# Handle the basic code points: Let b be the number of input code
# points before the last delimiter, or 0 if there is none, then
# copy the first b code points to the output.
b = 0
input_length.times do |j|
b = j if punycode_delimiter?(input[j])
end
if b > max_out
raise PunycodeBigOutput, "Output would exceed the space provided."
end
b.times do |j|
unless punycode_basic?(input[j])
raise PunycodeBadInput, "Input is invalid."
end
output[out] = input[j]
out+=1
end
# Main decoding loop: Start just after the last delimiter if any
# basic code points were copied; start at the beginning otherwise.
in_ = b > 0 ? b + 1 : 0
while in_ < input_length
# in_ is the index of the next character to be consumed, and
# out is the number of code points in the output array.
# Decode a generalized variable-length integer into delta,
# which gets added to i. The overflow checking is easier
# if we increase i as we go, then subtract off its starting
# value at the end to obtain delta.
oldi = i; w = 1; k = PUNYCODE_BASE
while true
if in_ >= input_length
raise PunycodeBadInput, "Input is invalid."
end
digit = punycode_decode_digit(input[in_])
in_+=1
if digit >= PUNYCODE_BASE
raise PunycodeBadInput, "Input is invalid."
end
if digit > (PUNYCODE_MAXINT - i) / w
raise PunycodeOverflow, "Input needs wider integers to process."
end
i += digit * w
t = (
if k <= bias
PUNYCODE_TMIN
elsif k >= bias + PUNYCODE_TMAX
PUNYCODE_TMAX
else
k - bias
end
)
break if digit < t
if w > PUNYCODE_MAXINT / (PUNYCODE_BASE - t)
raise PunycodeOverflow, "Input needs wider integers to process."
end
w *= PUNYCODE_BASE - t
k += PUNYCODE_BASE
end
bias = punycode_adapt(i - oldi, out + 1, oldi == 0)
# I was supposed to wrap around from out + 1 to 0,
# incrementing n each time, so we'll fix that now:
if i / (out + 1) > PUNYCODE_MAXINT - n
raise PunycodeOverflow, "Input needs wider integers to process."
end
n += i / (out + 1)
i %= out + 1
# Insert n at position i of the output:
# not needed for Punycode:
# raise PUNYCODE_INVALID_INPUT if decode_digit(n) <= base
if out >= max_out
raise PunycodeBigOutput, "Output would exceed the space provided."
end
#memmove(output + i + 1, output + i, (out - i) * sizeof *output)
output[i + 1, out - i] = output[i, out - i]
output[i] = n
i += 1
out += 1
end
output_length[0] = out
output.pack("U*")
end
private_class_method :punycode_decode
def self.punycode_basic?(codepoint)
codepoint < 0x80
end
private_class_method :punycode_basic?
def self.punycode_delimiter?(codepoint)
codepoint == PUNYCODE_DELIMITER
end
private_class_method :punycode_delimiter?
def self.punycode_encode_digit(d)
d + 22 + 75 * ((d < 26) ? 1 : 0)
end
private_class_method :punycode_encode_digit
# Returns the numeric value of a basic codepoint
# (for use in representing integers) in the range 0 to
# base - 1, or PUNYCODE_BASE if codepoint does not represent a value.
def self.punycode_decode_digit(codepoint)
if codepoint - 48 < 10
codepoint - 22
elsif codepoint - 65 < 26
codepoint - 65
elsif codepoint - 97 < 26
codepoint - 97
else
PUNYCODE_BASE
end
end
private_class_method :punycode_decode_digit
# Bias adaptation method
def self.punycode_adapt(delta, numpoints, firsttime)
delta = firsttime ? delta / PUNYCODE_DAMP : delta >> 1
# delta >> 1 is a faster way of doing delta / 2
delta += delta / numpoints
difference = PUNYCODE_BASE - PUNYCODE_TMIN
k = 0
while delta > (difference * PUNYCODE_TMAX) / 2
delta /= difference
k += PUNYCODE_BASE
end
k + (difference + 1) * delta / (delta + PUNYCODE_SKEW)
end
private_class_method :punycode_adapt
end
# :startdoc:
end
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,31 @@
# frozen_string_literal: true
#--
# Copyright (C) Bob Aman
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#++
# Used to prevent the class/module from being loaded more than once
if !defined?(Addressable::VERSION)
module Addressable
module VERSION
MAJOR = 2
MINOR = 8
TINY = 7
STRING = [MAJOR, MINOR, TINY].join('.')
end
end
end
@@ -0,0 +1,302 @@
# frozen_string_literal: true
# Copyright (C) Bob Aman
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
require "spec_helper"
# Have to use RubyGems to load the idn gem.
require "rubygems"
require "addressable/idna"
shared_examples_for "converting from unicode to ASCII" do
it "should convert 'www.google.com' correctly" do
expect(Addressable::IDNA.to_ascii("www.google.com")).to eq("www.google.com")
end
long = 'AcinusFallumTrompetumNullunCreditumVisumEstAtCuadLongumEtCefallum.com'
it "should convert '#{long}' correctly" do
expect(Addressable::IDNA.to_ascii(long)).to eq(long)
end
it "should convert 'www.詹姆斯.com' correctly" do
expect(Addressable::IDNA.to_ascii(
"www.詹姆斯.com"
)).to eq("www.xn--8ws00zhy3a.com")
end
it "also accepts unicode strings encoded as ascii-8bit" do
expect(Addressable::IDNA.to_ascii(
"www.詹姆斯.com".b
)).to eq("www.xn--8ws00zhy3a.com")
end
it "should convert 'www.Iñtërnâtiônàlizætiøn.com' correctly" do
"www.Iñtërnâtiônàlizætiøn.com"
expect(Addressable::IDNA.to_ascii(
"www.I\xC3\xB1t\xC3\xABrn\xC3\xA2ti\xC3\xB4" +
"n\xC3\xA0liz\xC3\xA6ti\xC3\xB8n.com"
)).to eq("www.xn--itrntinliztin-vdb0a5exd8ewcye.com")
end
it "should convert 'www.Iñtërnâtiônàlizætiøn.com' correctly" do
expect(Addressable::IDNA.to_ascii(
"www.In\xCC\x83te\xCC\x88rna\xCC\x82tio\xCC\x82n" +
"a\xCC\x80liz\xC3\xA6ti\xC3\xB8n.com"
)).to eq("www.xn--itrntinliztin-vdb0a5exd8ewcye.com")
end
it "should convert " +
"'www.ほんとうにながいわけのわからないどめいんめいのらべるまだながくしないとたりない.w3.mag.keio.ac.jp' " +
"correctly" do
expect(Addressable::IDNA.to_ascii(
"www.\343\201\273\343\202\223\343\201\250\343\201\206\343\201\253\343" +
"\201\252\343\201\214\343\201\204\343\202\217\343\201\221\343\201\256" +
"\343\202\217\343\201\213\343\202\211\343\201\252\343\201\204\343\201" +
"\251\343\202\201\343\201\204\343\202\223\343\202\201\343\201\204\343" +
"\201\256\343\202\211\343\201\271\343\202\213\343\201\276\343\201\240" +
"\343\201\252\343\201\214\343\201\217\343\201\227\343\201\252\343\201" +
"\204\343\201\250\343\201\237\343\202\212\343\201\252\343\201\204." +
"w3.mag.keio.ac.jp"
)).to eq(
"www.xn--n8jaaaaai5bhf7as8fsfk3jnknefdde3" +
"fg11amb5gzdb4wi9bya3kc6lra.w3.mag.keio.ac.jp"
)
end
it "should convert " +
"'www.ほんとうにながいわけのわからないどめいんめいのらべるまだながくしないとたりない.w3.mag.keio.ac.jp' " +
"correctly" do
expect(Addressable::IDNA.to_ascii(
"www.\343\201\273\343\202\223\343\201\250\343\201\206\343\201\253\343" +
"\201\252\343\201\213\343\202\231\343\201\204\343\202\217\343\201\221" +
"\343\201\256\343\202\217\343\201\213\343\202\211\343\201\252\343\201" +
"\204\343\201\250\343\202\231\343\202\201\343\201\204\343\202\223\343" +
"\202\201\343\201\204\343\201\256\343\202\211\343\201\270\343\202\231" +
"\343\202\213\343\201\276\343\201\237\343\202\231\343\201\252\343\201" +
"\213\343\202\231\343\201\217\343\201\227\343\201\252\343\201\204\343" +
"\201\250\343\201\237\343\202\212\343\201\252\343\201\204." +
"w3.mag.keio.ac.jp"
)).to eq(
"www.xn--n8jaaaaai5bhf7as8fsfk3jnknefdde3" +
"fg11amb5gzdb4wi9bya3kc6lra.w3.mag.keio.ac.jp"
)
end
it "should convert '点心和烤鸭.w3.mag.keio.ac.jp' correctly" do
expect(Addressable::IDNA.to_ascii(
"点心和烤鸭.w3.mag.keio.ac.jp"
)).to eq("xn--0trv4xfvn8el34t.w3.mag.keio.ac.jp")
end
it "should convert '가각갂갃간갅갆갇갈갉힢힣.com' correctly" do
expect(Addressable::IDNA.to_ascii(
"가각갂갃간갅갆갇갈갉힢힣.com"
)).to eq("xn--o39acdefghijk5883jma.com")
end
it "should convert " +
"'\347\242\274\346\250\231\346\272\226\350" +
"\220\254\345\234\213\347\242\274.com' correctly" do
expect(Addressable::IDNA.to_ascii(
"\347\242\274\346\250\231\346\272\226\350" +
"\220\254\345\234\213\347\242\274.com"
)).to eq("xn--9cs565brid46mda086o.com")
end
it "should convert 'リ宠퐱〹.com' correctly" do
expect(Addressable::IDNA.to_ascii(
"\357\276\230\345\256\240\355\220\261\343\200\271.com"
)).to eq("xn--eek174hoxfpr4k.com")
end
it "should convert 'リ宠퐱卄.com' correctly" do
expect(Addressable::IDNA.to_ascii(
"\343\203\252\345\256\240\355\220\261\345\215\204.com"
)).to eq("xn--eek174hoxfpr4k.com")
end
it "should convert 'ᆵ' correctly" do
expect(Addressable::IDNA.to_ascii(
"\341\206\265"
)).to eq("xn--4ud")
end
it "should convert 'ᆵ' correctly" do
expect(Addressable::IDNA.to_ascii(
"\357\276\257"
)).to eq("xn--4ud")
end
it "should convert '🌹🌹🌹.ws' correctly" do
expect(Addressable::IDNA.to_ascii(
"\360\237\214\271\360\237\214\271\360\237\214\271.ws"
)).to eq("xn--2h8haa.ws")
end
it "should handle two adjacent '.'s correctly" do
expect(Addressable::IDNA.to_ascii(
"example..host"
)).to eq("example..host")
end
end
shared_examples_for "converting from ASCII to unicode" do
long = 'AcinusFallumTrompetumNullunCreditumVisumEstAtCuadLongumEtCefallum.com'
it "should convert '#{long}' correctly" do
expect(Addressable::IDNA.to_unicode(long)).to eq(long)
end
it "should return the identity conversion when punycode decode fails" do
expect(Addressable::IDNA.to_unicode("xn--zckp1cyg1.sblo.jp")).to eq(
"xn--zckp1cyg1.sblo.jp")
end
it "should return the identity conversion when the ACE prefix has no suffix" do
expect(Addressable::IDNA.to_unicode("xn--...-")).to eq("xn--...-")
end
it "should convert 'www.google.com' correctly" do
expect(Addressable::IDNA.to_unicode("www.google.com")).to eq(
"www.google.com")
end
it "should convert 'www.詹姆斯.com' correctly" do
expect(Addressable::IDNA.to_unicode(
"www.xn--8ws00zhy3a.com"
)).to eq("www.詹姆斯.com")
end
it "should convert '詹姆斯.com' correctly" do
expect(Addressable::IDNA.to_unicode(
"xn--8ws00zhy3a.com"
)).to eq("詹姆斯.com")
end
it "should convert 'www.iñtërnâtiônàlizætiøn.com' correctly" do
expect(Addressable::IDNA.to_unicode(
"www.xn--itrntinliztin-vdb0a5exd8ewcye.com"
)).to eq("www.iñtërnâtiônàlizætiøn.com")
end
it "should convert 'iñtërnâtiônàlizætiøn.com' correctly" do
expect(Addressable::IDNA.to_unicode(
"xn--itrntinliztin-vdb0a5exd8ewcye.com"
)).to eq("iñtërnâtiônàlizætiøn.com")
end
it "should convert " +
"'www.ほんとうにながいわけのわからないどめいんめいのらべるまだながくしないとたりない.w3.mag.keio.ac.jp' " +
"correctly" do
expect(Addressable::IDNA.to_unicode(
"www.xn--n8jaaaaai5bhf7as8fsfk3jnknefdde3" +
"fg11amb5gzdb4wi9bya3kc6lra.w3.mag.keio.ac.jp"
)).to eq(
"www.ほんとうにながいわけのわからないどめいんめいのらべるまだながくしないとたりない.w3.mag.keio.ac.jp"
)
end
it "should convert '点心和烤鸭.w3.mag.keio.ac.jp' correctly" do
expect(Addressable::IDNA.to_unicode(
"xn--0trv4xfvn8el34t.w3.mag.keio.ac.jp"
)).to eq("点心和烤鸭.w3.mag.keio.ac.jp")
end
it "should convert '가각갂갃간갅갆갇갈갉힢힣.com' correctly" do
expect(Addressable::IDNA.to_unicode(
"xn--o39acdefghijk5883jma.com"
)).to eq("가각갂갃간갅갆갇갈갉힢힣.com")
end
it "should convert " +
"'\347\242\274\346\250\231\346\272\226\350" +
"\220\254\345\234\213\347\242\274.com' correctly" do
expect(Addressable::IDNA.to_unicode(
"xn--9cs565brid46mda086o.com"
)).to eq(
"\347\242\274\346\250\231\346\272\226\350" +
"\220\254\345\234\213\347\242\274.com"
)
end
it "should convert 'リ宠퐱卄.com' correctly" do
expect(Addressable::IDNA.to_unicode(
"xn--eek174hoxfpr4k.com"
)).to eq("\343\203\252\345\256\240\355\220\261\345\215\204.com")
end
it "should convert 'ᆵ' correctly" do
expect(Addressable::IDNA.to_unicode(
"xn--4ud"
)).to eq("\341\206\265")
end
it "should convert '🌹🌹🌹.ws' correctly" do
expect(Addressable::IDNA.to_unicode(
"xn--2h8haa.ws"
)).to eq("\360\237\214\271\360\237\214\271\360\237\214\271.ws")
end
it "should handle two adjacent '.'s correctly" do
expect(Addressable::IDNA.to_unicode(
"example..host"
)).to eq("example..host")
end
end
describe Addressable::IDNA, "when using the pure-Ruby implementation" do
before do
Addressable.send(:remove_const, :IDNA)
load "addressable/idna/pure.rb"
end
it_should_behave_like "converting from unicode to ASCII"
it_should_behave_like "converting from ASCII to unicode"
begin
require "fiber"
it "should not blow up inside fibers" do
f = Fiber.new do
Addressable.send(:remove_const, :IDNA)
load "addressable/idna/pure.rb"
end
f.resume
end
rescue LoadError
# Fibers aren't supported in this version of Ruby, skip this test.
warn('Fibers unsupported.')
end
end
begin
require "idn"
describe Addressable::IDNA, "when using the native-code implementation" do
before do
Addressable.send(:remove_const, :IDNA)
load "addressable/idna/native.rb"
end
it_should_behave_like "converting from unicode to ASCII"
it_should_behave_like "converting from ASCII to unicode"
end
rescue LoadError => error
raise error if ENV["CI"] && TestHelper.native_supported?
# Cannot test the native implementation without libidn support.
warn('Could not load native IDN implementation.')
end
@@ -0,0 +1,29 @@
# frozen_string_literal: true
# Copyright (C) Bob Aman
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
require "spec_helper"
require "addressable/uri"
require "net/http"
describe Net::HTTP do
it "should be compatible with Addressable" do
response_body =
Net::HTTP.get(Addressable::URI.parse('http://www.google.com/'))
expect(response_body).not_to be_nil
end
end
@@ -0,0 +1,58 @@
# frozen_string_literal: true
# Copyright (C) Bob Aman
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
require "spec_helper"
require "addressable/uri"
describe Addressable::URI, "when created with a URI known to cause crashes " +
"in certain browsers" do
it "should parse correctly" do
uri = Addressable::URI.parse('%%30%30')
expect(uri.path).to eq('%%30%30')
expect(uri.normalize.path).to eq('%2500')
end
it "should parse correctly as a full URI" do
uri = Addressable::URI.parse('http://www.example.com/%%30%30')
expect(uri.path).to eq('/%%30%30')
expect(uri.normalize.path).to eq('/%2500')
end
end
describe Addressable::URI, "when created with a URI known to cause crashes " +
"in certain browsers" do
it "should parse correctly" do
uri = Addressable::URI.parse('لُصّبُلُلصّبُررً ॣ ॣh ॣ ॣ 冗')
expect(uri.path).to eq('لُصّبُلُلصّبُررً ॣ ॣh ॣ ॣ 冗')
expect(uri.normalize.path).to eq(
'%D9%84%D9%8F%D8%B5%D9%91%D8%A8%D9%8F%D9%84%D9%8F%D9%84%D8%B5%D9%91' +
'%D8%A8%D9%8F%D8%B1%D8%B1%D9%8B%20%E0%A5%A3%20%E0%A5%A3h%20%E0%A5' +
'%A3%20%E0%A5%A3%20%E5%86%97'
)
end
it "should parse correctly as a full URI" do
uri = Addressable::URI.parse('http://www.example.com/لُصّبُلُلصّبُررً ॣ ॣh ॣ ॣ 冗')
expect(uri.path).to eq('/لُصّبُلُلصّبُررً ॣ ॣh ॣ ॣ 冗')
expect(uri.normalize.path).to eq(
'/%D9%84%D9%8F%D8%B5%D9%91%D8%A8%D9%8F%D9%84%D9%8F%D9%84%D8%B5%D9%91' +
'%D8%A8%D9%8F%D8%B1%D8%B1%D9%8B%20%E0%A5%A3%20%E0%A5%A3h%20%E0%A5' +
'%A3%20%E0%A5%A3%20%E5%86%97'
)
end
end
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,33 @@
# frozen_string_literal: true
require 'bundler/setup'
require 'rspec/its'
begin
require 'coveralls'
Coveralls.wear! do
add_filter "spec/"
add_filter "vendor/"
end
rescue LoadError
warn "warning: coveralls gem not found; skipping Coveralls"
require 'simplecov'
SimpleCov.start do
add_filter "spec/"
add_filter "vendor/"
end
end if Gem.loaded_specs.key?("simplecov")
class TestHelper
def self.native_supported?
mri = RUBY_ENGINE == "ruby"
windows = RUBY_PLATFORM.include?("mingw")
mri && !windows
end
end
RSpec.configure do |config|
config.warnings = true
config.filter_run_when_matching :focus
end
@@ -0,0 +1,4 @@
# frozen_string_literal: true
desc "Remove all build products"
task "clobber"
+95
View File
@@ -0,0 +1,95 @@
# frozen_string_literal: true
require "rubygems/package_task"
namespace :gem do
GEM_SPEC = Gem::Specification.new do |s|
s.name = PKG_NAME
s.version = PKG_VERSION
s.summary = PKG_SUMMARY
s.description = PKG_DESCRIPTION
s.files = PKG_FILES.to_a
s.extra_rdoc_files = %w( README.md )
s.rdoc_options.concat ["--main", "README.md"]
if !s.respond_to?(:add_development_dependency)
puts "Cannot build Gem with this version of RubyGems."
exit(1)
end
s.required_ruby_version = ">= 2.2"
s.add_runtime_dependency "public_suffix", ">= 2.0.2", "< 7.0"
s.add_development_dependency "bundler", ">= 1.0", "< 3.0"
s.require_path = "lib"
s.author = "Bob Aman"
s.email = "bob@sporkmonger.com"
s.homepage = "https://github.com/sporkmonger/addressable"
s.license = "Apache-2.0"
s.metadata = {
"changelog_uri" => "https://github.com/sporkmonger/addressable/blob/main/CHANGELOG.md#v#{PKG_VERSION}"
}
end
Gem::PackageTask.new(GEM_SPEC) do |p|
p.gem_spec = GEM_SPEC
p.need_tar = true
p.need_zip = true
end
desc "Generates .gemspec file"
task :gemspec do
spec_string = GEM_SPEC.to_ruby
File.open("#{GEM_SPEC.name}.gemspec", "w") do |file|
file.write spec_string
end
end
desc "Show information about the gem"
task :debug do
puts GEM_SPEC.to_ruby
end
desc "Install the gem"
task :install => ["clobber", "gem:package"] do
sh "gem install --local ./pkg/#{GEM_SPEC.full_name}.gem"
end
desc "Uninstall the gem"
task :uninstall do
installed_list = Gem.source_index.find_name(PKG_NAME)
if installed_list &&
(installed_list.collect { |s| s.version.to_s}.include?(PKG_VERSION))
sh(
"gem uninstall --version '#{PKG_VERSION}' " +
"--ignore-dependencies --executables #{PKG_NAME}"
)
end
end
desc "Reinstall the gem"
task :reinstall => [:uninstall, :install]
desc "Package for release"
task :release => ["gem:package", "gem:gemspec"] do |t|
v = ENV["VERSION"] or abort "Must supply VERSION=x.y.z"
abort "Versions don't match #{v} vs #{PROJ.version}" if v != PKG_VERSION
pkg = "pkg/#{GEM_SPEC.full_name}"
changelog = File.open("CHANGELOG.md") { |file| file.read }
puts "Releasing #{PKG_NAME} v. #{PKG_VERSION}"
Rake::Task["git:tag:create"].invoke
end
end
desc "Alias to gem:package"
task "gem" => "gem:package"
task "gem:release" => "gem:gemspec"
task "clobber" => ["gem:clobber_package"]
+47
View File
@@ -0,0 +1,47 @@
# frozen_string_literal: true
namespace :git do
namespace :tag do
desc "List tags from the Git repository"
task :list do
tags = `git tag -l`
tags.gsub!("\r", "")
tags = tags.split("\n").sort {|a, b| b <=> a }
puts tags.join("\n")
end
desc "Create a new tag in the Git repository"
task :create do
changelog = File.open("CHANGELOG.md", "r") { |file| file.read }
puts "-" * 80
puts changelog
puts "-" * 80
puts
v = ENV["VERSION"] or abort "Must supply VERSION=x.y.z"
abort "Versions don't match #{v} vs #{PKG_VERSION}" if v != PKG_VERSION
git_status = `git status`
if git_status !~ /^nothing to commit/
abort "Working directory isn't clean."
end
tag = "#{PKG_NAME}-#{PKG_VERSION}"
msg = "Release #{PKG_NAME}-#{PKG_VERSION}"
existing_tags = `git tag -l #{PKG_NAME}-*`.split('\n')
if existing_tags.include?(tag)
warn("Tag already exists, deleting...")
unless system "git tag -d #{tag}"
abort "Tag deletion failed."
end
end
puts "Creating git tag '#{tag}'..."
unless system "git tag -a -m \"#{msg}\" #{tag}"
abort "Tag creation failed."
end
end
end
end
task "gem:release" => "git:tag:create"
@@ -0,0 +1,24 @@
# frozen_string_literal: true
namespace :metrics do
task :lines do
lines, codelines, total_lines, total_codelines = 0, 0, 0, 0
for file_name in FileList["lib/**/*.rb"]
f = File.open(file_name)
while line = f.gets
lines += 1
next if line =~ /^\s*$/
next if line =~ /^\s*#/
codelines += 1
end
puts "L: #{sprintf("%4d", lines)}, " +
"LOC #{sprintf("%4d", codelines)} | #{file_name}"
total_lines += lines
total_codelines += codelines
lines, codelines = 0, 0
end
puts "Total: Lines #{total_lines}, LOC #{total_codelines}"
end
end
@@ -0,0 +1,72 @@
# frozen_string_literal: true
namespace :profile do
desc "Profile Template match memory allocations"
task :template_match_memory do
require "memory_profiler"
require "addressable/template"
start_at = Time.now.to_f
template = Addressable::Template.new("http://example.com/{?one,two,three}")
report = MemoryProfiler.report do
30_000.times do
template.match(
"http://example.com/?one=one&two=floo&three=me"
)
end
end
end_at = Time.now.to_f
print_options = { scale_bytes: true, normalize_paths: true }
puts "\n\n"
if ENV["CI"]
report.pretty_print(print_options)
else
t_allocated = report.scale_bytes(report.total_allocated_memsize)
t_retained = report.scale_bytes(report.total_retained_memsize)
puts "Total allocated: #{t_allocated} (#{report.total_allocated} objects)"
puts "Total retained: #{t_retained} (#{report.total_retained} objects)"
puts "Took #{end_at - start_at} seconds"
FileUtils.mkdir_p("tmp")
report.pretty_print(to_file: "tmp/memprof.txt", **print_options)
end
end
desc "Profile URI parse memory allocations"
task :memory do
require "memory_profiler"
require "addressable/uri"
if ENV["IDNA_MODE"] == "pure"
Addressable.send(:remove_const, :IDNA)
load "addressable/idna/pure.rb"
end
start_at = Time.now.to_f
report = MemoryProfiler.report do
30_000.times do
Addressable::URI.parse(
"http://google.com/stuff/../?with_lots=of&params=asdff#!stuff"
).normalize
end
end
end_at = Time.now.to_f
print_options = { scale_bytes: true, normalize_paths: true }
puts "\n\n"
if ENV["CI"]
report.pretty_print(**print_options)
else
t_allocated = report.scale_bytes(report.total_allocated_memsize)
t_retained = report.scale_bytes(report.total_retained_memsize)
puts "Total allocated: #{t_allocated} (#{report.total_allocated} objects)"
puts "Total retained: #{t_retained} (#{report.total_retained} objects)"
puts "Took #{end_at - start_at} seconds"
FileUtils.mkdir_p("tmp")
report.pretty_print(to_file: "tmp/memprof.txt", **print_options)
end
end
end
@@ -0,0 +1,23 @@
# frozen_string_literal: true
require "rspec/core/rake_task"
namespace :spec do
RSpec::Core::RakeTask.new(:simplecov) do |t|
t.pattern = FileList['spec/**/*_spec.rb']
t.rspec_opts = %w[--color --format documentation] unless ENV["CI"]
end
namespace :simplecov do
desc "Browse the code coverage report."
task :browse => "spec:simplecov" do
require "launchy"
Launchy.open("coverage/index.html")
end
end
end
desc "Alias to spec:simplecov"
task "spec" => "spec:simplecov"
task "clobber" => ["spec:clobber_simplecov"]
@@ -0,0 +1,29 @@
# frozen_string_literal: true
require "rake"
begin
require "yard"
require "yard/rake/yardoc_task"
namespace :doc do
desc "Generate Yardoc documentation"
YARD::Rake::YardocTask.new do |yardoc|
yardoc.name = "yard"
yardoc.options = ["--verbose", "--markup", "markdown"]
yardoc.files = FileList[
"lib/**/*.rb", "ext/**/*.c",
"README.md", "CHANGELOG.md", "LICENSE.txt"
].exclude(/idna/)
end
end
task "clobber" => ["doc:clobber_yard"]
desc "Alias to doc:yard"
task "doc" => "doc:yard"
rescue LoadError
# If yard isn't available, it's not the end of the world
desc "Alias to doc:rdoc"
task "doc" => "doc:rdoc"
end
+326
View File
@@ -0,0 +1,326 @@
# Changelog
## Addressable 2.9.0 <a name="v2.9.0">
- fixes ReDoS vulnerability in Addressable::Template#match (fixes incomplete
remediation in 2.8.10)
## Addressable 2.8.10 <a name="v2.8.10">
- fixes ReDoS vulnerability in Addressable::Template#match
## Addressable 2.8.9 <a name="v2.8.9">
- Reduce gem size by excluding test files ([#569])
- No need for bundler as development dependency ([#571], [5fc1d93](https://github.com/sporkmonger/addressable/commit/5fc1d93))
- idna/pure: stop building the useless `COMPOSITION_TABLE` (removes the `Addressable::IDNA::COMPOSITION_TABLE` constant) ([#564])
[#569]: https://github.com/sporkmonger/addressable/pull/569
[#571]: https://github.com/sporkmonger/addressable/pull/571
[#564]: https://github.com/sporkmonger/addressable/pull/564
## Addressable 2.8.8 <a name="v2.8.8">
- Replace the `unicode.data` blob by a ruby constant ([#561])
- Allow `public_suffix` 7 ([#558])
[#561]: https://github.com/sporkmonger/addressable/pull/561
[#558]: https://github.com/sporkmonger/addressable/pull/558
## Addressable 2.8.7 <a name="v2.8.7">
- Allow `public_suffix` 6 ([#535])
[#535]: https://github.com/sporkmonger/addressable/pull/535
## Addressable 2.8.6 <a name="v2.8.6">
- Memoize regexps for common character classes ([#524])
[#524]: https://github.com/sporkmonger/addressable/pull/524
## Addressable 2.8.5 <a name="v2.8.5">
- Fix thread safety issue with encoding tables ([#515])
- Define URI::NONE as a module to avoid serialization issues ([#509])
- Fix YAML serialization ([#508])
[#508]: https://github.com/sporkmonger/addressable/pull/508
[#509]: https://github.com/sporkmonger/addressable/pull/509
[#515]: https://github.com/sporkmonger/addressable/pull/515
## Addressable 2.8.4 <a name="v2.8.4">
- Restore `Addressable::IDNA.unicode_normalize_kc` as a deprecated method ([#504])
[#504]: https://github.com/sporkmonger/addressable/pull/504
## Addressable 2.8.3 <a name="v2.8.3">
- Fix template expand level 2 hash support for non-string objects ([#499], [#498])
[#499]: https://github.com/sporkmonger/addressable/pull/499
[#498]: https://github.com/sporkmonger/addressable/pull/498
## Addressable 2.8.2 <a name="v2.8.2">
- Improve cache hits and JIT friendliness ([#486](https://github.com/sporkmonger/addressable/pull/486))
- Improve code style and test coverage ([#482](https://github.com/sporkmonger/addressable/pull/482))
- Ensure reset of deferred validation ([#481](https://github.com/sporkmonger/addressable/pull/481))
- Resolve normalization differences between `IDNA::Native` and `IDNA::Pure` ([#408](https://github.com/sporkmonger/addressable/issues/408), [#492])
- Remove redundant colon in `Addressable::URI::CharacterClasses::AUTHORITY` regex ([#438](https://github.com/sporkmonger/addressable/pull/438)) (accidentally reverted by [#449] merge but [added back](https://github.com/sporkmonger/addressable/pull/492#discussion_r1105125280) in [#492])
[#492]: https://github.com/sporkmonger/addressable/pull/492
## Addressable 2.8.1 <a name="v2.8.1">
- refactor `Addressable::URI.normalize_path` to address linter offenses ([#430](https://github.com/sporkmonger/addressable/pull/430))
- update gemspec to reflect supported Ruby versions ([#466], [#464], [#463])
- compatibility w/ public_suffix 5.x ([#466], [#465], [#460])
- fixes "invalid byte sequence in UTF-8" exception when unencoding URLs containing non UTF-8 characters ([#459](https://github.com/sporkmonger/addressable/pull/459))
- `Ractor` compatibility ([#449])
- use the whole string instead of a single line for template match ([#431](https://github.com/sporkmonger/addressable/pull/431))
- force UTF-8 encoding only if needed ([#341](https://github.com/sporkmonger/addressable/pull/341))
[#449]: https://github.com/sporkmonger/addressable/pull/449
[#460]: https://github.com/sporkmonger/addressable/pull/460
[#463]: https://github.com/sporkmonger/addressable/pull/463
[#464]: https://github.com/sporkmonger/addressable/pull/464
[#465]: https://github.com/sporkmonger/addressable/pull/465
[#466]: https://github.com/sporkmonger/addressable/pull/466
## Addressable 2.8.0 <a name="v2.8.0">
- fixes ReDoS vulnerability in Addressable::Template#match
- no longer replaces `+` with spaces in queries for non-http(s) schemes
- fixed encoding ipv6 literals
- the `:compacted` flag for `normalized_query` now dedupes parameters
- fix broken `escape_component` alias
- dropping support for Ruby 2.0 and 2.1
- adding Ruby 3.0 compatibility for development tasks
- drop support for `rack-mount` and remove Addressable::Template#generate
- performance improvements
- switch CI/CD to GitHub Actions
## Addressable 2.7.0 <a name="v2.7.0">
- added `:compacted` flag to `normalized_query`
- `heuristic_parse` handles `mailto:` more intuitively
- dropped explicit support for JRuby 9.0.5.0
- compatibility w/ public_suffix 4.x
- performance improvements
## Addressable 2.6.0 <a name="v2.6.0">
- added `tld=` method to allow assignment to the public suffix
- most `heuristic_parse` patterns are now case-insensitive
- `heuristic_parse` handles more `file://` URI variations
- fixes bug in `heuristic_parse` when uri starts with digit
- fixes bug in `request_uri=` with query strings
- fixes template issues with `nil` and `?` operator
- `frozen_string_literal` pragmas added
- minor performance improvements in regexps
- fixes to eliminate warnings
## Addressable 2.5.2 <a name="v2.5.2">
- better support for frozen string literals
- fixed bug w/ uppercase characters in scheme
- IDNA errors w/ emoji URLs
- compatibility w/ public_suffix 3.x
## Addressable 2.5.1 <a name="v2.5.1">
- allow unicode normalization to be disabled for URI Template expansion
- removed duplicate test
## Addressable 2.5.0 <a name="v2.5.0">
- dropping support for Ruby 1.9
- adding support for Ruby 2.4 preview
- add support for public suffixes and tld; first runtime dependency
- hostname escaping should match RFC; underscores in hostnames no longer escaped
- paths beginning with // and missing an authority are now considered invalid
- validation now also takes place after setting a path
- handle backslashes in authority more like a browser for `heuristic_parse`
- unescaped backslashes in host now raise an `InvalidURIError`
- `merge!`, `join!`, `omit!` and `normalize!` don't disable deferred validation
- `heuristic_parse` now trims whitespace before parsing
- host parts longer than 63 bytes will be ignored and not passed to libidn
- normalized values always encoded as UTF-8
## Addressable 2.4.0 <a name="v2.4.0">
- support for 1.8.x dropped
- double quotes in a host now raises an error
- newlines in host will no longer get unescaped during normalization
- stricter handling of bogus scheme values
- stricter handling of encoded port values
- calling `require 'addressable'` will now load both the URI and Template files
- assigning to the `hostname` component with an `IPAddr` object is now supported
- assigning to the `origin` component is now supported
- fixed minor bug where an exception would be thrown for a missing ACE suffix
- better partial expansion of URI templates
## Addressable 2.3.8 <a name="v2.3.8">
- fix warnings
- update dependency gems
- support for 1.8.x officially deprecated
## Addressable 2.3.7 <a name="v2.3.7">
- fix scenario in which invalid URIs don't get an exception until inspected
- handle hostnames with two adjacent periods correctly
- upgrade of RSpec
## Addressable 2.3.6 <a name="v2.3.6">
- normalization drops empty query string
- better handling in template extract for missing values
- template modifier for `'?'` now treated as optional
- fixed issue where character class parameters were modified
- templates can now be tested for equality
- added `:sorted` option to normalization of query strings
- fixed issue with normalization of hosts given in `'example.com.'` form
## Addressable 2.3.5 <a name="v2.3.5">
- added Addressable::URI#empty? method
- Addressable::URI#hostname methods now strip square brackets from IPv6 hosts
- compatibility with Net::HTTP in Ruby 2.0.0
- Addressable::URI#route_from should always give relative URIs
## Addressable 2.3.4 <a name="v2.3.4">
- fixed issue with encoding altering its inputs
- query string normalization now leaves ';' characters alone
- FakeFS is detected before attempting to load unicode tables
- additional testing to ensure frozen objects don't cause problems
## Addressable 2.3.3 <a name="v2.3.3">
- fixed issue with converting common primitives during template expansion
- fixed port encoding issue
- removed a few warnings
- normalize should now ignore %2B in query strings
- the IDNA logic should now be handled by libidn in Ruby 1.9
- no template match should now result in nil instead of an empty MatchData
- added license information to gemspec
## Addressable 2.3.2 <a name="v2.3.2">
- added Addressable::URI#default_port method
- fixed issue with Marshalling Unicode data on Windows
- improved heuristic parsing to better handle IPv4 addresses
## Addressable 2.3.1 <a name="v2.3.1">
- fixed missing unicode data file
## Addressable 2.3.0 <a name="v2.3.0">
- updated Addressable::Template to use RFC 6570, level 4
- fixed compatibility problems with some versions of Ruby
- moved unicode tables into a data file for performance reasons
- removing support for multiple query value notations
## Addressable 2.2.8 <a name="v2.2.8">
- fixed issues with dot segment removal code
- form encoding can now handle multiple values per key
- updated development environment
## Addressable 2.2.7 <a name="v2.2.7">
- fixed issues related to Addressable::URI#query_values=
- the Addressable::URI.parse method is now polymorphic
## Addressable 2.2.6 <a name="v2.2.6">
- changed the way ambiguous paths are handled
- fixed bug with frozen URIs
- https supported in heuristic parsing
## Addressable 2.2.5 <a name="v2.2.5">
- 'parsing' a pre-parsed URI object is now a dup operation
- introduced conditional support for libidn
- fixed normalization issue on ampersands in query strings
- added additional tests around handling of query strings
## Addressable 2.2.4 <a name="v2.2.4">
- added origin support from draft-ietf-websec-origin-00
- resolved issue with attempting to navigate below root
- fixed bug with string splitting in query strings
## Addressable 2.2.3 <a name="v2.2.3">
- added :flat_array notation for query strings
## Addressable 2.2.2 <a name="v2.2.2">
- fixed issue with percent escaping of '+' character in query strings
## Addressable 2.2.1 <a name="v2.2.1">
- added support for application/x-www-form-urlencoded.
## Addressable 2.2.0 <a name="v2.2.0">
- added site methods
- improved documentation
## Addressable 2.1.2 <a name="v2.1.2">
- added HTTP request URI methods
- better handling of Windows file paths
- validation_deferred boolean replaced with defer_validation block
- normalization of percent-encoded paths should now be correct
- fixed issue with constructing URIs with relative paths
- fixed warnings
## Addressable 2.1.1 <a name="v2.1.1">
- more type checking changes
- fixed issue with unicode normalization
- added method to find template defaults
- symbolic keys are now allowed in template mappings
- numeric values and symbolic values are now allowed in template mappings
## Addressable 2.1.0 <a name="v2.1.0x">
- refactored URI template support out into its own class
- removed extract method due to being useless and unreliable
- removed Addressable::URI.expand_template
- removed Addressable::URI#extract_mapping
- added partial template expansion
- fixed minor bugs in the parse and heuristic_parse methods
- fixed incompatibility with Ruby 1.9.1
- fixed bottleneck in Addressable::URI#hash and Addressable::URI#to_s
- fixed unicode normalization exception
- updated query_values methods to better handle subscript notation
- worked around issue with freezing URIs
- improved specs
## Addressable 2.0.2 <a name="v2.0.2">
- fixed issue with URI template expansion
- fixed issue with percent escaping characters 0-15
## Addressable 2.0.1 <a name="v2.0.1">
- fixed issue with query string assignment
- fixed issue with improperly encoded components
## Addressable 2.0.0 <a name="v2.0.0">
- the initialize method now takes an options hash as its only parameter
- added query_values method to URI class
- completely replaced IDNA implementation with pure Ruby
- renamed Addressable::ADDRESSABLE_VERSION to Addressable::VERSION
- completely reworked the Rakefile
- changed the behavior of the port method significantly
- Addressable::URI.encode_segment, Addressable::URI.unencode_segment renamed
- documentation is now in YARD format
- more rigorous type checking
- to_str method implemented, implicit conversion to Strings now allowed
- Addressable::URI#omit method added, Addressable::URI#merge method replaced
- updated URI Template code to match v 03 of the draft spec
- added a bunch of new specifications
## Addressable 1.0.4 <a name="v1.0.4">
- switched to using RSpec's pending system for specs that rely on IDN
- fixed issue with creating URIs with paths that are not prefixed with '/'
## Addressable 1.0.3 <a name="v1.0.3">
- implemented a hash method
## Addressable 1.0.2 <a name="v1.0.2">
- fixed minor bug with the extract_mapping method
## Addressable 1.0.1 <a name="v1.0.1">
- fixed minor bug with the extract_mapping method
## Addressable 1.0.0 <a name="v1.0.0">
- heuristic parse method added
- parsing is slightly more strict
- replaced to_h with to_hash
- fixed routing methods
- improved specifications
- improved heckle rake task
- no surviving heckle mutations
## Addressable 0.1.2 <a name="v0.1.2">
- improved normalization
- fixed bug in joining algorithm
- updated specifications
## Addressable 0.1.1 <a name="v0.1.1">
- updated documentation
- added URI Template variable extraction
## Addressable 0.1.0 <a name="v0.1.0">
- initial release
- implementation based on RFC 3986, 3987
- support for IRIs via libidn
- support for the URI Template draft spec
+202
View File
@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
+121
View File
@@ -0,0 +1,121 @@
# Addressable
<dl>
<dt>Homepage</dt><dd><a href="https://github.com/sporkmonger/addressable">github.com/sporkmonger/addressable</a></dd>
<dt>Author</dt><dd><a href="mailto:bob@sporkmonger.com">Bob Aman</a></dd>
<dt>Copyright</dt><dd>Copyright © Bob Aman</dd>
<dt>License</dt><dd>Apache 2.0</dd>
</dl>
[![Gem Version](https://img.shields.io/gem/dt/addressable.svg)][gem]
[![Build Status](https://github.com/sporkmonger/addressable/workflows/CI/badge.svg)][actions]
[![Test Coverage Status](https://img.shields.io/coveralls/sporkmonger/addressable.svg)][coveralls]
[![Documentation Coverage Status](https://inch-ci.org/github/sporkmonger/addressable.svg?branch=master)][inch]
[gem]: https://rubygems.org/gems/addressable
[actions]: https://github.com/sporkmonger/addressable/actions
[coveralls]: https://coveralls.io/r/sporkmonger/addressable
[inch]: https://inch-ci.org/github/sporkmonger/addressable
## Description
Addressable is an alternative implementation to the URI implementation
that is part of Ruby's standard library. It is flexible, offers heuristic
parsing, and additionally provides extensive support for IRIs and URI templates.
Addressable closely conforms to RFC 3986, RFC 3987, and RFC 6570 (level 4).
## Reference
- {Addressable::URI}
- {Addressable::Template}
## Example usage
```ruby
require "addressable/uri"
uri = Addressable::URI.parse("http://example.com/path/to/resource/")
uri.scheme
#=> "http"
uri.host
#=> "example.com"
uri.path
#=> "/path/to/resource/"
uri = Addressable::URI.parse("http://www.詹姆斯.com/")
uri.normalize
#=> #<Addressable::URI:0xc9a4c8 URI:http://www.xn--8ws00zhy3a.com/>
```
## URI Templates
For more details, see [RFC 6570](https://www.rfc-editor.org/rfc/rfc6570.txt).
```ruby
require "addressable/template"
template = Addressable::Template.new("http://example.com/{?query*}")
template.expand({
"query" => {
'foo' => 'bar',
'color' => 'red'
}
})
#=> #<Addressable::URI:0xc9d95c URI:http://example.com/?foo=bar&color=red>
template = Addressable::Template.new("http://example.com/{?one,two,three}")
template.partial_expand({"one" => "1", "three" => 3}).pattern
#=> "http://example.com/?one=1{&two}&three=3"
template = Addressable::Template.new(
"http://{host}{/segments*}/{?one,two,bogus}{#fragment}"
)
uri = Addressable::URI.parse(
"http://example.com/a/b/c/?one=1&two=2#foo"
)
template.extract(uri)
#=>
# {
# "host" => "example.com",
# "segments" => ["a", "b", "c"],
# "one" => "1",
# "two" => "2",
# "fragment" => "foo"
# }
```
## Install
```console
$ gem install addressable
```
You may optionally turn on native IDN support by installing libidn and the
idn gem:
```console
$ sudo apt-get install libidn11-dev # Debian/Ubuntu
$ brew install libidn # OS X
$ gem install idn-ruby
```
## Semantic Versioning
This project uses [Semantic Versioning](https://semver.org/). You can (and should) specify your
dependency using a pessimistic version constraint covering the major and minor
values:
```ruby
spec.add_dependency 'addressable', '~> 2.7'
```
If you need a specific bug fix, you can also specify minimum tiny versions
without preventing updates to the latest minor release:
```ruby
spec.add_dependency 'addressable', '~> 2.3', '>= 2.3.7'
```
@@ -0,0 +1,4 @@
# frozen_string_literal: true
require 'addressable/uri'
require 'addressable/template'
@@ -0,0 +1,26 @@
# frozen_string_literal: true
#--
# Copyright (C) Bob Aman
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#++
begin
require "addressable/idna/native"
rescue LoadError
# libidn or the idn gem was not available, fall back on a pure-Ruby
# implementation...
require "addressable/idna/pure"
end
@@ -0,0 +1,66 @@
# frozen_string_literal: true
#--
# Copyright (C) Bob Aman
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#++
require "idn"
module Addressable
module IDNA
def self.punycode_encode(value)
IDN::Punycode.encode(value.to_s)
end
def self.punycode_decode(value)
IDN::Punycode.decode(value.to_s)
end
class << self
# @deprecated Use {String#unicode_normalize(:nfkc)} instead
def unicode_normalize_kc(value)
value.to_s.unicode_normalize(:nfkc)
end
extend Gem::Deprecate
deprecate :unicode_normalize_kc, "String#unicode_normalize(:nfkc)", 2023, 4
end
def self.to_ascii(value)
value.to_s.split('.', -1).map do |segment|
if segment.size > 0 && segment.size < 64
IDN::Idna.toASCII(segment, IDN::Idna::ALLOW_UNASSIGNED)
elsif segment.size >= 64
segment
else
''
end
end.join('.')
end
def self.to_unicode(value)
value.to_s.split('.', -1).map do |segment|
if segment.size > 0 && segment.size < 64
IDN::Idna.toUnicode(segment, IDN::Idna::ALLOW_UNASSIGNED)
elsif segment.size >= 64
segment
else
''
end
end.join('.')
end
end
end
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,31 @@
# frozen_string_literal: true
#--
# Copyright (C) Bob Aman
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#++
# Used to prevent the class/module from being loaded more than once
if !defined?(Addressable::VERSION)
module Addressable
module VERSION
MAJOR = 2
MINOR = 9
TINY = 0
STRING = [MAJOR, MINOR, TINY].join('.')
end
end
end
+9
View File
@@ -0,0 +1,9 @@
## 0.2.2
* The gem was missing
## 0.2.1
* Fixed rdoc tasks, replaced Test::Unit with Minitest (Thanks to @yob, @strzibny for changes)
* Removed jeweler dependency
+20
View File
@@ -0,0 +1,20 @@
Copyright (c) 2009 Jan Krutisch
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+20
View File
@@ -0,0 +1,20 @@
= afm
A very simple library to read Adobe Font Metrics files (afm).
Currently simply parses the file and saves it in a few attributes.
== Note on Patches/Pull Requests
* Fork the project.
* Make your feature addition or bug fix.
* Add tests for it. This is important so I don't break it in a
future version unintentionally.
* Commit, do not mess with rakefile, version, or history.
(if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
* Send me a pull request. Bonus points for topic branches.
== Copyright
Copyright (c) 2010 Jan Krutisch. See LICENSE for details.
+23
View File
@@ -0,0 +1,23 @@
require 'rubygems'
require 'bundler/setup'
require 'rake'
require 'rake/testtask'
Rake::TestTask.new(:test) do |test|
test.libs << 'lib' << 'test'
test.pattern = 'test/**/test_*.rb'
test.verbose = true
end
task :default => :test
require 'rdoc/task'
RDoc::Task.new do |rdoc|
version = File.exist?('VERSION') ? File.read('VERSION') : ""
rdoc.rdoc_dir = 'rdoc'
rdoc.title = "afm #{version}"
rdoc.rdoc_files.include('README*')
rdoc.rdoc_files.include('lib/**/*.rb')
end
+1
View File
@@ -0,0 +1 @@
0.2.2
+105
View File
@@ -0,0 +1,105 @@
module AFM
ISO_LATIN1_ENCODING = %w(
.notdef .notdef .notdef .notdef .notdef .notdef .notdef .notdef
.notdef .notdef .notdef .notdef .notdef .notdef .notdef .notdef
.notdef .notdef .notdef .notdef .notdef .notdef .notdef .notdef
.notdef .notdef .notdef .notdef .notdef .notdef .notdef .notdef space
exclam quotedbl numbersign dollar percent ampersand quoteright
parenleft parenright asterisk plus comma minus period slash zero one
two three four five six seven eight nine colon semicolon less equal
greater question at A B C D E F G H I J K L M N O P Q R S
T U V W X Y Z bracketleft backslash bracketright asciicircum
underscore quoteleft a b c d e f g h i j k l m n o p q r s
t u v w x y z braceleft bar braceright asciitilde .notdef .notdef
.notdef .notdef .notdef .notdef .notdef .notdef .notdef .notdef
.notdef .notdef .notdef .notdef .notdef .notdef .notdef dotlessi grave
acute circumflex tilde macron breve dotaccent dieresis .notdef ring
cedilla .notdef hungarumlaut ogonek caron space exclamdown cent
sterling currency yen brokenbar section dieresis copyright ordfeminine
guillemotleft logicalnot hyphen registered macron degree plusminus
twosuperior threesuperior acute mu paragraph periodcentered cedilla
onesuperior ordmasculine guillemotright onequarter onehalf threequarters
questiondown Agrave Aacute Acircumflex Atilde Adieresis Aring AE
Ccedilla Egrave Eacute Ecircumflex Edieresis Igrave Iacute Icircumflex
Idieresis Eth Ntilde Ograve Oacute Ocircumflex Otilde Odieresis
multiply Oslash Ugrave Uacute Ucircumflex Udieresis Yacute Thorn
germandbls agrave aacute acircumflex atilde adieresis aring ae
ccedilla egrave eacute ecircumflex edieresis igrave iacute icircumflex
idieresis eth ntilde ograve oacute ocircumflex otilde odieresis divide
oslash ugrave uacute ucircumflex udieresis yacute thorn ydieresis
)
class Font
attr_reader :metadata, :char_metrics, :char_metrics_by_code, :kern_pairs
# Loading a Font Metrics file by absolute path (no automatic font path resolution)
def initialize(filename)
@metadata = {}
@char_metrics = {}
@char_metrics_by_code = {}
@kern_pairs = []
File.open(filename) do |file|
mode = :meta
file.each_line do |line|
case(line)
when /^StartFontMetrics/ ; mode = :meta
when /^StartCharMetrics/ ; mode = :char_metrics
when /^EndCharMetrics/ ; mode = :meta
when /^StartKernData/ ; mode = :kern_data
when /^StartKernPairs/ ; mode = :kern_pairs
when /^EndKernPairs/ ; mode = :kern_data
when /^EndKernData/ ; mode = :meta
else
case(mode)
when :meta
if match = line.match(/^([\w]+) (.*)$/)
@metadata[match[1]] = match[2]
end
when :char_metrics
metrics = {}
metrics[:charcode] = match[1].to_i if match = line.match(/C (-?\d+) *?;/)
metrics[:wx] = match[1].to_i if match = line.match(/WX (-?\d+) *?;/)
metrics[:name] = match[1] if match = line.match(/N ([.\w]+) *?;/)
if match = line.match(/B (-?\d+) (-?\d+) (-?\d+) (-?\d+) *?;/)
metrics[:boundingbox] = [match[1].to_i, match[2].to_i, match[3].to_i, match[4].to_i]
end
@char_metrics[metrics[:name]] = metrics if metrics[:name]
@char_metrics_by_code[metrics[:charcode]] = metrics if metrics[:charcode] && metrics[:charcode] > 0
when :kern_pairs
if match = line.match(/^KPX ([.\w]+) ([.\w]+) (-?\d+)$/)
@kern_pairs << [match[1], match[2], match[3].to_i]
end
end
end
end
end
end
#
# alias for new()
def self.from_file(file)
self.new(file)
end
#
# Get metadata by key
def [](key)
@metadata[key]
end
#
# Get metrics for character. Takes an integer (charcode) or
# a one-char string. currently works only for Latin1 strings,
# since we only have a chartable for the Latin1 charset so far.
# (shamelessly stolen from AFM.pm by Gisle Aas)
def metrics_for(char)
glyph = if (char.kind_of?(Integer))
ISO_LATIN1_ENCODING[char]
else
ISO_LATIN1_ENCODING[char.unpack("C*").first]
end
@char_metrics[glyph]
end
end
end
File diff suppressed because it is too large Load Diff
+4
View File
@@ -0,0 +1,4 @@
require 'rubygems'
require 'bundler/setup'
require 'minitest/autorun'
require 'afm'
+32
View File
@@ -0,0 +1,32 @@
require 'helper'
class TestAfm < Minitest::Test
def setup
@font = AFM::Font.new(File.join(File.dirname(__FILE__), 'fixtures', 'Vera.afm'))
end
def test_should_set_metadata
assert_equal "BitstreamVeraSans-Roman", @font.metadata['FontName']
assert_equal "BitstreamVeraSans-Roman", @font['FontName']
end
def test_should_set_char_metrics
assert_equal 400, @font.char_metrics['exclam'][:wx]
assert_equal [85, -131, 310, 758], @font.char_metrics['parenleft'][:boundingbox]
end
def test_should_set_char_metrics_by_code
assert_equal 400, @font.char_metrics_by_code[33][:wx]
assert_equal [85, -131, 310, 758], @font.char_metrics_by_code[40][:boundingbox]
end
def test_should_get_char_metrics_by_char
assert_equal 400, @font.metrics_for("!")[:wx]
end
def test_open_font_with_alternative_method
assert !AFM::Font.from_file(File.join(File.dirname(__FILE__), 'fixtures', 'Vera.afm')).nil?
end
end
+15
View File
@@ -0,0 +1,15 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2017-2024, by Samuel Williams.
# Copyright, 2020, by Salim Semaoune.
require_relative "async/version"
require_relative "async/reactor"
require_relative "kernel/async"
require_relative "kernel/sync"
# Asynchronous programming framework.
module Async
end
@@ -0,0 +1,42 @@
A synchronization primitive, which allows one task to wait for a number of other tasks to complete. It can be used in conjunction with {Semaphore}.
## Example
~~~ ruby
require 'async'
require 'async/barrier'
barrier = Async::Barrier.new
Sync do
Console.info("Barrier Example: sleep sort.")
# Generate an array of 10 numbers:
numbers = 10.times.map{rand(10)}
sorted = []
# Sleep sort the numbers:
numbers.each do |number|
barrier.async do |task|
sleep(number)
sorted << number
end
end
# Wait for all the numbers to be sorted:
barrier.wait
Console.info("Sorted", sorted)
ensure
# Ensure all the tasks are stopped when we exit:
barrier.stop
end
~~~
### Output
~~~
0.0s info: Barrier Example: sleep sort.
9.0s info: Sorted
| [3, 3, 3, 4, 4, 5, 5, 5, 8, 9]
~~~
@@ -0,0 +1,78 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2019-2024, by Samuel Williams.
require_relative "list"
require_relative "task"
module Async
# A general purpose synchronisation primitive, which allows one task to wait for a number of other tasks to complete. It can be used in conjunction with {Semaphore}.
#
# @public Since *Async v1*.
class Barrier
# Initialize the barrier.
# @parameter parent [Task | Semaphore | Nil] The parent for holding any children tasks.
# @public Since *Async v1*.
def initialize(parent: nil)
@tasks = List.new
@parent = parent
end
class TaskNode < List::Node
def initialize(task)
@task = task
end
attr :task
end
private_constant :TaskNode
# Number of tasks being held by the barrier.
def size
@tasks.size
end
# All tasks which have been invoked into the barrier.
attr :tasks
# Execute a child task and add it to the barrier.
# @asynchronous Executes the given block concurrently.
def async(*arguments, parent: (@parent or Task.current), **options, &block)
task = parent.async(*arguments, **options, &block)
@tasks.append(TaskNode.new(task))
return task
end
# Whether there are any tasks being held by the barrier.
# @returns [Boolean]
def empty?
@tasks.empty?
end
# Wait for all tasks to complete by invoking {Task#wait} on each waiting task, which may raise an error. As long as the task has completed, it will be removed from the barrier.
# @asynchronous Will wait for tasks to finish executing.
def wait
@tasks.each do |waiting|
task = waiting.task
begin
task.wait
ensure
@tasks.remove?(waiting) unless task.alive?
end
end
end
# Stop all tasks held by the barrier.
# @asynchronous May wait for tasks to finish executing.
def stop
@tasks.each do |waiting|
waiting.task.stop
end
end
end
end
+74
View File
@@ -0,0 +1,74 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2018-2022, by Samuel Williams.
module Async
# A convenient wrapper around the internal monotonic clock.
# @public Since *Async v1*.
class Clock
# Get the current elapsed monotonic time.
def self.now
::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
end
# Measure the execution of a block of code.
# @yields {...} The block to execute.
# @returns [Numeric] The total execution time.
def self.measure
start_time = self.now
yield
return self.now - start_time
end
# Start measuring elapsed time from now.
# @returns [Clock]
def self.start
self.new.tap(&:start!)
end
# Create a new clock with the initial total time.
# @parameter total [Numeric] The initial clock duration.
def initialize(total = 0)
@total = total
@started = nil
end
# Start measuring a duration.
def start!
@started ||= Clock.now
end
# Stop measuring a duration and append the duration to the current total.
def stop!
if @started
@total += (Clock.now - @started)
@started = nil
end
return @total
end
# The total elapsed time including any current duration.
def total
total = @total
if @started
total += (Clock.now - @started)
end
return total
end
# Reset the total elapsed time. If the clock is currently running, reset the start time to now.
def reset!
@total = 0
if @started
@started = Clock.now
end
end
end
end
@@ -0,0 +1,31 @@
A synchronization primitive, which allows fibers to wait until a particular condition is (edge) triggered. Zero or more fibers can wait on a condition. When the condition is signalled, the fibers will be resumed in order.
## Example
~~~ ruby
require 'async'
Sync do
condition = Async::Condition.new
Async do
Console.info "Waiting for condition..."
value = condition.wait
Console.info "Condition was signalled: #{value}"
end
Async do |task|
sleep(1)
Console.info "Signalling condition..."
condition.signal("Hello World")
end
end
~~~
### Output
~~~
0.0s info: Waiting for condition...
1.0s info: Signalling condition...
1.0s info: Condition was signalled: Hello World
~~~
@@ -0,0 +1,75 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2017-2024, by Samuel Williams.
# Copyright, 2017, by Kent Gruber.
require "fiber"
require_relative "list"
module Async
# A synchronization primitive, which allows fibers to wait until a particular condition is (edge) triggered.
# @public Since *Async v1*.
class Condition
# Create a new condition.
def initialize
@waiting = List.new
end
class FiberNode < List::Node
def initialize(fiber)
@fiber = fiber
end
def transfer(*arguments)
@fiber.transfer(*arguments)
end
def alive?
@fiber.alive?
end
end
private_constant :FiberNode
# Queue up the current fiber and wait on yielding the task.
# @returns [Object]
def wait
@waiting.stack(FiberNode.new(Fiber.current)) do
Fiber.scheduler.transfer
end
end
# @deprecated Replaced by {#waiting?}
def empty?
@waiting.empty?
end
# @returns [Boolean] Is any fiber waiting on this notification?
def waiting?
@waiting.size > 0
end
# Signal to a given task that it should resume operations.
# @parameter value [Object | Nil] The value to return to the waiting fibers.
def signal(value = nil)
return if @waiting.empty?
waiting = self.exchange
waiting.each do |fiber|
Fiber.scheduler.resume(fiber, value) if fiber.alive?
end
return nil
end
protected
def exchange
waiting = @waiting
@waiting = List.new
return waiting
end
end
end
@@ -0,0 +1,42 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2024, by Samuel Williams.
module Async
# Shims for the console gem, redirecting warnings and above to `Kernel#warn`.
#
# If you require this file, the `async` library will not depend on the `console` gem.
#
# That includes any gems that sit within the `Async` namespace.
#
# This is an experimental feature.
module Console
# Log a message at the debug level. The shim is silent.
def self.debug(...)
end
# Log a message at the info level. The shim is silent.
def self.info(...)
end
# Log a message at the warn level. The shim redirects to `Kernel#warn`.
def self.warn(*arguments, exception: nil, **options)
if exception
super(*arguments, exception.full_message, **options)
else
super(*arguments, **options)
end
end
# Log a message at the error level. The shim redirects to `Kernel#warn`.
def self.error(...)
self.warn(...)
end
# Log a message at the fatal level. The shim redirects to `Kernel#warn`.
def self.fatal(...)
self.warn(...)
end
end
end
+59
View File
@@ -0,0 +1,59 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2024, by Samuel Williams.
module Async
# A load balancing mechanism that can be used process work when the system is idle.
class Idler
# Create a new idler.
#
# @public Since *Async v2*.
#
# @parameter maximum_load [Numeric] The maximum load before we start shedding work.
# @parameter backoff [Numeric] The initial backoff time, used for delaying work.
# @parameter parent [Interface(:async) | Nil] The parent task to use for async operations.
def initialize(maximum_load = 0.8, backoff: 0.01, parent: nil)
@maximum_load = maximum_load
@backoff = backoff
@parent = parent
end
# Wait until the system is idle, then execute the given block in a new task.
#
# @asynchronous Executes the given block concurrently.
#
# @parameter arguments [Array] The arguments to pass to the block.
# @parameter parent [Interface(:async) | Nil] The parent task to use for async operations.
# @parameter options [Hash] The options to pass to the task.
# @yields {|task| ...} When the system is idle, the block will be executed in a new task.
def async(*arguments, parent: (@parent or Task.current), **options, &block)
wait
# It is crucial that we optimistically execute the child task, so that we prevent a tight loop invoking this method from consuming all available resources.
parent.async(*arguments, **options, &block)
end
# Wait until the system is idle, according to the maximum load specified.
#
# If the scheduler is overloaded, this method will sleep for an exponentially increasing amount of time.
def wait
scheduler = Fiber.scheduler
backoff = nil
while true
load = scheduler.load
break if load < @maximum_load
if backoff
sleep(backoff)
backoff *= 2.0
else
scheduler.yield
backoff = @backoff
end
end
end
end
end
@@ -0,0 +1,7 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2024, by Samuel Williams.
# The implementation lives in `queue.rb` but later we may move it here for better autoload/inference.
require_relative "queue"
+311
View File
@@ -0,0 +1,311 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2022-2024, by Samuel Williams.
module Async
# A general doublely linked list. This is used internally by {Async::Barrier} and {Async::Condition} to manage child tasks.
class List
# Initialize a new, empty, list.
def initialize
@head = self
@tail = self
@size = 0
end
# @returns [String] A short summary of the list.
def to_s
sprintf("#<%s:0x%x size=%d>", self.class.name, object_id, @size)
end
alias inspect to_s
# Fast, safe, unbounded accumulation of children.
def to_a
items = []
current = self
while current.tail != self
unless current.tail.is_a?(Iterator)
items << current.tail
end
current = current.tail
end
return items
end
# @attribute [Node | Nil] Points at the end of the list.
attr_accessor :head
# @attribute [Node | Nil] Points at the start of the list.
attr_accessor :tail
# @attribute [Integer] The number of nodes in the list.
attr :size
# A callback that is invoked when an item is added to the list.
def added(node)
@size += 1
return node
end
# Append a node to the end of the list.
def append(node)
if node.head
raise ArgumentError, "Node is already in a list!"
end
node.tail = self
@head.tail = node
node.head = @head
@head = node
return added(node)
end
# Prepend a node to the start of the list.
def prepend(node)
if node.head
raise ArgumentError, "Node is already in a list!"
end
node.head = self
@tail.head = node
node.tail = @tail
@tail = node
return added(node)
end
# Add the node, yield, and the remove the node.
# @yields {|node| ...} Yields the node.
# @returns [Object] Returns the result of the block.
def stack(node, &block)
append(node)
return yield(node)
ensure
remove!(node)
end
# A callback that is invoked when an item is removed from the list.
def removed(node)
@size -= 1
return node
end
# Remove the node if it is in a list.
#
# You should be careful to only remove nodes that are part of this list.
#
# @returns [Node] Returns the node if it was removed, otherwise nil.
def remove?(node)
if node.head
return remove!(node)
end
return nil
end
# Remove the node. If it was already removed, this will raise an error.
#
# You should be careful to only remove nodes that are part of this list.
#
# @raises [ArgumentError] If the node is not part of this list.
# @returns [Node] Returns the node if it was removed, otherwise nil.
def remove(node)
# One downside of this interface is we don't actually check if the node is part of the list defined by `self`. This means that there is a potential for a node to be removed from a different list using this method, which in can throw off book-keeping when lists track size, etc.
unless node.head
raise ArgumentError, "Node is not in a list!"
end
remove!(node)
end
private def remove!(node)
node.head.tail = node.tail
node.tail.head = node.head
# This marks the node as being removed, and causes remove to fail if called a 2nd time.
node.head = nil
# node.tail = nil
return removed(node)
end
# @returns [Boolean] Returns true if the list is empty.
def empty?
@size == 0
end
# def validate!(node = nil)
# previous = self
# current = @tail
# found = node.equal?(self)
# while true
# break if current.equal?(self)
# if current.head != previous
# raise "Invalid previous linked list node!"
# end
# if current.is_a?(List) and !current.equal?(self)
# raise "Invalid list in list node!"
# end
# if node
# found ||= current.equal?(node)
# end
# previous = current
# current = current.tail
# end
# if node and !found
# raise "Node not found in list!"
# end
# end
# Iterate over each node in the linked list. It is generally safe to remove the current node, any previous node or any future node during iteration.
#
# @yields {|node| ...} Yields each node in the list.
# @returns [List] Returns self.
def each(&block)
return to_enum unless block_given?
Iterator.each(self, &block)
return self
end
# Determine whether the given node is included in the list.
#
# @parameter needle [Node] The node to search for.
# @returns [Boolean] Returns true if the node is in the list.
def include?(needle)
self.each do |item|
return true if needle.equal?(item)
end
return false
end
# @returns [Node] Returns the first node in the list, if it is not empty.
def first
# validate!
current = @tail
while !current.equal?(self)
if current.is_a?(Iterator)
current = current.tail
else
return current
end
end
return nil
end
# @returns [Node] Returns the last node in the list, if it is not empty.
def last
# validate!
current = @head
while !current.equal?(self)
if current.is_a?(Iterator)
current = current.head
else
return current
end
end
return nil
end
# Shift the first node off the list, if it is not empty.
def shift
if node = first
remove!(node)
end
end
# A linked list Node.
class Node
attr_accessor :head
attr_accessor :tail
alias inspect to_s
end
class Iterator < Node
def initialize(list)
@list = list
# Insert the iterator as the first item in the list:
@tail = list.tail
@tail.head = self
list.tail = self
@head = list
end
def remove!
@head.tail = @tail
@tail.head = @head
@head = nil
@tail = nil
@list = nil
end
def move_next
# Move to the next item (which could be an iterator or the end):
@tail.head = @head
@head.tail = @tail
@head = @tail
@tail = @tail.tail
@head.tail = self
@tail.head = self
end
def move_current
while true
# Are we at the end of the list?
if @tail.equal?(@list)
return nil
end
if @tail.is_a?(Iterator)
move_next
else
return @tail
end
end
end
def each
while current = move_current
yield current
if current.equal?(@tail)
move_next
end
end
end
def self.each(list, &block)
return if list.empty?
iterator = Iterator.new(list)
iterator.each(&block)
ensure
iterator&.remove!
end
end
private_constant :Iterator
end
end
+324
View File
@@ -0,0 +1,324 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2017-2024, by Samuel Williams.
# Copyright, 2017, by Kent Gruber.
# Copyright, 2022, by Shannon Skipper.
require "fiber/annotation"
require_relative "list"
module Async
# A list of children tasks.
class Children < List
# Create an empty list of children tasks.
def initialize
super
@transient_count = 0
end
# Some children may be marked as transient. Transient children do not prevent the parent from finishing.
# @returns [Boolean] Whether the node has transient children.
def transients?
@transient_count > 0
end
# Whether all children are considered finished. Ignores transient children.
def finished?
@size == @transient_count
end
# Whether the children is empty, preserved for compatibility.
def nil?
empty?
end
# Adjust the number of transient children, assuming it has changed.
#
# Despite being public, this is not intended to be called directly. It is used internally by {Node#transient=}.
#
# @parameter transient [Boolean] Whether to increment or decrement the transient count.
def adjust_transient_count(transient)
if transient
@transient_count += 1
else
@transient_count -= 1
end
end
private
def added(node)
if node.transient?
@transient_count += 1
end
return super
end
def removed(node)
if node.transient?
@transient_count -= 1
end
return super
end
end
# A node in a tree, used for implementing the task hierarchy.
class Node
# Create a new node in the tree.
# @parameter parent [Node | Nil] This node will attach to the given parent.
def initialize(parent = nil, annotation: nil, transient: false)
@parent = nil
@children = nil
@annotation = annotation
@object_name = nil
@transient = transient
@head = nil
@tail = nil
if parent
parent.add_child(self)
end
end
# @returns [Node] The root node in the hierarchy.
def root
@parent&.root || self
end
# @private
attr_accessor :head
# @private
attr_accessor :tail
# @attribute [Node] The parent node.
attr :parent
# @attribute [Children | Nil] Optional list of children.
attr :children
# @attribute [String | Nil] A useful identifier for the current node.
attr :annotation
# Whether this node has any children.
# @returns [Boolean]
def children?
@children && !@children.empty?
end
# Represents whether a node is transient. Transient nodes are not considered
# when determining if a node is finished. This is useful for tasks which are
# internal to an object rather than explicit user concurrency. For example,
# a child task which is pruning a connection pool is transient, because it
# is not directly related to the parent task, and should not prevent the
# parent task from finishing.
def transient?
@transient
end
# Change the transient state of the node.
#
# A transient node is not considered when determining if a node is finished, and propagates up if the parent is consumed.
#
# @parameter value [Boolean] Whether the node is transient.
def transient=(value)
if @transient != value
@transient = value
@parent&.children&.adjust_transient_count(value)
end
end
# Annotate the node with a description.
#
# @parameter annotation [String] The description to annotate the node with.
def annotate(annotation)
if block_given?
begin
current_annotation = @annotation
@annotation = annotation
return yield
ensure
@annotation = current_annotation
end
else
@annotation = annotation
end
end
# A description of the node, including the annotation and object name.
#
# @returns [String] The description of the node.
def description
@object_name ||= "#{self.class}:#{format '%#018x', object_id}#{@transient ? ' transient' : nil}"
if annotation = self.annotation
"#{@object_name} #{annotation}"
elsif line = self.backtrace(0, 1)&.first
"#{@object_name} #{line}"
else
@object_name
end
end
# Provides a backtrace for nodes that have an active execution context.
#
# @returns [Array(Thread::Backtrace::Locations) | Nil] The backtrace of the node, if available.
def backtrace(*arguments)
nil
end
# @returns [String] A description of the node.
def to_s
"\#<#{self.description}>"
end
alias inspect to_s
# Change the parent of this node.
#
# @parameter parent [Node | Nil] The parent to attach to, or nil to detach.
# @returns [Node] Itself.
def parent=(parent)
return if @parent.equal?(parent)
if @parent
@parent.remove_child(self)
@parent = nil
end
if parent
parent.add_child(self)
end
return self
end
protected def set_parent(parent)
@parent = parent
end
protected def add_child(child)
@children ||= Children.new
@children.append(child)
child.set_parent(self)
end
protected def remove_child(child)
@children.remove(child)
child.set_parent(nil)
end
# Whether the node can be consumed (deleted) safely. By default, checks if the children set is empty.
#
# @returns [Boolean]
def finished?
@children.nil? || @children.finished?
end
# If the node has a parent, and is {finished?}, then remove this node from
# the parent.
def consume
if parent = @parent and finished?
parent.remove_child(self)
# If we have children, then we need to move them to our the parent if they are not finished:
if @children
while child = @children.shift
if child.finished?
child.set_parent(nil)
else
parent.add_child(child)
end
end
@children = nil
end
parent.consume
end
end
# Traverse the task tree.
#
# @returns [Enumerator] An enumerator which will traverse the tree if no block is given.
# @yields {|node, level| ...} The node and the level relative to the given root.
def traverse(&block)
return enum_for(:traverse) unless block_given?
self.traverse_recurse(&block)
end
protected def traverse_recurse(level = 0, &block)
yield self, level
@children&.each do |child|
child.traverse_recurse(level + 1, &block)
end
end
# Immediately terminate all children tasks, including transient tasks. Internally invokes `stop(false)` on all children. This should be considered a last ditch effort and is used when closing the scheduler.
def terminate
# Attempt to stop the current task immediately, and all children:
stop(false)
# If that doesn't work, take more serious action:
@children&.each do |child|
child.terminate
end
return @children.nil?
end
# Attempt to stop the current node immediately, including all non-transient children. Invokes {#stop_children} to stop all children.
#
# @parameter later [Boolean] Whether to defer stopping until some point in the future.
def stop(later = false)
# The implementation of this method may defer calling `stop_children`.
stop_children(later)
end
# Attempt to stop all non-transient children.
private def stop_children(later = false)
@children&.each do |child|
child.stop(later) unless child.transient?
end
end
# Whether the node has been stopped.
def stopped?
@children.nil?
end
# Print the hierarchy of the task tree from the given node.
#
# @parameter out [IO] The output stream to write to.
# @parameter backtrace [Boolean] Whether to print the backtrace of each node.
def print_hierarchy(out = $stdout, backtrace: true)
self.traverse do |node, level|
indent = "\t" * level
out.puts "#{indent}#{node}"
print_backtrace(out, indent, node) if backtrace
end
end
private
def print_backtrace(out, indent, node)
if backtrace = node.backtrace
backtrace.each_with_index do |line, index|
out.puts "#{indent}#{index.zero? ? "→ " : " "}#{line}"
end
end
end
end
end
@@ -0,0 +1,35 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2018-2024, by Samuel Williams.
require_relative "condition"
module Async
# A synchronization primitive, which allows fibers to wait until a notification is received. Does not block the task which signals the notification. Waiting tasks are resumed on next iteration of the reactor.
# @public Since *Async v1*.
class Notification < Condition
# Signal to a given task that it should resume operations.
def signal(value = nil, task: Task.current)
return if @waiting.empty?
Fiber.scheduler.push Signal.new(self.exchange, value)
return nil
end
Signal = Struct.new(:waiting, :value) do
def alive?
true
end
def transfer
waiting.each do |fiber|
fiber.transfer(value) if fiber.alive?
end
end
end
private_constant :Signal
end
end
+171
View File
@@ -0,0 +1,171 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2018-2024, by Samuel Williams.
# Copyright, 2019, by Ryan Musgrave.
# Copyright, 2020-2022, by Bruno Sutic.
require_relative "notification"
module Async
# A queue which allows items to be processed in order.
#
# It has a compatible interface with {Notification} and {Condition}, except that it's multi-value.
#
# @public Since *Async v1*.
class Queue
# Create a new queue.
#
# @parameter parent [Interface(:async) | Nil] The parent task to use for async operations.
# @parameter available [Notification] The notification to use for signaling when items are available.
def initialize(parent: nil, available: Notification.new)
@items = []
@parent = parent
@available = available
end
# @attribute [Array] The items in the queue.
attr :items
# @returns [Integer] The number of items in the queue.
def size
@items.size
end
# @returns [Boolean] Whether the queue is empty.
def empty?
@items.empty?
end
# Add an item to the queue.
def push(item)
@items << item
@available.signal unless self.empty?
end
# Compatibility with {::Queue#push}.
def <<(item)
self.push(item)
end
# Add multiple items to the queue.
def enqueue(*items)
@items.concat(items)
@available.signal unless self.empty?
end
# Remove and return the next item from the queue.
def dequeue
while @items.empty?
@available.wait
end
@items.shift
end
# Compatibility with {::Queue#pop}.
def pop
self.dequeue
end
# Process each item in the queue.
#
# @asynchronous Executes the given block concurrently for each item.
#
# @parameter arguments [Array] The arguments to pass to the block.
# @parameter parent [Interface(:async) | Nil] The parent task to use for async operations.
# @parameter options [Hash] The options to pass to the task.
# @yields {|task| ...} When the system is idle, the block will be executed in a new task.
def async(parent: (@parent or Task.current), **options, &block)
while item = self.dequeue
parent.async(item, **options, &block)
end
end
# Enumerate each item in the queue.
def each
while item = self.dequeue
yield item
end
end
# Signal the queue with a value, the same as {#enqueue}.
def signal(value = nil)
self.enqueue(value)
end
# Wait for an item to be available, the same as {#dequeue}.
def wait
self.dequeue
end
end
# A queue which limits the number of items that can be enqueued.
# @public Since *Async v1*.
class LimitedQueue < Queue
# Create a new limited queue.
#
# @parameter limit [Integer] The maximum number of items that can be enqueued.
# @parameter full [Notification] The notification to use for signaling when the queue is full.
def initialize(limit = 1, full: Notification.new, **options)
super(**options)
@limit = limit
@full = full
end
# @attribute [Integer] The maximum number of items that can be enqueued.
attr :limit
# @returns [Boolean] Whether trying to enqueue an item would block.
def limited?
@items.size >= @limit
end
# Add an item to the queue.
#
# If the queue is full, this method will block until there is space available.
#
# @parameter item [Object] The item to add to the queue.
def push(item)
while limited?
@full.wait
end
super
end
# Add multiple items to the queue.
#
# If the queue is full, this method will block until there is space available.
#
# @parameter items [Array] The items to add to the queue.
def enqueue(*items)
while !items.empty?
while limited?
@full.wait
end
available = @limit - @items.size
@items.concat(items.shift(available))
@available.signal unless self.empty?
end
end
# Remove and return the next item from the queue.
#
# If the queue is empty, this method will block until an item is available.
#
# @returns [Object] The next item in the queue.
def dequeue
item = super
@full.signal
return item
end
end
end
@@ -0,0 +1,32 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2017-2024, by Samuel Williams.
# Copyright, 2017, by Kent Gruber.
# Copyright, 2018, by Sokolov Yura.
require_relative "scheduler"
module Async
# A wrapper around the the scheduler which binds it to the current thread automatically.
class Reactor < Scheduler
# @deprecated Replaced by {Kernel::Async}.
def self.run(...)
Async(...)
end
# Initialize the reactor and assign it to the current Fiber scheduler.
def initialize(...)
super
Fiber.set_scheduler(self)
end
# Close the reactor and remove it from the current Fiber scheduler.
def scheduler_close
self.close
end
public :sleep
end
end
@@ -0,0 +1,582 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2020-2024, by Samuel Williams.
# Copyright, 2020, by Jun Jiang.
# Copyright, 2021, by Julien Portalier.
require_relative "clock"
require_relative "task"
require_relative "worker_pool"
require "io/event"
require "console"
require "resolv"
module Async
begin
require "fiber/profiler"
Profiler = Fiber::Profiler
rescue LoadError
# Fiber::Profiler is not available.
Profiler = nil
end
# Handles scheduling of fibers. Implements the fiber scheduler interface.
class Scheduler < Node
WORKER_POOL = ENV.fetch("ASYNC_SCHEDULER_WORKER_POOL", nil).then do |value|
value == "true" ? true : nil
end
# Raised when an operation is attempted on a closed scheduler.
class ClosedError < RuntimeError
# Create a new error.
#
# @parameter message [String] The error message.
def initialize(message = "Scheduler is closed!")
super
end
end
# Whether the fiber scheduler is supported.
# @public Since *Async v1*.
def self.supported?
true
end
# Create a new scheduler.
#
# @public Since *Async v1*.
# @parameter parent [Node | Nil] The parent node to use for task hierarchy.
# @parameter selector [IO::Event::Selector] The selector to use for event handling.
def initialize(parent = nil, selector: nil, profiler: Profiler&.default, worker_pool: WORKER_POOL)
super(parent)
@selector = selector || ::IO::Event::Selector.new(Fiber.current)
@profiler = profiler
@interrupted = false
@blocked = 0
@busy_time = 0.0
@idle_time = 0.0
@timers = ::IO::Event::Timers.new
if worker_pool == true
@worker_pool = WorkerPool.new
else
@worker_pool = worker_pool
end
if @worker_pool
self.singleton_class.prepend(WorkerPool::BlockingOperationWait)
end
end
# Compute the scheduler load according to the busy and idle times that are updated by the run loop.
#
# @returns [Float] The load of the scheduler. 0.0 means no load, 1.0 means fully loaded or over-loaded.
def load
total_time = @busy_time + @idle_time
# If the total time is zero, then the load is zero:
return 0.0 if total_time.zero?
# We normalize to a 1 second window:
if total_time > 1.0
ratio = 1.0 / total_time
@busy_time *= ratio
@idle_time *= ratio
# We don't need to divide here as we've already normalised it to a 1s window:
return @busy_time
else
return @busy_time / total_time
end
end
# Invoked when the fiber scheduler is being closed.
#
# Executes the run loop until all tasks are finished, then closes the scheduler.
def scheduler_close(error = $!)
# If the execution context (thread) was handling an exception, we want to exit as quickly as possible:
unless error
self.run
end
ensure
self.close
end
# Terminate all child tasks.
def terminate
# If that doesn't work, take more serious action:
@children&.each do |child|
child.terminate
end
return @children.nil?
end
# Terminate all child tasks and close the scheduler.
# @public Since *Async v1*.
def close
self.run_loop do
until self.terminate
self.run_once!
end
end
Kernel.raise "Closing scheduler with blocked operations!" if @blocked > 0
ensure
# We want `@selector = nil` to be a visible side effect from this point forward, specifically in `#interrupt` and `#unblock`. If the selector is closed, then we don't want to push any fibers to it.
selector = @selector
@selector = nil
selector&.close
worker_pool = @worker_pool
@worker_pool = nil
worker_pool&.close
consume
end
# @returns [Boolean] Whether the scheduler has been closed.
# @public Since *Async v1*.
def closed?
@selector.nil?
end
# @returns [String] A description of the scheduler.
def to_s
"\#<#{self.description} #{@children&.size || 0} children (#{stopped? ? 'stopped' : 'running'})>"
end
# Interrupt the event loop and cause it to exit.
# @asynchronous May be called from any thread.
def interrupt
@interrupted = true
@selector&.wakeup
end
# Transfer from the calling fiber to the event loop.
def transfer
@selector.transfer
end
# Yield the current fiber and resume it on the next iteration of the event loop.
def yield
@selector.yield
end
# Schedule a fiber (or equivalent object) to be resumed on the next loop through the reactor.
# @parameter fiber [Fiber | Object] The object to be resumed on the next iteration of the run-loop.
def push(fiber)
@selector.push(fiber)
end
# Raise an exception on a specified fiber with the given arguments.
#
# This internally schedules the current fiber to be ready, before raising the exception, so that it will later resume execution.
#
# @parameter fiber [Fiber] The fiber to raise the exception on.
# @parameter *arguments [Array] The arguments to pass to the fiber.
def raise(...)
@selector.raise(...)
end
# Resume execution of the specified fiber.
#
# @parameter fiber [Fiber] The fiber to resume.
# @parameter arguments [Array] The arguments to pass to the fiber.
def resume(fiber, *arguments)
@selector.resume(fiber, *arguments)
end
# Invoked when a fiber tries to perform a blocking operation which cannot continue. A corresponding call {unblock} must be performed to allow this fiber to continue.
#
# @public Since *Async v2*.
# @asynchronous May only be called on same thread as fiber scheduler.
#
# @parameter blocker [Object] The object that is blocking the fiber.
# @parameter timeout [Float | Nil] The maximum time to block, or if nil, indefinitely.
def block(blocker, timeout)
# $stderr.puts "block(#{blocker}, #{Fiber.current}, #{timeout})"
fiber = Fiber.current
if timeout
timer = @timers.after(timeout) do
if fiber.alive?
fiber.transfer(false)
end
end
end
begin
@blocked += 1
@selector.transfer
ensure
@blocked -= 1
end
ensure
timer&.cancel!
end
# Unblock a fiber that was previously blocked.
#
# @public Since *Async v2* and *Ruby v3.1*.
# @asynchronous May be called from any thread.
#
# @parameter blocker [Object] The object that was blocking the fiber.
# @parameter fiber [Fiber] The fiber to unblock.
def unblock(blocker, fiber)
# $stderr.puts "unblock(#{blocker}, #{fiber})"
# This operation is protected by the GVL:
if selector = @selector
selector.push(fiber)
selector.wakeup
end
end
# Sleep for the specified duration.
#
# @public Since *Async v2* and *Ruby v3.1*.
# @asynchronous May be non-blocking.
#
# @parameter duration [Numeric | Nil] The time in seconds to sleep, or if nil, indefinitely.
def kernel_sleep(duration = nil)
if duration
self.block(nil, duration)
else
self.transfer
end
end
# Resolve the address of the given hostname.
#
# @public Since *Async v2*.
# @asynchronous May be non-blocking.
#
# @parameter hostname [String] The hostname to resolve.
def address_resolve(hostname)
# On some platforms, hostnames may contain a device-specific suffix (e.g. %en0). We need to strip this before resolving.
# See <https://github.com/socketry/async/issues/180> for more details.
hostname = hostname.split("%", 2).first
::Resolv.getaddresses(hostname)
end
if IO.method_defined?(:timeout)
private def get_timeout(io)
io.timeout
end
else
private def get_timeout(io)
nil
end
end
# Wait for the specified IO to become ready for the specified events.
#
# @public Since *Async v2*.
# @asynchronous May be non-blocking.
#
# @parameter io [IO] The IO object to wait on.
# @parameter events [Integer] The events to wait for, e.g. `IO::READABLE`, `IO::WRITABLE`, etc.
# @parameter timeout [Float | Nil] The maximum time to wait, or if nil, indefinitely.
def io_wait(io, events, timeout = nil)
fiber = Fiber.current
if timeout
# If an explicit timeout is specified, we expect that the user will handle it themselves:
timer = @timers.after(timeout) do
fiber.transfer
end
elsif timeout = get_timeout(io)
# Otherwise, if we default to the io's timeout, we raise an exception:
timer = @timers.after(timeout) do
fiber.raise(::IO::TimeoutError, "Timeout (#{timeout}s) while waiting for IO to become ready!")
end
end
return @selector.io_wait(fiber, io, events)
ensure
timer&.cancel!
end
if ::IO::Event::Support.buffer?
# Read from the specified IO into the buffer.
#
# @public Since *Async v2* and Ruby with `IO::Buffer` support.
# @asynchronous May be non-blocking.
#
# @parameter io [IO] The IO object to read from.
# @parameter buffer [IO::Buffer] The buffer to read into.
# @parameter length [Integer] The minimum number of bytes to read.
# @parameter offset [Integer] The offset within the buffer to read into.
def io_read(io, buffer, length, offset = 0)
fiber = Fiber.current
if timeout = get_timeout(io)
timer = @timers.after(timeout) do
fiber.raise(::IO::TimeoutError, "Timeout (#{timeout}s) while waiting for IO to become readable!")
end
end
@selector.io_read(fiber, io, buffer, length, offset)
ensure
timer&.cancel!
end
if RUBY_ENGINE != "ruby" || RUBY_VERSION >= "3.3.1"
# Write the specified buffer to the IO.
#
# @public Since *Async v2* and *Ruby v3.3.1* with `IO::Buffer` support.
# @asynchronous May be non-blocking.
#
# @parameter io [IO] The IO object to write to.
# @parameter buffer [IO::Buffer] The buffer to write from.
# @parameter length [Integer] The minimum number of bytes to write.
# @parameter offset [Integer] The offset within the buffer to write from.
def io_write(io, buffer, length, offset = 0)
fiber = Fiber.current
if timeout = get_timeout(io)
timer = @timers.after(timeout) do
fiber.raise(::IO::TimeoutError, "Timeout (#{timeout}s) while waiting for IO to become writable!")
end
end
@selector.io_write(fiber, io, buffer, length, offset)
ensure
timer&.cancel!
end
end
end
# Wait for the specified process ID to exit.
#
# @public Since *Async v2*.
# @asynchronous May be non-blocking.
#
# @parameter pid [Integer] The process ID to wait for.
# @parameter flags [Integer] A bit-mask of flags suitable for `Process::Status.wait`.
# @returns [Process::Status] A process status instance.
# @asynchronous May be non-blocking..
def process_wait(pid, flags)
return @selector.process_wait(Fiber.current, pid, flags)
end
# Run one iteration of the event loop.
#
# When terminating the event loop, we already know we are finished. So we don't need to check the task tree. This is a logical requirement because `run_once` ignores transient tasks. For example, a single top level transient task is not enough to keep the reactor running, but during termination we must still process it in order to terminate child tasks.
#
# @parameter timeout [Float | Nil] The maximum timeout, or if nil, indefinite.
# @returns [Boolean] Whether there is more work to do.
private def run_once!(timeout = nil)
start_time = Async::Clock.now
interval = @timers.wait_interval
# If there is no interval to wait (thus no timers), and no tasks, we could be done:
if interval.nil?
# Allow the user to specify a maximum interval if we would otherwise be sleeping indefinitely:
interval = timeout
elsif interval < 0
# We have timers ready to fire, don't sleep in the selctor:
interval = 0
elsif timeout and interval > timeout
interval = timeout
end
begin
@selector.select(interval)
rescue Errno::EINTR
# Ignore.
end
@timers.fire
# Compute load:
end_time = Async::Clock.now
total_duration = end_time - start_time
idle_duration = @selector.idle_duration
busy_duration = total_duration - idle_duration
@busy_time += busy_duration
@idle_time += idle_duration
# The reactor still has work to do:
return true
end
# Run one iteration of the event loop.
#
# @public Since *Async v1*.
# @asynchronous Must be invoked from blocking (root) fiber.
#
# @parameter timeout [Float | Nil] The maximum timeout, or if nil, indefinite.
# @returns [Boolean] Whether there is more work to do.
def run_once(timeout = nil)
Kernel.raise "Running scheduler on non-blocking fiber!" unless Fiber.blocking?
if self.finished?
self.stop
end
# If we are finished, we stop the task tree and exit:
if @children.nil?
return false
end
return run_once!(timeout)
end
# Checks and clears the interrupted state of the scheduler.
#
# @returns [Boolean] Whether the reactor has been interrupted.
private def interrupted?
if @interrupted
@interrupted = false
return true
end
if Thread.pending_interrupt?
return true
end
return false
end
# Stop all children, including transient children.
#
# @public Since *Async v1*.
def stop
@children&.each do |child|
child.stop
end
end
private def run_loop(&block)
interrupt = nil
begin
# In theory, we could use Exception here to be a little bit safer, but we've only shown the case for SignalException to be a problem, so let's not over-engineer this.
Thread.handle_interrupt(::SignalException => :never) do
until self.interrupted?
# If we are finished, we need to exit:
break unless yield
end
end
rescue Interrupt => interrupt
# If an interrupt did occur during an iteration of the event loop, we need to handle it. More specifically, `self.stop` is not safe to interrupt without potentially corrupting the task tree.
Thread.handle_interrupt(::SignalException => :never) do
Console.debug(self) do |buffer|
buffer.puts "Scheduler interrupted: #{interrupt.inspect}"
self.print_hierarchy(buffer)
end
self.stop
end
retry
end
# If the event loop was interrupted, and we finished exiting normally (due to the interrupt), we need to re-raise the interrupt so that the caller can handle it too.
if interrupt
Kernel.raise(interrupt)
end
end
# Run the reactor until all tasks are finished. Proxies arguments to {#async} immediately before entering the loop, if a block is provided.
#
# Forwards all parameters to {#async} if a block is given.
#
# @public Since *Async v1*.
#
# @yields {|task| ...} The top level task, if a block is given.
# @returns [Task] The initial task that was scheduled into the reactor.
def run(...)
Kernel.raise ClosedError if @selector.nil?
begin
@profiler&.start
initial_task = self.async(...) if block_given?
self.run_loop do
run_once
end
return initial_task
ensure
@profiler&.stop
end
end
# Start an asynchronous task within the specified reactor. The task will be executed until the first blocking call, at which point it will yield and and this method will return.
#
# @public Since *Async v1*.
# @asynchronous May context switch immediately to new task.
# @deprecated Use {#run} or {Task#async} instead.
#
# @yields {|task| ...} Executed within the task.
# @returns [Task] The task that was scheduled into the reactor.
def async(*arguments, **options, &block)
# warn "Async::Scheduler#async is deprecated. Use `run` or `Task#async` instead.", uplevel: 1, category: :deprecated
Kernel.raise ClosedError if @selector.nil?
task = Task.new(Task.current? || self, **options, &block)
task.run(*arguments)
return task
end
def fiber(...)
return async(...).fiber
end
# Invoke the block, but after the specified timeout, raise {TimeoutError} in any currenly blocking operation. If the block runs to completion before the timeout occurs or there are no non-blocking operations after the timeout expires, the code will complete without any exception.
#
# @public Since *Async v1*.
# @asynchronous May raise an exception at any interruption point (e.g. blocking operations).
#
# @parameter duration [Numeric] The time in seconds, in which the task should complete.
# @parameter exception [Class] The exception class to raise.
# @parameter message [String] The message to pass to the exception.
# @yields {|duration| ...} The block to execute with a timeout.
def with_timeout(duration, exception = TimeoutError, message = "execution expired", &block)
fiber = Fiber.current
timer = @timers.after(duration) do
if fiber.alive?
fiber.raise(exception, message)
end
end
yield timer
ensure
timer&.cancel!
end
# Invoke the block, but after the specified timeout, raise the specified exception with the given message. If the block runs to completion before the timeout occurs or there are no non-blocking operations after the timeout expires, the code will complete without any exception.
#
# @public Since *Async v1* and *Ruby v3.1*. May be invoked from `Timeout.timeout`.
# @asynchronous May raise an exception at any interruption point (e.g. blocking operations).
#
# @parameter duration [Numeric] The time in seconds, in which the task should complete.
# @parameter exception [Class] The exception class to raise.
# @parameter message [String] The message to pass to the exception.
# @yields {|duration| ...} The block to execute with a timeout.
def timeout_after(duration, exception, message, &block)
with_timeout(duration, exception, message) do |timer|
yield duration
end
end
end
end
@@ -0,0 +1,41 @@
A synchronization primitive, which limits access to a given resource, such as a limited number of database connections, open files, or network connections.
## Example
~~~ ruby
require 'async'
require 'async/semaphore'
require 'net/http'
Sync do
# Only allow two concurrent tasks at a time:
semaphore = Async::Semaphore.new(2)
# Generate an array of 10 numbers:
terms = ['ruby', 'python', 'go', 'java', 'c++']
# Search for the terms:
terms.each do |term|
semaphore.async do |task|
Console.info("Searching for #{term}...")
response = Net::HTTP.get(URI "https://www.google.com/search?q=#{term}")
Console.info("Got response #{response.size} bytes.")
end
end
end
~~~
### Output
~~~
0.0s info: Searching for ruby... [ec=0x3c] [pid=50523]
0.04s info: Searching for python... [ec=0x21c] [pid=50523]
1.7s info: Got response 182435 bytes. [ec=0x3c] [pid=50523]
1.71s info: Searching for go... [ec=0x834] [pid=50523]
3.0s info: Got response 204854 bytes. [ec=0x21c] [pid=50523]
3.0s info: Searching for java... [ec=0xf64] [pid=50523]
4.32s info: Got response 103235 bytes. [ec=0x834] [pid=50523]
4.32s info: Searching for c++... [ec=0x12d4] [pid=50523]
4.65s info: Got response 109697 bytes. [ec=0xf64] [pid=50523]
6.64s info: Got response 87249 bytes. [ec=0x12d4] [pid=50523]
~~~
@@ -0,0 +1,127 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2018-2024, by Samuel Williams.
require_relative "list"
module Async
# A synchronization primitive, which limits access to a given resource.
# @public Since *Async v1*.
class Semaphore
# @parameter limit [Integer] The maximum number of times the semaphore can be acquired before it blocks.
# @parameter parent [Task | Semaphore | Nil] The parent for holding any children tasks.
def initialize(limit = 1, parent: nil)
@count = 0
@limit = limit
@waiting = List.new
@parent = parent
end
# The current number of tasks that have acquired the semaphore.
attr :count
# The maximum number of tasks that can acquire the semaphore.
attr :limit
# The tasks waiting on this semaphore.
attr :waiting
# Allow setting the limit. This is useful for cases where the semaphore is used to limit the number of concurrent tasks, but the number of tasks is not known in advance or needs to be modified.
#
# On increasing the limit, some tasks may be immediately resumed. On decreasing the limit, some tasks may execute until the count is < than the limit.
#
# @parameter limit [Integer] The new limit.
def limit= limit
difference = limit - @limit
@limit = limit
# We can't suspend
if difference > 0
difference.times do
break unless node = @waiting.first
node.resume
end
end
end
# Is the semaphore currently acquired?
def empty?
@count.zero?
end
# Whether trying to acquire this semaphore would block.
def blocking?
@count >= @limit
end
# Run an async task. Will wait until the semaphore is ready until spawning and running the task.
def async(*arguments, parent: (@parent or Task.current), **options)
wait
parent.async(**options) do |task|
@count += 1
begin
yield task, *arguments
ensure
self.release
end
end
end
# Acquire the semaphore, block if we are at the limit.
# If no block is provided, you must call release manually.
# @yields {...} When the semaphore can be acquired.
# @returns The result of the block if invoked.
def acquire
wait
@count += 1
return unless block_given?
begin
return yield
ensure
self.release
end
end
# Release the semaphore. Must match up with a corresponding call to `acquire`. Will release waiting fibers in FIFO order.
def release
@count -= 1
while (@limit - @count) > 0 and node = @waiting.first
node.resume
end
end
private
class FiberNode < List::Node
def initialize(fiber)
@fiber = fiber
end
def resume
if @fiber.alive?
Fiber.scheduler.resume(@fiber)
end
end
end
private_constant :FiberNode
# Wait until the semaphore becomes available.
def wait
return unless blocking?
@waiting.stack(FiberNode.new(Fiber.current)) do
Fiber.scheduler.transfer while blocking?
end
end
end
end
+30
View File
@@ -0,0 +1,30 @@
A sequence of instructions, defined by a block, which is executed sequentially and managed by the scheduler. A task can be in one of the following states: `initialized`, `running`, `completed`, `failed`, `cancelled` or `stopped`.
```mermaid
stateDiagram-v2
[*] --> Initialized
Initialized --> Running : Run
Running --> Completed : Return Value
Running --> Failed : Exception
Completed --> [*]
Failed --> [*]
Running --> Stopped : Stop
Stopped --> [*]
Completed --> Stopped : Stop
Failed --> Stopped : Stop
Initialized --> Stopped : Stop
```
## Example
```ruby
require 'async'
# Create an asynchronous task that sleeps for 1 second:
Async do |task|
sleep(1)
end
```
+459
View File
@@ -0,0 +1,459 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2017-2024, by Samuel Williams.
# Copyright, 2017, by Kent Gruber.
# Copyright, 2017, by Devin Christensen.
# Copyright, 2020, by Patrik Wenger.
# Copyright, 2023, by Math Ieu.
require "fiber"
require "console"
require_relative "node"
require_relative "condition"
Fiber.attr_accessor :async_task
module Async
# Raised when a task is explicitly stopped.
class Stop < Exception
# Used to defer stopping the current task until later.
class Later
# Create a new stop later operation.
#
# @parameter task [Task] The task to stop later.
def initialize(task)
@task = task
end
# @returns [Boolean] Whether the task is alive.
def alive?
true
end
# Transfer control to the operation - this will stop the task.
def transfer
@task.stop
end
end
end
# Raised if a timeout occurs on a specific Fiber. Handled gracefully by `Task`.
# @public Since *Async v1*.
class TimeoutError < StandardError
# Create a new timeout error.
#
# @parameter message [String] The error message.
def initialize(message = "execution expired")
super
end
end
# @public Since *Async v1*.
class Task < Node
# Raised when a child task is created within a task that has finished execution.
class FinishedError < RuntimeError
# Create a new finished error.
#
# @parameter message [String] The error message.
def initialize(message = "Cannot create child task within a task that has finished execution!")
super
end
end
# @deprecated With no replacement.
def self.yield
Fiber.scheduler.transfer
end
# Run the given block of code in a task, asynchronously, in the given scheduler.
def self.run(scheduler, *arguments, **options, &block)
self.new(scheduler, **options, &block).tap do |task|
task.run(*arguments)
end
end
# Create a new task.
# @parameter reactor [Reactor] the reactor this task will run within.
# @parameter parent [Task] the parent task.
def initialize(parent = Task.current?, finished: nil, **options, &block)
super(parent, **options)
# These instance variables are critical to the state of the task.
# In the initialized state, the @block should be set, but the @fiber should be nil.
# In the running state, the @fiber should be set.
# In a finished state, the @block should be nil, and the @fiber should be nil.
@block = block
@fiber = nil
@status = :initialized
@result = nil
@finished = finished
@defer_stop = nil
end
# @returns [Scheduler] The scheduler for this task.
def reactor
self.root
end
# @returns [Array(Thread::Backtrace::Location) | Nil] The backtrace of the task, if available.
def backtrace(*arguments)
@fiber&.backtrace(*arguments)
end
# Annotate the task with a description.
#
# This will internally try to annotate the fiber if it is running, otherwise it will annotate the task itself.
#
# @parameter annotation [String] The description to annotate the task with.
def annotate(annotation, &block)
if @fiber
@fiber.annotate(annotation, &block)
else
super
end
end
# @returns [Object] The annotation of the task.
def annotation
if @fiber
@fiber.annotation
else
super
end
end
# @returns [String] A description of the task and it's current status.
def to_s
"\#<#{self.description} (#{@status})>"
end
# @deprecated Prefer {Kernel#sleep} except when compatibility with `stable-v1` is required.
def sleep(duration = nil)
super
end
# Execute the given block of code, raising the specified exception if it exceeds the given duration during a non-blocking operation.
def with_timeout(duration, exception = TimeoutError, message = "execution expired", &block)
Fiber.scheduler.with_timeout(duration, exception, message, &block)
end
# Yield back to the reactor and allow other fibers to execute.
def yield
Fiber.scheduler.yield
end
# @attribute [Fiber] The fiber which is being used for the execution of this task.
attr :fiber
# @returns [Boolean] Whether the internal fiber is alive, i.e. it is actively executing.
def alive?
@fiber&.alive?
end
# Whether we can remove this node from the reactor graph.
# @returns [Boolean]
def finished?
# If the block is nil and the fiber is nil, it means the task has finished execution. This becomes true after `finish!` is called.
super && @block.nil? && @fiber.nil?
end
# @returns [Boolean] Whether the task is running.
def running?
@status == :running
end
# @returns [Boolean] Whether the task failed with an exception.
def failed?
@status == :failed
end
# @returns [Boolean] Whether the task has been stopped.
def stopped?
@status == :stopped
end
# @returns [Boolean] Whether the task has completed execution and generated a result.
def completed?
@status == :completed
end
# Alias for {#completed?}.
def complete?
self.completed?
end
# @attribute [Symbol] The status of the execution of the task, one of `:initialized`, `:running`, `:complete`, `:stopped` or `:failed`.
attr :status
# Begin the execution of the task.
#
# @raises [RuntimeError] If the task is already running.
def run(*arguments)
if @status == :initialized
@status = :running
schedule do
@block.call(self, *arguments)
rescue => error
# I'm not completely happy with this overhead, but the alternative is to not log anything which makes debugging extremely difficult. Maybe we can introduce a debug wrapper which adds extra logging.
if @finished.nil?
warn(self, "Task may have ended with unhandled exception.", exception: error)
end
raise
end
else
raise RuntimeError, "Task already running!"
end
end
# Run an asynchronous task as a child of the current task.
#
# @public Since *Async v1*.
# @asynchronous May context switch immediately to the new task.
#
# @yields {|task| ...} in the context of the new task.
# @raises [FinishedError] If the task has already finished.
# @returns [Task] The child task.
def async(*arguments, **options, &block)
raise FinishedError if self.finished?
task = Task.new(self, **options, &block)
# When calling an async block, we deterministically execute it until the first blocking operation. We don't *have* to do this - we could schedule it for later execution, but it's useful to:
#
# - Fail at the point of the method call where possible.
# - Execute determinstically where possible.
# - Avoid scheduler overhead if no blocking operation is performed.
#
# There are different strategies (greedy vs non-greedy). We are currently using a greedy strategy.
task.run(*arguments)
return task
end
# Retrieve the current result of the task. Will cause the caller to wait until result is available. If the task resulted in an unhandled error (derived from `StandardError`), this will be raised. If the task was stopped, this will return `nil`.
#
# Conceptually speaking, waiting on a task should return a result, and if it throws an exception, this is certainly an exceptional case that should represent a failure in your program, not an expected outcome. In other words, you should not design your programs to expect exceptions from `#wait` as a normal flow control, and prefer to catch known exceptions within the task itself and return a result that captures the intention of the failure, e.g. a `TimeoutError` might simply return `nil` or `false` to indicate that the operation did not generate a valid result (as a timeout was an expected outcome of the internal operation in this case).
#
# @raises [RuntimeError] If the task's fiber is the current fiber.
# @returns [Object] The final expression/result of the task's block.
def wait
raise "Cannot wait on own fiber!" if Fiber.current.equal?(@fiber)
# `finish!` will set both of these to nil before signaling the condition:
if @block || @fiber
@finished ||= Condition.new
@finished.wait
end
if @status == :failed
raise @result
else
return @result
end
end
# Access the result of the task without waiting. May be nil if the task is not completed. Does not raise exceptions.
attr :result
# Stop the task and all of its children.
#
# If `later` is false, it means that `stop` has been invoked directly. When `later` is true, it means that `stop` is invoked by `stop_children` or some other indirect mechanism. In that case, if we encounter the "current" fiber, we can't stop it right away, as it's currently performing `#stop`. Stopping it immediately would interrupt the current stop traversal, so we need to schedule the stop to occur later.
#
# @parameter later [Boolean] Whether to stop the task later, or immediately.
def stop(later = false)
if self.stopped?
# If the task is already stopped, a `stop` state transition re-enters the same state which is a no-op. However, we will also attempt to stop any running children too. This can happen if the children did not stop correctly the first time around. Doing this should probably be considered a bug, but it's better to be safe than sorry.
return stopped!
end
# If the fiber is alive, we need to stop it:
if @fiber&.alive?
# As the task is now exiting, we want to ensure the event loop continues to execute until the task finishes.
self.transient = false
# If we are deferring stop...
if @defer_stop == false
# Don't stop now... but update the state so we know we need to stop later.
@defer_stop = true
return false
end
if self.current?
# If the fiber is current, and later is `true`, we need to schedule the fiber to be stopped later, as it's currently invoking `stop`:
if later
# If the fiber is the current fiber and we want to stop it later, schedule it:
Fiber.scheduler.push(Stop::Later.new(self))
else
# Otherwise, raise the exception directly:
raise Stop, "Stopping current task!"
end
else
# If the fiber is not curent, we can raise the exception directly:
begin
# There is a chance that this will stop the fiber that originally called stop. If that happens, the exception handling in `#stopped` will rescue the exception and re-raise it later.
Fiber.scheduler.raise(@fiber, Stop)
rescue FiberError => error
# In some cases, this can cause a FiberError (it might be resumed already), so we schedule it to be stopped later:
Fiber.scheduler.push(Stop::Later.new(self))
end
end
else
# We are not running, but children might be, so transition directly into stopped state:
stop!
end
end
# Defer the handling of stop. During the execution of the given block, if a stop is requested, it will be deferred until the block exits. This is useful for ensuring graceful shutdown of servers and other long-running tasks. You should wrap the response handling code in a defer_stop block to ensure that the task is stopped when the response is complete but not before.
#
# You can nest calls to defer_stop, but the stop will only be deferred until the outermost block exits.
#
# If stop is invoked a second time, it will be immediately executed.
#
# @yields {} The block of code to execute.
# @public Since *Async v1*.
def defer_stop
# Tri-state variable for controlling stop:
# - nil: defer_stop has not been called.
# - false: defer_stop has been called and we are not stopping.
# - true: defer_stop has been called and we will stop when exiting the block.
if @defer_stop.nil?
begin
# If we are not deferring stop already, we can defer it now:
@defer_stop = false
yield
rescue Stop
# If we are exiting due to a stop, we shouldn't try to invoke stop again:
@defer_stop = nil
raise
ensure
defer_stop = @defer_stop
# We need to ensure the state is reset before we exit the block:
@defer_stop = nil
# If we were asked to stop, we should do so now:
if defer_stop
raise Stop, "Stopping current task (was deferred)!"
end
end
else
# If we are deferring stop already, entering it again is a no-op.
yield
end
end
# @returns [Boolean] Whether stop has been deferred.
def stop_deferred?
@defer_stop
end
# Lookup the {Task} for the current fiber. Raise `RuntimeError` if none is available.
# @returns [Task]
# @raises[RuntimeError] If task was not {set!} for the current fiber.
def self.current
Fiber.current.async_task or raise RuntimeError, "No async task available!"
end
# Check if there is a task defined for the current fiber.
# @returns [Interface(:async) | Nil]
def self.current?
Fiber.current.async_task
end
# @returns [Boolean] Whether this task is the currently executing task.
def current?
Fiber.current.equal?(@fiber)
end
private
def warn(...)
Console.warn(...)
end
# Finish the current task, moving any children to the parent.
def finish!
# Don't hold references to the fiber or block after the task has finished:
@fiber = nil
@block = nil # If some how we went directly from initialized to finished.
# Attempt to remove this node from the task tree.
consume
# If this task was being used as a future, signal completion here:
if @finished
@finished.signal(self)
@finished = nil
end
end
# State transition into the completed state.
def completed!(result)
@result = result
@status = :completed
end
# State transition into the failed state.
def failed!(exception = false)
@result = exception
@status = :failed
end
def stopped!
# Console.info(self, status:) {"Task #{self} was stopped with #{@children&.size.inspect} children!"}
@status = :stopped
stopped = false
begin
# We are not running, but children might be so we should stop them:
stop_children(true)
rescue Stop
stopped = true
# If we are stopping children, and one of them tries to stop the current task, we should ignore it. We will be stopped later.
retry
end
if stopped
raise Stop, "Stopping current task!"
end
end
def stop!
stopped!
finish!
end
def schedule(&block)
@fiber = Fiber.new(annotation: self.annotation) do
begin
completed!(yield)
rescue Stop
stopped!
rescue StandardError => error
failed!(error)
rescue Exception => exception
failed!(exception)
# This is a critical failure, we should stop the reactor:
raise
ensure
# Console.info(self) {"Task ensure $! = #{$!} with #{@children&.size.inspect} children!"}
finish!
end
end
@fiber.async_task = self
self.root.resume(@fiber)
end
end
end
@@ -0,0 +1,59 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2021-2024, by Samuel Williams.
require_relative "condition"
module Async
# A synchronization primitive that allows one task to wait for another task to resolve a value.
class Variable
# Create a new variable.
#
# @parameter condition [Condition] The condition to use for synchronization.
def initialize(condition = Condition.new)
@condition = condition
@value = nil
end
# Resolve the value.
#
# Signals all waiting tasks.
#
# @parameter value [Object] The value to resolve.
def resolve(value = true)
@value = value
condition = @condition
@condition = nil
self.freeze
condition.signal(value)
end
# Alias for {#resolve}.
def value=(value)
self.resolve(value)
end
# Whether the value has been resolved.
#
# @returns [Boolean] Whether the value has been resolved.
def resolved?
@condition.nil?
end
# Wait for the value to be resolved.
#
# @returns [Object] The resolved value.
def wait
@condition&.wait
return @value
end
# Alias for {#wait}.
def value
self.wait
end
end
end
@@ -0,0 +1,8 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2017-2024, by Samuel Williams.
module Async
VERSION = "2.23.0"
end
+50
View File
@@ -0,0 +1,50 @@
A synchronization primitive, which allows you to wait for tasks to complete in order of completion. This is useful for implementing a task pool, where you want to wait for the first task to complete, and then cancel the rest.
If you try to wait for more things than you have added, you will deadlock.
## Example
~~~ ruby
require 'async'
require 'async/semaphore'
require 'async/barrier'
require 'async/waiter'
Sync do
barrier = Async::Barrier.new
waiter = Async::Waiter.new(parent: barrier)
semaphore = Async::Semaphore.new(2, parent: waiter)
# Sleep sort the numbers:
generator = Async do
while true
semaphore.async do |task|
number = rand(1..10)
sleep(number)
end
end
end
numbers = []
4.times do
# Wait for all the numbers to be sorted:
numbers << waiter.wait
end
# Don't generate any more numbers:
generator.stop
# Stop all tasks which we don't care about:
barrier.stop
Console.info("Smallest", numbers)
end
~~~
### Output
~~~
0.0s info: Smallest
| [3, 3, 1, 2]
~~~
+56
View File
@@ -0,0 +1,56 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2022-2024, by Samuel Williams.
# Copyright, 2024, by Patrik Wenger.
module Async
# A composable synchronization primitive, which allows one task to wait for a number of other tasks to complete. It can be used in conjunction with {Semaphore} and/or {Barrier}.
class Waiter
# Create a waiter instance.
#
# @parameter parent [Interface(:async) | Nil] The parent task to use for asynchronous operations.
# @parameter finished [Async::Condition] The condition to signal when a task completes.
def initialize(parent: nil, finished: Async::Condition.new)
@finished = finished
@done = []
@parent = parent
end
# Execute a child task and add it to the waiter.
# @asynchronous Executes the given block concurrently.
def async(parent: (@parent or Task.current), **options, &block)
parent.async(**options) do |task|
yield(task)
ensure
@done << task
@finished.signal
end
end
# Wait for the first `count` tasks to complete.
# @parameter count [Integer | Nil] The number of tasks to wait for.
# @returns [Array(Async::Task)] If an integer is given, the tasks which have completed.
# @returns [Async::Task] Otherwise, the first task to complete.
def first(count = nil)
minimum = count || 1
while @done.size < minimum
@finished.wait
end
return @done.shift(*count)
end
# Wait for the first `count` tasks to complete.
# @parameter count [Integer | Nil] The number of tasks to wait for.
def wait(count = nil)
if count
first(count).map(&:wait)
else
first.wait
end
end
end
end
@@ -0,0 +1,182 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2024, by Samuel Williams.
require "etc"
module Async
# A simple work pool that offloads work to a background thread.
#
# @private
class WorkerPool
# Used to augment the scheduler to add support for blocking operations.
module BlockingOperationWait
# Wait for the given work to be executed.
#
# @public Since *Async v2.19* and *Ruby v3.4*.
# @asynchronous May be non-blocking.
#
# @parameter work [Proc] The work to execute on a background thread.
# @returns [Object] The result of the work.
def blocking_operation_wait(work)
@worker_pool.call(work)
end
end
# Execute the given work in a background thread.
class Promise
# Create a new promise.
#
# @parameter work [Proc] The work to be done.
def initialize(work)
@work = work
@state = :pending
@value = nil
@guard = ::Mutex.new
@condition = ::ConditionVariable.new
@thread = nil
end
# Execute the work and resolve the promise.
def call
work = nil
@guard.synchronize do
@thread = ::Thread.current
return unless work = @work
end
resolve(work.call)
rescue Exception => error
reject(error)
end
private def resolve(value)
@guard.synchronize do
@work = nil
@thread = nil
@value = value
@state = :resolved
@condition.broadcast
end
end
private def reject(error)
@guard.synchronize do
@work = nil
@thread = nil
@value = error
@state = :failed
@condition.broadcast
end
end
# Cancel the work and raise an exception in the background thread.
def cancel
return unless @work
@guard.synchronize do
@work = nil
@state = :cancelled
@thread&.raise(Interrupt)
end
end
# Wait for the work to be done.
#
# @returns [Object] The result of the work.
def wait
@guard.synchronize do
while @state == :pending
@condition.wait(@guard)
end
if @state == :failed
raise @value
else
return @value
end
end
end
end
# A background worker thread.
class Worker
# Create a new worker.
def initialize
@work = ::Thread::Queue.new
@thread = ::Thread.new(&method(:run))
end
# Execute work until the queue is closed.
def run
while work = @work.pop
work.call
end
end
# Close the worker thread.
def close
if thread = @thread
@thread = nil
thread.kill
end
end
# Call the work and notify the scheduler when it is done.
def call(work)
promise = Promise.new(work)
@work.push(promise)
begin
return promise.wait
ensure
promise.cancel
end
end
end
# Create a new work pool.
#
# @parameter size [Integer] The number of threads to use.
def initialize(size: Etc.nprocessors)
@ready = ::Thread::Queue.new
size.times do
@ready.push(Worker.new)
end
end
# Close the work pool. Kills all outstanding work.
def close
if ready = @ready
@ready = nil
ready.close
while worker = ready.pop
worker.close
end
end
end
# Offload work to a thread.
#
# @parameter work [Proc] The work to be done.
def call(work)
if ready = @ready
worker = ready.pop
begin
worker.call(work)
ensure
ready.push(worker)
end
else
raise RuntimeError, "No worker available!"
end
end
end
end
@@ -0,0 +1,67 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2017-2024, by Samuel Williams.
# Copyright, 2017, by Kent Gruber.
warn "Async::Wrapper is deprecated and will be removed on 2025-03-31. Please use native interfaces instead.", uplevel: 1, category: :deprecated
module Async
# Represents an asynchronous IO within a reactor.
# @deprecated With no replacement. Prefer native interfaces.
class Wrapper
# An exception that occurs when the asynchronous operation was cancelled.
class Cancelled < StandardError
end
# @parameter io the native object to wrap.
# @parameter reactor [Reactor] the reactor that is managing this wrapper, or not specified, it's looked up by way of {Task.current}.
def initialize(io, reactor = nil)
@io = io
@reactor = reactor
@timeout = nil
end
attr_accessor :reactor
# Dup the underlying IO.
def dup
self.class.new(@io.dup)
end
# The underlying native `io`.
attr :io
# Wait for the io to become readable.
def wait_readable(timeout = @timeout)
@io.to_io.wait_readable(timeout) or raise TimeoutError
end
# Wait for the io to become writable.
def wait_priority(timeout = @timeout)
@io.to_io.wait_priority(timeout) or raise TimeoutError
end
# Wait for the io to become writable.
def wait_writable(timeout = @timeout)
@io.to_io.wait_writable(timeout) or raise TimeoutError
end
# Wait fo the io to become either readable or writable.
# @parameter duration [Float] timeout after the given duration if not `nil`.
def wait_any(timeout = @timeout)
@io.to_io.wait(::IO::READABLE|::IO::WRITABLE|::IO::PRIORITY, timeout) or raise TimeoutError
end
# Close the underlying IO.
def close
@io.close
end
# Whether the underlying IO is closed.
def closed?
@io.closed?
end
end
end
+40
View File
@@ -0,0 +1,40 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2019-2024, by Samuel Williams.
require_relative "../async/reactor"
module Kernel
# Run the given block of code in a task, asynchronously, creating a reactor if necessary.
#
# The preferred method to invoke asynchronous behavior at the top level.
#
# - When invoked within an existing reactor task, it will run the given block
# asynchronously. Will return the task once it has been scheduled.
# - When invoked at the top level, will create and run a reactor, and invoke
# the block as an asynchronous task. Will block until the reactor finishes
# running.
#
# @yields {|task| ...} The block that will execute asynchronously.
# @parameter task [Async::Task] The task that is executing the given block.
#
# @public Since *Async v1*.
# @asynchronous May block until given block completes executing.
def Async(...)
if current = ::Async::Task.current?
return current.async(...)
elsif scheduler = Fiber.scheduler
::Async::Task.run(scheduler, ...)
else
# This calls Fiber.set_scheduler(self):
reactor = ::Async::Reactor.new
begin
return reactor.run(...)
ensure
Fiber.set_scheduler(nil)
end
end
end
end
+39
View File
@@ -0,0 +1,39 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2019-2024, by Samuel Williams.
# Copyright, 2020, by Brian Morearty.
# Copyright, 2024, by Patrik Wenger.
require_relative "../async/reactor"
# Extensions to all Ruby objects.
module Kernel
# Run the given block of code synchronously, but within a reactor if not already in one.
#
# @yields {|task| ...} The block that will execute asynchronously.
# @parameter task [Async::Task] The task that is executing the given block.
#
# @public Since *Async v1*.
# @asynchronous Will block until given block completes executing.
def Sync(annotation: nil, &block)
if task = ::Async::Task.current?
if annotation
task.annotate(annotation) {yield task}
else
yield task
end
elsif scheduler = Fiber.scheduler
::Async::Task.run(scheduler, &block).wait
else
# This calls Fiber.set_scheduler(self):
reactor = Async::Reactor.new
begin
return reactor.run(annotation: annotation, finished: ::Async::Condition.new, &block).wait
ensure
Fiber.set_scheduler(nil)
end
end
end
end
@@ -0,0 +1,6 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2024, by Samuel Williams.
require_relative "async/task"
@@ -0,0 +1,20 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2024, by Samuel Williams.
require_relative "../../../async/task"
require "metrics/provider"
Metrics::Provider(Async::Task) do
ASYNC_TASK_SCHEDULED = Metrics.metric("async.task.scheduled", :counter, description: "The number of tasks scheduled.")
ASYNC_TASK_FINISHED = Metrics.metric("async.task.finished", :counter, description: "The number of tasks finished.")
def schedule(&block)
ASYNC_TASK_SCHEDULED.emit(1)
super(&block)
ensure
ASYNC_TASK_FINISHED.emit(1)
end
end
@@ -0,0 +1,7 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2024, by Samuel Williams.
require_relative "async/task"
require_relative "async/barrier"
@@ -0,0 +1,17 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2022, by Samuel Williams.
require_relative "../../../async/barrier"
require "traces/provider"
Traces::Provider(Async::Barrier) do
def wait
attributes = {
"size" => self.size
}
Traces.trace("async.barrier.wait", attributes: attributes) {super}
end
end
@@ -0,0 +1,40 @@
# frozen_string_literal: true
# Released under the MIT License.
# Copyright, 2022, by Samuel Williams.
require_relative "../../../async/task"
require "traces/provider"
Traces::Provider(Async::Task) do
def schedule(&block)
# If we are not actively tracing anything, then we can skip this:
unless Traces.active?
return super(&block)
end
unless self.transient?
trace_context = Traces.trace_context
end
attributes = {
# We use the instance variable as it corresponds to the user-provided block.
"block" => @block,
"transient" => self.transient?,
}
# Run the trace in the context of the child task:
super do
Traces.trace_context = trace_context
if annotation = self.annotation
attributes["annotation"] = annotation
end
Traces.trace("async.task", attributes: attributes) do
# Yes, this is correct, we already called super above:
yield
end
end
end
end
+47
View File
@@ -0,0 +1,47 @@
# MIT License
Copyright, 2017-2024, by Samuel Williams.
Copyright, 2017, by Kent Gruber.
Copyright, 2017, by Devin Christensen.
Copyright, 2018, by Sokolov Yura.
Copyright, 2018, by Jiang Jinyang.
Copyright, 2019, by Jeremy Jung.
Copyright, 2019, by Ryan Musgrave.
Copyright, 2020-2023, by Olle Jonsson.
Copyright, 2020, by Salim Semaoune.
Copyright, 2020, by Brian Morearty.
Copyright, 2020, by Stefan Wrobel.
Copyright, 2020-2024, by Patrik Wenger.
Copyright, 2020, by Ken Muryoi.
Copyright, 2020, by Jun Jiang.
Copyright, 2020-2022, by Bruno Sutic.
Copyright, 2021, by Julien Portalier.
Copyright, 2022, by Shannon Skipper.
Copyright, 2022, by Masafumi Okura.
Copyright, 2022, by Trevor Turk.
Copyright, 2022, by Masayuki Yamamoto.
Copyright, 2023, by Leon Löchner.
Copyright, 2023, by Colin Kelley.
Copyright, 2023, by Math Ieu.
Copyright, 2023, by Emil Tin.
Copyright, 2023, by Gert Goet.
Copyright, 2024, by Dimitar Peychinov.
Copyright, 2024, by Jamie McCarthy.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+94
View File
@@ -0,0 +1,94 @@
# ![Async](assets/logo.webp)
Async is a composable asynchronous I/O framework for Ruby based on [io-event](https://github.com/socketry/io-event).
> "Lately I've been looking into `async`, as one of my projects –
> [tus-ruby-server](https://github.com/janko/tus-ruby-server) – would really benefit from non-blocking I/O. It's really
> beautifully designed." *– [janko](https://github.com/janko)*
[![Development Status](https://github.com/socketry/async/workflows/Test/badge.svg)](https://github.com/socketry/async/actions?workflow=Test)
[<img src="https://api.gitsponsors.com/api/badge/img?id=87380483" height="20"/>](https://api.gitsponsors.com/api/badge/link?p=U4gCxvzG7eUksiJSe0MSlPHWWhBYryqj6i48tx5L7/r/2NgkAToKb6dEm31bAftU3H+7BVwk3VhUBtE4GHqHJTPfWPR6xo2BQVoT15rFAGAsLFgdT2kKopIfCGV/QDOm7BrkodS2//R7NUMksAdaCQ==)
## Features
- Scalable event-driven I/O for Ruby. Thousands of clients per process\!
- Light weight fiber-based concurrency. No need for callbacks\!
- Multi-thread/process containers for parallelism.
- Growing eco-system of event-driven components.
## Usage
Please see the [project documentation](https://socketry.github.io/async/) for more details.
- [Getting Started](https://socketry.github.io/async/guides/getting-started/index) - This guide shows how to add async to your project and run code asynchronously.
- [Asynchronous Tasks](https://socketry.github.io/async/guides/asynchronous-tasks/index) - This guide explains how asynchronous tasks work and how to use them.
- [Scheduler](https://socketry.github.io/async/guides/scheduler/index) - This guide gives an overview of how the scheduler is implemented.
- [Compatibility](https://socketry.github.io/async/guides/compatibility/index) - This guide gives an overview of the compatibility of Async with Ruby and other frameworks.
- [Best Practices](https://socketry.github.io/async/guides/best-practices/index) - This guide gives an overview of best practices for using Async.
- [Debugging](https://socketry.github.io/async/guides/debugging/index) - This guide explains how to debug issues with programs that use Async.
## Releases
Please see the [project releases](https://socketry.github.io/async/releases/index) for all releases.
### v2.23.0
- Rename `ASYNC_SCHEDULER_DEFAULT_WORKER_POOL` to `ASYNC_SCHEDULER_WORKER_POOL`.
- [Fiber Stall Profiler](https://socketry.github.io/async/releases/index#fiber-stall-profiler)
### v2.21.1
- [Worker Pool](https://socketry.github.io/async/releases/index#worker-pool)
### v2.20.0
- [Traces and Metrics Providers](https://socketry.github.io/async/releases/index#traces-and-metrics-providers)
### v2.19.0
- [Async::Scheduler Debugging](https://socketry.github.io/async/releases/index#async::scheduler-debugging)
- [Console Shims](https://socketry.github.io/async/releases/index#console-shims)
### v2.18.0
- Add support for `Sync(annotation:)`, so that you can annotate the block with a description of what it does, even if it doesn't create a new task.
### v2.17.0
- Introduce `Async::Queue#push` and `Async::Queue#pop` for compatibility with `::Queue`.
### v2.16.0
- [Better Handling of Async and Sync in Nested Fibers](https://socketry.github.io/async/releases/index#better-handling-of-async-and-sync-in-nested-fibers)
## See Also
- [async-http](https://github.com/socketry/async-http) — Asynchronous HTTP client/server.
- [async-websocket](https://github.com/socketry/async-websocket) — Asynchronous client and server websockets.
- [async-dns](https://github.com/socketry/async-dns) — Asynchronous DNS resolver and server.
- [falcon](https://github.com/socketry/falcon) — A rack compatible server built on top of `async-http`.
- [rubydns](https://github.com/ioquatix/rubydns) — An easy to use Ruby DNS server.
- [slack-ruby-bot](https://github.com/slack-ruby/slack-ruby-bot) — A client for making slack bots.
## Contributing
We welcome contributions to this project.
1. Fork it.
2. Create your feature branch (`git checkout -b my-new-feature`).
3. Commit your changes (`git commit -am 'Add some feature'`).
4. Push to the branch (`git push origin my-new-feature`).
5. Create new Pull Request.
### Developer Certificate of Origin
In order to protect users of this project, we require all contributors to comply with the [Developer Certificate of Origin](https://developercertificate.org/). This ensures that all contributions are properly licensed and attributed.
### Community Guidelines
This project is best served by a collaborative and respectful environment. Treat each other professionally, respect differing viewpoints, and engage constructively. Harassment, discrimination, or harmful behavior is not tolerated. Communicate clearly, listen actively, and support one another. If any issues arise, please inform the project maintainers.
+121
View File
@@ -0,0 +1,121 @@
# Releases
## v2.23.0
- Rename `ASYNC_SCHEDULER_DEFAULT_WORKER_POOL` to `ASYNC_SCHEDULER_WORKER_POOL`.
### Fiber Stall Profiler
After several iterations of experimentation, we are officially introducing the fiber stall profiler, implemented using the optional `fiber-profiler` gem. This gem is not included by default, but can be added to your project:
``` bash
$ bundle add fiber-profiler
```
After adding the gem, you can enable the fiber stall profiler by setting the `FIBER_PROFILER_CAPTURE=true` environment variable:
``` bash
$ FIBER_PROFILER_CAPTURE=true bundle exec ruby -rasync -e 'Async{Fiber.blocking{sleep 0.1}}'
Fiber stalled for 0.105 seconds
-e:1 in c-call '#<Class:Fiber>#blocking' (0.105s)
-e:1 in c-call 'Kernel#sleep' (0.105s)
Skipped 1 calls that were too short to be meaningful.
```
The fiber profiler will help you find problems with your code that cause the event loop to stall, which can be a common source of performance issues in asynchronous code.
## v2.21.1
### Worker Pool
Ruby 3.4 will feature a new fiber scheduler hook, `blocking_operation_wait` which allows the scheduler to redirect the work given to `rb_nogvl` to a worker pool.
The Async scheduler optionally supports this feature using a worker pool, by using the following environment variable:
ASYNC_SCHEDULER_WORKER_POOL=true
This will cause the scheduler to use a worker pool for general blocking operations, rather than blocking the event loop.
It should be noted that this isn't a net win, as the overhead of using a worker pool can be significant compared to the `rb_nogvl` work. As such, it is recommended to benchmark your application with and without the worker pool to determine if it is beneficial.
## v2.20.0
### Traces and Metrics Providers
Async now has [traces](https://github.com/socketry/traces) and [metrics](https://github.com/socketry/metrics) providers for various core classes. This allows you to emit traces and metrics to a suitable backend (including DataDog, New Relic, OpenTelemetry, etc.) for monitoring and debugging purposes.
To take advantage of this feature, you will need to introduce your own `config/traces.rb` and `config/metrics.rb`. Async's own repository includes these files for testing purposes, you could copy them into your own project and modify them as needed.
## v2.19.0
### Async::Scheduler Debugging
Occasionally on issues, I encounter people asking for help and I need more information. Pressing Ctrl-C to exit a hung program is common, but it usually doesn't provide enough information to diagnose the problem. Setting the `CONSOLE_LEVEL=debug` environment variable will now print additional information about the scheduler when you interrupt it, including a backtrace of the current tasks.
> CONSOLE_LEVEL=debug bundle exec ruby ./test.rb
^C 0.0s debug: Async::Reactor [oid=0x974] [ec=0x988] [pid=9116] [2024-11-08 14:12:03 +1300]
| Scheduler interrupted: Interrupt
| #<Async::Reactor:0x0000000000000974 1 children (running)>
| #<Async::Task:0x000000000000099c /Users/samuel/Developer/socketry/async/lib/async/scheduler.rb:185:in `transfer' (running)>
| → /Users/samuel/Developer/socketry/async/lib/async/scheduler.rb:185:in `transfer'
| /Users/samuel/Developer/socketry/async/lib/async/scheduler.rb:185:in `block'
| /Users/samuel/Developer/socketry/async/lib/async/scheduler.rb:207:in `kernel_sleep'
| /Users/samuel/Developer/socketry/async/test.rb:7:in `sleep'
| /Users/samuel/Developer/socketry/async/test.rb:7:in `sleepy'
| /Users/samuel/Developer/socketry/async/test.rb:12:in `block in <top (required)>'
| /Users/samuel/Developer/socketry/async/lib/async/task.rb:197:in `block in run'
| /Users/samuel/Developer/socketry/async/lib/async/task.rb:420:in `block in schedule'
/Users/samuel/Developer/socketry/async/lib/async/scheduler.rb:317:in `select': Interrupt
... (backtrace continues) ...
This gives better visibility into what the scheduler is doing, and should help diagnose issues.
### Console Shims
The `async` gem depends on `console` gem, because my goal was to have good logging by default without thinking about it too much. However, some users prefer to avoid using the `console` gem for logging, so I've added an experimental set of shims which should allow you to bypass the `console` gem entirely.
``` ruby
require 'async/console'
require 'async'
Async{raise "Boom"}
```
Will now use `Kernel#warn` to print the task failure warning:
#<Async::Task:0x00000000000012d4 /home/samuel/Developer/socketry/async/lib/async/task.rb:104:in `backtrace' (running)>
Task may have ended with unhandled exception.
(irb):4:in `block in <top (required)>': Boom (RuntimeError)
from /home/samuel/Developer/socketry/async/lib/async/task.rb:197:in `block in run'
from /home/samuel/Developer/socketry/async/lib/async/task.rb:420:in `block in schedule'
## v2.18.0
- Add support for `Sync(annotation:)`, so that you can annotate the block with a description of what it does, even if it doesn't create a new task.
## v2.17.0
- Introduce `Async::Queue#push` and `Async::Queue#pop` for compatibility with `::Queue`.
## v2.16.0
### Better Handling of Async and Sync in Nested Fibers
Interleaving bare fibers within `Async` and `Sync` blocks should not cause problems, but it presents a number of issues in the current implementation. Tracking the parent-child relationship between tasks, when they are interleaved with bare fibers, is difficult. The current implementation assumes that if there is no parent task, then it should create a new reactor. This is not always the case, as the parent task might not be visible due to nested Fibers. As a result, `Async` will create a new reactor, trying to stop the existing one, causing major internal consistency issues.
I encountered this issue when trying to use `Async` within a streaming response in Rails. The `protocol-rack` [uses a normal fiber to wrap streaming responses](https://github.com/socketry/protocol-rack/blob/cb1ca44e9deadb9369bdb2ea03416556aa927c5c/lib/protocol/rack/body/streaming.rb#L24-L28), and if you try to use `Async` within it, it will create a new reactor, causing the server to lock up.
Ideally, `Async` and `Sync` helpers should work when any `Fiber.scheduler` is defined. Right now, it's unrealistic to expect `Async::Task` to work in any scheduler, but at the very least, the following should work:
``` ruby
reactor = Async::Reactor.new # internally calls Fiber.set_scheduler
# This should run in the above reactor, rather than creating a new one.
Async do
puts "Hello World"
end
```
In order to do this, bare `Async` and `Sync` blocks should use `Fiber.scheduler` as a parent if possible.
See <https://github.com/socketry/async/pull/340> for more details.
+22
View File
@@ -0,0 +1,22 @@
Copyright (C) 1993-2013 Yukihiro Matsumoto. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.
+48
View File
@@ -0,0 +1,48 @@
# Base64
The Base64 module provides for the encoding (`#encode64`, `#strict_encode64`,
`#urlsafe_encode64`) and decoding (`#decode64`, `#strict_decode64`,
`#urlsafe_decode64`) of binary data using a Base64 representation.
## Installation
Add this line to your application's Gemfile:
```ruby
gem 'base64'
```
And then execute:
$ bundle install
Or install it yourself as:
$ gem install base64
## Usage
A simple encoding and decoding.
```ruby
require "base64"
enc = Base64.encode64('Send reinforcements')
# -> "U2VuZCByZWluZm9yY2VtZW50cw==\n"
plain = Base64.decode64(enc)
# -> "Send reinforcements"
```
The purpose of using base64 to encode data is that it translates any
binary data into purely printable characters.
## Development
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
## Contributing
Bug reports and pull requests are welcome on GitHub at https://github.com/ruby/base64.
+363
View File
@@ -0,0 +1,363 @@
# frozen_string_literal: true
#
# \Module \Base64 provides methods for:
#
# - Encoding a binary string (containing non-ASCII characters)
# as a string of printable ASCII characters.
# - Decoding such an encoded string.
#
# \Base64 is commonly used in contexts where binary data
# is not allowed or supported:
#
# - Images in HTML or CSS files, or in URLs.
# - Email attachments.
#
# A \Base64-encoded string is about one-third larger that its source.
# See the {Wikipedia article}[https://en.wikipedia.org/wiki/Base64]
# for more information.
#
# This module provides three pairs of encode/decode methods.
# Your choices among these methods should depend on:
#
# - Which character set is to be used for encoding and decoding.
# - Whether "padding" is to be used.
# - Whether encoded strings are to contain newlines.
#
# Note: Examples on this page assume that the including program has executed:
#
# require 'base64'
#
# == Encoding Character Sets
#
# A \Base64-encoded string consists only of characters from a 64-character set:
#
# - <tt>('A'..'Z')</tt>.
# - <tt>('a'..'z')</tt>.
# - <tt>('0'..'9')</tt>.
# - <tt>=</tt>, the 'padding' character.
# - Either:
# - <tt>%w[+ /]</tt>:
# {RFC-2045-compliant}[https://datatracker.ietf.org/doc/html/rfc2045];
# _not_ safe for URLs.
# - <tt>%w[- _]</tt>:
# {RFC-4648-compliant}[https://datatracker.ietf.org/doc/html/rfc4648];
# safe for URLs.
#
# If you are working with \Base64-encoded strings that will come from
# or be put into URLs, you should choose this encoder-decoder pair
# of RFC-4648-compliant methods:
#
# - Base64.urlsafe_encode64 and Base64.urlsafe_decode64.
#
# Otherwise, you may choose any of the pairs in this module,
# including the pair above, or the RFC-2045-compliant pairs:
#
# - Base64.encode64 and Base64.decode64.
# - Base64.strict_encode64 and Base64.strict_decode64.
#
# == Padding
#
# \Base64-encoding changes a triplet of input bytes
# into a quartet of output characters.
#
# <b>Padding in Encode Methods</b>
#
# Padding -- extending an encoded string with zero, one, or two trailing
# <tt>=</tt> characters -- is performed by methods Base64.encode64,
# Base64.strict_encode64, and, by default, Base64.urlsafe_encode64:
#
# Base64.encode64('s') # => "cw==\n"
# Base64.strict_encode64('s') # => "cw=="
# Base64.urlsafe_encode64('s') # => "cw=="
# Base64.urlsafe_encode64('s', padding: false) # => "cw"
#
# When padding is performed, the encoded string is always of length <em>4n</em>,
# where +n+ is a non-negative integer:
#
# - Input bytes of length <em>3n</em> generate unpadded output characters
# of length <em>4n</em>:
#
# # n = 1: 3 bytes => 4 characters.
# Base64.strict_encode64('123') # => "MDEy"
# # n = 2: 6 bytes => 8 characters.
# Base64.strict_encode64('123456') # => "MDEyMzQ1"
#
# - Input bytes of length <em>3n+1</em> generate padded output characters
# of length <em>4(n+1)</em>, with two padding characters at the end:
#
# # n = 1: 4 bytes => 8 characters.
# Base64.strict_encode64('1234') # => "MDEyMw=="
# # n = 2: 7 bytes => 12 characters.
# Base64.strict_encode64('1234567') # => "MDEyMzQ1Ng=="
#
# - Input bytes of length <em>3n+2</em> generate padded output characters
# of length <em>4(n+1)</em>, with one padding character at the end:
#
# # n = 1: 5 bytes => 8 characters.
# Base64.strict_encode64('12345') # => "MDEyMzQ="
# # n = 2: 8 bytes => 12 characters.
# Base64.strict_encode64('12345678') # => "MDEyMzQ1Njc="
#
# When padding is suppressed, for a positive integer <em>n</em>:
#
# - Input bytes of length <em>3n</em> generate unpadded output characters
# of length <em>4n</em>:
#
# # n = 1: 3 bytes => 4 characters.
# Base64.urlsafe_encode64('123', padding: false) # => "MDEy"
# # n = 2: 6 bytes => 8 characters.
# Base64.urlsafe_encode64('123456', padding: false) # => "MDEyMzQ1"
#
# - Input bytes of length <em>3n+1</em> generate unpadded output characters
# of length <em>4n+2</em>, with two padding characters at the end:
#
# # n = 1: 4 bytes => 6 characters.
# Base64.urlsafe_encode64('1234', padding: false) # => "MDEyMw"
# # n = 2: 7 bytes => 10 characters.
# Base64.urlsafe_encode64('1234567', padding: false) # => "MDEyMzQ1Ng"
#
# - Input bytes of length <em>3n+2</em> generate unpadded output characters
# of length <em>4n+3</em>, with one padding character at the end:
#
# # n = 1: 5 bytes => 7 characters.
# Base64.urlsafe_encode64('12345', padding: false) # => "MDEyMzQ"
# # m = 2: 8 bytes => 11 characters.
# Base64.urlsafe_encode64('12345678', padding: false) # => "MDEyMzQ1Njc"
#
# <b>Padding in Decode Methods</b>
#
# All of the \Base64 decode methods support (but do not require) padding.
#
# \Method Base64.decode64 does not check the size of the padding:
#
# Base64.decode64("MDEyMzQ1Njc") # => "01234567"
# Base64.decode64("MDEyMzQ1Njc=") # => "01234567"
# Base64.decode64("MDEyMzQ1Njc==") # => "01234567"
#
# \Method Base64.strict_decode64 strictly enforces padding size:
#
# Base64.strict_decode64("MDEyMzQ1Njc") # Raises ArgumentError
# Base64.strict_decode64("MDEyMzQ1Njc=") # => "01234567"
# Base64.strict_decode64("MDEyMzQ1Njc==") # Raises ArgumentError
#
# \Method Base64.urlsafe_decode64 allows padding in +str+,
# which if present, must be correct:
# see {Padding}[Base64.html#module-Base64-label-Padding], above:
#
# Base64.urlsafe_decode64("MDEyMzQ1Njc") # => "01234567"
# Base64.urlsafe_decode64("MDEyMzQ1Njc=") # => "01234567"
# Base64.urlsafe_decode64("MDEyMzQ1Njc==") # Raises ArgumentError.
#
# == Newlines
#
# An encoded string returned by Base64.encode64 or Base64.urlsafe_encode64
# has an embedded newline character
# after each 60-character sequence, and, if non-empty, at the end:
#
# # No newline if empty.
# encoded = Base64.encode64("\x00" * 0)
# encoded.index("\n") # => nil
#
# # Newline at end of short output.
# encoded = Base64.encode64("\x00" * 1)
# encoded.size # => 4
# encoded.index("\n") # => 4
#
# # Newline at end of longer output.
# encoded = Base64.encode64("\x00" * 45)
# encoded.size # => 60
# encoded.index("\n") # => 60
#
# # Newlines embedded and at end of still longer output.
# encoded = Base64.encode64("\x00" * 46)
# encoded.size # => 65
# encoded.rindex("\n") # => 65
# encoded.split("\n").map {|s| s.size } # => [60, 4]
#
# The string to be encoded may itself contain newlines,
# which are encoded as \Base64:
#
# # Base64.encode64("\n\n\n") # => "CgoK\n"
# s = "This is line 1\nThis is line 2\n"
# Base64.encode64(s) # => "VGhpcyBpcyBsaW5lIDEKVGhpcyBpcyBsaW5lIDIK\n"
#
module Base64
VERSION = "0.2.0"
module_function
# Returns a string containing the RFC-2045-compliant \Base64-encoding of +bin+.
#
# Per RFC 2045, the returned string may contain the URL-unsafe characters
# <tt>+</tt> or <tt>/</tt>;
# see {Encoding Character Set}[Base64.html#module-Base64-label-Encoding+Character+Sets] above:
#
# Base64.encode64("\xFB\xEF\xBE") # => "++++\n"
# Base64.encode64("\xFF\xFF\xFF") # => "////\n"
#
# The returned string may include padding;
# see {Padding}[Base64.html#module-Base64-label-Padding] above.
#
# Base64.encode64('*') # => "Kg==\n"
#
# The returned string ends with a newline character, and if sufficiently long
# will have one or more embedded newline characters;
# see {Newlines}[Base64.html#module-Base64-label-Newlines] above:
#
# Base64.encode64('*') # => "Kg==\n"
# Base64.encode64('*' * 46)
# # => "KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq\nKg==\n"
#
# The string to be encoded may itself contain newlines,
# which will be encoded as ordinary \Base64:
#
# Base64.encode64("\n\n\n") # => "CgoK\n"
# s = "This is line 1\nThis is line 2\n"
# Base64.encode64(s) # => "VGhpcyBpcyBsaW5lIDEKVGhpcyBpcyBsaW5lIDIK\n"
#
def encode64(bin)
[bin].pack("m")
end
# Returns a string containing the decoding of an RFC-2045-compliant
# \Base64-encoded string +str+:
#
# s = "VGhpcyBpcyBsaW5lIDEKVGhpcyBpcyBsaW5lIDIK\n"
# Base64.decode64(s) # => "This is line 1\nThis is line 2\n"
#
# Non-\Base64 characters in +str+ are ignored;
# see {Encoding Character Set}[Base64.html#module-Base64-label-Encoding+Character+Sets] above:
# these include newline characters and characters <tt>-</tt> and <tt>/</tt>:
#
# Base64.decode64("\x00\n-_") # => ""
#
# Padding in +str+ (even if incorrect) is ignored:
#
# Base64.decode64("MDEyMzQ1Njc") # => "01234567"
# Base64.decode64("MDEyMzQ1Njc=") # => "01234567"
# Base64.decode64("MDEyMzQ1Njc==") # => "01234567"
#
def decode64(str)
str.unpack1("m")
end
# Returns a string containing the RFC-2045-compliant \Base64-encoding of +bin+.
#
# Per RFC 2045, the returned string may contain the URL-unsafe characters
# <tt>+</tt> or <tt>/</tt>;
# see {Encoding Character Set}[Base64.html#module-Base64-label-Encoding+Character+Sets] above:
#
# Base64.strict_encode64("\xFB\xEF\xBE") # => "++++\n"
# Base64.strict_encode64("\xFF\xFF\xFF") # => "////\n"
#
# The returned string may include padding;
# see {Padding}[Base64.html#module-Base64-label-Padding] above.
#
# Base64.strict_encode64('*') # => "Kg==\n"
#
# The returned string will have no newline characters, regardless of its length;
# see {Newlines}[Base64.html#module-Base64-label-Newlines] above:
#
# Base64.strict_encode64('*') # => "Kg=="
# Base64.strict_encode64('*' * 46)
# # => "KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKg=="
#
# The string to be encoded may itself contain newlines,
# which will be encoded as ordinary \Base64:
#
# Base64.strict_encode64("\n\n\n") # => "CgoK"
# s = "This is line 1\nThis is line 2\n"
# Base64.strict_encode64(s) # => "VGhpcyBpcyBsaW5lIDEKVGhpcyBpcyBsaW5lIDIK"
#
def strict_encode64(bin)
[bin].pack("m0")
end
# Returns a string containing the decoding of an RFC-2045-compliant
# \Base64-encoded string +str+:
#
# s = "VGhpcyBpcyBsaW5lIDEKVGhpcyBpcyBsaW5lIDIK"
# Base64.strict_decode64(s) # => "This is line 1\nThis is line 2\n"
#
# Non-\Base64 characters in +str+ not allowed;
# see {Encoding Character Set}[Base64.html#module-Base64-label-Encoding+Character+Sets] above:
# these include newline characters and characters <tt>-</tt> and <tt>/</tt>:
#
# Base64.strict_decode64("\n") # Raises ArgumentError
# Base64.strict_decode64('-') # Raises ArgumentError
# Base64.strict_decode64('_') # Raises ArgumentError
#
# Padding in +str+, if present, must be correct:
#
# Base64.strict_decode64("MDEyMzQ1Njc") # Raises ArgumentError
# Base64.strict_decode64("MDEyMzQ1Njc=") # => "01234567"
# Base64.strict_decode64("MDEyMzQ1Njc==") # Raises ArgumentError
#
def strict_decode64(str)
str.unpack1("m0")
end
# Returns the RFC-4648-compliant \Base64-encoding of +bin+.
#
# Per RFC 4648, the returned string will not contain the URL-unsafe characters
# <tt>+</tt> or <tt>/</tt>,
# but instead may contain the URL-safe characters
# <tt>-</tt> and <tt>_</tt>;
# see {Encoding Character Set}[Base64.html#module-Base64-label-Encoding+Character+Sets] above:
#
# Base64.urlsafe_encode64("\xFB\xEF\xBE") # => "----"
# Base64.urlsafe_encode64("\xFF\xFF\xFF") # => "____"
#
# By default, the returned string may have padding;
# see {Padding}[Base64.html#module-Base64-label-Padding], above:
#
# Base64.urlsafe_encode64('*') # => "Kg=="
#
# Optionally, you can suppress padding:
#
# Base64.urlsafe_encode64('*', padding: false) # => "Kg"
#
# The returned string will have no newline characters, regardless of its length;
# see {Newlines}[Base64.html#module-Base64-label-Newlines] above:
#
# Base64.urlsafe_encode64('*') # => "Kg=="
# Base64.urlsafe_encode64('*' * 46)
# # => "KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKg=="
#
def urlsafe_encode64(bin, padding: true)
str = strict_encode64(bin)
str.chomp!("==") or str.chomp!("=") unless padding
str.tr!("+/", "-_")
str
end
# Returns the decoding of an RFC-4648-compliant \Base64-encoded string +str+:
#
# +str+ may not contain non-Base64 characters;
# see {Encoding Character Set}[Base64.html#module-Base64-label-Encoding+Character+Sets] above:
#
# Base64.urlsafe_decode64('+') # Raises ArgumentError.
# Base64.urlsafe_decode64('/') # Raises ArgumentError.
# Base64.urlsafe_decode64("\n") # Raises ArgumentError.
#
# Padding in +str+, if present, must be correct:
# see {Padding}[Base64.html#module-Base64-label-Padding], above:
#
# Base64.urlsafe_decode64("MDEyMzQ1Njc") # => "01234567"
# Base64.urlsafe_decode64("MDEyMzQ1Njc=") # => "01234567"
# Base64.urlsafe_decode64("MDEyMzQ1Njc==") # Raises ArgumentError.
#
def urlsafe_decode64(str)
# NOTE: RFC 4648 does say nothing about unpadded input, but says that
# "the excess pad characters MAY also be ignored", so it is inferred that
# unpadded input is also acceptable.
if !str.end_with?("=") && str.length % 4 != 0
str = str.ljust((str.length + 3) & ~3, "=")
str.tr!("-_", "+/")
else
str = str.tr("-_", "+/")
end
strict_decode64(str)
end
end
+22
View File
@@ -0,0 +1,22 @@
Copyright (C) 1993-2013 Yukihiro Matsumoto. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.
+56
View File
@@ -0,0 +1,56 @@
Ruby is copyrighted free software by Yukihiro Matsumoto <matz@netlab.jp>.
You can redistribute it and/or modify it under either the terms of the
2-clause BSDL (see the file BSDL), or the conditions below:
1. You may make and give away verbatim copies of the source form of the
software without restriction, provided that you duplicate all of the
original copyright notices and associated disclaimers.
2. You may modify your copy of the software in any way, provided that
you do at least ONE of the following:
a. place your modifications in the Public Domain or otherwise
make them Freely Available, such as by posting said
modifications to Usenet or an equivalent medium, or by allowing
the author to include your modifications in the software.
b. use the modified software only within your corporation or
organization.
c. give non-standard binaries non-standard names, with
instructions on where to get the original software distribution.
d. make other distribution arrangements with the author.
3. You may distribute the software in object code or binary form,
provided that you do at least ONE of the following:
a. distribute the binaries and library files of the software,
together with instructions (in the manual page or equivalent)
on where to get the original distribution.
b. accompany the distribution with the machine-readable source of
the software.
c. give non-standard binaries non-standard names, with
instructions on where to get the original software distribution.
d. make other distribution arrangements with the author.
4. You may modify and include the part of the software into any other
software (possibly commercial). But some files in the distribution
are not written by the author, so that they are not under these terms.
For the list of those files and their copying conditions, see the
file LEGAL.
5. The scripts and library files supplied as input to or produced as
output from the software do not automatically fall under the
copyright of the software, but belong to whomever generated them,
and may be sold commercially, and may be aggregated with this
software.
6. THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE.
+60
View File
@@ -0,0 +1,60 @@
# -*- rdoc -*-
= LEGAL NOTICE INFORMATION
--------------------------
All the files in this distribution are covered under either the Ruby's
license (see the file COPYING) or public-domain except some files
mentioned below.
== MIT License
>>>
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
== Old-style BSD license
>>>
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
3. Neither the name of the University nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.
IMPORTANT NOTE::
From ftp://ftp.cs.berkeley.edu/pub/4bsd/README.Impt.License.Change
paragraph 3 above is now null and void.
+48
View File
@@ -0,0 +1,48 @@
# Base64
The Base64 module provides for the encoding (`#encode64`, `#strict_encode64`,
`#urlsafe_encode64`) and decoding (`#decode64`, `#strict_decode64`,
`#urlsafe_decode64`) of binary data using a Base64 representation.
## Installation
Add this line to your application's Gemfile:
```ruby
gem 'base64'
```
And then execute:
$ bundle install
Or install it yourself as:
$ gem install base64
## Usage
A simple encoding and decoding.
```ruby
require "base64"
enc = Base64.encode64('Send reinforcements')
# -> "U2VuZCByZWluZm9yY2VtZW50cw==\n"
plain = Base64.decode64(enc)
# -> "Send reinforcements"
```
The purpose of using base64 to encode data is that it translates any
binary data into purely printable characters.
## Development
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
## Contributing
Bug reports and pull requests are welcome on GitHub at https://github.com/ruby/base64.
+381
View File
@@ -0,0 +1,381 @@
# frozen_string_literal: true
#
# \Module \Base64 provides methods for:
#
# - \Encoding a binary string (containing non-ASCII characters)
# as a string of printable ASCII characters.
# - Decoding such an encoded string.
#
# \Base64 is commonly used in contexts where binary data
# is not allowed or supported:
#
# - Images in HTML or CSS files, or in URLs.
# - Email attachments.
#
# A \Base64-encoded string is about one-third larger that its source.
# See the {Wikipedia article}[https://en.wikipedia.org/wiki/Base64]
# for more information.
#
# This module provides three pairs of encode/decode methods.
# Your choices among these methods should depend on:
#
# - Which character set is to be used for encoding and decoding.
# - Whether "padding" is to be used.
# - Whether encoded strings are to contain newlines.
#
# Note: Examples on this page assume that the including program has executed:
#
# require 'base64'
#
# == \Encoding Character Sets
#
# A \Base64-encoded string consists only of characters from a 64-character set:
#
# - <tt>('A'..'Z')</tt>.
# - <tt>('a'..'z')</tt>.
# - <tt>('0'..'9')</tt>.
# - <tt>=</tt>, the 'padding' character.
# - Either:
# - <tt>%w[+ /]</tt>:
# {RFC-2045-compliant}[https://datatracker.ietf.org/doc/html/rfc2045];
# _not_ safe for URLs.
# - <tt>%w[- _]</tt>:
# {RFC-4648-compliant}[https://datatracker.ietf.org/doc/html/rfc4648];
# safe for URLs.
#
# If you are working with \Base64-encoded strings that will come from
# or be put into URLs, you should choose this encoder-decoder pair
# of RFC-4648-compliant methods:
#
# - Base64.urlsafe_encode64 and Base64.urlsafe_decode64.
#
# Otherwise, you may choose any of the pairs in this module,
# including the pair above, or the RFC-2045-compliant pairs:
#
# - Base64.encode64 and Base64.decode64.
# - Base64.strict_encode64 and Base64.strict_decode64.
#
# == Padding
#
# \Base64-encoding changes a triplet of input bytes
# into a quartet of output characters.
#
# <b>Padding in Encode Methods</b>
#
# Padding -- extending an encoded string with zero, one, or two trailing
# <tt>=</tt> characters -- is performed by methods Base64.encode64,
# Base64.strict_encode64, and, by default, Base64.urlsafe_encode64:
#
# Base64.encode64('s') # => "cw==\n"
# Base64.strict_encode64('s') # => "cw=="
# Base64.urlsafe_encode64('s') # => "cw=="
# Base64.urlsafe_encode64('s', padding: false) # => "cw"
#
# When padding is performed, the encoded string is always of length <em>4n</em>,
# where +n+ is a non-negative integer:
#
# - Input bytes of length <em>3n</em> generate unpadded output characters
# of length <em>4n</em>:
#
# # n = 1: 3 bytes => 4 characters.
# Base64.strict_encode64('123') # => "MDEy"
# # n = 2: 6 bytes => 8 characters.
# Base64.strict_encode64('123456') # => "MDEyMzQ1"
#
# - Input bytes of length <em>3n+1</em> generate padded output characters
# of length <em>4(n+1)</em>, with two padding characters at the end:
#
# # n = 1: 4 bytes => 8 characters.
# Base64.strict_encode64('1234') # => "MDEyMw=="
# # n = 2: 7 bytes => 12 characters.
# Base64.strict_encode64('1234567') # => "MDEyMzQ1Ng=="
#
# - Input bytes of length <em>3n+2</em> generate padded output characters
# of length <em>4(n+1)</em>, with one padding character at the end:
#
# # n = 1: 5 bytes => 8 characters.
# Base64.strict_encode64('12345') # => "MDEyMzQ="
# # n = 2: 8 bytes => 12 characters.
# Base64.strict_encode64('12345678') # => "MDEyMzQ1Njc="
#
# When padding is suppressed, for a positive integer <em>n</em>:
#
# - Input bytes of length <em>3n</em> generate unpadded output characters
# of length <em>4n</em>:
#
# # n = 1: 3 bytes => 4 characters.
# Base64.urlsafe_encode64('123', padding: false) # => "MDEy"
# # n = 2: 6 bytes => 8 characters.
# Base64.urlsafe_encode64('123456', padding: false) # => "MDEyMzQ1"
#
# - Input bytes of length <em>3n+1</em> generate unpadded output characters
# of length <em>4n+2</em>, with two padding characters at the end:
#
# # n = 1: 4 bytes => 6 characters.
# Base64.urlsafe_encode64('1234', padding: false) # => "MDEyMw"
# # n = 2: 7 bytes => 10 characters.
# Base64.urlsafe_encode64('1234567', padding: false) # => "MDEyMzQ1Ng"
#
# - Input bytes of length <em>3n+2</em> generate unpadded output characters
# of length <em>4n+3</em>, with one padding character at the end:
#
# # n = 1: 5 bytes => 7 characters.
# Base64.urlsafe_encode64('12345', padding: false) # => "MDEyMzQ"
# # m = 2: 8 bytes => 11 characters.
# Base64.urlsafe_encode64('12345678', padding: false) # => "MDEyMzQ1Njc"
#
# <b>Padding in Decode Methods</b>
#
# All of the \Base64 decode methods support (but do not require) padding.
#
# \Method Base64.decode64 does not check the size of the padding:
#
# Base64.decode64("MDEyMzQ1Njc") # => "01234567"
# Base64.decode64("MDEyMzQ1Njc=") # => "01234567"
# Base64.decode64("MDEyMzQ1Njc==") # => "01234567"
#
# \Method Base64.strict_decode64 strictly enforces padding size:
#
# Base64.strict_decode64("MDEyMzQ1Njc") # Raises ArgumentError
# Base64.strict_decode64("MDEyMzQ1Njc=") # => "01234567"
# Base64.strict_decode64("MDEyMzQ1Njc==") # Raises ArgumentError
#
# \Method Base64.urlsafe_decode64 allows padding in the encoded string,
# which if present, must be correct:
# see {Padding}[Base64.html#module-Base64-label-Padding], above:
#
# Base64.urlsafe_decode64("MDEyMzQ1Njc") # => "01234567"
# Base64.urlsafe_decode64("MDEyMzQ1Njc=") # => "01234567"
# Base64.urlsafe_decode64("MDEyMzQ1Njc==") # Raises ArgumentError.
#
# == Newlines
#
# An encoded string returned by Base64.encode64 or Base64.urlsafe_encode64
# has an embedded newline character
# after each 60-character sequence, and, if non-empty, at the end:
#
# # No newline if empty.
# encoded = Base64.encode64("\x00" * 0)
# encoded.index("\n") # => nil
#
# # Newline at end of short output.
# encoded = Base64.encode64("\x00" * 1)
# encoded.size # => 4
# encoded.index("\n") # => 4
#
# # Newline at end of longer output.
# encoded = Base64.encode64("\x00" * 45)
# encoded.size # => 60
# encoded.index("\n") # => 60
#
# # Newlines embedded and at end of still longer output.
# encoded = Base64.encode64("\x00" * 46)
# encoded.size # => 65
# encoded.rindex("\n") # => 65
# encoded.split("\n").map {|s| s.size } # => [60, 4]
#
# The string to be encoded may itself contain newlines,
# which are encoded as \Base64:
#
# # Base64.encode64("\n\n\n") # => "CgoK\n"
# s = "This is line 1\nThis is line 2\n"
# Base64.encode64(s) # => "VGhpcyBpcyBsaW5lIDEKVGhpcyBpcyBsaW5lIDIK\n"
#
module Base64
VERSION = "0.3.0"
module_function
# :call-seq:
# Base64.encode64(string) -> encoded_string
#
# Returns a string containing the RFC-2045-compliant \Base64-encoding of +string+.
#
# Per RFC 2045, the returned string may contain the URL-unsafe characters
# <tt>+</tt> or <tt>/</tt>;
# see {Encoding Character Set}[Base64.html#module-Base64-label-Encoding+Character+Sets] above:
#
# Base64.encode64("\xFB\xEF\xBE") # => "++++\n"
# Base64.encode64("\xFF\xFF\xFF") # => "////\n"
#
# The returned string may include padding;
# see {Padding}[Base64.html#module-Base64-label-Padding] above.
#
# Base64.encode64('*') # => "Kg==\n"
#
# The returned string ends with a newline character, and if sufficiently long
# will have one or more embedded newline characters;
# see {Newlines}[Base64.html#module-Base64-label-Newlines] above:
#
# Base64.encode64('*') # => "Kg==\n"
# Base64.encode64('*' * 46)
# # => "KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq\nKg==\n"
#
# The string to be encoded may itself contain newlines,
# which will be encoded as ordinary \Base64:
#
# Base64.encode64("\n\n\n") # => "CgoK\n"
# s = "This is line 1\nThis is line 2\n"
# Base64.encode64(s) # => "VGhpcyBpcyBsaW5lIDEKVGhpcyBpcyBsaW5lIDIK\n"
#
def encode64(bin)
[bin].pack("m")
end
# :call-seq:
# Base64.decode(encoded_string) -> decoded_string
#
# Returns a string containing the decoding of an RFC-2045-compliant
# \Base64-encoded string +encoded_string+:
#
# s = "VGhpcyBpcyBsaW5lIDEKVGhpcyBpcyBsaW5lIDIK\n"
# Base64.decode64(s) # => "This is line 1\nThis is line 2\n"
#
# Non-\Base64 characters in +encoded_string+ are ignored;
# see {Encoding Character Set}[Base64.html#module-Base64-label-Encoding+Character+Sets] above:
# these include newline characters and characters <tt>-</tt> and <tt>/</tt>:
#
# Base64.decode64("\x00\n-_") # => ""
#
# Padding in +encoded_string+ (even if incorrect) is ignored:
#
# Base64.decode64("MDEyMzQ1Njc") # => "01234567"
# Base64.decode64("MDEyMzQ1Njc=") # => "01234567"
# Base64.decode64("MDEyMzQ1Njc==") # => "01234567"
#
def decode64(str)
str.unpack1("m")
end
# :call-seq:
# Base64.strict_encode64(string) -> encoded_string
#
# Returns a string containing the RFC-2045-compliant \Base64-encoding of +string+.
#
# Per RFC 2045, the returned string may contain the URL-unsafe characters
# <tt>+</tt> or <tt>/</tt>;
# see {Encoding Character Set}[Base64.html#module-Base64-label-Encoding+Character+Sets] above:
#
# Base64.strict_encode64("\xFB\xEF\xBE") # => "++++\n"
# Base64.strict_encode64("\xFF\xFF\xFF") # => "////\n"
#
# The returned string may include padding;
# see {Padding}[Base64.html#module-Base64-label-Padding] above.
#
# Base64.strict_encode64('*') # => "Kg==\n"
#
# The returned string will have no newline characters, regardless of its length;
# see {Newlines}[Base64.html#module-Base64-label-Newlines] above:
#
# Base64.strict_encode64('*') # => "Kg=="
# Base64.strict_encode64('*' * 46)
# # => "KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKg=="
#
# The string to be encoded may itself contain newlines,
# which will be encoded as ordinary \Base64:
#
# Base64.strict_encode64("\n\n\n") # => "CgoK"
# s = "This is line 1\nThis is line 2\n"
# Base64.strict_encode64(s) # => "VGhpcyBpcyBsaW5lIDEKVGhpcyBpcyBsaW5lIDIK"
#
def strict_encode64(bin)
[bin].pack("m0")
end
# :call-seq:
# Base64.strict_decode64(encoded_string) -> decoded_string
#
# Returns a string containing the decoding of an RFC-2045-compliant
# \Base64-encoded string +encoded_string+:
#
# s = "VGhpcyBpcyBsaW5lIDEKVGhpcyBpcyBsaW5lIDIK"
# Base64.strict_decode64(s) # => "This is line 1\nThis is line 2\n"
#
# Non-\Base64 characters in +encoded_string+ are not allowed;
# see {Encoding Character Set}[Base64.html#module-Base64-label-Encoding+Character+Sets] above:
# these include newline characters and characters <tt>-</tt> and <tt>/</tt>:
#
# Base64.strict_decode64("\n") # Raises ArgumentError
# Base64.strict_decode64('-') # Raises ArgumentError
# Base64.strict_decode64('_') # Raises ArgumentError
#
# Padding in +encoded_string+, if present, must be correct:
#
# Base64.strict_decode64("MDEyMzQ1Njc") # Raises ArgumentError
# Base64.strict_decode64("MDEyMzQ1Njc=") # => "01234567"
# Base64.strict_decode64("MDEyMzQ1Njc==") # Raises ArgumentError
#
def strict_decode64(str)
str.unpack1("m0")
end
# :call-seq:
# Base64.urlsafe_encode64(string) -> encoded_string
#
# Returns the RFC-4648-compliant \Base64-encoding of +string+.
#
# Per RFC 4648, the returned string will not contain the URL-unsafe characters
# <tt>+</tt> or <tt>/</tt>,
# but instead may contain the URL-safe characters
# <tt>-</tt> and <tt>_</tt>;
# see {Encoding Character Set}[Base64.html#module-Base64-label-Encoding+Character+Sets] above:
#
# Base64.urlsafe_encode64("\xFB\xEF\xBE") # => "----"
# Base64.urlsafe_encode64("\xFF\xFF\xFF") # => "____"
#
# By default, the returned string may have padding;
# see {Padding}[Base64.html#module-Base64-label-Padding], above:
#
# Base64.urlsafe_encode64('*') # => "Kg=="
#
# Optionally, you can suppress padding:
#
# Base64.urlsafe_encode64('*', padding: false) # => "Kg"
#
# The returned string will have no newline characters, regardless of its length;
# see {Newlines}[Base64.html#module-Base64-label-Newlines] above:
#
# Base64.urlsafe_encode64('*') # => "Kg=="
# Base64.urlsafe_encode64('*' * 46)
# # => "KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKg=="
#
def urlsafe_encode64(bin, padding: true)
str = strict_encode64(bin)
str.chomp!("==") or str.chomp!("=") unless padding
str.tr!("+/", "-_")
str
end
# :call-seq:
# Base64.urlsafe_decode64(encoded_string) -> decoded_string
#
# Returns the decoding of an RFC-4648-compliant \Base64-encoded string +encoded_string+:
#
# +encoded_string+ may not contain non-Base64 characters;
# see {Encoding Character Set}[Base64.html#module-Base64-label-Encoding+Character+Sets] above:
#
# Base64.urlsafe_decode64('+') # Raises ArgumentError.
# Base64.urlsafe_decode64('/') # Raises ArgumentError.
# Base64.urlsafe_decode64("\n") # Raises ArgumentError.
#
# Padding in +encoded_string+, if present, must be correct:
# see {Padding}[Base64.html#module-Base64-label-Padding], above:
#
# Base64.urlsafe_decode64("MDEyMzQ1Njc") # => "01234567"
# Base64.urlsafe_decode64("MDEyMzQ1Njc=") # => "01234567"
# Base64.urlsafe_decode64("MDEyMzQ1Njc==") # Raises ArgumentError.
#
def urlsafe_decode64(str)
# NOTE: RFC 4648 does say nothing about unpadded input, but says that
# "the excess pad characters MAY also be ignored", so it is inferred that
# unpadded input is also acceptable.
if !str.end_with?("=") && str.length % 4 != 0
str = str.ljust((str.length + 3) & ~3, "=")
str.tr!("-_", "+/")
else
str = str.tr("-_", "+/")
end
strict_decode64(str)
end
end

Some files were not shown because too many files have changed in this diff Show More