6. SNAPPY#

group SNAPPY_API

A light-weight compression algorithm. It is designed for speed of compression and decompression, rather than for the utmost in space savings.

For getting better compression ratios when you are compressing data with long repeated sequences or compressing data that is similar to other data, while still compressing fast, you might look at first using BMDiff and then compressing the output of BMDiff with Snappy.

Generic compression/decompression routines.

size_t Compress(Source *source, Sink *sink)#

Compress the bytes read from “*source” and append to “*sink”. Return the number of bytes written.

Parameters

Direction

Description

source

in,out

A Source is an interface that yields a sequence of bytes, you can initialize it by calling snappy::ByteArraySource(inBuf,inBufLen);where inBuf is the pointer to original data and inBufLen is the size of inBuf

sink

in,out

A Sink is an interface that consumes a sequence of bytes, you can initialize it by calling snappy::UncheckedByteArraySink(dest); where dest is the pointer to the destination buffer.

Returns:

Result

Description

Success

Return the number of bytes written.

Failure

Return 0 upon failure or NULL parameters are passed

bool GetUncompressedLength(Source *source, uint32_t *result)#

Find the uncompressed length of the given stream, as given by the header. Note that the true length could deviate from this; the stream could e.g. be truncated.

Parameters

Direction

Description

source

in,out

A Source is an interface that yields a sequence of bytes, you can initialize it by calling snappy::ByteArraySource(inBuf,inBufLen); where inBuf is the pointer to original data and inBufLen is the size of inBuf

result

out

Uncompressed length of the given stream is stored here.

Note

Also note that this leaves “*source” in a state that is unsuitable for further operations, such as RawUncompress(). You will need to rewind or recreate the source yourself before attempting any further calls.

Returns:

Result

Description

Success

If the data inside the source is uncorrupted it will return true.

Failure

It will return false if the data inside the source is corrupted.

bool Uncompress(Source *compressed, Sink *uncompressed)#

Decompresses “compressed” to “*uncompressed”.

Parameters

Direction

Description

compressed

in,out

A Source is an interface that yields a sequence of bytes, you can initialize it by calling snappy::ByteArraySource(inBuf,inBufLen); where inBuf is the pointer to original data and inBufLen is the size of inBuf .

uncompressed

in,out

A Sink is an interface that consumes a sequence of bytes, you can initialize it by calling snappy::UncheckedByteArraySink(dest); where dest is the pointer to the destination buffer.

Returns:

Result

Description

Success

Returns true if successful.

Failure

Returns false if the decompression fails.

size_t UncompressAsMuchAsPossible(Source *compressed, Sink *uncompressed)#

This routine decompresses as much of the “compressed” as possible into sink. It returns the number of valid bytes added to sink (extra invalid bytes may have been added due to errors; the caller should ignore those). The emitted data typically has length GetUncompressedLength(), but may be shorter if an error is encountered.

Parameters

Direction

Description

compressed

in,out

A Source is an interface that yields a sequence of bytes, you can initialize it by calling snappy::ByteArraySource(inBuf,inBufLen); where inBuf is the pointer to original data and inBufLen is the size of inBuf .

uncompressed

in,out

A Sink is an interface that consumes a sequence of bytes, you can initialize it by calling snappy::UncheckedByteArraySink(dest); where dest is the pointer to the destination buffer.

Returns:

Result

Description

Success

It returns the number of valid bytes added to sink (extra invalid bytes may have been added due to errors; the caller should ignore those)

Failure

Returns 0 if the message is corrupted and could not be decompressed or NULL parameters are passed.

Higher-level string based routines.

Higher-level string based routines (should be sufficient for most users)

size_t Compress(const char *input, size_t input_length, std::string *compressed)#

Sets “*compressed” to the compressed version of “input[0,input_length-1]”. Original contents of *compressed are lost.

Parameters

Direction

Description

input

in

This is the buffer where the data we want to compress is accessible.

input_length

in

Length of the input buffer.

compressed

in,out

This is a buffer in which compressed data is stored.

Attention

REQUIRES: “input[]” is not an alias of “*compressed”.

Returns:

Result

Description

Success

Return the number of bytes written.

Failure

Return 0 upon failure or NULL parameters are passed

bool Uncompress(const char *compressed, size_t compressed_length, std::string *uncompressed)#

Decompresses “compressed[0,compressed_length-1]” to “*uncompressed”. Original contents of “*uncompressed” are lost.

Parameters

Direction

Description

compressed

in

This is a buffer which contains compressed data.

compressed_length

in

This is the length of the compressed buffer.

uncompressed

out

Uncompressed data is stored in this buffer.

Attention

REQUIRES: “compressed[]” is not an alias of “*uncompressed”.

Returns:

Result

Description

Success

If the data inside the compressed is successfully decompressed it will return true.

Failure

It will return false if the decompression fails.

Lower-level character array based routines.

These May be useful for efficiency reasons in certain circumstances.

void RawCompress(const char *input, size_t input_length, char *compressed, size_t *compressed_length)#

Takes the data stored in “input[0..input_length]” and stores it in the array pointed to by “compressed”.

“*compressed_length” is set to the length of the compressed output.

Parameters

Direction

Description

input

in

This is the buffer where the data we want to compress is accessible.

input_length

in

Length of the input buffer.

compressed

out

This is a buffer in which compressed data is stored.

compressed_length

out

The length of the data after compression is stored in this.

Attention

REQUIRES: “compressed” must point to an area of memory that is at least “MaxCompressedLength(input_length)” bytes in length.

Note

- Example:

char  output = new char[snappy::MaxCompressedLength(input_length)];\n
size_t output_length;\n
RawCompress(input, input_length, output, &output_length);\n
... Process(output, output_length) ...\n
delete [] output;\n

Returns:

void

bool RawUncompress(const char *compressed, size_t compressed_length, char *uncompressed)#

Given data in “compressed[0..compressed_length-1]” generated by calling the Snappy::Compress routine, this routine stores the uncompressed data to uncompressed[0..GetUncompressedLength(compressed)-1] .

Parameters

Direction

Description

compressed

in

This is a buffer which contains compressed data.

compressed_length

in

This is the length of the compressed buffer.

uncompressed

out

Uncompressed data is stored in this buffer.

Returns:

Result

Description

Success

Returns true if successful.

Failure

Returns false if the message is corrupted and could not be decrypted.

bool RawUncompress(Source *compressed, char *uncompressed)#

Given data from the byte source ‘compressed’ generated by calling the Snappy::Compress routine, this routine stores the uncompressed data to uncompressed[0..GetUncompressedLength(compressed,compressed_length)-1] .

Parameters

Direction

Description

compressed

in,out

A Source is an interface that yields a sequence of bytes, you can initialize it by calling snappy::ByteArraySource(inBuf,inBufLen); where inBuf is the pointer to original data and inBufLen is the size of inBuf .

uncompressed

out

Uncompressed data is stored in this buffer.

Returns:

Result

Description

Success

Returns true if successful.

Failure

Returns false if the message is corrupted and could not be decrypted.

bool RawUncompressToIOVec(const char *compressed, size_t compressed_length, const struct iovec *iov, size_t iov_cnt)#

Given data in “compressed[0..compressed_length-1]” generated by calling the Snappy::Compress routine, this routine stores the uncompressed data to the iovec “iov”. The number of physical buffers in “iov” is given by iov_cnt and their cumulative size must be at least GetUncompressedLength(compressed). The individual buffers in “iov” must not overlap with each other.

Parameters

Direction

Description

compressed

in

This is a buffer which contains compressed data.

compressed_length

in

This is the length of the compressed buffer.

iov

in,out

The struct iovec defines one vector element. Normally, this structure is used as an array of multiple elements.

iov_cnt

in,out

This is the number of iovec structures in the array of iov .

Returns:

Result

Description

Success

Returns true if successful.

Failure

Returns false if the message is corrupted and could not be decrypted.

bool RawUncompressToIOVec(Source *compressed, const struct iovec *iov, size_t iov_cnt)#

Given data from the byte source ‘compressed’ generated by calling the Snappy::Compress routine, this routine stores the uncompressed data to the iovec “iov”. The number of physical buffers in “iov” is given by iov_cnt and their cumulative size must be at least GetUncompressedLength(compressed). The individual buffers in “iov” must not overlap with each other.

Parameters

Direction

Description

compressed

in,out

A Source is an interface that yields a sequence of bytes, you can initialize it by calling snappy::ByteArraySource(inBuf,inBufLen); where inBuf is the pointer to original data and inBufLen is the size of inBuf .

iov

in,out

The struct iovec defines one vector element. Normally, this structure is used as an array of multiple elements.

iov_cnt

out

This is the number of iovec structures in the array of iov .

Returns:

Result

Description

Success

Returns true if successful.

Failure

Returns false if the message is corrupted and could not be decrypted.

Helper Functions.

size_t MaxCompressedLength(size_t source_bytes)#

This function determines the maximal size of the compressed representation of input data that is “source_bytes” bytes in length.

Parameters

Direction

Description

source_bytes

in

The size of source in bytes.

Returns:

Result

Description

Success

Returns the maximal size of the compressed representation of input data that is “source_bytes” bytes in length.

bool GetUncompressedLength(const char *compressed, size_t compressed_length, size_t *result)#

Get the Uncompressed Length object.

This operation takes O(1) time.

Parameters

Direction

Description

compressed

in

This is a buffer which contains compressed data.

compressed_length

in

This is the length of the compressed buffer.

result

out

This is the pointer to type size_t where the uncompressed length is stored.

Attention

REQUIRES: “compressed[]” was produced by RawCompress() or Compress().

Returns:

Result

Description

Success

Returns true on successful parsing.

Failure

Returns false on parsing error.

bool GetUncompressedLengthFromMTCompressedBuffer(const char *compressed, size_t compressed_length, size_t *result)#

Get the Uncompressed Length object from the AOCL multithreaded compressor’s compressed buffer.

This operation takes O(1) time.

Parameters

Direction

Description

compressed

in

This is a buffer which contains compressed data. (along with the RAP frame)

compressed_length

in

This is the length of the compressed buffer (including the RAP frame).

result

out

This is the pointer to type size_t where the uncompressed length is stored.

Attention

REQUIRES: “compressed[]” was produced by RawCompress() or Compress() IN AOCL’s MULTITHREADED MODE.

Returns:

Result

Description

Success

Returns true on successful parsing.

Failure

Returns false on parsing error.

bool IsValidCompressedBuffer(const char *compressed, size_t compressed_length)#

Returns true iff the contents of “compressed[]” can be uncompressed successfully. Does not return the uncompressed data. Takes time proportional to compressed_length, but is usually at least a factor of four faster than actual decompression.

Parameters

Direction

Description

compressed

in

This is a buffer which contains compressed data.

compressed_length

in

This is the length of the compressed buffer.

Returns:

Result

Description

Success

Returns true iff the contents of “compressed[]” can be uncompressed successfully.

Failure

Returns false if error.

bool IsValidCompressed(Source *compressed)#

Returns true iff the contents of “compressed” can be uncompressed successfully. Does not return the uncompressed data. Takes time proportional to *compressed length, but is usually at least a factor of four faster than actual decompression. On success, consumes all of *compressed. On failure, consumes an unspecified prefix of *compressed.

Parameters

Direction

Description

compressed

in,out

A Source is an interface that yields a sequence of bytes, you can initialize it by calling snappy::ByteArraySource(inBuf,inBufLen); where inBuf is the pointer to original data and inBufLen is the size of inBuf .

Returns:

Result

Description

Success

Returns true iff the contents of “compressed” can be uncompressed successfully.

Failure

Returns false if error.

Variables

static int kBlockLog = 16#
static size_t kBlockSize = 1 << kBlockLog#
static int kMaxHashTableBits = 14#
static size_t kMaxHashTableSize = 1 << kMaxHashTableBits#
static int kMinHashTableBits = 8#
static size_t kMinHashTableSize = 1 << kMinHashTableBits#

6.1. SNAPPY C APIs#

group SNAPPY_C_API

Plain C interface (a wrapper around the C++ implementation).

Enums

enum snappy_status#

Return values; see the documentation for each function to know what each can return.

Values:

enumerator SNAPPY_OK#
enumerator SNAPPY_INVALID_INPUT#
enumerator SNAPPY_BUFFER_TOO_SMALL#

Functions

snappy_status snappy_compress(const char *input, size_t input_length, char *compressed, size_t *compressed_length)#

Takes the data stored in “input[0..input_length-1]” and stores it in the array pointed to by “compressed”.

Parameters

Direction

Description

input

in

This is the buffer where the data we want to compress is accessible.

input_length

in

Length of the input buffer.

compressed

out

This is a buffer in which compressed data is stored.

compressed_length

in,out

compressed_length signals the space available in “compressed” buffer. After successful compression compressed_length contains the true length of the compressed output.

Note

- Example:

size_t output_length = snappy_max_compressed_length(input_length);
char* output = (char*)malloc(output_length);
if (snappy_compress(input, input_length, output, &output_length)
    == SNAPPY_OK) {
  ... Process(output, output_length) ...
}
free(output);

Returns:

Result

Description

Success

Returns SNAPPY_OK if successful.

Failure

While calling this function if compressed_length is not at least equal to “snappy_max_compressed_length(input_length)”, SNAPPY_BUFFER_TOO_SMALL is returned.

size_t snappy_max_compressed_length(size_t source_length)#

This function determines the maximal size of the compressed representation of input data that is “source_length” bytes in length.

Parameters

Direction

Description

source_length

in

The size of source in bytes.

Returns:

Result

Description

Success

Returns the maximal size of the compressed representation of input data that is “source_bytes” bytes in length.

snappy_status snappy_uncompress(const char *compressed, size_t compressed_length, char *uncompressed, size_t *uncompressed_length)#

Given data in “compressed[0..compressed_length-1]” generated by calling the snappy_compress routine, this routine stores the uncompressed data to uncompressed[0..uncompressed_length-1].

Parameters

Direction

Description

compressed

in

This is a buffer which contains compressed data.

compressed_length

in

This is the length of the compressed buffer.

uncompressed

out

Uncompressed data is stored in this buffer.

uncompressed_length

in,out

uncompressed_length signals the space available in “uncompressed”. After successful decompression, uncompressed_length contains the true length of the decompressed output.

Note

- Example:

size_t output_length;
if (snappy_uncompressed_length(input, input_length, &output_length)
    != SNAPPY_OK) {
  ... fail ...
}
char* output = (char*)malloc(output_length);
if (snappy_uncompress(input, input_length, output, &output_length)
    == SNAPPY_OK) {
  ... Process(output, output_length) ...
}
free(output);

Returns:

Result

Description

Success

Returns SNAPPY_OK if successful.

Failure

While calling this function if uncompressed_length is not at least equal to the value returned by snappy_uncompressed_length for this stream, SNAPPY_BUFFER_TOO_SMALL is returned Returns (a value not equal to SNAPPY_OK) if the message is corrupted and could not be decrypted.

snappy_status snappy_uncompressed_length(const char *compressed, size_t compressed_length, size_t *result)#

Get the Uncompressed Length object.

This operation takes O(1) time.

Parameters

Direction

Description

compressed

in

This is a buffer which contains compressed data.

compressed_length

in

This is the length of the compressed buffer.

result

out

This is the pointer to type size_t where the uncompressed length is stored.

Attention

REQUIRES: “compressed[]” was produced by RawCompress() or Compress().

Returns:

Result

Description

Success

Returns true on successful parsing.

Failure

Returns false on parsing error.

snappy_status snappy_validate_compressed_buffer(const char *compressed, size_t compressed_length)#

Returns true iff the contents of “compressed[]” can be uncompressed successfully. Does not return the uncompressed data. Takes time proportional to compressed_length, but is usually at least a factor of four faster than actual decompression.

Parameters

Direction

Description

compressed

in

This is a buffer which contains compressed data.

compressed_length

in

This is the length of the compressed buffer.

Returns:

Result

Description

Success

Returns true iff the contents of “compressed[]” can be uncompressed successfully.

Failure

Returns false if error.