Module compression

Record known compressors

Includes utilities for determining whether or not to compress

Functions

▶ def byte_sample(b, size, n)
Sample a bytestring from many locations
source link
Parameters
- b : bytes or memoryview
- size : int
```
size of each sample to collect
```
- n : int
```
number of samples to collect
```
▶ def decompress(header, frames)
Decompress frames according to information in the header
source link
Reexports
- Imported in distributed.protocol.core.
- Imported in distributed.protocol.
- Imported in distributed.protocol.serialize.
- Imported in distributed.protocol.tests.test_numpy.
- Imported in distributed.protocol.tests.test_pandas.
▶ def get_default_compression()
source link
Reexports
- Imported in distributed.comm.core.
▶ def maybe_compress(payload, min_size=1e4, sample_size=1e4, nsamples=5, ...)
Maybe compress payload
def maybe_compress(
payload,
min_size=1e4,
sample_size=1e4,
nsamples=5,
compression=dask.config.get("distributed.comm.compression"),
)
source link
```
1.  We don't compress small messages
2.  We sample the payload in a few spots, compress that, and if it doesn't
    do any good we return the original
3.  We then compress the full original, it it doesn't compress well then we
    return the original
4.  We return the compressed result
```
Reexports
- Imported in distributed.protocol.core.
- Imported in distributed.protocol.
- Imported in distributed.protocol.serialize.
- Imported in distributed.protocol.tests.test_numpy.
- Imported in distributed.protocol.tests.test_protocol.