90 Commits

Author SHA1 Message Date
Yann Collet
07c526c0af Merge pull request #1564 from lz4/filecompress_fix
fixed minor static analyzer warning in fileCompress example
2025-01-16 12:33:06 -08:00
Yann Collet
b8af3e2f74 fix fuzzer test
even the _destSize*() variants need a targetDstSize >= 1 to be successful,
since it's the smallest valid payload allowed by LZ4.
2025-01-16 11:47:15 -08:00
Yann Collet
555d294b9b fix flushing single byte 2025-01-13 19:13:22 -08:00
Yann Collet
d5362329cd simplified LZ4F_localSaveDict() call pattern
and improves traces a bit
2025-01-13 18:51:58 -08:00
Yann Collet
b35b24fb78 improved lz4frame corruption error reporting 2025-01-13 04:45:41 -08:00
Yann Collet
923839bbc9 fixed minor static analyzer warning in fileCompress example 2025-01-12 23:30:55 -08:00
Yann Collet
205fd851dd Merge pull request #1556 from lz4/lz4opt_src0
fix lz4_opt behavior when srcSize == 0
2024-12-23 18:08:48 -08:00
Yann Collet
527431e832 fix minor scan-build warning 2024-12-23 13:30:37 -08:00
Yann Collet
de629c38e9 fix modulo 0, part 2 2024-12-23 13:08:39 -08:00
Yann Collet
eba7d05b8e fix modulo operation in tests 2024-12-23 11:31:28 -08:00
Yann Collet
9b9e03ff09 lz4hc: centralized input sanitization 2024-12-23 11:27:00 -08:00
Yann Collet
710ed4f0d7 no good reason to FORCE_INLINE 2024-12-23 11:03:24 -08:00
Yann Collet
5c2068c606 fix lz4_opt behavior when srcSize == 0
reported by @rinsuki
2024-12-23 10:46:33 -08:00
Yann Collet
6869ab67d7 Merge pull request #1555 from lz4/dependabot/github_actions/actions/upload-artifact-4.5.0
Bump actions/upload-artifact from 4.4.3 to 4.5.0
2024-12-23 10:16:20 -08:00
dependabot[bot]
7e41965fbb Bump actions/upload-artifact from 4.4.3 to 4.5.0
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4.4.3 to 4.5.0.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](b4b15b8c7c...6f51ac03b9)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-12-23 06:19:55 +00:00
Yann Collet
6cf42afbea Merge pull request #1545 from dearblue/repology
Specify parameters for repology status
2024-11-25 09:00:50 -08:00
Yann Collet
17a8cf523c Merge pull request #1547 from lz4/dependabot/github_actions/github/codeql-action-3.27.5
Bump github/codeql-action from 3.26.2 to 3.27.5
2024-11-25 00:26:46 -08:00
dependabot[bot]
3f7a031e62 Bump github/codeql-action from 3.26.2 to 3.27.5
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.26.2 to 3.27.5.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](429e197704...f09c1c0a94)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-11-25 06:48:28 +00:00
Yann Collet
8da5d43433 Merge pull request #1544 from t-mat/fix-examples-dictionaryRandomAccess
Fix examples dictionary random access
2024-11-24 08:28:13 -08:00
Yann Collet
0c841277a4 Merge pull request #1546 from lz4/gh_x32_disable
disable x32 tests on Github Actions CI
2024-11-24 07:54:20 -08:00
Yann Collet
11aceb0896 disable x32 tests on Github Actions CI
Since the support was already restricted to older compilers,
it seems this is going away for good.

Testing x32, if deemed necessary, will have to employ other CI.
2024-11-24 06:45:01 -08:00
dearblue
17c36d9c9e Specify parameters for repology status 2024-11-24 18:56:12 +09:00
t-mat
855a097886 chore: Fix dictionaryRandomAccess's issue for small file
When size of the given file is less than 1024 (DICTIONARY_BYTES), dict[] may contain garbage.
Therefore, test_decompress() fails with garbage dictionary.

This change set fixes the failure of test for dictionaryRandomAccess with TESTFILE_SMALL.
2024-11-18 16:43:13 +09:00
t-mat
184ffbd686 chore: Add test for dictionaryRandomAccess
dictionaryRandomAccess doesn't pass this test.
This issue will be fixed in the follow up change set.

Actual failure log:
./dictionaryRandomAccess .gitignore .gitignore 0 32
inp    = [.gitignore]
lz4    = [.gitignore.lz4s-1024]
dec    = [.gitignore.lz4s-1024.dec]
dict   = [.gitignore]
offset = [0]
length = [32]
compress : .gitignore -> .gitignore.lz4s-1024
compress : done
decompress : .gitignore.lz4s-1024 -> .gitignore.lz4s-1024.dec
decompress : done
verify : .gitignore <-> .gitignore.lz4s-1024.dec
verify : NG
2024-11-18 16:42:50 +09:00
Yann Collet
7887022e5d Merge pull request #1510 from ds-sloth/respect-global-pic
CMakeLists.txt: respect global CMAKE_POSITION_INDEPENDENT_CODE flag
2024-11-05 10:59:46 -08:00
Yann Collet
c505a48525 Merge pull request #1537 from lz4/dependabot/github_actions/actions/setup-python-5.3.0
Bump actions/setup-python from 5.2.0 to 5.3.0
2024-10-28 16:16:19 -07:00
dependabot[bot]
7f51552ca4 Bump actions/setup-python from 5.2.0 to 5.3.0
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5.2.0 to 5.3.0.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](f677139bbe...0b93645e9f)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-28 18:57:35 +00:00
Yann Collet
389c153911 Merge pull request #1538 from lz4/dependabot/github_actions/actions/checkout-4.2.2
Bump actions/checkout from 4.2.1 to 4.2.2
2024-10-28 11:48:45 -07:00
dependabot[bot]
5994fdab5c Bump actions/checkout from 4.2.1 to 4.2.2
Bumps [actions/checkout](https://github.com/actions/checkout) from 4.2.1 to 4.2.2.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](eef61447b9...11bd71901b)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-28 06:43:42 +00:00
Yann Collet
d9c01a3d49 Merge pull request #1531 from lz4/dependabot/github_actions/actions/upload-artifact-4.4.3
Bump actions/upload-artifact from 4.4.0 to 4.4.3
2024-10-15 09:36:14 -07:00
Yann Collet
3a90748da5 Merge pull request #1533 from lz4/dependabot/github_actions/actions/checkout-4.2.1
Bump actions/checkout from 4.2.0 to 4.2.1
2024-10-14 18:41:36 -07:00
dependabot[bot]
48f8fe1636 Bump actions/checkout from 4.2.0 to 4.2.1
Bumps [actions/checkout](https://github.com/actions/checkout) from 4.2.0 to 4.2.1.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](d632683dd7...eef61447b9)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-14 06:38:49 +00:00
dependabot[bot]
15833a8260 Bump actions/upload-artifact from 4.4.0 to 4.4.3
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4.4.0 to 4.4.3.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](50769540e7...b4b15b8c7c)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-14 06:38:39 +00:00
Yann Collet
c612292343 Merge pull request #1524 from bgilbert/freestanding
build/meson: force `LZ4_DEBUG=0` for `tests/freestanding`
2024-10-06 14:37:03 -07:00
Yann Collet
b33d5e8e2e Merge pull request #1522 from bgilbert/meson
build/meson: avoid passing long options to `ln`
2024-10-06 12:11:29 -07:00
Yann Collet
dc29c67e98 Merge pull request #1523 from bgilbert/uname
build: use `uname -m` to detect processor
2024-10-06 11:05:03 -07:00
Benjamin Gilbert
b8f1cc1f07 build/meson: force LZ4_DEBUG=0 for tests/freestanding
Match the Makefile, which never sets LZ4_DEBUG for this test.  If
LZ4_DEBUG is non-zero, lib/lz4.c #includes <assert.h>, the musl version of
which has a conflicting definition for __assert_fail:

    https://www.openwall.com/lists/musl/2019/03/04/6

Fixes build failure:

    ../../../tests/freestanding.c:173:6: error: conflicting types for '__assert_fail'; have 'void(const char *, const char *, unsigned int,  const char *)'
      173 | void __assert_fail(const char * assertion, const char * file, unsigned int line, const char * function) {
          |      ^~~~~~~~~~~~~
    In file included from ../../../tests/../lib/lz4.c:270,
                     from ../../../tests/freestanding.c:49:
    /usr/include/assert.h:19:16: note: previous declaration of '__assert_fail' with type 'void(const char *, const char *, int,  const char *)'
       19 | _Noreturn void __assert_fail (const char *, const char *, int, const char *);
          |                ^~~~~~~~~~~~~
2024-10-06 04:11:36 -07:00
Benjamin Gilbert
bc8003bc64 build: use uname -m to detect processor
Ubuntu patches uname -p to return the processor type:

    https://git.launchpad.net/ubuntu/+source/coreutils/tree/debian/patches/80_fedora_sysinfo.patch?h=ubuntu/oracular

and Fedora also did before Fedora 38:

    cd953e11dd

but in general, uname -p returns "unknown".  Use uname -m instead.

Fixes build error:

    /usr/bin/ld: /tmp/ccgv9E4e.o: in function `_start':
    freestanding.c:(.text+0x6255a): multiple definition of `_start'; /usr/lib/gcc/x86_64-redhat-linux/14/../../../../lib64/crt1.o:(.text+0x0): first defined here
    collect2: error: ld returned 1 exit status
2024-10-06 03:44:41 -07:00
Benjamin Gilbert
6d17921245 build/meson: avoid passing long options to ln
On macOS, ln only supports short options.
2024-10-06 03:18:27 -07:00
Benjamin Gilbert
ac3e03c55c build/meson: update directory path in README 2024-10-06 03:15:38 -07:00
Yann Collet
c470f4dae6 Merge pull request #1499 from lz4/dependabot/github_actions/actions/upload-artifact-4.4.0
Bump actions/upload-artifact from 4.3.6 to 4.4.0
2024-10-04 09:43:38 -07:00
Yann Collet
c11bf7733b Merge pull request #1497 from rpurdie/dev
lib/Makefile: Fix static library reproducibility
2024-10-01 09:15:55 -07:00
Yann Collet
54f253c1da Merge pull request #1500 from lz4/dependabot/github_actions/actions/setup-python-5.2.0
Bump actions/setup-python from 5.1.1 to 5.2.0
2024-10-01 08:10:01 -07:00
Yann Collet
def5c3bd0e Merge pull request #1506 from lz4/clevel_doc
update -H documentation regarding compression levels
2024-10-01 06:46:35 -07:00
dependabot[bot]
c14d595b3a Bump actions/setup-python from 5.1.1 to 5.2.0
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5.1.1 to 5.2.0.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](39cd14951b...f677139bbe)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-01 06:19:15 +00:00
Yann Collet
824f7a7ad0 Merge pull request #1508 from lz4/fix_tname
minor: fix test name
2024-09-30 23:12:03 -07:00
Yann Collet
1d99913f5e Merge pull request #1518 from lz4/dependabot/github_actions/actions/checkout-4.2.0
Bump actions/checkout from 4.1.7 to 4.2.0
2024-09-30 23:01:03 -07:00
Yann Collet
c5bec79b69 Merge pull request #1520 from dearblue/freebsd-14.1
Update FreeBSD VM image to 14.1
2024-09-30 21:15:26 -07:00
dependabot[bot]
42f0995cd5 Bump actions/checkout from 4.1.7 to 4.2.0
Bumps [actions/checkout](https://github.com/actions/checkout) from 4.1.7 to 4.2.0.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](692973e3d9...d632683dd7)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-10-01 03:40:51 +00:00
Yann Collet
0b33a695e6 Merge pull request #1521 from lz4/fix_vs2022
Fix vs2022
2024-09-30 20:32:46 -07:00
Yann Collet
696aeb1449 fix vs2022 in github actions CI 2024-09-30 17:43:59 -07:00
Yann Collet
5e29e3cb6d switch win32 test to debug
in the hope that optimization errors will not be there.
2024-09-30 17:04:43 -07:00
Yann Collet
032029ccdd enable vs2019 in github actions
see if it works better than current vs2022.
2024-09-30 16:59:59 -07:00
dearblue
4101102391 Update FreeBSD VM image to 14.1
FreeBSD 14.0 will reach the end of life on 2024-09-30.
The updated 14.1 is scheduled to end-of-life on 2025-03-31.

ref. https://www.freebsd.org/releases/14.2R/schedule/
2024-09-30 23:24:24 +09:00
ds-sloth
0e44420380 CMakeLists.txt: use CMAKE_POSITION_INDEPENDENT_CODE to set default only 2024-09-16 19:21:35 -04:00
ds-sloth
80d4c6215a CMakeLists.txt: respect global CMAKE_POSITION_INDEPENDENT_CODE flag 2024-09-11 21:37:35 -04:00
Yann Collet
452f35c3c0 minor: fix test name
obvious insertion mishap
2024-09-07 15:50:10 -07:00
Yann Collet
217bdb2f2f update -H documentation regarding compression levels 2024-09-04 11:03:28 -07:00
dependabot[bot]
1a744ea740 Bump actions/upload-artifact from 4.3.6 to 4.4.0
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4.3.6 to 4.4.0.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](834a144ee9...50769540e7)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-02 06:47:28 +00:00
Richard Purdie
e0b1747860 lib/Makefile: Fix static library reproducibility
The static library contents varies depending of the order of the object
files on disk meaning it isn't reproducible.

To avoid this, use the SRCFILES values which are already sorted, mapped
to the object names instead.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-08-29 11:01:11 +01:00
Yann Collet
e0781fbe4b Merge pull request #1493 from lz4/dependabot/github_actions/github/codeql-action-3.26.2
Bump github/codeql-action from 3.25.12 to 3.26.2
2024-08-19 09:34:06 -07:00
dependabot[bot]
fdfa68c752 Bump github/codeql-action from 3.25.12 to 3.26.2
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.25.12 to 3.26.2.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](4fa2a79536...429e197704)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-08-19 06:51:15 +00:00
Yann Collet
cde811e3ef Merge pull request #1491 from lz4/dependabot/github_actions/actions/upload-artifact-4.3.6
Bump actions/upload-artifact from 4.3.5 to 4.3.6
2024-08-12 09:29:11 -07:00
dependabot[bot]
776a8fcd7a Bump actions/upload-artifact from 4.3.5 to 4.3.6
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4.3.5 to 4.3.6.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](89ef406dd8...834a144ee9)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-08-12 06:19:47 +00:00
Yann Collet
208b2c7d58 Merge pull request #1488 from lz4/dependabot/github_actions/actions/upload-artifact-4.3.5
Bump actions/upload-artifact from 4.3.4 to 4.3.5
2024-08-07 09:45:03 -07:00
dependabot[bot]
f6370cf1b3 Bump actions/upload-artifact from 4.3.4 to 4.3.5
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4.3.4 to 4.3.5.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](0b2256b8c0...89ef406dd8)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-08-05 06:08:38 +00:00
Yann Collet
45fda541f0 Merge pull request #1485 from grobian/meson-no-stdc99
meson: do not force c99 mode
2024-08-03 23:05:47 -07:00
Yann Collet
d35f779858 Merge pull request #1487 from lz4/better_ptime
slighly better cpu time measurement for Windows
2024-08-03 10:44:56 -07:00
Yann Collet
6410f177c6 slighly better cpu time measurement for Windows
correctly skips the initialization part of the programs,
so that it's now on par with posix's clock().

In the future, would allow to measure cpu time between arbitrary points in the program execution, though for the time being, it's used for full session.
2024-08-03 09:51:55 -07:00
Yann Collet
7e79c1da1f Merge pull request #1486 from lz4/win_ptime
fix process time counter on Windows for very large files
2024-08-03 00:21:34 -07:00
Yann Collet
f7e65802dc fix process time counter on Windows for very large files 2024-08-02 23:39:45 -07:00
Yann Collet
3d946c68aa Merge pull request #1479 from heitbaum/cmake
add back lz4c target but default to OFF
2024-07-30 13:10:42 -07:00
Yann Collet
1c0a090eef Merge pull request #1481 from lz4/doc_T
-h only documents -T# when multithreading is enabled
2024-07-30 13:10:26 -07:00
Yann Collet
e8ca3bac3d Merge pull request #1482 from lz4/dependabot/github_actions/ossf/scorecard-action-2.4.0
Bump ossf/scorecard-action from 2.3.3 to 2.4.0
2024-07-30 13:09:46 -07:00
Fabian Groffen
bce6a94c25 meson: do not force c99 mode
On Solaris/OpenIndiana hosts forcing (old) C99 means disabling POSIX
2001 functionality, resulting in errors like

In file included from ../../../net/ptah/export/gentoo/working-repos/lz4/programs/bench.c:39:
../../../net/ptah/export/gentoo/working-repos/lz4/programs/util.h: In function  UTIL_getOpenFileSize’:
../../../net/ptah/export/gentoo/working-repos/lz4/programs/util.h:156:23: error: implicit declaration of function ‘fileno’ [-Wimplicit-function-declaration]
  156 | #  define UTIL_fileno fileno
      |                       ^~~~~~
../../../net/ptah/export/gentoo/working-repos/lz4/programs/util.h:325:10: note: in expansion of macro ‘UTIL_fileno’
  325 |     fd = UTIL_fileno(file);
      |          ^~~~~~~~~~~

These can be fixed either by forcing a standard to be applied in programs/platform.h or by not forcing the compiler to use an old standard.

Since CMake and Makefile don't force C99 by default either, just drop it from
meson.build.

Signed-off-by: Fabian Groffen <grobian@gentoo.org>
2024-07-30 21:50:06 +02:00
dependabot[bot]
451f83b33d Bump ossf/scorecard-action from 2.3.3 to 2.4.0
Bumps [ossf/scorecard-action](https://github.com/ossf/scorecard-action) from 2.3.3 to 2.4.0.
- [Release notes](https://github.com/ossf/scorecard-action/releases)
- [Changelog](https://github.com/ossf/scorecard-action/blob/main/RELEASE.md)
- [Commits](dc50aa9510...62b2cac7ed)

---
updated-dependencies:
- dependency-name: ossf/scorecard-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-07-29 06:45:00 +00:00
Yann Collet
fe399506bd Merge pull request #1469 from hhoffstaette/no-sp
Fix building freestanding test with clang & enabled stack protector
2024-07-28 21:32:24 -07:00
Yann Collet
0f273e1868 Merge pull request #1472 from axiqia/dev
Fix file size check in LZ4F_readOpen
2024-07-28 21:32:04 -07:00
Yann Collet
9d298e9540 -h only documents -T# when multithreading is enabled
fixes #1480
2024-07-28 21:30:29 -07:00
Rudi Heitbaum
6969661757 add back lz4c target but default to OFF
Partially revert "removed lz4c target"
This reverts commit 65998fecaf.
2024-07-28 01:26:49 +00:00
Yann Collet
d5276bd8f1 Merge pull request #1475 from eltociear/patch-1
docs: update lz4conf.h
2024-07-25 14:53:08 -07:00
Ikko Eltociear Ashimine
7739164446 docs: update lz4conf.h
overriden -> overridden
2024-07-26 01:24:23 +09:00
Shuai Xue
5ec9143e1e Fix file size check in LZ4F_readOpen
When the size of the opened compressed file is less than
LZ4F_HEADER_SIZE_MAX (19) bytes, LZ4F_readOpen() will return an io_read
error code. The minimum compressed file size is LZ4F_HEADER_SIZE_MIN +
LZ4F_ENDMARK_SIZE.

Fix to check the file size with minimum size.

Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
2024-07-24 19:40:52 +08:00
Yann Collet
708fe99cc4 Merge pull request #1471 from lz4/appveyor_nodeploy
remove "deploy" paragraph from Appveyor CI script
2024-07-24 01:18:24 -07:00
Yann Collet
7c2a2606fc Merge pull request #1468 from hhoffstaette/start-alignment
Fix stack alignment of _start() in freestanding test
2024-07-24 01:18:13 -07:00
Yann Collet
58b943cf1e Merge pull request #1470 from dg0yt/cmake
Add an lz4::lz4 target to the exported package
2024-07-24 01:18:00 -07:00
Yann Collet
780f80733a remove "deploy" paragraph from Appveyor CI script
it isn't safe and doesn't work anyway
2024-07-23 14:16:46 -07:00
Kai Pastor
ca197e67a0 Add an lz4::lz4 target to the exported package 2024-07-23 20:32:43 +02:00
Holger Hoffstätte
d08826c4fd Fix building freestanding test with clang & enabled stack protector
The freestanding test is special as it requires reimplementation of
runtime/toolchain functionality. When building with clang where the
compiler is configured to enable the stack-protector by default,
building fails due to missing __stack_chk_fail symbols.
Thefore disable stack protection specifically for this test.
2024-07-23 13:36:41 +02:00
Holger Hoffstätte
63267a77d8 Fix stack alignment of _start() in freestanding test
When the freestanding test is built with any kind of optimization
that enables vectorized loops, special care must be taken to align
the stack for _start() at a 16-byte boundary.
2024-07-22 22:16:16 +02:00
30 changed files with 490 additions and 315 deletions

View File

@@ -2,7 +2,7 @@ task:
name: FreeBSD
freebsd_instance:
matrix:
image_family: freebsd-14-0
image_family: freebsd-14-1
install_script: pkg install -y gmake
script: |
cc -v

View File

@@ -32,8 +32,9 @@ jobs:
# pkgs : apt-get package names. It can include multiple package names which are delimited by space.
# cc : C compiler executable.
# cxx : C++ compiler executable for targets `cxxtest` and `ctocxxtest`.
# x32 : Set 'true' if compiler supports x32. Otherwise, set 'false'.
# Set 'fail' if it supports x32 but fails for now. 'fail' cases must be removed.
# x32 : Note: x32 tests do no longer work on github ci.
# Set 'true' if compiler & os support x32. Otherwise, set 'false'.
# Set 'fail' if it supports x32 but fails for now. 'fail' cases must be removed.
# x86 : Set 'true' if compiler supports x86 (-m32). Otherwise, set 'false'.
# Set 'fail' if it supports x86 but fails for now. 'fail' cases must be removed.
# cxxtest : Set 'true' if it can be compiled as C++ code. Otherwise, set 'false'.
@@ -51,8 +52,8 @@ jobs:
{ pkgs: 'gcc-11 g++-11 lib32gcc-11-dev libx32gcc-11-dev', cc: gcc-11, cxx: g++-11, x32: 'false', x86: 'true', cxxtest: 'true', freestanding: 'true', os: ubuntu-22.04, },
{ pkgs: 'gcc-10 g++-10 lib32gcc-10-dev libx32gcc-10-dev', cc: gcc-10, cxx: g++-10, x32: 'false', x86: 'true', cxxtest: 'true', freestanding: 'true', os: ubuntu-22.04, },
{ pkgs: 'gcc-9 g++-9 lib32gcc-9-dev libx32gcc-9-dev', cc: gcc-9, cxx: g++-9, x32: 'false', x86: 'true', cxxtest: 'true', freestanding: 'true', os: ubuntu-22.04, },
{ pkgs: 'gcc-8 g++-8 lib32gcc-8-dev libx32gcc-8-dev', cc: gcc-8, cxx: g++-8, x32: 'true', x86: 'true', cxxtest: 'true', freestanding: 'true', os: ubuntu-20.04, },
{ pkgs: 'gcc-7 g++-7 lib32gcc-7-dev libx32gcc-7-dev', cc: gcc-7, cxx: g++-7, x32: 'true', x86: 'true', cxxtest: 'true', freestanding: 'true', os: ubuntu-20.04, },
{ pkgs: 'gcc-8 g++-8 lib32gcc-8-dev libx32gcc-8-dev', cc: gcc-8, cxx: g++-8, x32: 'false', x86: 'true', cxxtest: 'true', freestanding: 'true', os: ubuntu-20.04, },
{ pkgs: 'gcc-7 g++-7 lib32gcc-7-dev libx32gcc-7-dev', cc: gcc-7, cxx: g++-7, x32: 'false', x86: 'true', cxxtest: 'true', freestanding: 'true', os: ubuntu-20.04, },
# clang
{ pkgs: 'lib32gcc-12-dev libx32gcc-12-dev', cc: clang, cxx: clang++, x32: 'false', x86: 'true', cxxtest: 'true', freestanding: 'false', os: ubuntu-latest, },
@@ -61,11 +62,11 @@ jobs:
{ pkgs: 'clang-13 lib32gcc-12-dev libx32gcc-12-dev', cc: clang-13, cxx: clang++-13, x32: 'false', x86: 'true', cxxtest: 'true', freestanding: 'false', os: ubuntu-22.04, },
{ pkgs: 'clang-12 lib32gcc-12-dev libx32gcc-12-dev', cc: clang-12, cxx: clang++-12, x32: 'false', x86: 'true', cxxtest: 'true', freestanding: 'false', os: ubuntu-22.04, },
{ pkgs: 'clang-11 lib32gcc-12-dev libx32gcc-12-dev', cc: clang-11, cxx: clang++-11, x32: 'false', x86: 'true', cxxtest: 'true', freestanding: 'false', os: ubuntu-22.04, },
{ pkgs: 'clang-10 lib32gcc-10-dev libx32gcc-10-dev', cc: clang-10, cxx: clang++-10, x32: 'fail', x86: 'true', cxxtest: 'true', freestanding: 'false', os: ubuntu-20.04, },
{ pkgs: 'clang-9 lib32gcc-10-dev libx32gcc-10-dev', cc: clang-9, cxx: clang++-9, x32: 'fail', x86: 'true', cxxtest: 'true', freestanding: 'false', os: ubuntu-20.04, },
{ pkgs: 'clang-8 lib32gcc-10-dev libx32gcc-10-dev', cc: clang-8, cxx: clang++-8, x32: 'fail', x86: 'true', cxxtest: 'true', freestanding: 'false', os: ubuntu-20.04, },
{ pkgs: 'clang-7 lib32gcc-7-dev libx32gcc-7-dev', cc: clang-7, cxx: clang++-7, x32: 'fail', x86: 'true', cxxtest: 'true', freestanding: 'false', os: ubuntu-20.04, },
{ pkgs: 'clang-6.0 lib32gcc-10-dev libx32gcc-10-dev', cc: clang-6.0, cxx: clang++-6.0, x32: 'fail', x86: 'true', cxxtest: 'true', freestanding: 'false', os: ubuntu-20.04, },
{ pkgs: 'clang-10 lib32gcc-10-dev libx32gcc-10-dev', cc: clang-10, cxx: clang++-10, x32: 'false', x86: 'true', cxxtest: 'true', freestanding: 'false', os: ubuntu-20.04, },
{ pkgs: 'clang-9 lib32gcc-10-dev libx32gcc-10-dev', cc: clang-9, cxx: clang++-9, x32: 'false', x86: 'true', cxxtest: 'true', freestanding: 'false', os: ubuntu-20.04, },
{ pkgs: 'clang-8 lib32gcc-10-dev libx32gcc-10-dev', cc: clang-8, cxx: clang++-8, x32: 'false', x86: 'true', cxxtest: 'true', freestanding: 'false', os: ubuntu-20.04, },
{ pkgs: 'clang-7 lib32gcc-7-dev libx32gcc-7-dev', cc: clang-7, cxx: clang++-7, x32: 'false', x86: 'true', cxxtest: 'true', freestanding: 'false', os: ubuntu-20.04, },
{ pkgs: 'clang-6.0 lib32gcc-10-dev libx32gcc-10-dev', cc: clang-6.0, cxx: clang++-6.0, x32: 'false', x86: 'true', cxxtest: 'true', freestanding: 'false', os: ubuntu-20.04, },
]
runs-on: ${{ matrix.os }}
@@ -75,7 +76,7 @@ jobs:
CXX: ${{ matrix.cxx }}
FIXME__LZ4_CI_IGNORE : ' echo Error. But we ignore it for now.'
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: apt-get install
run: |
@@ -164,7 +165,7 @@ jobs:
name: Fullbench
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: apt-get install
run: |
@@ -191,7 +192,7 @@ jobs:
name: Fuzzer test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: apt-get install
run: |
@@ -212,7 +213,7 @@ jobs:
name: LZ4 Makefile - support for standard variables
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: make standard_variables
run: make standard_variables V=1
@@ -222,7 +223,7 @@ jobs:
name: LZ4 versions test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: apt-get install
run: |
@@ -237,7 +238,7 @@ jobs:
name: LZ4 inter-versions ABI compatibility test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: apt-get install
run: |
@@ -252,7 +253,7 @@ jobs:
name: LZ4 frame test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: apt-get install
run: |
@@ -269,7 +270,7 @@ jobs:
name: test different values of LZ4_MEMORY_USAGE
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: LZ4_MEMORY_USAGE
run: make V=1 -C tests test-compile-with-lz4-memory-usage
@@ -278,7 +279,7 @@ jobs:
name: Custom LZ4_DISTANCE_MAX
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: custom LZ4_DISTANCE_MAX; test LZ4_USER_MEMORY_FUNCTIONS
run: |
CPPFLAGS='-DLZ4_DISTANCE_MAX=8000' make V=1 check
@@ -294,7 +295,7 @@ jobs:
name: Test lz4 compression on a block device
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: create a block device, compress it with lz4 # alternative : blindly use /dev/loop0, seems to always exist
run: |
make lz4
@@ -321,7 +322,7 @@ jobs:
name: make cppcheck
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: apt-get install
run: |
sudo apt-get update
@@ -340,7 +341,7 @@ jobs:
name: make staticAnalyze
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: apt-get install
run: |
sudo apt-get update
@@ -362,7 +363,7 @@ jobs:
name: valgrind
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: apt-get install
run: |
sudo apt-get update
@@ -381,7 +382,7 @@ jobs:
name: Linux x64 ubsan
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: ubsan
run: |
@@ -393,7 +394,7 @@ jobs:
name: Linux x86 ubsan
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: apt-get install
run: |
@@ -411,7 +412,7 @@ jobs:
name: Linux x64 ASAN
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: setup
run: sudo sysctl -w vm.mmap_min_addr=4096
@@ -429,7 +430,7 @@ jobs:
name: Linux x64 MSAN
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: check lz4
run: make clean; CC=clang CFLAGS=-fsanitize=memory LDFLAGS=-fsanitize=memory make -j check V=1
@@ -444,7 +445,7 @@ jobs:
name: Linux x64 TSAN
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: lz4 cli
run: make clean; CC=clang CPPFLAGS="-fsanitize=thread" make -j -C tests test-lz4 V=1
@@ -454,7 +455,7 @@ jobs:
name: lint unicode in ./lib/, ./tests/ and ./programs/
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: unicode lint
run: bash ./tests/unicode_lint.sh
@@ -462,7 +463,7 @@ jobs:
name: make examples
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: apt-get install
run: |
sudo apt-get update
@@ -498,7 +499,7 @@ jobs:
dry-run: false
sanitizer: ${{ matrix.sanitizer }}
- name: Upload Crash
uses: actions/upload-artifact@0b2256b8c012f0828dc542b3febcab082c67f72b # https://github.com/actions/upload-artifact v4.3.4
uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # https://github.com/actions/upload-artifact v4.5.0
if: failure() && steps.build.outcome == 'success'
with:
name: ${{ matrix.sanitizer }}-artifacts
@@ -547,7 +548,7 @@ jobs:
XEMU: ${{ matrix.xemu }}
MAKEVAR: ${{ matrix.makevar }}
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: apt-get install
run: |
@@ -581,7 +582,7 @@ jobs:
name: macOS
runs-on: macos-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: Environment info
run: |
@@ -613,7 +614,7 @@ jobs:
name: cmake
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: Environment info
run: |
@@ -631,7 +632,7 @@ jobs:
name: cmake (static lib)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: Environment info
run: |
@@ -650,7 +651,7 @@ jobs:
name: make cmakebuild
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: make cmakebuild
# V=1 for lz4 Makefile, VERBOSE=1 for cmake Makefile.
run: make clean; make cmakebuild V=1 VERBOSE=1
@@ -666,18 +667,20 @@ jobs:
{ os: windows-2022, build_path: ".\\build\\VS2022" },
]
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: Build ${{ matrix.system.os }}, Win32
# Switching to debug mode, since vs2022 optimizations seem to introduce a bug since July 2024
run: |
pushd ${{ matrix.system.build_path }}
.\\build-and-test-win32-release.bat
.\\build-and-test-win32-debug.bat
popd
- name: Build ${{ matrix.system.os }}, x64
# Switching to debug mode, since vs2022 optimizations seem to introduce a bug since July 2024
run: |
pushd ${{ matrix.system.build_path }}
.\\build-and-test-x64-release.bat
.\\build-and-test-x64-debug.bat
popd
- name: VS solution generation
@@ -687,7 +690,7 @@ jobs:
popd
- name: Upload Generated VS2022 Directory
uses: actions/upload-artifact@0b2256b8c012f0828dc542b3febcab082c67f72b # https://github.com/actions/upload-artifact v4.3.4
uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # https://github.com/actions/upload-artifact v4.5.0
with:
name: VS2022-Build-Dir
path: "build/visual/Visual Studio 17 2022"
@@ -708,8 +711,8 @@ jobs:
name: Meson + Ninja
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/setup-python@39cd14951b08e74b54015e9e001cdefcf80e669f # https://github.com/actions/setup-python v5.1.1
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # https://github.com/actions/setup-python v5.3.0
with:
python-version: '3.x'
@@ -759,7 +762,7 @@ jobs:
name: git version tag checking for release
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: make -C tests checkTag
if: startsWith(github.ref, 'refs/tags/v') # If git tag name starts with 'v'
run: |
@@ -785,7 +788,7 @@ jobs:
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # https://github.com/actions/checkout v4.1.7
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # https://github.com/actions/checkout v4.2.2
- name: init
run: |

View File

@@ -29,12 +29,12 @@ jobs:
steps:
- name: "Checkout code"
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- name: "Run analysis"
uses: ossf/scorecard-action@dc50aa9510b46c811795eb24b2f1ba02a914e534 # v2.3.3
uses: ossf/scorecard-action@62b2cac7ed8198b15735ed49ab1e5cf35480ba46 # v2.4.0
with:
results_file: results.sarif
results_format: sarif
@@ -52,7 +52,7 @@ jobs:
# Upload the results as artifacts (optional). Commenting out will disable uploads of run results in SARIF
# format to the repository Actions tab.
- name: "Upload artifact"
uses: actions/upload-artifact@0b2256b8c012f0828dc542b3febcab082c67f72b # v4.3.4
uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
with:
name: SARIF file
path: results.sarif
@@ -60,6 +60,6 @@ jobs:
# Upload the results to GitHub's code scanning dashboard.
- name: "Upload to code-scanning"
uses: github/codeql-action/upload-sarif@4fa2a7953630fd2f3fb380f21be14ede0169dd4f # v3.25.12
uses: github/codeql-action/upload-sarif@f09c1c0a94de965c15400f5634aa42fac8fb8f88 # v3.27.5
with:
sarif_file: results.sarif

1
.gitignore vendored
View File

@@ -46,6 +46,7 @@ ld.exe*
# test artifacts
*.lz4
tmp*
builddir/
# generated Windows resource files
lib/*.rc

View File

@@ -123,7 +123,7 @@ Most distributions are bundled with a package manager
which allows easy installation of both the `liblz4` library
and the `lz4` command line interface.
[![Packaging status](https://repology.org/badge/vertical-allrepos/lz4.svg)](https://repology.org/project/lz4/versions)
[![Packaging status](https://repology.org/badge/vertical-allrepos/lz4.svg?columns=4&exclude_unsupported=1)](https://repology.org/project/lz4/versions)
### Special Thanks

View File

@@ -116,26 +116,3 @@ test_script:
artifacts:
- path: bin\lz4_x64.zip
- path: bin\lz4_x86.zip
deploy:
- provider: GitHub
artifact: bin\lz4_x64.zip
auth_token:
secure: w6UJaGie0qbZvffr/fqyhO/Vj8rMiQWnv9a8qm3gxfngdHDTMT42wYupqJpIExId
force_update: true
prerelease: true
on:
COMPILER: gcc
PLATFORM: "mingw64"
appveyor_repo_tag: true
- provider: GitHub
artifact: bin\lz4_x86.zip
auth_token:
secure: w6UJaGie0qbZvffr/fqyhO/Vj8rMiQWnv9a8qm3gxfngdHDTMT42wYupqJpIExId
force_update: true
prerelease: true
on:
COMPILER: gcc
PLATFORM: "mingw32"
appveyor_repo_tag: true

View File

@@ -32,6 +32,7 @@ project(LZ4 VERSION ${LZ4_VERSION_STRING} LANGUAGES C)
option(LZ4_BUILD_CLI "Build lz4 program" ON)
option(LZ4_BUILD_LEGACY_LZ4C "Build lz4c program with legacy argument support" OFF)
# Determine if LZ4 is being built as part of another project.
@@ -84,7 +85,13 @@ list(APPEND LZ4_CLI_SOURCES ${LZ4_SOURCES}) # LZ4_CLI always use liblz4 sources
# Whether to use position independent code for the static library. If
# we're building a shared library this is ignored and PIC is always
# used.
option(LZ4_POSITION_INDEPENDENT_LIB "Use position independent code for static library (if applicable)" ON)
if(NOT DEFINED CMAKE_POSITION_INDEPENDENT_CODE OR CMAKE_POSITION_INDEPENDENT_CODE)
set(LZ4_POSITION_INDEPENDENT_LIB_DEFAULT ON)
else()
set(LZ4_POSITION_INDEPENDENT_LIB_DEFAULT OFF)
endif()
option(LZ4_POSITION_INDEPENDENT_LIB "Use position independent code for static library (if applicable)" ${LZ4_POSITION_INDEPENDENT_LIB_DEFAULT})
# liblz4
include(GNUInstallDirs)
@@ -144,6 +151,13 @@ if (LZ4_BUILD_CLI)
set_target_properties(lz4cli PROPERTIES OUTPUT_NAME lz4)
endif()
# lz4c
if (LZ4_BUILD_LEGACY_LZ4C)
list(APPEND LZ4_PROGRAMS_BUILT lz4c)
add_executable(lz4c ${LZ4_CLI_SOURCES})
set_target_properties(lz4c PROPERTIES COMPILE_DEFINITIONS "ENABLE_LZ4C_LEGACY_OPTIONS")
endif()
# Extra warning flags
if(MSVC)
set(CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG} /W4")

View File

@@ -1,2 +1,10 @@
@PACKAGE_INIT@
include( "${CMAKE_CURRENT_LIST_DIR}/lz4Targets.cmake" )
include( "${CMAKE_CURRENT_LIST_DIR}/lz4Targets.cmake" )
if(NOT TARGET lz4::lz4)
add_library(lz4::lz4 INTERFACE IMPORTED)
if("@BUILD_SHARED_LIBS@")
set_target_properties(lz4::lz4 PROPERTIES INTERFACE_LINK_LIBRARIES LZ4::lz4_shared)
else()
set_target_properties(lz4::lz4 PROPERTIES INTERFACE_LINK_LIBRARIES LZ4::lz4_static)
endif()
endif()

View File

@@ -10,7 +10,7 @@ This Meson build system is provided with no guarantee.
## How to build
`cd` to this meson directory (`contrib/meson`)
`cd` to this meson directory (`build/meson`)
```sh
meson setup --buildtype=release -Ddefault_library=shared -Dprograms=true builddir

View File

@@ -16,7 +16,6 @@ project(
'c',
license: 'BSD-2-Clause-Patent AND GPL-2.0-or-later',
default_options: [
'c_std=c99',
'buildtype=release',
'warning_level=3'
],

View File

@@ -49,8 +49,8 @@ lz4cat = custom_target(
output: 'lz4cat',
command: [
'ln',
'--symbolic',
'--force',
'-s',
'-f',
fs.name(lz4.full_path()),
'@OUTPUT@'
]
@@ -62,8 +62,8 @@ unlz4 = custom_target(
output: 'unlz4',
command: [
'ln',
'--symbolic',
'--force',
'-s',
'-f',
fs.name(lz4.full_path()),
'@OUTPUT@'
]

View File

@@ -47,7 +47,7 @@ test_exes = {
},
'freestanding': {
'sources': files(lz4_source_root / 'tests/freestanding.c'),
'c_args': ['-ffreestanding', '-Wno-unused-parameter', '-Wno-declaration-after-statement'],
'c_args': ['-ffreestanding', '-fno-stack-protector', '-Wno-unused-parameter', '-Wno-declaration-after-statement', '-DLZ4_DEBUG=0'],
'link_args': ['-nostdlib'],
'build': cc.get_id() in ['gcc', 'clang'] and
host_machine.system() == 'linux' and host_machine.cpu_family() == 'x86_64',

View File

@@ -35,6 +35,7 @@ WFLAGS = -std=gnu99 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Wstrict-pro
CFLAGS = $(WFLAGS) -O2 $(USERCFLAGS)
TESTFILE = Makefile
TESTFILE_SMALL= .gitignore
SLIBLZ4 := $(LIBDIR)/liblz4.a
LZ4DIR = ../programs
LZ4 = $(LZ4DIR)/lz4
@@ -80,6 +81,7 @@ test : all $(LZ4)
./blockStreaming_lineByLine$(EXT) $(TESTFILE)
@echo "\n=== Dictionary Random Access ==="
./dictionaryRandomAccess$(EXT) $(TESTFILE) $(TESTFILE) 1100 1400
./dictionaryRandomAccess$(EXT) $(TESTFILE_SMALL) $(TESTFILE_SMALL) 0 32
@echo "\n=== Frame compression ==="
./frameCompress$(EXT) $(TESTFILE)
$(LZ4) -vt $(TESTFILE).lz4

View File

@@ -251,7 +251,7 @@ int main(int argc, char* argv[])
FILE* outFp = fopen(decFilename, "wb");
printf("decompress : %s -> %s\n", lz4Filename, decFilename);
test_decompress(outFp, inpFp, dict, DICTIONARY_BYTES, offset, length);
test_decompress(outFp, inpFp, dict, dictSize, offset, length);
printf("decompress : done\n");
fclose(outFp);

View File

@@ -28,7 +28,7 @@ static size_t get_file_size(char *filename)
return 0;
}
return statbuf.st_size;
return (size_t)statbuf.st_size;
}
static int compress_file(FILE* f_in, FILE* f_out)
@@ -36,7 +36,6 @@ static int compress_file(FILE* f_in, FILE* f_out)
assert(f_in != NULL); assert(f_out != NULL);
LZ4F_errorCode_t ret = LZ4F_OK_NoError;
size_t len;
LZ4_writeFile_t* lz4fWrite;
void* const buf = malloc(CHUNK_SIZE);
if (!buf) {
@@ -56,7 +55,7 @@ static int compress_file(FILE* f_in, FILE* f_out)
}
while (1) {
len = fread(buf, 1, CHUNK_SIZE, f_in);
size_t len = fread(buf, 1, CHUNK_SIZE, f_in);
if (ferror(f_in)) {
printf("fread error\n");
@@ -94,6 +93,7 @@ static int decompress_file(FILE* f_in, FILE* f_out)
void* const buf= malloc(CHUNK_SIZE);
if (!buf) {
printf("error: memory allocation failed \n");
return 1;
}
ret = LZ4F_readOpen(&lz4fRead, f_in);
@@ -135,9 +135,11 @@ out:
return 0;
}
/* @return 0 if both FILE* have identical content */
int compareFiles(FILE* fp0, FILE* fp1)
{
int result = 0;
assert(fp0 != NULL); assert(fp1 != NULL);
while (result==0) {
char b0[1024];
@@ -147,7 +149,7 @@ int compareFiles(FILE* fp0, FILE* fp1)
result = (r0 != r1);
if (!r0 || !r1) break;
if (!result) result = memcmp(b0, b1, r0);
if (r0 == r1) result = memcmp(b0, b1, r0);
}
return result;
@@ -175,7 +177,7 @@ int main(int argc, const char **argv) {
{ FILE* const inpFp = fopen(inpFilename, "rb");
FILE* const outFp = fopen(lz4Filename, "wb");
printf("compress : %s -> %s\n", inpFilename, lz4Filename);
LZ4F_errorCode_t ret = compress_file(inpFp, outFp);
int ret = compress_file(inpFp, outFp);
fclose(inpFp);
fclose(outFp);
@@ -188,7 +190,7 @@ int main(int argc, const char **argv) {
inpFilename,
get_file_size(inpFilename),
get_file_size(lz4Filename), /* might overflow is size_t is 32 bits and size_{in,out} > 4 GB */
(double)get_file_size(lz4Filename) / get_file_size(inpFilename) * 100);
(double)get_file_size(lz4Filename) / (double)get_file_size(inpFilename) * 100);
printf("compress : done\n");
}
@@ -198,8 +200,11 @@ int main(int argc, const char **argv) {
FILE* const inpFp = fopen(lz4Filename, "rb");
FILE* const outFp = fopen(decFilename, "wb");
if (inpFp == NULL) exit(1);
if (outFp == NULL) exit(1);
printf("decompress : %s -> %s\n", lz4Filename, decFilename);
LZ4F_errorCode_t ret = decompress_file(inpFp, outFp);
int ret = decompress_file(inpFp, outFp);
fclose(outFp);
fclose(inpFp);
@@ -216,6 +221,9 @@ int main(int argc, const char **argv) {
{ FILE* const inpFp = fopen(inpFilename, "rb");
FILE* const decFp = fopen(decFilename, "rb");
if (inpFp == NULL) exit(1);
if (decFp == NULL) exit(1);
printf("verify : %s <-> %s\n", inpFilename, decFilename);
int const cmp = compareFiles(inpFp, decFp);

View File

@@ -55,6 +55,7 @@ CFLAGS = $(DEBUGFLAGS) $(USERCFLAGS)
ALLFLAGS = $(CFLAGS) $(CPPFLAGS) $(LDFLAGS)
SRCFILES := $(sort $(wildcard *.c))
OBJFILES = $(SRCFILES:.c=.o)
include ../Makefile.inc
@@ -102,7 +103,7 @@ liblz4.a: $(SRCFILES)
ifeq ($(BUILD_STATIC),yes) # can be disabled on command line
@echo compiling static library
$(COMPILE.c) $^
$(AR) rcs $@ *.o
$(AR) rcs $@ $(OBJFILES)
endif
ifeq ($(WINBASED),yes)

View File

@@ -258,10 +258,12 @@ LZ4LIB_API int LZ4_compress_fast_extState (void* state, const char* src, char* d
* @return : Nb bytes written into 'dst' (necessarily <= dstCapacity)
* or 0 if compression fails.
*
* Note : from v1.8.2 to v1.9.1, this function had a bug (fixed in v1.9.2+):
* the produced compressed content could, in specific circumstances,
* require to be decompressed into a destination buffer larger
* by at least 1 byte than the content to decompress.
* Note : 'targetDstSize' must be >= 1, because it's the smallest valid lz4 payload.
*
* Note 2:from v1.8.2 to v1.9.1, this function had a bug (fixed in v1.9.2+):
* the produced compressed content could, in rare circumstances,
* require to be decompressed into a destination buffer
* larger by at least 1 byte than decompressesSize.
* If an application uses `LZ4_compress_destSize()`,
* it's highly recommended to update liblz4 to v1.9.2 or better.
* If this can't be done or ensured,

View File

@@ -93,7 +93,7 @@ LZ4F_errorCode_t LZ4F_readOpen(LZ4_readFile_t** lz4fRead, FILE* fp)
(*lz4fRead)->fp = fp;
consumedSize = fread(buf, 1, sizeof(buf), (*lz4fRead)->fp);
if (consumedSize != sizeof(buf)) {
if (consumedSize < LZ4F_HEADER_SIZE_MIN + LZ4F_ENDMARK_SIZE) {
LZ4F_freeAndNullReadFile(lz4fRead);
RETURN_ERROR(io_read);
}

View File

@@ -155,7 +155,7 @@ static void LZ4F_free(void* p, LZ4F_CustomMem cmem)
static int g_debuglog_enable = 1;
# define DEBUGLOG(l, ...) { \
if ((g_debuglog_enable) && (l<=LZ4_DEBUG)) { \
fprintf(stderr, __FILE__ " (%i): ", __LINE__ ); \
fprintf(stderr, __FILE__ " %i: ", __LINE__ ); \
fprintf(stderr, __VA_ARGS__); \
fprintf(stderr, " \n"); \
} }
@@ -700,6 +700,7 @@ size_t LZ4F_compressBegin_internal(LZ4F_cctx* cctx,
RETURN_ERROR_IF(dstCapacity < maxFHSize, dstMaxSize_tooSmall);
if (preferencesPtr == NULL) preferencesPtr = &prefNull;
cctx->prefs = *preferencesPtr;
DEBUGLOG(5, "LZ4F_compressBegin_internal: Independent_blocks=%u", cctx->prefs.frameInfo.blockMode);
/* cctx Management */
{ U16 const ctxTypeID = (cctx->prefs.compressionLevel < LZ4HC_CLEVEL_MIN) ? 1 : 2;
@@ -887,10 +888,11 @@ static size_t LZ4F_makeBlock(void* dst,
LZ4F_blockChecksum_t crcFlag)
{
BYTE* const cSizePtr = (BYTE*)dst;
int dstCapacity = (srcSize > 1) ? (int)srcSize - 1 : 1;
U32 cSize;
assert(compress != NULL);
cSize = (U32)compress(lz4ctx, (const char*)src, (char*)(cSizePtr+BHSize),
(int)(srcSize), (int)(srcSize-1),
(int)srcSize, dstCapacity,
level, cdict);
if (cSize == 0 || cSize >= srcSize) {
@@ -961,12 +963,13 @@ static compressFunc_t LZ4F_selectCompression(LZ4F_blockMode_t blockMode, int lev
return LZ4F_compressBlockHC_continue;
}
/* Save history (up to 64KB) into @tmpBuff */
static int LZ4F_localSaveDict(LZ4F_cctx_t* cctxPtr)
/* Save or shorten history (up to 64KB) into @tmpBuff */
static void LZ4F_localSaveDict(LZ4F_cctx_t* cctxPtr)
{
if (cctxPtr->prefs.compressionLevel < LZ4HC_CLEVEL_MIN)
return LZ4_saveDict ((LZ4_stream_t*)(cctxPtr->lz4CtxPtr), (char*)(cctxPtr->tmpBuff), 64 KB);
return LZ4_saveDictHC ((LZ4_streamHC_t*)(cctxPtr->lz4CtxPtr), (char*)(cctxPtr->tmpBuff), 64 KB);
int const dictSize = (cctxPtr->prefs.compressionLevel < LZ4HC_CLEVEL_MIN) ?
LZ4_saveDict ((LZ4_stream_t*)(cctxPtr->lz4CtxPtr), (char*)(cctxPtr->tmpBuff), 64 KB) :
LZ4_saveDictHC ((LZ4_streamHC_t*)(cctxPtr->lz4CtxPtr), (char*)(cctxPtr->tmpBuff), 64 KB);
cctxPtr->tmpIn = cctxPtr->tmpBuff + dictSize;
}
typedef enum { notDone, fromTmpBuffer, fromSrcBuffer } LZ4F_lastBlockStatus;
@@ -974,7 +977,7 @@ typedef enum { notDone, fromTmpBuffer, fromSrcBuffer } LZ4F_lastBlockStatus;
static const LZ4F_compressOptions_t k_cOptionsNull = { 0, { 0, 0, 0 } };
/*! LZ4F_compressUpdateImpl() :
/*! LZ4F_compressUpdateImpl() :
* LZ4F_compressUpdate() can be called repetitively to compress as much data as necessary.
* When successful, the function always entirely consumes @srcBuffer.
* src data is either buffered or compressed into @dstBuffer.
@@ -1072,9 +1075,7 @@ static size_t LZ4F_compressUpdateImpl(LZ4F_cctx* cctxPtr,
if (compressOptionsPtr->stableSrc) {
cctxPtr->tmpIn = cctxPtr->tmpBuff; /* src is stable : dictionary remains in src across invocations */
} else {
int const realDictSize = LZ4F_localSaveDict(cctxPtr);
assert(0 <= realDictSize && realDictSize <= 64 KB);
cctxPtr->tmpIn = cctxPtr->tmpBuff + realDictSize;
LZ4F_localSaveDict(cctxPtr);
}
}
@@ -1084,8 +1085,7 @@ static size_t LZ4F_compressUpdateImpl(LZ4F_cctx* cctxPtr,
{
/* only preserve 64KB within internal buffer. Ensures there is enough room for next block.
* note: this situation necessarily implies lastBlockCompressed==fromTmpBuffer */
int const realDictSize = LZ4F_localSaveDict(cctxPtr);
cctxPtr->tmpIn = cctxPtr->tmpBuff + realDictSize;
LZ4F_localSaveDict(cctxPtr);
assert((cctxPtr->tmpIn + blockSize) <= (cctxPtr->tmpBuff + cctxPtr->maxBufferSize));
}
@@ -1150,7 +1150,7 @@ size_t LZ4F_uncompressedUpdate(LZ4F_cctx* cctxPtr,
/*! LZ4F_flush() :
* When compressed data must be sent immediately, without waiting for a block to be filled,
* invoke LZ4_flush(), which will immediately compress any remaining data stored within LZ4F_cctx.
* invoke LZ4F_flush(), which will immediately compress any remaining data stored within LZ4F_cctx.
* The result of the function is the number of bytes written into dstBuffer.
* It can be zero, this means there was no data left within LZ4F_cctx.
* The function outputs an error code if it fails (can be tested using LZ4F_isError())
@@ -1164,6 +1164,8 @@ size_t LZ4F_flush(LZ4F_cctx* cctxPtr,
BYTE* dstPtr = dstStart;
compressFunc_t compress;
DEBUGLOG(5, "LZ4F_flush: %zu buffered bytes (saved dict size = %i) (dstCapacity=%u)",
cctxPtr->tmpInSize, (int)(cctxPtr->tmpIn - cctxPtr->tmpBuff), (unsigned)dstCapacity);
if (cctxPtr->tmpInSize == 0) return 0; /* nothing to flush */
RETURN_ERROR_IF(cctxPtr->cStage != 1, compressionState_uninitialized);
RETURN_ERROR_IF(dstCapacity < (cctxPtr->tmpInSize + BHSize + BFSize), dstMaxSize_tooSmall);
@@ -1185,9 +1187,9 @@ size_t LZ4F_flush(LZ4F_cctx* cctxPtr,
cctxPtr->tmpInSize = 0;
/* keep tmpIn within limits */
if ((cctxPtr->tmpIn + cctxPtr->maxBlockSize) > (cctxPtr->tmpBuff + cctxPtr->maxBufferSize)) { /* necessarily LZ4F_blockLinked */
int const realDictSize = LZ4F_localSaveDict(cctxPtr);
cctxPtr->tmpIn = cctxPtr->tmpBuff + realDictSize;
if ((cctxPtr->tmpIn + cctxPtr->maxBlockSize) > (cctxPtr->tmpBuff + cctxPtr->maxBufferSize)) {
assert(cctxPtr->prefs.frameInfo.blockMode == LZ4F_blockLinked);
LZ4F_localSaveDict(cctxPtr);
}
return (size_t)(dstPtr - dstStart);

View File

@@ -289,6 +289,9 @@ LZ4FLIB_API LZ4F_errorCode_t LZ4F_freeCompressionContext(LZ4F_cctx* cctx);
/* Size in bytes of the content checksum. */
#define LZ4F_CONTENT_CHECKSUM_SIZE 4
/* Size in bytes of the endmark. */
#define LZ4F_ENDMARK_SIZE 4
/*! LZ4F_compressBegin() :
* will write the frame header into dstBuffer.
* dstCapacity must be >= LZ4F_HEADER_SIZE_MAX bytes.

View File

@@ -262,6 +262,30 @@ static void LZ4HC_init_internal (LZ4HC_CCtx_internal* hc4, const BYTE* start)
/**************************************
* Encode
**************************************/
#if defined(LZ4_DEBUG) && (LZ4_DEBUG >= 2)
# define RAWLOG(...) fprintf(stderr, __VA_ARGS__)
void LZ4HC_hexOut(const void* src, size_t len)
{
const BYTE* p = (const BYTE*)src;
size_t n;
for (n=0; n<len; n++) {
RAWLOG("%02X ", p[n]);
}
RAWLOG(" \n");
}
# define HEX_CMP(_lev, _ptr, _ref, _len) \
if (LZ4_DEBUG >= _lev) { \
RAWLOG("match bytes: "); \
LZ4HC_hexOut(_ptr, _len); \
RAWLOG("ref bytes: "); \
LZ4HC_hexOut(_ref, _len); \
}
#else
# define HEX_CMP(l,p,r,_l)
#endif
/* LZ4HC_encodeSequence() :
* @return : 0 if ok,
* 1 if buffer issue detected */
@@ -278,47 +302,49 @@ LZ4_FORCE_INLINE int LZ4HC_encodeSequence (
#define op (*_op)
#define anchor (*_anchor)
size_t length;
BYTE* const token = op++;
#if defined(LZ4_DEBUG) && (LZ4_DEBUG >= 6)
static const BYTE* start = NULL;
static U32 totalCost = 0;
U32 const pos = (start==NULL) ? 0 : (U32)(anchor - start);
U32 const pos = (start==NULL) ? 0 : (U32)(anchor - start); /* only works for single segment */
U32 const ll = (U32)(ip - anchor);
U32 const llAdd = (ll>=15) ? ((ll-15) / 255) + 1 : 0;
U32 const mlAdd = (matchLength>=19) ? ((matchLength-19) / 255) + 1 : 0;
U32 const cost = 1 + llAdd + ll + 2 + mlAdd;
if (start==NULL) start = anchor; /* only works for single segment */
/* g_debuglog_enable = (pos >= 2228) & (pos <= 2262); */
DEBUGLOG(6, "pos:%7u -- literals:%4u, match:%4i, offset:%5i, cost:%4u + %5u",
pos,
(U32)(ip - anchor), matchLength, offset,
cost, totalCost);
# if 1 /* only works on single segment data */
HEX_CMP(7, ip, ip-offset, matchLength);
# endif
totalCost += cost;
#endif
/* Encode Literal length */
length = (size_t)(ip - anchor);
LZ4_STATIC_ASSERT(notLimited == 0);
/* Check output limit */
if (limit && ((op + (length / 255) + length + (2 + 1 + LASTLITERALS)) > oend)) {
DEBUGLOG(6, "Not enough room to write %i literals (%i bytes remaining)",
(int)length, (int)(oend - op));
return 1;
}
if (length >= RUN_MASK) {
size_t len = length - RUN_MASK;
*token = (RUN_MASK << ML_BITS);
for(; len >= 255 ; len -= 255) *op++ = 255;
*op++ = (BYTE)len;
} else {
*token = (BYTE)(length << ML_BITS);
}
{ size_t litLen = (size_t)(ip - anchor);
LZ4_STATIC_ASSERT(notLimited == 0);
/* Check output limit */
if (limit && ((op + (litLen / 255) + litLen + (2 + 1 + LASTLITERALS)) > oend)) {
DEBUGLOG(6, "Not enough room to write %i literals (%i bytes remaining)",
(int)litLen, (int)(oend - op));
return 1;
}
if (litLen >= RUN_MASK) {
size_t len = litLen - RUN_MASK;
*token = (RUN_MASK << ML_BITS);
for(; len >= 255 ; len -= 255) *op++ = 255;
*op++ = (BYTE)len;
} else {
*token = (BYTE)(litLen << ML_BITS);
}
/* Copy Literals */
LZ4_wildCopy8(op, anchor, op + length);
op += length;
/* Copy Literals */
LZ4_wildCopy8(op, anchor, op + litLen);
op += litLen;
}
/* Encode Offset */
assert(offset <= LZ4_DISTANCE_MAX );
@@ -327,20 +353,20 @@ LZ4_FORCE_INLINE int LZ4HC_encodeSequence (
/* Encode MatchLength */
assert(matchLength >= MINMATCH);
length = (size_t)matchLength - MINMATCH;
if (limit && (op + (length / 255) + (1 + LASTLITERALS) > oend)) {
DEBUGLOG(6, "Not enough room to write match length");
return 1; /* Check output limit */
}
if (length >= ML_MASK) {
*token += ML_MASK;
length -= ML_MASK;
for(; length >= 510 ; length -= 510) { *op++ = 255; *op++ = 255; }
if (length >= 255) { length -= 255; *op++ = 255; }
*op++ = (BYTE)length;
} else {
*token += (BYTE)(length);
}
{ size_t mlCode = (size_t)matchLength - MINMATCH;
if (limit && (op + (mlCode / 255) + (1 + LASTLITERALS) > oend)) {
DEBUGLOG(6, "Not enough room to write match length");
return 1; /* Check output limit */
}
if (mlCode >= ML_MASK) {
*token += ML_MASK;
mlCode -= ML_MASK;
for(; mlCode >= 510 ; mlCode -= 510) { *op++ = 255; *op++ = 255; }
if (mlCode >= 255) { mlCode -= 255; *op++ = 255; }
*op++ = (BYTE)mlCode;
} else {
*token += (BYTE)(mlCode);
} }
/* Prepare next loop */
ip += matchLength;
@@ -519,6 +545,12 @@ static LZ4MID_searchIntoDict_f select_searchDict_function(const LZ4HC_CCtx_inter
return LZ4MID_searchHCDict;
}
/* preconditions:
* - *srcSizePtr within [1, LZ4_MAX_INPUT_SIZE]
* - src is valid
* - maxOutputSize >= 1
* - dst is valid
*/
static int LZ4MID_compress (
LZ4HC_CCtx_internal* const ctx,
const char* const src,
@@ -550,18 +582,16 @@ static int LZ4MID_compress (
unsigned matchLength;
unsigned matchDistance;
/* input sanitization */
DEBUGLOG(5, "LZ4MID_compress (%i bytes)", *srcSizePtr);
/* preconditions verifications */
if (dict == usingDictCtxHc) DEBUGLOG(5, "usingDictCtxHc");
assert(*srcSizePtr >= 0);
if (*srcSizePtr) assert(src != NULL);
if (maxOutputSize) assert(dst != NULL);
if (*srcSizePtr < 0) return 0; /* invalid */
if (maxOutputSize < 0) return 0; /* invalid */
if (*srcSizePtr > LZ4_MAX_INPUT_SIZE) {
/* forbidden: no input is allowed to be that large */
return 0;
}
assert(*srcSizePtr > 0);
assert(*srcSizePtr <= LZ4_MAX_INPUT_SIZE);
assert(src != NULL);
assert(maxOutputSize >= 1);
assert(dst != NULL);
if (limit == fillOutput) oend -= LASTLITERALS; /* Hack for support LZ4 format restriction */
if (*srcSizePtr < LZ4_minLength)
goto _lz4mid_last_literals; /* Input too small, no compression (all literals) */
@@ -940,6 +970,7 @@ LZ4HC_InsertAndGetWiderMatch (
offset = (int)(ipIndex - matchIndex);
sBack = back;
DEBUGLOG(7, "Found match of len=%i within prefix, offset=%i, back=%i", longest, offset, -back);
HEX_CMP(7, ip + back, ip + back - offset, (size_t)matchLength);
} } }
} else { /* lowestMatchIndex <= matchIndex < dictLimit : within Ext Dict */
const BYTE* const matchPtr = dictStart + (matchIndex - dictIdx);
@@ -959,6 +990,7 @@ LZ4HC_InsertAndGetWiderMatch (
offset = (int)(ipIndex - matchIndex);
sBack = back;
DEBUGLOG(7, "Found match of len=%i within dict, offset=%i, back=%i", longest, offset, -back);
HEX_CMP(7, ip + back, matchPtr + back, (size_t)matchLength);
} } }
if (chainSwap && matchLength==longest) { /* better match => select a better chain */
@@ -1118,10 +1150,16 @@ LZ4HC_InsertAndFindBestMatch(LZ4HC_CCtx_internal* const hc4, /* Index table wi
}
/* preconditions:
* - *srcSizePtr within [1, LZ4_MAX_INPUT_SIZE]
* - src is valid
* - maxOutputSize >= 1
* - dst is valid
*/
LZ4_FORCE_INLINE int LZ4HC_compress_hashChain (
LZ4HC_CCtx_internal* const ctx,
const char* const source,
char* const dest,
const char* const src,
char* const dst,
int* srcSizePtr,
int const maxOutputSize,
int maxNbAttempts,
@@ -1132,14 +1170,14 @@ LZ4_FORCE_INLINE int LZ4HC_compress_hashChain (
const int inputSize = *srcSizePtr;
const int patternAnalysis = (maxNbAttempts > 128); /* levels 9+ */
const BYTE* ip = (const BYTE*) source;
const BYTE* ip = (const BYTE*)src;
const BYTE* anchor = ip;
const BYTE* const iend = ip + inputSize;
const BYTE* const mflimit = iend - MFLIMIT;
const BYTE* const matchlimit = (iend - LASTLITERALS);
BYTE* optr = (BYTE*) dest;
BYTE* op = (BYTE*) dest;
BYTE* optr = (BYTE*) dst;
BYTE* op = (BYTE*) dst;
BYTE* oend = op + maxOutputSize;
const BYTE* start0;
@@ -1150,6 +1188,13 @@ LZ4_FORCE_INLINE int LZ4HC_compress_hashChain (
/* init */
DEBUGLOG(5, "LZ4HC_compress_hashChain (dict?=>%i)", dict);
/* preconditions verifications */
assert(*srcSizePtr >= 1);
assert(src != NULL);
assert(maxOutputSize >= 1);
assert(dst != NULL);
*srcSizePtr = 0;
if (limit == fillOutput) oend -= LASTLITERALS; /* Hack for support LZ4 format restriction */
if (inputSize < LZ4_minLength) goto _last_literals; /* Input too small, no compression (all literals) */
@@ -1334,8 +1379,8 @@ _last_literals:
}
/* End */
*srcSizePtr = (int) (((const char*)ip) - source);
return (int) (((char*)op)-dest);
*srcSizePtr = (int) (((const char*)ip) - src);
return (int) (((char*)op)-dst);
_dest_overflow:
if (limit == fillOutput) {
@@ -1370,7 +1415,7 @@ static int LZ4HC_compress_optimal( LZ4HC_CCtx_internal* ctx,
const dictCtx_directive dict,
const HCfavor_e favorDecSpeed);
LZ4_FORCE_INLINE int
static int
LZ4HC_compress_generic_internal (
LZ4HC_CCtx_internal* const ctx,
const char* const src,
@@ -1382,11 +1427,15 @@ LZ4HC_compress_generic_internal (
const dictCtx_directive dict
)
{
DEBUGLOG(5, "LZ4HC_compress_generic_internal(src=%p, srcSize=%d)",
src, *srcSizePtr);
DEBUGLOG(5, "LZ4HC_compress_generic_internal(src=%p, srcSize=%d, dstCapacity=%d)",
src, *srcSizePtr, dstCapacity);
if (limit == fillOutput && dstCapacity < 1) return 0; /* Impossible to store anything */
/* input sanitization */
if ((U32)*srcSizePtr > (U32)LZ4_MAX_INPUT_SIZE) return 0; /* Unsupported input size (too large or negative) */
if (dstCapacity < 1) return 0; /* Invalid: impossible to store anything */
assert(dst); /* since dstCapacity >= 1, dst must be valid */
if (*srcSizePtr == 0) { *dst = 0; return 1; }
assert(src != NULL); /* since *srcSizePtr >= 1, src must be valid */
ctx->end += *srcSizePtr;
{ cParams_t const cParam = LZ4HC_getCLevelParams(cLevel);
@@ -1820,6 +1869,13 @@ LZ4HC_FindLongerMatch(LZ4HC_CCtx_internal* const ctx,
}
/* preconditions:
* - *srcSizePtr within [1, LZ4_MAX_INPUT_SIZE]
* - src is valid
* - maxOutputSize >= 1
* - dst is valid
*/
static int LZ4HC_compress_optimal ( LZ4HC_CCtx_internal* ctx,
const char* const source,
char* dst,
@@ -1837,7 +1893,7 @@ static int LZ4HC_compress_optimal ( LZ4HC_CCtx_internal* ctx,
#if defined(LZ4HC_HEAPMODE) && LZ4HC_HEAPMODE==1
LZ4HC_optimal_t* const opt = (LZ4HC_optimal_t*)ALLOC(sizeof(LZ4HC_optimal_t) * (LZ4_OPT_NUM + TRAILING_LITERALS));
#else
LZ4HC_optimal_t opt[LZ4_OPT_NUM + TRAILING_LITERALS]; /* ~64 KB, which is a bit large for stack... */
LZ4HC_optimal_t opt[LZ4_OPT_NUM + TRAILING_LITERALS]; /* ~64 KB, which can be a bit large for some stacks... */
#endif
const BYTE* ip = (const BYTE*) source;
@@ -1852,10 +1908,17 @@ static int LZ4HC_compress_optimal ( LZ4HC_CCtx_internal* ctx,
int ovoff = 0;
/* init */
DEBUGLOG(5, "LZ4HC_compress_optimal(dst=%p, dstCapa=%u)", dst, (unsigned)dstCapacity);
#if defined(LZ4HC_HEAPMODE) && LZ4HC_HEAPMODE==1
if (opt == NULL) goto _return_label;
#endif
DEBUGLOG(5, "LZ4HC_compress_optimal(dst=%p, dstCapa=%u)", dst, (unsigned)dstCapacity);
/* preconditions verifications */
assert(dstCapacity > 0);
assert(dst != NULL);
assert(*srcSizePtr > 0);
assert(source != NULL);
*srcSizePtr = 0;
if (limit == fillOutput) oend -= LASTLITERALS; /* Hack for support LZ4 format restriction */
if (sufficient_len >= LZ4_OPT_NUM) sufficient_len = LZ4_OPT_NUM-1;

View File

@@ -131,7 +131,9 @@ static int usage(const char* exeName)
DISPLAY( "Arguments : \n");
DISPLAY( " -1 : fast compression (default) \n");
DISPLAY( " -%2d : slowest compression level \n", LZ4HC_CLEVEL_MAX);
#if LZ4IO_MULTITHREAD
DISPLAY( " -T# : use # threads for compression (default:%i==auto) \n", LZ4_NBWORKERS_DEFAULT);
#endif
DISPLAY( " -d : decompression (default for %s extension)\n", LZ4_EXTENSION);
DISPLAY( " -f : overwrite output without prompting \n");
DISPLAY( " -k : preserve source files(s) (default) \n");
@@ -206,8 +208,11 @@ static int usage_longhelp(const char* exeName)
DISPLAY( "\n");
DISPLAY( "Compression levels : \n");
DISPLAY( "---------------------\n");
DISPLAY( "-0 ... -2 => Fast compression, all identical\n");
DISPLAY( "-3 ... -%d => High compression; higher number == more compression but slower\n", LZ4HC_CLEVEL_MAX);
DISPLAY( "-0 => Default level, identical to -%u \n", LZ4_CLEVEL_DEFAULT);
DISPLAY( "-1 => Fast compression \n");
DISPLAY( "-2 .. -%d => High compression; higher number == more compression but slower\n", LZ4HC_CLEVEL_MAX);
DISPLAY( "--best => Highest available compression level (-%d) \n", LZ4HC_CLEVEL_MAX);
DISPLAY( "--fast=# => Faster compression; higher number == faster but compress less\n");
DISPLAY( "\n");
DISPLAY( "stdin, stdout and the console : \n");
DISPLAY( "--------------------------------\n");

View File

@@ -28,20 +28,21 @@
/* Default compression level.
* Can be overridden by environment variable LZ4_CLEVEL.
* Can be overridden at runtime using -# command */
* Is overridden at runtime by command -# */
#ifndef LZ4_CLEVEL_DEFAULT
# define LZ4_CLEVEL_DEFAULT 1
#endif
/* Determines if multithreading is enabled or not
* Default: disabled */
* Default: enabled on Windows, disabled on other platforms */
#ifndef LZ4IO_MULTITHREAD
# ifdef _WIN32
/* Windows support Completion Ports */
/* Windows supports Completion Ports */
# define LZ4IO_MULTITHREAD 1
# else
/* Requires <pthread> support.
* Can't be reliably and portably tested at source code level */
* Can't be reliably and portably tested at source code level
* so must be set a build level */
# define LZ4IO_MULTITHREAD 0
# endif
#endif
@@ -49,7 +50,7 @@
/* Determines default nb of threads for compression
* Default value is 0, which means "auto" :
* nb of threads is determined from detected local cpu.
* Can be overriden by Environment Variable LZ4_NBWORKERS.
* Can be overridden by Environment Variable LZ4_NBWORKERS.
* Can be overridden at runtime using -T# command */
#ifndef LZ4_NBWORKERS_DEFAULT
# define LZ4_NBWORKERS_DEFAULT 0

View File

@@ -109,21 +109,31 @@ static int g_displayLevel = 0; /* 0 : no display ; 1: errors ; 2 : + result
static const Duration_ns refreshRate = 200000000;
static TIME_t g_time = { 0 };
static double cpuLoad_sec(clock_t cpuStart)
/* Gives the cpu time,
* i.e. the sum of kernel time + user time spent in all active threads since the beginning of the program.
* When compared with time duration, it gives a sense of cpu load.
* On Windows: uses GetProcessTimes() counters.
* On posix: relies on the fact that `clock()` measures cpu time.
*/
static double cpuTime_sec(void)
{
#ifdef _WIN32
FILETIME creationTime, exitTime, kernelTime, userTime;
(void)cpuStart;
GetProcessTimes(GetCurrentProcess(), &creationTime, &exitTime, &kernelTime, &userTime);
assert(kernelTime.dwHighDateTime == 0);
assert(userTime.dwHighDateTime == 0);
return ((double)kernelTime.dwLowDateTime + (double)userTime.dwLowDateTime) * 100. / 1000000000.;
return ( ((double)kernelTime.dwLowDateTime + (double)userTime.dwLowDateTime)
+ ((double)kernelTime.dwHighDateTime + (double)userTime.dwHighDateTime) * (double)(1ULL << 32) )
* 100. / 1000000000.;
#else
return (double)(clock() - cpuStart) / CLOCKS_PER_SEC;
return (double)clock() / (double)CLOCKS_PER_SEC;
#endif
}
static void LZ4IO_finalTimeDisplay(TIME_t timeStart, clock_t cpuStart, unsigned long long size)
static double cpuTimeSpan_sec(double startTime)
{
return cpuTime_sec() - startTime;
}
static void LZ4IO_finalTimeDisplay(TIME_t timeStart, double cpuTimeStart, unsigned long long size)
{
#if LZ4IO_MULTITHREAD
if (!TIME_support_MT_measurements()) {
@@ -131,9 +141,9 @@ static void LZ4IO_finalTimeDisplay(TIME_t timeStart, clock_t cpuStart, unsigned
} else
#endif
{
Duration_ns duration_ns = TIME_clockSpan_ns(timeStart);
Duration_ns const duration_ns = TIME_clockSpan_ns(timeStart);
double const seconds = (double)(duration_ns + !duration_ns) / (double)1000000000.;
double const cpuLoad_s = cpuLoad_sec(cpuStart);
double const cpuLoad_s = cpuTimeSpan_sec(cpuTimeStart);
DISPLAYLEVEL(3,"Done in %.2f s ==> %.2f MiB/s (cpu load : %.0f%%)\n", seconds,
(double)size / seconds / 1024. / 1024.,
(cpuLoad_s / seconds) * 100.);
@@ -910,7 +920,7 @@ int LZ4IO_compressFilename_Legacy(const char* input_filename,
const LZ4IO_prefs_t* prefs)
{
TIME_t const timeStart = TIME_getTime();
clock_t const cpuStart = clock();
double const cpuStart = cpuTime_sec();
unsigned long long processed = 0;
int r = LZ4IO_compressLegacy_internal(&processed, input_filename, output_filename, compressionlevel, prefs);
LZ4IO_finalTimeDisplay(timeStart, cpuStart, processed);
@@ -927,7 +937,7 @@ int LZ4IO_compressMultipleFilenames_Legacy(
int compressionLevel, const LZ4IO_prefs_t* prefs)
{
TIME_t const timeStart = TIME_getTime();
clock_t const cpuStart = clock();
double const cpuStart = cpuTime_sec();
unsigned long long totalProcessed = 0;
int i;
int missed_files = 0;
@@ -1503,7 +1513,7 @@ LZ4IO_compressFilename_extRess(unsigned long long* inStreamSize,
int LZ4IO_compressFilename(const char* srcFileName, const char* dstFileName, int compressionLevel, const LZ4IO_prefs_t* prefs)
{
TIME_t const timeStart = TIME_getTime();
clock_t const cpuStart = clock();
double const cpuStart = cpuTime_sec();
cRess_t ress = LZ4IO_createCResources(prefs);
unsigned long long processed;
@@ -1532,7 +1542,7 @@ int LZ4IO_compressMultipleFilenames(
cRess_t ress;
unsigned long long totalProcessed = 0;
TIME_t timeStart = TIME_getTime();
clock_t cpuStart = clock();
double const cpuStart = cpuTime_sec();
if (dstFileName == NULL) return ifntSize; /* not enough memory */
ress = LZ4IO_createCResources(prefs);
@@ -2484,7 +2494,7 @@ int LZ4IO_decompressFilename(const char* input_filename, const char* output_file
{
dRess_t const ress = LZ4IO_createDResources(prefs);
TIME_t const timeStart = TIME_getTime();
clock_t const cpuStart = clock();
double const cpuStart = cpuTime_sec();
unsigned long long processed = 0;
int const errStat = LZ4IO_decompressDstFile(&processed, ress, input_filename, output_filename, prefs);
@@ -2509,7 +2519,7 @@ int LZ4IO_decompressMultipleFilenames(
size_t const suffixSize = strlen(suffix);
dRess_t ress = LZ4IO_createDResources(prefs);
TIME_t timeStart = TIME_getTime();
clock_t cpuStart = clock();
double const cpuStart = cpuTime_sec();
if (outFileName==NULL) END_PROCESS(70, "Memory allocation error");
if (prefs->blockChecksum==0 && prefs->streamChecksum==0) {

View File

@@ -367,7 +367,7 @@ test-decompress-partial : decompress-partial decompress-partial-usingDict
# freestanding test only for Linux x86_64
#-----------------------------------------------------------------------------
UNAME_S ?= $(if $(filter Windows_NT,$(OS)),Windows,$(shell uname -s))
UNAME_P ?= $(if $(filter Windows_NT,$(OS)),Unknown,$(shell uname -p))
UNAME_M ?= $(if $(filter Windows_NT,$(OS)),Unknown,$(shell uname -m))
FREESTANDING_CFLAGS := -ffreestanding -nostdlib
@@ -375,7 +375,7 @@ ifneq ($(UNAME_S), Linux)
FREESTANDING_CFLAGS :=
endif
ifneq ($(UNAME_P), x86_64)
ifneq ($(UNAME_M), x86_64)
FREESTANDING_CFLAGS :=
endif

View File

@@ -26,3 +26,8 @@ endif()
set(LZ4_TESTS_SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/..")
add_executable(decompress-partial "${LZ4_TESTS_SOURCE_DIR}/decompress-partial.c")
target_link_libraries(decompress-partial LZ4::lz4_shared)
# Add a test that builds against the lz4::lz4 target
set(LZ4_TESTS_SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/..")
add_executable(decompress-partial-lz4 "${LZ4_TESTS_SOURCE_DIR}/decompress-partial.c")
target_link_libraries(decompress-partial-lz4 lz4::lz4)

View File

@@ -75,6 +75,7 @@ static const U32 nbTestsDefault = 256 KB;
static const U32 prime1 = 2654435761U;
static const U32 prime2 = 2246822519U;
static const U64 extCrcSeed = 1;
/*-************************************
* Macros
@@ -298,7 +299,7 @@ static int unitTests(U32 seed, double compressibility)
goto _output_error;
}
FUZ_fillCompressibleNoiseBuffer(CNBuffer, COMPRESSIBLE_NOISE_LENGTH, compressibility, randState);
crcOrig = XXH64(CNBuffer, COMPRESSIBLE_NOISE_LENGTH, 1);
crcOrig = XXH64(CNBuffer, COMPRESSIBLE_NOISE_LENGTH, extCrcSeed);
/* LZ4F_compressBound() : special case : srcSize == 0 */
DISPLAYLEVEL(3, "LZ4F_compressBound(0) = ");
@@ -371,7 +372,7 @@ static int unitTests(U32 seed, double compressibility)
DISPLAYLEVEL(3, "Single Pass decompression : ");
CHECK( LZ4F_decompress(dCtx, decodedBuffer, &decodedBufferSize, compressedBuffer, &compressedBufferSize, NULL) );
{ U64 const crcDest = XXH64(decodedBuffer, decodedBufferSize, 1);
{ U64 const crcDest = XXH64(decodedBuffer, decodedBufferSize, extCrcSeed);
if (crcDest != crcOrig) goto _output_error; }
DISPLAYLEVEL(3, "Regenerated %u bytes \n", (U32)decodedBufferSize);
@@ -398,7 +399,7 @@ static int unitTests(U32 seed, double compressibility)
if (decResult != 0) goto _output_error; /* should finish now */
op += oSize;
if (op>oend) { DISPLAY("decompression write overflow \n"); goto _output_error; }
{ U64 const crcDest = XXH64(decodedBuffer, (size_t)(op-ostart), 1);
{ U64 const crcDest = XXH64(decodedBuffer, (size_t)(op-ostart), extCrcSeed);
if (crcDest != crcOrig) goto _output_error;
} }
@@ -471,7 +472,7 @@ static int unitTests(U32 seed, double compressibility)
op += oSize;
ip += iSize;
}
{ U64 const crcDest = XXH64(decodedBuffer, COMPRESSIBLE_NOISE_LENGTH, 1);
{ U64 const crcDest = XXH64(decodedBuffer, COMPRESSIBLE_NOISE_LENGTH, extCrcSeed);
if (crcDest != crcOrig) goto _output_error;
}
DISPLAYLEVEL(3, "Regenerated %u/%u bytes \n", (unsigned)(op-ostart), (unsigned)COMPRESSIBLE_NOISE_LENGTH);
@@ -515,7 +516,7 @@ static int unitTests(U32 seed, double compressibility)
ip += iSize;
}
{ size_t const decodedSize = (size_t)(op - ostart);
U64 const crcDest = XXH64(decodedBuffer, decodedSize, 1);
U64 const crcDest = XXH64(decodedBuffer, decodedSize, extCrcSeed);
if (crcDest != crcOrig) goto _output_error;
DISPLAYLEVEL(3, "Regenerated %u bytes \n", (U32)decodedSize);
}
@@ -571,8 +572,8 @@ static int unitTests(U32 seed, double compressibility)
CHECK( LZ4F_decompress(dctx, decodedBuffer, &decodedSize, compressedBuffer, &iSize, NULL) );
if (decodedSize != testSize) goto _output_error;
if (iSize != cSize) goto _output_error;
{ U64 const crcDest = XXH64(decodedBuffer, decodedSize, 1);
U64 const crcSrc = XXH64(CNBuffer, testSize, 1);
{ U64 const crcDest = XXH64(decodedBuffer, decodedSize, extCrcSeed);
U64 const crcSrc = XXH64(CNBuffer, testSize, extCrcSeed);
if (crcDest != crcSrc) goto _output_error;
}
DISPLAYLEVEL(3, "Regenerated %u bytes \n", (U32)decodedSize);
@@ -994,15 +995,28 @@ static void locateBuffDiff(const void* buff1, const void* buff2, size_t size, o_
#undef CHECK
#define CHECK(cond, ...) { if (cond) { EXIT_MSG(__VA_ARGS__); } }
static LZ4F_errorCode_t LZ4F_returnErrorCode(LZ4F_errorCodes code)
{
return (LZ4F_errorCode_t)-(ptrdiff_t)code;
}
#define RETURN_ERROR(e) return LZ4F_returnErrorCode(LZ4F_ERROR_ ## e)
/* @o_scenario: output behavior: contiguous, non-contiguous, or overwrite
* @crcOrig: xxh64 of original data using extCrcSeed
* @return : 0 on success, or LZ4F_ErrorCode
* */
size_t test_lz4f_decompression_wBuffers(
const void* cSrc, size_t cSize,
void* dst, size_t dstCapacity, o_scenario_e o_scenario,
void* dst, size_t dstCapacity,
o_scenario_e o_scenario,
const void* srcRef, size_t decompressedSize,
U64 crcOrig,
U32* const randState,
LZ4F_dctx* const dCtx,
U32 seed, U32 testNb,
int findErrorPos)
int shouldSucceed)
{
const BYTE* ip = (const BYTE*)cSrc;
const BYTE* const iend = ip + cSize;
@@ -1015,7 +1029,8 @@ size_t test_lz4f_decompression_wBuffers(
size_t totalOut = 0;
size_t moreToFlush = 0;
XXH64_state_t xxh64;
XXH64_reset(&xxh64, 1);
XXH64_reset(&xxh64, extCrcSeed);
o_scenario = o_contiguous;
assert(ip < iend);
while (ip < iend) {
unsigned const nbBitsI = (FUZ_rand(randState) % (maxBits-1)) + 1;
@@ -1047,7 +1062,7 @@ size_t test_lz4f_decompression_wBuffers(
moreToFlush = LZ4F_decompress(dCtx, tmpop, &oSize, tmpip, &iSize, &dOptions);
free(iBuffer);
}
DISPLAYLEVEL(7, "oSize=%u, readSize=%u \n", (unsigned)oSize, (unsigned)iSize);
DISPLAYLEVEL(7, "oSize=%u, readSize=%u \n", (unsigned)oSize, (unsigned)iSize);
if (sentinelTest) {
CHECK(op[oSizeMax] != mark, "op[oSizeMax] = %02X != %02X : "
@@ -1055,17 +1070,21 @@ size_t test_lz4f_decompression_wBuffers(
op[oSizeMax], mark);
}
if (LZ4F_getErrorCode(moreToFlush) == LZ4F_ERROR_contentChecksum_invalid) {
if (findErrorPos) DISPLAYLEVEL(2, "checksum error detected \n");
if (findErrorPos) locateBuffDiff(srcRef, dst, decompressedSize, o_scenario);
if (shouldSucceed) DISPLAYLEVEL(1, "checksum error reported after decompressing segment with LZ4F_decompress() \n");
if (shouldSucceed) locateBuffDiff(srcRef, dst, decompressedSize, o_scenario);
RETURN_ERROR(contentChecksum_invalid);
}
if (LZ4F_isError(moreToFlush)) {
if (shouldSucceed) DISPLAYLEVEL(1, "error reported after decompressing segment with LZ4F_decompress() \n");
return moreToFlush;
}
if (LZ4F_isError(moreToFlush)) return moreToFlush;
XXH64_update(&xxh64, op, oSize);
totalOut += oSize;
op += oSize;
ip += iSize;
if (o_scenario == o_noncontiguous) {
if (op == oend) return LZ4F_ERROR_GENERIC; /* can theoretically happen with bogus data */
if (op == oend) RETURN_ERROR(GENERIC); /* can theoretically happen with bogus data */
op++; /* create a gap between consecutive output */
}
if (o_scenario==o_overwrite) op = (BYTE*)dst; /* overwrite destination */
@@ -1073,17 +1092,19 @@ size_t test_lz4f_decompression_wBuffers(
&& (iSize == 0)) /* no input consumed */
break;
}
if (moreToFlush != 0) return LZ4F_ERROR_decompressionFailed;
if (moreToFlush != 0) RETURN_ERROR(decompressionFailed);
if (totalOut) { /* otherwise, it's a skippable frame */
U64 const crcDecoded = XXH64_digest(&xxh64);
if (crcDecoded != crcOrig) {
if (findErrorPos) locateBuffDiff(srcRef, dst, decompressedSize, o_scenario);
return LZ4F_ERROR_contentChecksum_invalid;
if (shouldSucceed) DISPLAYLEVEL(1, "checksum error detected at end of decompression \n");
if (shouldSucceed) locateBuffDiff(srcRef, dst, decompressedSize, o_scenario);
RETURN_ERROR(contentChecksum_invalid);
} }
return 0;
}
/* @return : 0 on success, or LZ4F_ErrorCode */
size_t test_lz4f_decompression(const void* cSrc, size_t cSize,
const void* srcRef, size_t decompressedSize,
U64 crcOrig,
@@ -1095,7 +1116,7 @@ size_t test_lz4f_decompression(const void* cSrc, size_t cSize,
o_scenario_e const o_scenario = (o_scenario_e)(FUZ_rand(randState) % 3); /* 0 : contiguous; 1 : non-contiguous; 2 : dst overwritten */
/* tighten dst buffer conditions */
size_t const dstCapacity = (o_scenario == o_noncontiguous) ?
(decompressedSize * 2) + 128 :
(decompressedSize * 2) + 128 : /* should provide enough room to insert spaces beween blocks */
decompressedSize;
size_t result;
void* const dstBuffer = malloc(dstCapacity);
@@ -1152,7 +1173,7 @@ int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressi
size_t const srcStartId = FUZ_rand(&randState) % (CNBufferLength - srcSize);
const BYTE* const srcStart = (const BYTE*)CNBuffer + srcStartId;
unsigned const neverFlush = (FUZ_rand(&randState) & 15) == 1;
U64 const crcOrig = XXH64(srcStart, srcSize, 1);
U64 const crcOrig = XXH64(srcStart, srcSize, extCrcSeed);
LZ4F_preferences_t prefs;
const LZ4F_preferences_t* prefsPtr = &prefs;
size_t cSize;
@@ -1200,11 +1221,9 @@ int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressi
size_t iSize = MIN(sampleMax, (size_t)(iend-ip));
size_t const oSize = LZ4F_compressBound(iSize, prefsPtr);
cOptions.stableSrc = ((FUZ_rand(&randState) & 3) == 1);
DISPLAYLEVEL(6, "Sending %u bytes to compress (stableSrc:%u) \n",
(unsigned)iSize, cOptions.stableSrc);
#if 1
/* insert uncompressed segment */
/* test inserting an uncompressed block */
if ( (iSize>0)
&& !neverFlush /* do not mess with compressBound when neverFlush is set */
&& prefsPtr != NULL /* prefs are set */
@@ -1212,6 +1231,7 @@ int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressi
&& (FUZ_rand(&randState) & 15) == 1 ) {
size_t const uSize = FUZ_rand(&randState) % iSize;
size_t const flushedSize = LZ4F_uncompressedUpdate(cCtx, op, (size_t)(oend-op), ip, uSize, &cOptions);
DISPLAYLEVEL(6, "Actually sending %u bytes as uncompressed \n", (unsigned)uSize);
CHECK(LZ4F_isError(flushedSize), "Insert uncompressed data failed (error %i : %s)",
(int)flushedSize, LZ4F_getErrorName(flushedSize));
op += flushedSize;
@@ -1220,6 +1240,8 @@ int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressi
}
#endif
DISPLAYLEVEL(6, "Sending %u bytes to compress or buffer (stableSrc:%u) \n",
(unsigned)iSize, cOptions.stableSrc);
{ size_t const flushedSize = LZ4F_compressUpdate(cCtx, op, oSize, ip, iSize, &cOptions);
CHECK(LZ4F_isError(flushedSize), "Compression failed (error %i : %s)",
(int)flushedSize, LZ4F_getErrorName(flushedSize));
@@ -1230,7 +1252,7 @@ int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressi
{ unsigned const forceFlush = neverFlush ? 0 : ((FUZ_rand(&randState) & 3) == 1);
if (forceFlush) {
size_t const flushSize = LZ4F_flush(cCtx, op, (size_t)(oend-op), &cOptions);
DISPLAYLEVEL(6,"flushing %u bytes \n", (unsigned)flushSize);
DISPLAYLEVEL(6, "flushing %u bytes \n", (unsigned)flushSize);
CHECK(LZ4F_isError(flushSize), "Compression failed (error %i)", (int)flushSize);
op += flushSize;
if ((FUZ_rand(&randState) % 1024) == 3) {
@@ -1244,7 +1266,10 @@ int fuzzerTests(U32 seed, unsigned nbTests, unsigned startTest, double compressi
op += 4;
} } } }
} /* while (ip<iend) */
CHECK(op>=oend, "LZ4F_compressFrameBound overflow");
/* check compressBound guarantees */
if (neverFlush) CHECK(op>=oend, "LZ4F_compressFrameBound overflow");
{ size_t const dstEndSafeSize = LZ4F_compressBound(0, prefsPtr);
int const tooSmallDstEnd = ((FUZ_rand(&randState) & 31) == 3);
size_t const dstEndTooSmallSize = (FUZ_rand(&randState) % dstEndSafeSize) + 1;

View File

@@ -225,7 +225,7 @@ EXTERN_C int memcmp(const void *s1, const void *s2, size_t n) {
//
EXTERN_C void _start(void) {
EXTERN_C void __attribute__((force_align_arg_pointer)) _start(void) {
test();
MY_exit(0);
}

View File

@@ -551,7 +551,7 @@ static const DecompressionDesc decDescArray[] = {
{ "LZ4_decompress_safe_forceExtDict", local_LZ4_decompress_safe_forceExtDict, 1, 0 },
#endif
{ "LZ4F_decompress", local_LZ4F_decompress, 1, 1 },
{ "LZ4F_decompLZ4F_decompress_followHintress", local_LZ4F_decompress_followHint, 1, 1 },
{ "LZ4F_decompress_followHint", local_LZ4F_decompress_followHint, 1, 1 },
{ "LZ4F_decompress_noHint", local_LZ4F_decompress_noHint, 1, 1 },
};

View File

@@ -309,6 +309,10 @@ static void FUZ_findDiff(const void* buff1, const void* buff2)
const BYTE* const b1 = (const BYTE*)buff1;
const BYTE* const b2 = (const BYTE*)buff2;
size_t u = 0;
if ((buff1 == NULL) || (buff2 == NULL)) {
DISPLAY(" one of the buffers is NULL \n");
return;
}
while (b1[u]==b2[u]) u++;
DISPLAY("\nWrong Byte at position %u \n", (unsigned)u);
}
@@ -375,13 +379,13 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c
cycleNb++) {
U32 testNb = 0;
U32 randState = FUZ_rand(&coreRandState) ^ PRIME3;
int const blockSize = (FUZ_rand(&randState) % (FUZ_MAX_BLOCK_SIZE-1)) + 1;
int const blockSize = (FUZ_rand(&randState) % (FUZ_MAX_BLOCK_SIZE-1));
int const blockStart = (int)(FUZ_rand(&randState) % (U32)(COMPRESSIBLE_NOISE_LENGTH - blockSize - 1)) + 1;
int const dictSizeRand = FUZ_rand(&randState) % FUZ_MAX_DICT_SIZE;
int const dictSize = MIN(dictSizeRand, blockStart - 1);
int const compressionLevel = FUZ_rand(&randState) % (LZ4HC_CLEVEL_MAX+1);
const char* block = ((char*)CNBuffer) + blockStart;
const char* dict = block - dictSize;
const char* block = blockSize ? ((char*)CNBuffer) + blockStart : NULL;
const char* dict = (char*)CNBuffer + blockStart - dictSize;
int compressedSize, HCcompressedSize;
int blockContinueCompressedSize;
U32 const crcOrig = XXH32(block, (size_t)blockSize, 0);
@@ -392,7 +396,7 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c
/* Compression tests */
if ( ((FUZ_rand(&randState) & 63) == 2)
&& ((size_t)blockSize < labSize) ) {
memcpy(lowAddrBuffer, block, blockSize);
if (blockSize) { assert(block); memcpy(lowAddrBuffer, block, blockSize); }
block = (const char*)lowAddrBuffer;
}
@@ -448,7 +452,7 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c
FUZ_DISPLAYTEST();
decodedBuffer[srcSize] = canary;
{ int const dSize = LZ4_decompress_safe(compressedBuffer, decodedBuffer, cSize, srcSize);
FUZ_CHECKTEST(dSize<0, "LZ4_decompress_safe failed (%i) on data compressed by LZ4_compressHC_destSize", dSize);
FUZ_CHECKTEST(dSize<0, "LZ4_decompress_safe failed (%i) on data compressed by LZ4_compress_HC_destSize", dSize);
FUZ_CHECKTEST(dSize!=srcSize, "LZ4_decompress_safe failed : decompressed %i bytes, was supposed to decompress %i bytes", dSize, srcSize);
}
FUZ_CHECKTEST(decodedBuffer[srcSize] != canary, "LZ4_decompress_safe overwrite dst buffer !");
@@ -487,16 +491,20 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c
FUZ_DISPLAYTEST("test LZ4_compress_destSize_extState() with too small dest buffer (must succeed, compress less than full input)");
{ int inputSize = blockSize;
int const r3 = LZ4_compress_destSize_extState(stateLZ4, block, compressedBuffer, &inputSize, r-1, 8);
/* note: LZ4_compress_destSize() still requires a targetDstSize > 0, since it's the minimum valid output size produced by LZ4 */
int targetDstSize = r-1 < 1 ? 1 : r-1;
int const r3 = LZ4_compress_destSize_extState(stateLZ4, block, compressedBuffer, &inputSize, targetDstSize, 8);
FUZ_CHECKTEST(r3==0, "LZ4_compress_destSize_extState() failed");
FUZ_CHECKTEST(inputSize>=blockSize, "LZ4_compress_destSize_extState() should consume less than full input");
if (targetDstSize < r)
FUZ_CHECKTEST(inputSize>=blockSize, "LZ4_compress_destSize_extState() should consume less than full input");
}
}
/* Test compression using fast reset external state*/
FUZ_DISPLAYTEST("test LZ4_compress_fast_extState_fastReset()");
{ int const r = LZ4_compress_fast_extState_fastReset(stateLZ4, block, compressedBuffer, blockSize, (int)compressedBufferSize, 8);
FUZ_CHECKTEST(r==0, "LZ4_compress_fast_extState_fastReset() failed"); }
FUZ_CHECKTEST(r==0, "LZ4_compress_fast_extState_fastReset() failed");
}
/* Test compression */
FUZ_DISPLAYTEST("test LZ4_compress_default()");
@@ -529,12 +537,14 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c
}
/* Test decoding with one byte missing => must fail */
FUZ_DISPLAYTEST("LZ4_decompress_fast() with output buffer 1-byte too short");
decodedBuffer[blockSize-1] = 0;
{ int const r = LZ4_decompress_fast(cBuffer_exact, decodedBuffer, blockSize-1);
FUZ_CHECKTEST(r>=0, "LZ4_decompress_fast should have failed, due to Output Size being too small");
if (blockSize >= 1) {
FUZ_DISPLAYTEST("LZ4_decompress_fast() with output buffer 1-byte too short");
decodedBuffer[blockSize-1] = 0;
{ int const r = LZ4_decompress_fast(cBuffer_exact, decodedBuffer, blockSize-1);
FUZ_CHECKTEST(r>=0, "LZ4_decompress_fast should have failed, due to Output Size being too small");
}
FUZ_CHECKTEST(decodedBuffer[blockSize-1]!=0, "LZ4_decompress_fast overrun specified output buffer");
}
FUZ_CHECKTEST(decodedBuffer[blockSize-1]!=0, "LZ4_decompress_fast overrun specified output buffer");
/* Test decoding with one byte too much => must fail */
FUZ_DISPLAYTEST();
@@ -568,12 +578,14 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c
}
/* Test decoding with output size being one byte too short => must fail */
FUZ_DISPLAYTEST();
decodedBuffer[blockSize-1] = 0;
{ int const r = LZ4_decompress_safe(cBuffer_exact, decodedBuffer, compressedSize, blockSize-1);
FUZ_CHECKTEST(r>=0, "LZ4_decompress_safe should have failed, due to Output Size being one byte too short");
if (blockSize >= 1) {
FUZ_DISPLAYTEST();
decodedBuffer[blockSize-1] = 0;
{ int const r = LZ4_decompress_safe(cBuffer_exact, decodedBuffer, compressedSize, blockSize-1);
FUZ_CHECKTEST(r>=0, "LZ4_decompress_safe should have failed, due to Output Size being one byte too short");
}
FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_safe overrun specified output buffer size");
}
FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_safe overrun specified output buffer size");
/* Test decoding with output size being 10 bytes too short => must fail */
FUZ_DISPLAYTEST();
@@ -638,58 +650,60 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c
}
FUZ_CHECKTEST(decodedBuffer[blockSize], "LZ4_decompress_safe overrun specified output buffer size");
/* Test partial decoding => must work */
FUZ_DISPLAYTEST("test LZ4_decompress_safe_partial");
{ size_t const missingOutBytes = FUZ_rand(&randState) % (unsigned)blockSize;
int const targetSize = (int)((size_t)blockSize - missingOutBytes);
size_t const extraneousInBytes = FUZ_rand(&randState) % 2;
int const inCSize = (int)((size_t)compressedSize + extraneousInBytes);
char const sentinel = decodedBuffer[targetSize] = block[targetSize] ^ 0x5A;
int const decResult = LZ4_decompress_safe_partial(compressedBuffer, decodedBuffer, inCSize, targetSize, blockSize);
FUZ_CHECKTEST(decResult<0, "LZ4_decompress_safe_partial failed despite valid input data (error:%i)", decResult);
FUZ_CHECKTEST(decResult != targetSize, "LZ4_decompress_safe_partial did not regenerated required amount of data (%i < %i <= %i)", decResult, targetSize, blockSize);
FUZ_CHECKTEST(decodedBuffer[targetSize] != sentinel, "LZ4_decompress_safe_partial overwrite beyond requested size (though %i <= %i <= %i)", decResult, targetSize, blockSize);
FUZ_CHECKTEST(memcmp(block, decodedBuffer, (size_t)targetSize), "LZ4_decompress_safe_partial: corruption detected in regenerated data");
}
if (blockSize) {
/* Test partial decoding => must work */
FUZ_DISPLAYTEST("test LZ4_decompress_safe_partial");
{ size_t const missingOutBytes = FUZ_rand(&randState) % (unsigned)blockSize;
int const targetSize = (int)((size_t)blockSize - missingOutBytes);
size_t const extraneousInBytes = FUZ_rand(&randState) % 2;
int const inCSize = (int)((size_t)compressedSize + extraneousInBytes);
char const sentinel = decodedBuffer[targetSize] = block[targetSize] ^ 0x5A;
int const decResult = LZ4_decompress_safe_partial(compressedBuffer, decodedBuffer, inCSize, targetSize, blockSize);
FUZ_CHECKTEST(decResult<0, "LZ4_decompress_safe_partial failed despite valid input data (error:%i)", decResult);
FUZ_CHECKTEST(decResult != targetSize, "LZ4_decompress_safe_partial did not regenerated required amount of data (%i < %i <= %i)", decResult, targetSize, blockSize);
FUZ_CHECKTEST(decodedBuffer[targetSize] != sentinel, "LZ4_decompress_safe_partial overwrite beyond requested size (though %i <= %i <= %i)", decResult, targetSize, blockSize);
FUZ_CHECKTEST(memcmp(block, decodedBuffer, (size_t)targetSize), "LZ4_decompress_safe_partial: corruption detected in regenerated data");
}
/* Partial decompression using dictionary. */
FUZ_DISPLAYTEST("test LZ4_decompress_safe_partial_usingDict using no dict");
{ size_t const missingOutBytes = FUZ_rand(&randState) % (unsigned)blockSize;
int const targetSize = (int)((size_t)blockSize - missingOutBytes);
size_t const extraneousInBytes = FUZ_rand(&randState) % 2;
int const inCSize = (int)((size_t)compressedSize + extraneousInBytes);
char const sentinel = decodedBuffer[targetSize] = block[targetSize] ^ 0x5A;
int const decResult = LZ4_decompress_safe_partial_usingDict(compressedBuffer, decodedBuffer, inCSize, targetSize, blockSize, NULL, 0);
FUZ_CHECKTEST(decResult<0, "LZ4_decompress_safe_partial_usingDict failed despite valid input data (error:%i)", decResult);
FUZ_CHECKTEST(decResult != targetSize, "LZ4_decompress_safe_partial_usingDict did not regenerated required amount of data (%i < %i <= %i)", decResult, targetSize, blockSize);
FUZ_CHECKTEST(decodedBuffer[targetSize] != sentinel, "LZ4_decompress_safe_partial_usingDict overwrite beyond requested size (though %i <= %i <= %i)", decResult, targetSize, blockSize);
FUZ_CHECKTEST(memcmp(block, decodedBuffer, (size_t)targetSize), "LZ4_decompress_safe_partial_usingDict: corruption detected in regenerated data");
}
/* Partial decompression using dictionary. */
FUZ_DISPLAYTEST("test LZ4_decompress_safe_partial_usingDict using no dict");
{ size_t const missingOutBytes = FUZ_rand(&randState) % (unsigned)blockSize;
int const targetSize = (int)((size_t)blockSize - missingOutBytes);
size_t const extraneousInBytes = FUZ_rand(&randState) % 2;
int const inCSize = (int)((size_t)compressedSize + extraneousInBytes);
char const sentinel = decodedBuffer[targetSize] = block[targetSize] ^ 0x5A;
int const decResult = LZ4_decompress_safe_partial_usingDict(compressedBuffer, decodedBuffer, inCSize, targetSize, blockSize, NULL, 0);
FUZ_CHECKTEST(decResult<0, "LZ4_decompress_safe_partial_usingDict failed despite valid input data (error:%i)", decResult);
FUZ_CHECKTEST(decResult != targetSize, "LZ4_decompress_safe_partial_usingDict did not regenerated required amount of data (%i < %i <= %i)", decResult, targetSize, blockSize);
FUZ_CHECKTEST(decodedBuffer[targetSize] != sentinel, "LZ4_decompress_safe_partial_usingDict overwrite beyond requested size (though %i <= %i <= %i)", decResult, targetSize, blockSize);
FUZ_CHECKTEST(memcmp(block, decodedBuffer, (size_t)targetSize), "LZ4_decompress_safe_partial_usingDict: corruption detected in regenerated data");
}
FUZ_DISPLAYTEST("test LZ4_decompress_safe_partial_usingDict() using prefix as dict");
{ size_t const missingOutBytes = FUZ_rand(&randState) % (unsigned)blockSize;
int const targetSize = (int)((size_t)blockSize - missingOutBytes);
size_t const extraneousInBytes = FUZ_rand(&randState) % 2;
int const inCSize = (int)((size_t)compressedSize + extraneousInBytes);
char const sentinel = decodedBuffer[targetSize] = block[targetSize] ^ 0x5A;
int const decResult = LZ4_decompress_safe_partial_usingDict(compressedBuffer, decodedBuffer, inCSize, targetSize, blockSize, decodedBuffer, dictSize);
FUZ_CHECKTEST(decResult<0, "LZ4_decompress_safe_partial_usingDict failed despite valid input data (error:%i)", decResult);
FUZ_CHECKTEST(decResult != targetSize, "LZ4_decompress_safe_partial_usingDict did not regenerated required amount of data (%i < %i <= %i)", decResult, targetSize, blockSize);
FUZ_CHECKTEST(decodedBuffer[targetSize] != sentinel, "LZ4_decompress_safe_partial_usingDict overwrite beyond requested size (though %i <= %i <= %i)", decResult, targetSize, blockSize);
FUZ_CHECKTEST(memcmp(block, decodedBuffer, (size_t)targetSize), "LZ4_decompress_safe_partial_usingDict: corruption detected in regenerated data");
}
FUZ_DISPLAYTEST("test LZ4_decompress_safe_partial_usingDict() using prefix as dict");
{ size_t const missingOutBytes = FUZ_rand(&randState) % (unsigned)blockSize;
int const targetSize = (int)((size_t)blockSize - missingOutBytes);
size_t const extraneousInBytes = FUZ_rand(&randState) % 2;
int const inCSize = (int)((size_t)compressedSize + extraneousInBytes);
char const sentinel = decodedBuffer[targetSize] = block[targetSize] ^ 0x5A;
int const decResult = LZ4_decompress_safe_partial_usingDict(compressedBuffer, decodedBuffer, inCSize, targetSize, blockSize, decodedBuffer, dictSize);
FUZ_CHECKTEST(decResult<0, "LZ4_decompress_safe_partial_usingDict failed despite valid input data (error:%i)", decResult);
FUZ_CHECKTEST(decResult != targetSize, "LZ4_decompress_safe_partial_usingDict did not regenerated required amount of data (%i < %i <= %i)", decResult, targetSize, blockSize);
FUZ_CHECKTEST(decodedBuffer[targetSize] != sentinel, "LZ4_decompress_safe_partial_usingDict overwrite beyond requested size (though %i <= %i <= %i)", decResult, targetSize, blockSize);
FUZ_CHECKTEST(memcmp(block, decodedBuffer, (size_t)targetSize), "LZ4_decompress_safe_partial_usingDict: corruption detected in regenerated data");
}
FUZ_DISPLAYTEST("test LZ4_decompress_safe_partial_usingDict() using external dict");
{ size_t const missingOutBytes = FUZ_rand(&randState) % (unsigned)blockSize;
int const targetSize = (int)((size_t)blockSize - missingOutBytes);
size_t const extraneousInBytes = FUZ_rand(&randState) % 2;
int const inCSize = (int)((size_t)compressedSize + extraneousInBytes);
char const sentinel = decodedBuffer[targetSize] = block[targetSize] ^ 0x5A;
int const decResult = LZ4_decompress_safe_partial_usingDict(compressedBuffer, decodedBuffer, inCSize, targetSize, blockSize, dict, dictSize);
FUZ_CHECKTEST(decResult<0, "LZ4_decompress_safe_partial_usingDict failed despite valid input data (error:%i)", decResult);
FUZ_CHECKTEST(decResult != targetSize, "LZ4_decompress_safe_partial_usingDict did not regenerated required amount of data (%i < %i <= %i)", decResult, targetSize, blockSize);
FUZ_CHECKTEST(decodedBuffer[targetSize] != sentinel, "LZ4_decompress_safe_partial_usingDict overwrite beyond requested size (though %i <= %i <= %i)", decResult, targetSize, blockSize);
FUZ_CHECKTEST(memcmp(block, decodedBuffer, (size_t)targetSize), "LZ4_decompress_safe_partial_usingDict: corruption detected in regenerated data");
FUZ_DISPLAYTEST("test LZ4_decompress_safe_partial_usingDict() using external dict");
{ size_t const missingOutBytes = FUZ_rand(&randState) % (unsigned)blockSize;
int const targetSize = (int)((size_t)blockSize - missingOutBytes);
size_t const extraneousInBytes = FUZ_rand(&randState) % 2;
int const inCSize = (int)((size_t)compressedSize + extraneousInBytes);
char const sentinel = decodedBuffer[targetSize] = block[targetSize] ^ 0x5A;
int const decResult = LZ4_decompress_safe_partial_usingDict(compressedBuffer, decodedBuffer, inCSize, targetSize, blockSize, dict, dictSize);
FUZ_CHECKTEST(decResult<0, "LZ4_decompress_safe_partial_usingDict failed despite valid input data (error:%i)", decResult);
FUZ_CHECKTEST(decResult != targetSize, "LZ4_decompress_safe_partial_usingDict did not regenerated required amount of data (%i < %i <= %i)", decResult, targetSize, blockSize);
FUZ_CHECKTEST(decodedBuffer[targetSize] != sentinel, "LZ4_decompress_safe_partial_usingDict overwrite beyond requested size (though %i <= %i <= %i)", decResult, targetSize, blockSize);
FUZ_CHECKTEST(memcmp(block, decodedBuffer, (size_t)targetSize), "LZ4_decompress_safe_partial_usingDict: corruption detected in regenerated data");
}
}
/* Test Compression with limited output size */
@@ -754,7 +768,7 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c
/* Decompress with dictionary as prefix */
FUZ_DISPLAYTEST("test LZ4_decompress_fast_usingDict() with dictionary as prefix");
memcpy(decodedBuffer, dict, dictSize);
if (dictSize) memcpy(decodedBuffer, dict, dictSize);
ret = LZ4_decompress_fast_usingDict(compressedBuffer, decodedBuffer+dictSize, blockSize, decodedBuffer, dictSize);
FUZ_CHECKTEST(ret!=blockContinueCompressedSize, "LZ4_decompress_fast_usingDict did not read all compressed block input");
{ U32 const crcCheck = XXH32(decodedBuffer+dictSize, (size_t)blockSize, 0);
@@ -780,6 +794,7 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c
FUZ_DISPLAYTEST("LZ4_compress_fast_continue() with dictionary and output buffer too short by one byte");
LZ4_loadDict(&LZ4dictBody, dict, dictSize);
assert(blockContinueCompressedSize >= 1);
ret = LZ4_compress_fast_continue(&LZ4dictBody, block, compressedBuffer, blockSize, blockContinueCompressedSize-1, 1);
FUZ_CHECKTEST(ret>0, "LZ4_compress_fast_continue using ExtDict should fail : one missing byte for output buffer : %i written, %i buffer", ret, blockContinueCompressedSize);
@@ -814,17 +829,19 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c
FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe_usingDict corrupted decoded data");
}
FUZ_DISPLAYTEST();
decodedBuffer[blockSize-1] = 0;
ret = LZ4_decompress_fast_usingDict(compressedBuffer, decodedBuffer, blockSize-1, dict, dictSize);
FUZ_CHECKTEST(ret>=0, "LZ4_decompress_fast_usingDict should have failed : wrong original size (-1 byte)");
FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_fast_usingDict overrun specified output buffer size");
if (blockSize >= 1) {
FUZ_DISPLAYTEST();
decodedBuffer[blockSize-1] = 0;
ret = LZ4_decompress_fast_usingDict(compressedBuffer, decodedBuffer, blockSize-1, dict, dictSize);
FUZ_CHECKTEST(ret>=0, "LZ4_decompress_fast_usingDict should have failed : wrong original size (-1 byte)");
FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_fast_usingDict overrun specified output buffer size");
FUZ_DISPLAYTEST();
decodedBuffer[blockSize-1] = 0;
ret = LZ4_decompress_safe_usingDict(compressedBuffer, decodedBuffer, blockContinueCompressedSize, blockSize-1, dict, dictSize);
FUZ_CHECKTEST(ret>=0, "LZ4_decompress_safe_usingDict should have failed : not enough output size (-1 byte)");
FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_safe_usingDict overrun specified output buffer size");
FUZ_DISPLAYTEST();
decodedBuffer[blockSize-1] = 0;
ret = LZ4_decompress_safe_usingDict(compressedBuffer, decodedBuffer, blockContinueCompressedSize, blockSize-1, dict, dictSize);
FUZ_CHECKTEST(ret>=0, "LZ4_decompress_safe_usingDict should have failed : not enough output size (-1 byte)");
FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_safe_usingDict overrun specified output buffer size");
}
FUZ_DISPLAYTEST();
{ int const missingBytes = (FUZ_rand(&randState) & 0xF) + 2;
@@ -864,6 +881,7 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c
FUZ_DISPLAYTEST("LZ4_compress_fast_continue() after LZ4_attach_dictionary(), but output buffer is 1 byte too short");
LZ4_resetStream_fast(&LZ4_stream);
LZ4_attach_dictionary(&LZ4_stream, &LZ4dictBody);
assert(blockContinueCompressedSize >= 1);
ret = LZ4_compress_fast_continue(&LZ4_stream, block, compressedBuffer, blockSize, blockContinueCompressedSize-1, 1);
FUZ_CHECKTEST(ret>0, "LZ4_compress_fast_continue using extDictCtx should fail : one missing byte for output buffer : %i written, %i buffer", ret, blockContinueCompressedSize);
/* note : context is no longer dirty after a failed compressed block */
@@ -908,17 +926,19 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c
FUZ_CHECKTEST(crcCheck!=crcOrig, "LZ4_decompress_safe_usingDict corrupted decoded data");
}
FUZ_DISPLAYTEST();
decodedBuffer[blockSize-1] = 0;
ret = LZ4_decompress_fast_usingDict(compressedBuffer, decodedBuffer, blockSize-1, dict, dictSize);
FUZ_CHECKTEST(ret>=0, "LZ4_decompress_fast_usingDict should have failed : wrong original size (-1 byte)");
FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_fast_usingDict overrun specified output buffer size");
if (blockSize >= 1) {
FUZ_DISPLAYTEST();
decodedBuffer[blockSize-1] = 0;
ret = LZ4_decompress_fast_usingDict(compressedBuffer, decodedBuffer, blockSize-1, dict, dictSize);
FUZ_CHECKTEST(ret>=0, "LZ4_decompress_fast_usingDict should have failed : wrong original size (-1 byte)");
FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_fast_usingDict overrun specified output buffer size");
FUZ_DISPLAYTEST();
decodedBuffer[blockSize-1] = 0;
ret = LZ4_decompress_safe_usingDict(compressedBuffer, decodedBuffer, blockContinueCompressedSize, blockSize-1, dict, dictSize);
FUZ_CHECKTEST(ret>=0, "LZ4_decompress_safe_usingDict should have failed : not enough output size (-1 byte)");
FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_safe_usingDict overrun specified output buffer size");
FUZ_DISPLAYTEST();
decodedBuffer[blockSize-1] = 0;
ret = LZ4_decompress_safe_usingDict(compressedBuffer, decodedBuffer, blockContinueCompressedSize, blockSize-1, dict, dictSize);
FUZ_CHECKTEST(ret>=0, "LZ4_decompress_safe_usingDict should have failed : not enough output size (-1 byte)");
FUZ_CHECKTEST(decodedBuffer[blockSize-1], "LZ4_decompress_safe_usingDict overrun specified output buffer size");
}
FUZ_DISPLAYTEST("LZ4_decompress_safe_usingDict with a too small output buffer");
{ int const missingBytes = (FUZ_rand(&randState) & 0xF) + 2;
@@ -941,6 +961,7 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c
FUZ_DISPLAYTEST("LZ4_compress_HC_continue with same external dictionary, but output buffer 1 byte too short");
LZ4_loadDictHC(LZ4dictHC, dict, dictSize);
assert(blockContinueCompressedSize>=1);
ret = LZ4_compress_HC_continue(LZ4dictHC, block, compressedBuffer, blockSize, blockContinueCompressedSize-1);
FUZ_CHECKTEST(ret>0, "LZ4_compress_HC_continue using ExtDict should fail : one missing byte for output buffer (expected %i, but result=%i)", blockContinueCompressedSize, ret);
/* note : context is no longer dirty after a failed compressed block */
@@ -977,6 +998,7 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c
FUZ_DISPLAYTEST();
LZ4_resetStreamHC_fast (LZ4_streamHC, compressionLevel);
LZ4_attach_HC_dictionary(LZ4_streamHC, LZ4dictHC);
assert(blockContinueCompressedSize>=1);
ret = LZ4_compress_HC_continue(LZ4_streamHC, block, compressedBuffer, blockSize, blockContinueCompressedSize-1);
FUZ_CHECKTEST(ret>0, "LZ4_compress_HC_continue using ExtDictCtx should fail : one missing byte for output buffer (%i != %i)", ret, blockContinueCompressedSize);
/* note : context is no longer dirty after a failed compressed block */
@@ -1014,7 +1036,7 @@ static int FUZ_test(U32 seed, U32 nbCycles, const U32 startCycle, const double c
/* Compress HC continue destSize */
FUZ_DISPLAYTEST();
{ int const availableSpace = (int)(FUZ_rand(&randState) % (U32)blockSize) + 5;
{ int const availableSpace = (blockSize ? (int)(FUZ_rand(&randState) % (U32)blockSize) : 0) + 5;
int consumedSize = blockSize;
FUZ_DISPLAYTEST();
LZ4_loadDictHC(LZ4dictHC, dict, dictSize);
@@ -1297,6 +1319,30 @@ static void FUZ_unitTests(int compressionLevel)
} }
}
DISPLAYLEVEL(3, "LZ4_compress_HC() with NULL input:");
{ int const maxCSize = LZ4_compressBound(0);
int level;
for (level=LZ4HC_CLEVEL_MIN; level <= LZ4HC_CLEVEL_MAX; level++) {
int const cSize = LZ4_compress_HC(NULL, testCompressed, 0, maxCSize, level);
FUZ_CHECKTEST(!(cSize==1 && testCompressed[0]==0),
"compressing empty should give byte 0"
" (maxCSize == %i) (cSize == %i) (byte == 0x%02X)",
maxCSize, cSize, testCompressed[0]);
}
}
DISPLAYLEVEL(3, " OK \n");
DISPLAYLEVEL(3, "LZ4_compress_HC() with both NULL input and output:");
{ int level;
for (level=LZ4HC_CLEVEL_MIN; level <= LZ4HC_CLEVEL_MAX; level++) {
int const cSize = LZ4_compress_HC(NULL, NULL, 0, 0, level);
FUZ_CHECKTEST(cSize != 0,
"compressing into NULL must fail"
" (cSize == %i != 0)", cSize);
}
}
DISPLAYLEVEL(3, " OK \n");
DISPLAYLEVEL(3, "LZ4_initStreamHC with multiple valid alignments : ");
{ typedef struct {
LZ4_streamHC_t hc1;
@@ -1420,10 +1466,10 @@ static void FUZ_unitTests(int compressionLevel)
LZ4_resetStreamHC_fast(&sHC, compressionLevel);
LZ4_loadDictHC(&sHC, testInput, segSize);
result1 = LZ4_compress_HC_continue(&sHC, testInput + segSize, testCompressed, segSize, segSize -1);
FUZ_CHECKTEST(result1==0, "LZ4_compressHC_limitedOutput_continue() dictionary compression failed : result = %i", result1);
FUZ_CHECKTEST(result1==0, "LZ4_compress_HC_continue() dictionary compression failed : result = %i", result1);
FUZ_CHECKTEST(sHC.internal_donotuse.dirty, "Context should be clean");
result2 = LZ4_compress_HC_continue(&sHC, testInput + 2*(size_t)segSize, testCompressed+result1, segSize, segSize-1);
FUZ_CHECKTEST(result2==0, "LZ4_compressHC_limitedOutput_continue() dictionary compression failed : result = %i", result2);
FUZ_CHECKTEST(result2==0, "LZ4_compress_HC_continue() dictionary compression failed : result = %i", result2);
FUZ_CHECKTEST(sHC.internal_donotuse.dirty, "Context should be clean");
result = LZ4_decompress_safe_usingDict(testCompressed, testVerify, result1, segSize, testInput, segSize);
@@ -1439,7 +1485,7 @@ static void FUZ_unitTests(int compressionLevel)
LZ4_resetStreamHC_fast(&sHC, compressionLevel);
LZ4_loadDictHC(&sHC, testInput, 32 KB);
result = LZ4_compress_HC_continue(&sHC, testInput + 64 KB, testCompressed, testCompressedSize, testCompressedSize-1);
FUZ_CHECKTEST(result==0, "LZ4_compressHC_limitedOutput_continue() remote dictionary failed : result = %i", result);
FUZ_CHECKTEST(result==0, "LZ4_compress_HC_continue() remote dictionary failed : result = %i", result);
FUZ_CHECKTEST(sHC.internal_donotuse.dirty, "Context should be clean");
result = LZ4_decompress_safe_usingDict(testCompressed, testVerify, result, testCompressedSize, testInput, 32 KB);
@@ -1471,7 +1517,7 @@ static void FUZ_unitTests(int compressionLevel)
crcOrig = XXH64_digest(&crcOrigState);
assert(segSize <= INT_MAX);
result = LZ4_compress_HC_continue(&sHC, testInput + segStart, testCompressed, (int)segSize, LZ4_compressBound((int)segSize));
FUZ_CHECKTEST(result==0, "LZ4_compressHC_limitedOutput_continue() dictionary compression failed : result = %i", result);
FUZ_CHECKTEST(result==0, "LZ4_compress_HC_continue() dictionary compression failed : result = %i", result);
FUZ_CHECKTEST(sHC.internal_donotuse.dirty, "Context should be clean");
result = LZ4_decompress_safe_usingDict(testCompressed, dst, result, (int)segSize, dict, (int)dictSize);