[jbig2dec] Limit allocator to common memory use, avoiding timeouts (#6184)

The JBIG2 format allows for ridiculously large images (4G x 4G
pixels!), which means that jbig2dec may use an enormous amount of
memory when it tries to decode them. OSS-fuzz currently restricts test
cases to 2.5Gbyte of memory per run, so the jbig2dec fuzzer
implemented a custom allocator to limit jbig2dec's memory usage to
1Gbyte. When the custom allocator runs out of memory jbig2dec
indicates an error, frees its resources and returns.

Limiting memory usage to 1Gbyte, below OSS-fuzz's limit of 2.5Gbyte,
eliminated the entire class of false positive OSS-fuzz issues
concerning out of memory situations. These were false positives in the
sense that a program using jbig2dec is in control of how much memory
jbig2dec uses, but the program must implement a custom jbig2dec
allocator and limit it to the desired amount of memory.

Another class of false positive OSS-fuzz issues remain; issues where
the image data still takes more than 25 seconds to process, causing an
OSS-fuzz timeout. These cases use less than 1Gbyte of memory, but
processing that amount of data may still take a long time. Since
processing time and data size are related, a program may limit the
amount of memory allotted to jbigdec's custom allocator to something
less than 1Gbyte to reduce processing time.

Running through a set of real world JBIG2 images shows that no more
than 20MByte is used to decode any of them and none take more then 25
seconds to decode on a desktop machine. To eliminate the class of
false positive OSS-fuzz timeout issues the fuzzer will now limit the
amount of memory to 32Mbyte with the hope that their processing time
will be reduced below 25 seconds.

Of course OSS-fuzz may still detect issues where jbig2dec gets caught
in an endless loop (or the processing time is long for a reason other
than data size). These are the issues we want OSS-fuzz to identify and
get fixed, since the parameters causing those timeouts are not in
control by a program using jbig2dec.

projects/jbig2dec/jbig2_fuzzer.cc

@@ -27,7 +27,7 @@
 #define KBYTE ((size_t) 1024)
 #define MBYTE (1024 * KBYTE)
 #define GBYTE (1024 * MBYTE)
-#define MAX_ALLOCATION (1 * GBYTE)
+#define MAX_ALLOCATION (32 * MBYTE)
 
 static size_t used;