html2
[ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
This chapter describes how flex handles dynamic memory, and how you can override the default behavior.
21.1 The Default Memory Management | ||
21.2 Overriding The Default Memory Management | ||
21.3 A Note About yytext And Memory |
[ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Flex allocates dynamic memory during initialization, and once in a while from
within a call to yylex(). Initialization takes place during the first call to
yylex(). Thereafter, flex may reallocate more memory if it needs to enlarge a
buffer. As of version 2.5.9 Flex will clean up all memory when you call yylex_destroy
See faq-memory-leak.
Flex allocates dynamic memory for four purposes, listed below (2)
%option stack
is not
specified. You will rarely need to tune this buffer. The ideal size for this
stack is the maximum depth expected. The memory for this stack is
automatically destroyed when you call yylex_destroy(). See option-stack.
[ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Flex calls the functions yyalloc
, yyrealloc
, and yyfree
when it needs to allocate or free memory. By default, these functions are
wrappers around the standard C functions, malloc
, realloc
, and
free
, respectively. You can override the default implementations by telling
flex that you will provide your own implementations.
To override the default implementations, you must do two things:
@verbatim // For a non-reentrant scanner void * yyalloc (size_t bytes); void * yyrealloc (void * ptr, size_t bytes); void yyfree (void * ptr); // For a reentrant scanner void * yyalloc (size_t bytes, void * yyscanner); void * yyrealloc (void * ptr, size_t bytes, void * yyscanner); void yyfree (void * ptr, void * yyscanner); |
In the following example, we will override all three memory routines. We assume
that there is a custom allocator with garbage collection. In order to make this
example interesting, we will use a reentrant scanner, passing a pointer to the
custom allocator through yyextra
.
@verbatim %{ #include "some_allocator.h" %} /* Suppress the default implementations. */ %option noyyalloc noyyrealloc noyyfree %option reentrant /* Initialize the allocator. */ #define YY_EXTRA_TYPE struct allocator* #define YY_USER_INIT yyextra = allocator_create(); %% .|\n ; %% /* Provide our own implementations. */ void * yyalloc (size_t bytes, void* yyscanner) { return allocator_alloc (yyextra, bytes); } void * yyrealloc (void * ptr, size_t bytes, void* yyscanner) { return allocator_realloc (yyextra, bytes); } void yyfree (void * ptr, void * yyscanner) { /* Do nothing -- we leave it to the garbage collector. */ } |
[ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
When flex finds a match, yytext
points to the first character of the
match in the input buffer. The string itself is part of the input buffer, and
is NOT allocated separately. The value of yytext will be overwritten the next
time yylex() is called. In short, the value of yytext is only valid from within
the matched rule's action.
Often, you want the value of yytext to persist for later processing, i.e., by a parser with non-zero lookahead. In order to preserve yytext, you will have to copy it with strdup() or a similar function. But this introduces some headache because your parser is now responsible for freeing the copy of yytext. If you use a yacc or bison parser, (commonly used with flex), you will discover that the error recovery mechanisms can cause memory to be leaked.
To prevent memory leaks from strdup'd yytext, you will have to track the memory somehow. Our experience has shown that a garbage collection mechanism or a pooled memory mechanism will save you a lot of grief when writing parsers.
[ << ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |