4cpp Lexing Library

Cpp_Get_Token_Result cpp_get_token()

A Cpp_Get_Token_Result struct is returned containing the index of a token and a flag indicating whether the pos is contained in the token or in whitespace after the token.

This call performs a binary search over all of the tokens looking for the token that contains the specified position. If the position is in whitespace between the tokens, the returned token index is the index of the token immediately before the provided position. The returned index can be -1 if the position is before the first token.

Cpp_Lex_Result cpp_lex_step()

This call is the primary interface of the lexing system. It is quite general so it can be used in a lot of different ways. I will explain the general rules first, and then give some examples of common ways it might be used.

First a lexing state, Cpp_Lex_Data, must be initialized. The file to lex must be read into N contiguous chunks of memory. An output Cpp_Token_Array must be allocated and initialized with the appropriate count and max_count values. Then each chunk of the file must be passed to cpp_lex_step in order using the same lexing state for each call. Every time a call to cpp_lex_step returns LexResult_NeedChunk, the next call to cpp_lex_step should use the next chunk. If the return is some other value, the lexer hasn't finished with the current chunk and it sopped for some other reason, so the same chunk should be used again in the next call.

If the file chunks contain a null terminator the lexer will return LexResult_Finished when it finds this character. At this point calling the lexer again with the same state will result in an error. If you do not have a null terminated chunk to end the file, you may instead pass the exact size in bytes of the entire file to the full_size parameter and it will automatically handle the termination of the lexing state when it has read that many bytes. If a full_size is specified and the system terminates for having seen that many bytes, it will return LexResult_Finished. If a full_size is specified and a null character is read before the total number of bytes have been read the system will still terminate as usual and return LexResult_Finished.

If the system has filled the entire output array it will return LexResult_NeedTokenMemory. When this happens if you want to continue lexing the file you can grow the token array, or switch to a new output array and then call cpp_lex_step again with the chunk that was being lexed and the new output. You can also specify a max_tokens_out which is limits how many new tokens will be added to the token array. Even if token_array_out still had more space to hold tokens, if the max_tokens_out limit is hit, the lexer will stop and return LexResult_HitTokenLimit. If this happens there is still space left in the token array, so you can resume simply by calling cpp_lex_step again with the same chunk and the same output array. Also note that, unlike the chunks which must only be replaced when the system says it needs a chunk. You may switch to or modify the output array in between calls as much as you like.

The most basic use of this system is to get it all done in one big chunk and try to allocate a nearly "infinite" output array so that it will not run out of memory. This way you can get the entire job done in one call and then just assert to make sure it returns LexResult_Finished to you:

Cpp_Lex_Data cpp_lex_data_init()

A brand new lex state ready to begin lexing a file from the beginning.

Creates a new lex state in the form of a Cpp_Lex_Data struct and returns the struct. The system needs a temporary buffer that is as long as the longest token. 4096 is usually enough but the buffer is not checked, so to be 100% bullet proof it has to be the same length as the file being lexed.

int32_t cpp_lex_data_temp_size()

This call gets the current size of the temporary buffer in the lexer state so that you can move to a new temporary buffer by copying the data over.

void cpp_lex_data_temp_read()

This call reads the current contents of the temporary buffer.

void cpp_lex_data_new_temp_DEP()

Cpp_Relex_Range cpp_get_relex_range()

Cpp_Relex_Data cpp_relex_init()

Returns a partially initialized relex state.

This call does the first setup step of initializing a relex state. To finish initializing the relex state you must tell the state about the positioning of the first chunk it will be fed. There are two methods of doing this, the direct method is with cpp_relex_declare_first_chunk_position, the method that is often more convenient is with cpp_relex_is_start_chunk. If the file is not chunked the second step of initialization can be skipped.

int32_t cpp_relex_start_position()

Returns the first position in the file the relexer wants to read. This is usually a position slightly earlier than the start_pos provided as the edit range.

After doing the first stage of initialization this call is useful for figuring out what chunk of the file to feed to the lexer first. It should be a chunk that contains the position returned by this call.

void cpp_relex_declare_first_chunk_position()

To initialize the relex system completely, the system needs to know how the characters in the first file line up with the file's absolute layout. This call declares where the first chunk's start position is in the absolute file layout, and the system infers the alignment from that. For this method to work the starting position of the relexing needs to be inside the first chunk. To get the relexers starting position call cpp_relex_start_position.

int32_t cpp_relex_is_start_chunk()

Returns non-zero if the passed in chunk should be used as the first chunk for lexing.

With this method, once a state is initialized, each chunk can be fed in one after the other in the order they appear in the absolute file layout. When this call returns non-zero it means that the chunk that was passed in on that call should be used in the first call to cpp_relex_step. If, after trying all of the chunks, they all return zero, pass in NULL for chunk and 0 for chunk_size to tell the system that all possible chunks have already been tried, and then use those values again in the one and only call to cpp_relex_step.

Cpp_Lex_Result cpp_relex_step()

When a file has already been lexed, and then it is edited in a small local way, rather than lexing the new file all over again, cpp_relex_step can try to find just the range of tokens that need to be updated and fix them in.

First the lex state must be initialized (cpp_relex_init). Then one or more calls to cpp_relex_step will start editing the array and filling out the relex_array. The return value of cpp_relex_step indicates whether the relex was successful or was interrupted and if it was interrupted, what the system needs to resume.

LexResult_Finished indicates that the relex engine finished successfully.

LexResult_NeedChunk indicates that the system needs the next chunk of the file.

LexResult_NeedTokenMemory indicates that the relex_array has reached capacity, and that it needs to be extended if it is going to continue. Sometimes in this case it is better to stop and just lex the entire file normally, because there are a few cases where a small local change effects a long range of the lexers output.

The relex operation can be closed in one of two ways. If the LexResult_Finished value has been returned by this call, then to complete the edits to the array make sure the original array has enough capacity to store the final result by calling cpp_relex_get_new_count. Then the operation can be finished successfully by calling cpp_relex_complete.

Whether or not the relex process finished with LexResult_Finished the process can be finished by calling cpp_relex_abort, which puts the array back into it's original state. No close is necessary if getting the original array state back is not necessary.

int32_t cpp_relex_get_new_count()

After getting a LexResult_Finished from cpp_relex_step, this call can be used to get the size the new array will have. If the original array doesn't have enough capacity to store the new array, it's capacity should be increased before passing to cpp_relex_complete.

void cpp_relex_complete()

After getting a LexResult_Finished from cpp_relex_step, and ensuring that array has a large enough capacity by calling cpp_relex_get_new_count, this call does the necessary replacement of tokens in the array to make it match the new file.

void cpp_relex_abort()

After the first call to cpp_relex_step, the array's contents may have been changed, this call assures the array is in it's original state. After this call the relex state is dead.

Cpp_Token_Array cpp_make_token_array()

An empty Cpp_Token_Array with memory malloc'd for storing tokens.

This call allocates a Cpp_Token_Array with malloc for use in other convenience functions. Stacks that are not allocated this way should not be used in the convenience functions.

void cpp_free_token_array()

This call frees a Cpp_Token_Array.

void cpp_resize_token_array()

This call allocates a new memory chunk and moves the existing tokens in the array over to the new chunk.

void cpp_lex_file()

Lexes an entire file and manages the interaction with the lexer system so that it is quick and convenient to lex files.

enum Cpp_Token_Type;

A Cpp_Token_Type classifies a token to make parsing easier. Some types are not actually output by the lexer, but exist because parsers will also make use of token types in their own output.

struct Cpp_Token {
};

Cpp_Token represents a single lexed token. It is the primary output of the lexing system.

enum Cpp_Token_Flag;

The Cpp_Token_Flags are used to mark up tokens with additional information.

struct Cpp_Token_Array {
};

Cpp_Token_Array is used to bundle together the common elements of a growing array of Cpp_Tokens. To initialize it the tokens field should point to a block of memory with a size equal to max_count*sizeof(Cpp_Token) and the count should be initialized to zero.

struct Cpp_Get_Token_Result {
};

Cpp_Get_Token_Result is the return result of the cpp_get_token call.

struct Cpp_Relex_Range {
};

Cpp_Relex_Range is the return result of the cpp_get_relex_range call.

struct Cpp_Lex_Data { /* non-public internals */ } ;

Cpp_Lex_Data represents the state of the lexer so that the system may be resumable and the user can manage the lexer state and decide when to resume lexing with it. To create a new lexer state call cpp_lex_data_init.

The internals of the lex state should not be treated as a part of the public API.

enum Cpp_Lex_Result;

Cpp_Lex_Result is returned from the lexing engine to indicate why it stopped lexing.

struct Cpp_Relex_Data { /* non-public internals */ } ;

Cpp_Relex_Data represents the state of the relexer so that the system may be resumable. To create a new relex state call cpp_relex_init.