mr4th-public
/
4coder
1
Fork 0
Code Issues Releases Activity
4coder/4coder_API.html

135 lines
90 KiB
HTML
Raw Normal View History

converted portion of the HTML doc generator to string append instead of fprintf
2016-09-04 05:08:36 +00:00
<html lang="en-US"><head><title>4coder API Docs</title><style>body { background: #FAFAFA; color: #0D0D0D; }h1,h2,h3,h4 { color: #309030; margin: 0; }h2 { margin-top: 6mm; }h3 { margin-top: 5mm; margin-bottom: 5mm; }h4 { font-size: 1.1em; }a { color: #309030; text-decoration: none; }a:visited { color: #A0C050; }a:hover { background: #E0FFD0; }ul { list-style: none; padding: 0; margin: 0; }</style></head>
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
<body><div style='font-family:Arial; margin: 0 auto; width: 800px; text-align: justify; line-height: 1.25;'><h1 style='margin-top: 5mm; margin-bottom: 5mm;'>4cpp Lexing Library</h1><h3 style='margin:0;'>Table of Contents</h3><ul><li><a href='#section_introduction'>&sect;1 Introduction</a></li><li><a href='#section_lexer_library'>&sect;2 Lexer Library</a></li></ul>
<h2 id='section_introduction'>&sect;1 Introduction</h2><div><p>This is the documentation for the 4cpp lexer version 1.1. The documentation is the newest piece of this lexer project so it may still have problems. What is here should be correct and mostly complete.</p><p>If you have questions or discover errors please contact <span style='font-family: "Courier New", Courier, monospace; text-align: left;'>editor@4coder.net</span> or to get help from community members you can post on the 4coder forums hosted on handmade.network at <span style='font-family: "Courier New", Courier, monospace; text-align: left;'>4coder.handmade.network</span></p></div>
<h2 id='section_lexer_library'>&sect;2 Lexer Library</h2><h3>&sect;2.1 Lexer Intro</h3><div>The 4cpp lexer system provides a polished, fast, flexible system that takes in C/C++ and outputs a tokenization of the text data. There are two API levels. One level is setup to let you easily get a tokenization of the file. This level manages memory for you with malloc to make it as fast as possible to start getting your tokens. The second level enables deep integration by allowing control over allocation, data chunking, and output rate control.<br><br>To use the quick setup API you simply include 4cpp_lexer.h and read the documentation at <a href='#cpp_lex_file_doc'>cpp_lex_file</a>.<br><br>To use the the fancier API include 4cpp_lexer.h and read the documentation at <a href='#cpp_lex_step_doc'>cpp_lex_step</a>. If you want to be absolutely sure you are not including malloc into your program you can define FCPP_FORBID_MALLOC before the include and the "step" API will continue to work.<br><br>There are a few more features in 4cpp that are not documented yet. You are free to try to use these, but I am not totally sure they are ready yet, and when they are they will be documented.</div><h3>&sect;2.2 Lexer Function List</h3><ul><li><a href='#cpp_get_token_doc'>cpp_get_token</a></li><li><a href='#cpp_lex_step_doc'>cpp_lex_step</a></li><li><a href='#cpp_lex_data_init_doc'>cpp_lex_data_init</a></li><li><a href='#cpp_lex_data_temp_size_doc'>cpp_lex_data_temp_size</a></li><li><a href='#cpp_lex_data_temp_read_doc'>cpp_lex_data_temp_read</a></li><li><a href='#cpp_lex_data_new_temp_DEP_doc'>cpp_lex_data_new_temp_DEP</a></li><li><a href='#cpp_get_relex_range_doc'>cpp_get_relex_range</a></li><li><a href='#cpp_relex_init_doc'>cpp_relex_init</a></li><li><a href='#cpp_relex_start_position_doc'>cpp_relex_start_position</a></li><li><a href='#cpp_relex_declare_first_chunk_position_doc'>cpp_relex_declare_first_chunk_position</a></li><li><a href='#cpp_relex_is_start_chunk_doc'>cpp_relex_is_start_chunk</a></li><li><a href='#cpp_relex_step_doc'>cpp_relex_step</a></li><li><a href='#cpp_relex_get_new_count_doc'>cpp_relex_get_new_count</a></li><li><a href='#cpp_relex_complete_doc'>cpp_relex_complete</a></li><li><a href='#cpp_relex_abort_doc'>cpp_relex_abort</a></li><li><a href='#cpp_make_token_array_doc'>cpp_make_token_array</a></li><li><a href='#cpp_free_token_array_doc'>cpp_free_token_array</a></li><li><a href='#cpp_resize_token_array_doc'>cpp_resize_token_array</a></li><li><a href='#cpp_lex_file_doc'>cpp_lex_file</a></li></ul><h3>&sect;2.3 Lexer Types List</h3><ul><li><a href='#Cpp_Token_Type_doc'>Cpp_Token_Type</a></li><li><a href='#Cpp_Token_doc'>Cpp_Token</a></li><li><a href='#Cpp_Token_Flag_doc'>Cpp_Token_Flag</a></li><li><a href='#Cpp_Token_Array_doc'>Cpp_Token_Array</a></li><li><a href='#Cpp_Get_Token_Result_doc'>Cpp_Get_Token_Result</a></li><li><a href='#Cpp_Relex_Range_doc'>Cpp_Relex_Range</a></li><li><a href='#Cpp_Lex_Data_doc'>Cpp_Lex_Data</a></li><li><a href='#Cpp_Lex_Result_doc'>Cpp_Lex_Result</a></li><li><a href='#Cpp_Relex_Data_doc'>Cpp_Relex_Data</a></li></ul><h3>&sect;2.4 Lexer Function Descriptions</h3><div id='cpp_get_token_doc' style='margin-bottom: 1cm;'><h4>&sect;2.4.1: cpp_get_token</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>Cpp_Get_Token_Result cpp_get_token(<div style='margin-left: 4mm;'>Cpp_Token_Array *token_array_in,<br>int32_t pos<br></div>)</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><div style='font-weight: 600;'>token_array</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The array of tokens from which to get a token.</div></div></div><div><div style='font-weight: 600;'>pos</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The position, measured in bytes, to get the token for.</div></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #30
wrote lexer function docs, expanded doc generator
2016-09-06 21:22:35 +00:00
of a token and a flag indicating whether the pos is contained in the token
or in whitespace after the token.</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>This call performs a binary search over all of the tokens looking
for the token that contains the specified position. If the position
is in whitespace between the tokens, the returned token index is the
index of the token immediately before the provided position. The returned
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
index can be -1 if the position is before the first token.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#Cpp_Get_Token_Result_doc'>Cpp_Get_Token_Result</a></div></div><hr><div id='cpp_lex_step_doc' style='margin-bottom: 1cm;'><h4>&sect;2.4.2: cpp_lex_step</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>Cpp_Lex_Result cpp_lex_step(<div style='margin-left: 4mm;'>Cpp_Lex_Data *S_ptr,<br>char *chunk,<br>int32_t size,<br>int32_t full_size,<br>Cpp_Token_Array *token_array_out,<br>int32_t max_tokens_out<br></div>)</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><div style='font-weight: 600;'>S_ptr</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The lexer state. Go to the Cpp_Lex_Data section to see how to initialize the state.</div></div></div><div><div style='font-weight: 600;'>chunk</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The first or next chunk of the file being lexed.</div></div></div><div><div style='font-weight: 600;'>size</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The number of bytes in the chunk including the null terminator if the chunk ends in a null terminator.
wrote lexer function docs, expanded doc generator
2016-09-06 21:22:35 +00:00
If the chunk ends in a null terminator the system will interpret it as the end of the file.</div></div></div><div><div style='font-weight: 600;'>full_size</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>If the final chunk is not null terminated this parameter should specify the length of the
file in bytes. To rely on an eventual null terminator use HAS_NULL_TERM for this parameter.</div></div></div><div><div style='font-weight: 600;'>token_array_out</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The token array structure that will receive the tokens output by the lexer.</div></div></div><div><div style='font-weight: 600;'>max_tokens_out</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The maximum number of tokens to be output to the token array. To rely on the
max built into the token array pass NO_OUT_LIMIT here.</div></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>This call is the primary interface of the lexing system. It is quite general so it can be used in
a lot of different ways. I will explain the general rules first, and then give some examples of common
ways it might be used.<br><br>
First a lexing state, Cpp_Lex_Data, must be initialized. The file to lex must be read into N contiguous chunks
got 4cpp ready to ship
2016-09-07 03:39:19 +00:00
of memory. An output Cpp_Token_Array must be allocated and initialized with the appropriate count and max_count
values. Then each chunk of the file must be passed to cpp_lex_step in order using the same lexing state for each call.
Every time a call to cpp_lex_step returns LexResult_NeedChunk, the next call to cpp_lex_step should use the
wrote lexer function docs, expanded doc generator
2016-09-06 21:22:35 +00:00
next chunk. If the return is some other value, the lexer hasn't finished with the current chunk and it sopped for some
other reason, so the same chunk should be used again in the next call.<br><br>
If the file chunks contain a null terminator the lexer will return LexResult_Finished when it finds this character.
At this point calling the lexer again with the same state will result in an error. If you do not have a null
terminated chunk to end the file, you may instead pass the exact size in bytes of the entire file to the full_size
parameter and it will automatically handle the termination of the lexing state when it has read that many bytes.
got 4cpp ready to ship
2016-09-07 03:39:19 +00:00
If a full_size is specified and the system terminates for having seen that many bytes, it will return
LexResult_Finished. If a full_size is specified and a null character is read before the total number of bytes have
been read the system will still terminate as usual and return LexResult_Finished.<br><br>
wrote lexer function docs, expanded doc generator
2016-09-06 21:22:35 +00:00
If the system has filled the entire output array it will return LexResult_NeedTokenMemory. When this happens if you
want to continue lexing the file you can grow the token array, or switch to a new output array and then call
got 4cpp ready to ship
2016-09-07 03:39:19 +00:00
cpp_lex_step again with the chunk that was being lexed and the new output. You can also specify a max_tokens_out
wrote lexer function docs, expanded doc generator
2016-09-06 21:22:35 +00:00
which is limits how many new tokens will be added to the token array. Even if token_array_out still had more space
to hold tokens, if the max_tokens_out limit is hit, the lexer will stop and return LexResult_HitTokenLimit. If this
got 4cpp ready to ship
2016-09-07 03:39:19 +00:00
happens there is still space left in the token array, so you can resume simply by calling cpp_lex_step again with
the same chunk and the same output array. Also note that, unlike the chunks which must only be replaced when the
system says it needs a chunk. You may switch to or modify the output array in between calls as much as you like.<br><br>
wrote lexer function docs, expanded doc generator
2016-09-06 21:22:35 +00:00
The most basic use of this system is to get it all done in one big chunk and try to allocate a nearly "infinite" output
array so that it will not run out of memory. This way you can get the entire job done in one call and then just assert
got 4cpp ready to ship
2016-09-07 03:39:19 +00:00
to make sure it returns LexResult_Finished to you:<br><br>
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
<br><br><div style='font-family: "Courier New", Courier, monospace; text-align: left;margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #EFEFDF; padding: 0.25em;'>Cpp_Token_Array lex_file(char *file_name){<br>&nbsp;&nbsp;&nbsp;&nbsp;File_Data file = read_whole_file(file_name);<br>&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;char *temp = (char*)malloc(4096); // hopefully big enough<br>&nbsp;&nbsp;&nbsp;&nbsp;Cpp_Lex_Data lex_state = cpp_lex_data_init(temp); <br>&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;Cpp_Token_Array array = {0};<br>&nbsp;&nbsp;&nbsp;&nbsp;array.tokens = (Cpp_Token*)malloc(1 << 20); // hopefully big enough<br>&nbsp;&nbsp;&nbsp;&nbsp;array.max_count = (1 << 20)/sizeof(Cpp_Token);<br>&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;Cpp_Lex_Result result = <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;cpp_lex_step(&lex_state, file.data, file.size, file.size,<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&array, NO_OUT_LIMIT);<br>&nbsp;&nbsp;&nbsp;&nbsp;Assert(result == LexResult_Finished);<br>&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;free(temp);<br>&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;return(array);<br>}<br></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#Cpp_Lex_Data_doc'>Cpp_Lex_Data</a></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#Cpp_Lex_Result_doc'>Cpp_Lex_Result</a></div></div><hr><div id='cpp_lex_data_init_doc' style='margin-bottom: 1cm;'><h4>&sect;2.4.3: cpp_lex_data_init</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>Cpp_Lex_Data cpp_lex_data_init(<div style='margin-left: 4mm;'><br></div>)</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Return</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>A brand new lex state ready to begin lexing a file from the beginning.</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>Creates a new lex state in the form of a Cpp_Lex_Data struct and returns the struct.
got 4cpp ready to ship
2016-09-07 03:39:19 +00:00
The system needs a temporary buffer that is as long as the longest token. 4096 is usually
enough but the buffer is not checked, so to be 100% bullet proof it has to be the same length
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
as the file being lexed.<br><br></div></div><hr><div id='cpp_lex_data_temp_size_doc' style='margin-bottom: 1cm;'><h4>&sect;2.4.4: cpp_lex_data_temp_size</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>int32_t cpp_lex_data_temp_size(<div style='margin-left: 4mm;'>Cpp_Lex_Data *lex_data<br></div>)</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><div style='font-weight: 600;'>lex_data</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The lex state from which to get the temporary buffer size.</div></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>This call gets the current size of the temporary buffer in the lexer state so
that you can move to a new temporary buffer by copying the data over.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_lex_data_temp_read_doc'>cpp_lex_data_temp_read</a></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_lex_data_new_temp_doc'>cpp_lex_data_new_temp</a></div></div><hr><div id='cpp_lex_data_temp_read_doc' style='margin-bottom: 1cm;'><h4>&sect;2.4.5: cpp_lex_data_temp_read</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>void cpp_lex_data_temp_read(<div style='margin-left: 4mm;'>Cpp_Lex_Data *lex_data,<br>char *out_buffer<br></div>)</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><div style='font-weight: 600;'>lex_data</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The lex state from which to read the temporary buffer.</div></div></div><div><div style='font-weight: 600;'>out_buffer</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The buffer into which the contents of the temporary buffer will be written.
The size of the buffer must be at least the size as returned by cpp_lex_data_temp_size.</div></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>This call reads the current contents of the temporary buffer.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_lex_data_temp_size_doc'>cpp_lex_data_temp_size</a></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_lex_data_new_temp_doc'>cpp_lex_data_new_temp</a></div></div><hr><div id='cpp_lex_data_new_temp_DEP_doc' style='margin-bottom: 1cm;'><h4>&sect;2.4.6: cpp_lex_data_new_temp_DEP</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>void cpp_lex_data_new_temp_DEP(<div style='margin-left: 4mm;'>Cpp_Lex_Data *lex_data,<br>char *new_buffer<br></div>)</div></div><hr><div id='cpp_get_relex_range_doc' style='margin-bottom: 1cm;'><h4>&sect;2.4.7: cpp_get_relex_range</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>Cpp_Relex_Range cpp_get_relex_range(<div style='margin-left: 4mm;'>Cpp_Token_Array *array,<br>int32_t start_pos,<br>int32_t end_pos<br></div>)</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><div style='font-weight: 600;'>array</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>A pointer to the token array that will be modified by the relex,
documentation for relex system, and some cleanup on buffer rendering
2016-09-20 00:51:28 +00:00
this array should already contain the tokens for the previous state of the file.</div></div></div><div><div style='font-weight: 600;'>start_pos</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The start position of the edited region of the file.
The start and end points are based on the edited region of the file before the edit.</div></div></div><div><div style='font-weight: 600;'>end_pos</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The end position of the edited region of the file.
In particular, end_pos is the first character after the edited region not effected by the edit.
Thus if the edited region contained one character end_pos - start_pos should equal 1.
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
The start and end points are based on the edited region of the file before the edit.</div></div></div></div><hr><div id='cpp_relex_init_doc' style='margin-bottom: 1cm;'><h4>&sect;2.4.8: cpp_relex_init</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>Cpp_Relex_Data cpp_relex_init(<div style='margin-left: 4mm;'>Cpp_Token_Array *array,<br>int32_t start_pos,<br>int32_t end_pos,<br>int32_t character_shift_amount<br></div>)</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><div style='font-weight: 600;'>array</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>A pointer to the token array that will be modified by the relex,
documentation for relex system, and some cleanup on buffer rendering
2016-09-20 00:51:28 +00:00
this array should already contain the tokens for the previous state of the file.</div></div></div><div><div style='font-weight: 600;'>start_pos</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The start position of the edited region of the file.
The start and end points are based on the edited region of the file before the edit.</div></div></div><div><div style='font-weight: 600;'>end_pos</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The end position of the edited region of the file.
In particular, end_pos is the first character after the edited region not effected by the edit.
Thus if the edited region contained one character end_pos - start_pos should equal 1.
added clparams to app step on linux; got the lexer API working with a fixed with tb
2016-09-20 18:10:05 +00:00
The start and end points are based on the edited region of the file before the edit.</div></div></div><div><div style='font-weight: 600;'>character_shift_amount</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The shift in the characters after the edited region.</div></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Return</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>Returns a partially initialized relex state.</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>This call does the first setup step of initializing a relex state. To finish initializing the relex state
documentation for relex system, and some cleanup on buffer rendering
2016-09-20 00:51:28 +00:00
you must tell the state about the positioning of the first chunk it will be fed. There are two methods of doing
this, the direct method is with cpp_relex_declare_first_chunk_position, the method that is often more convenient
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
is with cpp_relex_is_start_chunk. If the file is not chunked the second step of initialization can be skipped.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_relex_declare_first_chunk_position_doc'>cpp_relex_declare_first_chunk_position</a></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_relex_is_start_chunk_doc'>cpp_relex_is_start_chunk</a></div></div><hr><div id='cpp_relex_start_position_doc' style='margin-bottom: 1cm;'><h4>&sect;2.4.9: cpp_relex_start_position</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>int32_t cpp_relex_start_position(<div style='margin-left: 4mm;'>Cpp_Relex_Data *S_ptr<br></div>)</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><div style='font-weight: 600;'>S_ptr</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Return</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>Returns the first position in the file the relexer wants to read. This is usually a position slightly
documentation for relex system, and some cleanup on buffer rendering
2016-09-20 00:51:28 +00:00
earlier than the start_pos provided as the edit range.</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>After doing the first stage of initialization this call is useful for figuring out what chunk
of the file to feed to the lexer first. It should be a chunk that contains the position returned
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
by this call.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_relex_init_doc'>cpp_relex_init</a></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_relex_declare_first_chunk_position_doc'>cpp_relex_declare_first_chunk_position</a></div></div><hr><div id='cpp_relex_declare_first_chunk_position_doc' style='margin-bottom: 1cm;'><h4>&sect;2.4.10: cpp_relex_declare_first_chunk_position</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>void cpp_relex_declare_first_chunk_position(<div style='margin-left: 4mm;'>Cpp_Relex_Data *S_ptr,<br>int32_t position<br></div>)</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><div style='font-weight: 600;'>S_ptr</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div style='font-weight: 600;'>position</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The start position of the first chunk that will be fed to the relex process.</div></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>To initialize the relex system completely, the system needs to know how the characters in the
documentation for relex system, and some cleanup on buffer rendering
2016-09-20 00:51:28 +00:00
first file line up with the file's absolute layout. This call declares where the first chunk's start
position is in the absolute file layout, and the system infers the alignment from that. For this method
to work the starting position of the relexing needs to be inside the first chunk. To get the relexers
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
starting position call cpp_relex_start_position.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_relex_init_doc'>cpp_relex_init</a></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_relex_start_position_doc'>cpp_relex_start_position</a></div></div><hr><div id='cpp_relex_is_start_chunk_doc' style='margin-bottom: 1cm;'><h4>&sect;2.4.11: cpp_relex_is_start_chunk</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>int32_t cpp_relex_is_start_chunk(<div style='margin-left: 4mm;'>Cpp_Relex_Data *S_ptr,<br>char *chunk,<br>int32_t chunk_size<br></div>)</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><div style='font-weight: 600;'>S_ptr</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div style='font-weight: 600;'>chunk</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The chunk to check.</div></div></div><div><div style='font-weight: 600;'>chunk_size</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The size of the chunk to check.</div></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Return</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>Returns non-zero if the passed in chunk should be used as the first chunk for lexing.</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>With this method, once a state is initialized, each chunk can be fed in one after the other in
documentation for relex system, and some cleanup on buffer rendering
2016-09-20 00:51:28 +00:00
the order they appear in the absolute file layout. When this call returns non-zero it means that
the chunk that was passed in on that call should be used in the first call to cpp_relex_step. If,
after trying all of the chunks, they all return zero, pass in NULL for chunk and 0 for chunk_size
to tell the system that all possible chunks have already been tried, and then use those values again
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
in the one and only call to cpp_relex_step.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_relex_init_doc'>cpp_relex_init</a></div></div><hr><div id='cpp_relex_step_doc' style='margin-bottom: 1cm;'><h4>&sect;2.4.12: cpp_relex_step</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>Cpp_Lex_Result cpp_relex_step(<div style='margin-left: 4mm;'>Cpp_Relex_Data *S_ptr,<br>char *chunk,<br>int32_t chunk_size,<br>int32_t full_size,<br>Cpp_Token_Array *array,<br>Cpp_Token_Array *relex_array<br></div>)</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><div style='font-weight: 600;'>S_ptr</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>A pointer to a fully initiazed relex state.</div></div></div><div><div style='font-weight: 600;'>chunk</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>A chunk of the edited file being relexed.</div></div></div><div><div style='font-weight: 600;'>chunk_size</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The size of the current chunk.</div></div></div><div><div style='font-weight: 600;'>full_size</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The full size of the edited file.</div></div></div><div><div style='font-weight: 600;'>array</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>A pointer to a token array that contained the original tokens before the edit.</div></div></div><div><div style='font-weight: 600;'>relex_array</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>A pointer to a token array for spare space. The capacity of the
documentation for relex system, and some cleanup on buffer rendering
2016-09-20 00:51:28 +00:00
relex_array determines how far the relex process can go. If it runs out, the process
can be continued if the same relex_array is extended without losing the tokens it contains.
To get an appropriate capacity for relex_array, you can get the range of tokens that the relex
operation is likely to traverse by looking at the result from cpp_get_relex_range.</div></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>When a file has already been lexed, and then it is edited in a small local way,
rather than lexing the new file all over again, cpp_relex_step can try to find just
the range of tokens that need to be updated and fix them in.<br><br>
First the lex state must be initialized (cpp_relex_init). Then one or more calls to
cpp_relex_step will start editing the array and filling out the relex_array. The return
value of cpp_relex_step indicates whether the relex was successful or was interrupted
and if it was interrupted, what the system needs to resume.<br><br>
LexResult_Finished indicates that the relex engine finished successfully.<br><br>
LexResult_NeedChunk indicates that the system needs the next chunk of the file.<br><br>
LexResult_NeedTokenMemory indicates that the relex_array has reached capacity, and that
it needs to be extended if it is going to continue. Sometimes in this case it is better
to stop and just lex the entire file normally, because there are a few cases where a small
local change effects a long range of the lexers output.<br><br>
The relex operation can be closed in one of two ways. If the LexResult_Finished
value has been returned by this call, then to complete the edits to the array make
sure the original array has enough capacity to store the final result by calling
cpp_relex_get_new_count. Then the operation can be finished successfully by calling
cpp_relex_complete.<br><br>
Whether or not the relex process finished with LexResult_Finished the process can be
finished by calling cpp_relex_abort, which puts the array back into it's original state.
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
No close is necessary if getting the original array state back is not necessary.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_relex_init_doc'>cpp_relex_init</a></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_get_relex_range_doc'>cpp_get_relex_range</a></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#Cpp_Lex_Result_doc'>Cpp_Lex_Result</a></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_relex_get_new_count_doc'>cpp_relex_get_new_count</a></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_relex_complete_doc'>cpp_relex_complete</a></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_relex_abort_doc'>cpp_relex_abort</a></div></div><hr><div id='cpp_relex_get_new_count_doc' style='margin-bottom: 1cm;'><h4>&sect;2.4.13: cpp_relex_get_new_count</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>int32_t cpp_relex_get_new_count(<div style='margin-left: 4mm;'>Cpp_Relex_Data *S_ptr,<br>int32_t current_count,<br>Cpp_Token_Array *relex_array<br></div>)</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><div style='font-weight: 600;'>S_ptr</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>A pointer to a state that has gone through cpp_relex_step with a LexResult_Finished return.</div></div></div><div><div style='font-weight: 600;'>current_count</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The count of tokens in the original array before the edit.</div></div></div><div><div style='font-weight: 600;'>relex_array</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The relex_array that was used in the cpp_relex_step call/calls.</div></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>After getting a LexResult_Finished from cpp_relex_step, this call can be used to get
documentation for relex system, and some cleanup on buffer rendering
2016-09-20 00:51:28 +00:00
the size the new array will have. If the original array doesn't have enough capacity to store
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
the new array, it's capacity should be increased before passing to cpp_relex_complete.<br><br></div></div><hr><div id='cpp_relex_complete_doc' style='margin-bottom: 1cm;'><h4>&sect;2.4.14: cpp_relex_complete</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>void cpp_relex_complete(<div style='margin-left: 4mm;'>Cpp_Relex_Data *S_ptr,<br>Cpp_Token_Array *array,<br>Cpp_Token_Array *relex_array<br></div>)</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><div style='font-weight: 600;'>S_ptr</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>A pointer to a state that has gone through cpp_relex_step with a LexResult_Finished return.</div></div></div><div><div style='font-weight: 600;'>array</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The original array being edited by cpp_relex_step calls.</div></div></div><div><div style='font-weight: 600;'>relex_array</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The relex_array that was filled by cpp_relex_step.</div></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>After getting a LexResult_Finished from cpp_relex_step, and ensuring that
documentation for relex system, and some cleanup on buffer rendering
2016-09-20 00:51:28 +00:00
array has a large enough capacity by calling cpp_relex_get_new_count, this call
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
does the necessary replacement of tokens in the array to make it match the new file.<br><br></div></div><hr><div id='cpp_relex_abort_doc' style='margin-bottom: 1cm;'><h4>&sect;2.4.15: cpp_relex_abort</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>void cpp_relex_abort(<div style='margin-left: 4mm;'>Cpp_Relex_Data *S_ptr,<br>Cpp_Token_Array *array<br></div>)</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><div style='font-weight: 600;'>S_ptr</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>A pointer to a state that has gone through at least one cpp_relex_step.</div></div></div><div><div style='font-weight: 600;'>array</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The original array that went through cpp_relex_step to be edited.</div></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>After the first call to cpp_relex_step, the array's contents may have been changed,
documentation for relex system, and some cleanup on buffer rendering
2016-09-20 00:51:28 +00:00
this call assures the array is in it's original state. After this call the relex state
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
is dead.<br><br></div></div><hr><div id='cpp_make_token_array_doc' style='margin-bottom: 1cm;'><h4>&sect;2.4.16: cpp_make_token_array</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>Cpp_Token_Array cpp_make_token_array(<div style='margin-left: 4mm;'>int32_t starting_max<br></div>)</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><div style='font-weight: 600;'>starting_max</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The number of tokens to initialize the array with.</div></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Return</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>An empty Cpp_Token_Array with memory malloc'd for storing tokens.</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>This call allocates a Cpp_Token_Array with malloc for use in other
wrote lexer function docs, expanded doc generator
2016-09-06 21:22:35 +00:00
convenience functions. Stacks that are not allocated this way should not be
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
used in the convenience functions.<br><br></div></div><hr><div id='cpp_free_token_array_doc' style='margin-bottom: 1cm;'><h4>&sect;2.4.17: cpp_free_token_array</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>void cpp_free_token_array(<div style='margin-left: 4mm;'>Cpp_Token_Array token_array<br></div>)</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><div style='font-weight: 600;'>token_array</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>An array previously allocated by cpp_make_token_array</div></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>This call frees a Cpp_Token_Array.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_make_token_array_doc'>cpp_make_token_array</a></div></div><hr><div id='cpp_resize_token_array_doc' style='margin-bottom: 1cm;'><h4>&sect;2.4.18: cpp_resize_token_array</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>void cpp_resize_token_array(<div style='margin-left: 4mm;'>Cpp_Token_Array *token_array,<br>int32_t new_max<br></div>)</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><div style='font-weight: 600;'>token_array</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>An array previously allocated by cpp_make_token_array.</div></div></div><div><div style='font-weight: 600;'>new_max</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The new maximum size the array should support. If this is not greater
wrote lexer function docs, expanded doc generator
2016-09-06 21:22:35 +00:00
than the current size of the array the operation is ignored.</div></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>This call allocates a new memory chunk and moves the existing tokens in the array
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
over to the new chunk.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_make_token_array_doc'>cpp_make_token_array</a></div></div><hr><div id='cpp_lex_file_doc' style='margin-bottom: 1cm;'><h4>&sect;2.4.19: cpp_lex_file</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>void cpp_lex_file(<div style='margin-left: 4mm;'>char *data,<br>int32_t size,<br>Cpp_Token_Array *token_array_out<br></div>)</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><div style='font-weight: 600;'>data</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The file data to be lexed in a single contiguous block.</div></div></div><div><div style='font-weight: 600;'>size</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The number of bytes in data.</div></div></div><div><div style='font-weight: 600;'>token_array_out</div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The token array where the output tokens will be pushed.
wrote lexer function docs, expanded doc generator
2016-09-06 21:22:35 +00:00
This token array must be previously allocated with cpp_make_token_array</div></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>Lexes an entire file and manages the interaction with the lexer system so that
it is quick and convenient to lex files.<br><br>
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
<br><br><div style='font-family: "Courier New", Courier, monospace; text-align: left;margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #EFEFDF; padding: 0.25em;'>Cpp_Token_Array lex_file(char *file_name){<br>&nbsp;&nbsp;&nbsp;&nbsp;File_Data file = read_whole_file(file_name);<br>&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;// This array will be automatically grown if it runs<br>&nbsp;&nbsp;&nbsp;&nbsp;// out of memory.<br>&nbsp;&nbsp;&nbsp;&nbsp;Cpp_Token_Array array = cpp_make_token_array(100);<br>&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;cpp_lex_file(file.data, file.size, &array);<br>&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;return(array);<br>}<br></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_make_token_array_doc'>cpp_make_token_array</a></div></div><hr><h3>&sect;2.5 Lexer Type Descriptions</h3><div id='Cpp_Token_Type_doc' style='margin-bottom: 1cm;'><h4>&sect;2.5.1: Cpp_Token_Type</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>enum Cpp_Token_Type;</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>A Cpp_Token_Type classifies a token to make parsing easier. Some types are not
lexer doc section setup
2016-09-05 15:44:19 +00:00
actually output by the lexer, but exist because parsers will also make use of token
new code manipulating experiments, fix to relex rule
2016-09-14 23:39:33 +00:00
types in their own output.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Values</i></b></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_JUNK</span> = 0</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_COMMENT</span> = 1</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_PP_INCLUDE</span> = 2</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_PP_DEFINE</span> = 3</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_PP_UNDEF</span> = 4</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_PP_IF</span> = 5</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_PP_IFDEF</span> = 6</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_PP_IFNDEF</span> = 7</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_PP_ELSE</span> = 8</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_PP_ELIF</span> = 9</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_PP_ENDIF</span> = 10</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_PP_ERROR</span> = 11</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_PP_IMPORT</span> = 12</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_PP_USING</span> = 13</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_PP_LINE</span> = 14</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospa
parsing to determine the full nature of the token.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_AMPERSAND</span> = 48</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>This is an 'ambiguous' token type because it requires
parsing to determine the full nature of the token.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_TILDE</span> = 49</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>This is an 'ambiguous' token type because it requires
parsing to determine the full nature of the token.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_PLUS</span> = 50</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>This is an 'ambiguous' token type because it requires
parsing to determine the full nature of the token.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_MINUS</span> = 51</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>This is an 'ambiguous' token type because it requires
parsing to determine the full nature of the token.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_INCREMENT</span> = 52</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>This is an 'ambiguous' token type because it requires
parsing to determine the full nature of the token.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_DECREMENT</span> = 53</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>This is an 'ambiguous' token type because it requires
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
parsing to determine the full nature of the token.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_SCOPE</span> = 54</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_POSTINC</span> = 55</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>This type is for parser use, it is not output by the lexer.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_POSTDEC</span> = 56</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>This type is for parser use, it is not output by the lexer.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_FUNC_STYLE_CAST</span> = 57</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>This type is for parser use, it is not output by the lexer.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_CPP_STYLE_CAST</span> = 58</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_CALL</span> = 59</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>This type is for parser use, it is not output by the lexer.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_INDEX</span> = 60</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>This type is for parser use, it is not output by the lexer.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_DOT</span> = 61</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_ARROW</span> = 62</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_PREINC</span> = 63</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>This token is for parser use, it is not output by the lexer.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_PREDEC</span> = 64</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>This token is for parser use, it is not output by the lexer.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_POSITIVE</span> = 65</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>This token is for parser use, it is not output by the lexer.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TOKEN_NEGAITVE</span> = 66</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>This token is for parser use, it is not output
got 4cpp ready to ship
2016-09-07 03:39:19 +00:00
It is the primary output of the lexing system.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Fields</i></b></div><div><div style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>type</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The type field indicates the type of the token.
All tokens have a type no matter the circumstances.<br><br></div></div></div><div><div style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>start</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The start field indicates the index of the first character
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
of this token's lexeme.<br><br></div></div></div><div><div style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>size</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The size field indicates the number of bytes in this token's lexeme.<br><br></div></div></div><div><div style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>state_flags</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The state_flags should not be used outside of the lexer's implementation.<br><br></div></div></div><div><div style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>flags</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The flags field contains extra useful information about the token.<br><br></div></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#Cpp_Token_Flag_doc'>Cpp_Token_Flag</a></div></div><hr><div id='Cpp_Token_Flag_doc' style='margin-bottom: 1cm;'><h4>&sect;2.5.3: Cpp_Token_Flag</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>enum Cpp_Token_Flag;</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>The Cpp_Token_Flags are used to mark up tokens with additional information.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Values</i></b></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TFLAG_PP_DIRECTIVE</span> = 0x1</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>Indicates that the token is a preprocessor directive.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TFLAG_PP_BODY</span> = 0x2</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>Indicates that the token is on the line of a preprocessor directive.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TFLAG_MULTILINE</span> = 0x4</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>Indicates that the token spans across multiple lines. This can show up
on line comments and string literals with back slash line continuation.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TFLAG_IS_OPERATOR</span> = 0x8</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>Indicates that the token is some kind of operator or punctuation like braces.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>CPP_TFLAG_IS_KEYWORD</span> = 0x10</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>Indicates that the token is a keyword.<br><br></div></div></div></div><hr><div id='Cpp_Token_Array_doc' style='margin-bottom: 1cm;'><h4>&sect;2.5.4: Cpp_Token_Array</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>struct Cpp_Token_Array {<br><div style='margin-left: 8mm;'>Cpp_Token * tokens;<br>int32_t count;<br>int32_t max_count;<br></div>};<br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>Cpp_Token_Array is used to bundle together the common elements
got 4cpp ready to ship
2016-09-07 03:39:19 +00:00
of a growing array of Cpp_Tokens. To initialize it the tokens field should
point to a block of memory with a size equal to max_count*sizeof(Cpp_Token)
and the count should be initialized to zero.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Fields</i></b></div><div><div style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>tokens</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The tokens field points to the memory used to store the array of tokens.<br><br></div></div></div><div><div style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>count</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The count field counts how many tokens in the array are currently used.<br><br></div></div></div><div><div style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>max_count</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The max_count field specifies the maximum size the count field may grow to before
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
the tokens array is out of space.<br><br></div></div></div></div><hr><div id='Cpp_Get_Token_Result_doc' style='margin-bottom: 1cm;'><h4>&sect;2.5.5: Cpp_Get_Token_Result</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>struct Cpp_Get_Token_Result {<br><div style='margin-left: 8mm;'>int32_t token_index;<br>int32_t in_whitespace;<br></div>};<br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>Cpp_Get_Token_Result is the return result of the cpp_get_token call.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Fields</i></b></div><div><div style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>token_index</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The token_index field indicates which token answers the query. To get the token from
got 4cpp ready to ship
2016-09-07 03:39:19 +00:00
the source array <br><br><div style='font-family: "Courier New", Courier, monospace; text-align: left;margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #EFEFDF; padding: 0.25em;'>array.tokens[result.token_index]<br></div></div></div></div><div><div style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>in_whitespace</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The in_whitespace field is true when the query position was actually in whitespace
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
after the result token.<br><br></div></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_get_token_doc'>cpp_get_token</a></div></div><hr><div id='Cpp_Relex_Range_doc' style='margin-bottom: 1cm;'><h4>&sect;2.5.6: Cpp_Relex_Range</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>struct Cpp_Relex_Range {<br><div style='margin-left: 8mm;'>int32_t start_token_index;<br>int32_t end_token_index;<br></div>};<br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>Cpp_Relex_Range is the return result of the cpp_get_relex_range call.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Fields</i></b></div><div><div style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>start_token_index</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The index of the first token in the unedited array that needs to be relexed.<br><br></div></div></div><div><div style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>end_token_index</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The index of the first token in the unedited array after the edited range
documentation for relex system, and some cleanup on buffer rendering
2016-09-20 00:51:28 +00:00
that may not need to be relexed. Sometimes a relex operation has to lex past this
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
position to find a token that is not effected by the edit.<br><br></div></div></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_get_relex_range_doc'>cpp_get_relex_range</a></div></div><hr><div id='Cpp_Lex_Data_doc' style='margin-bottom: 1cm;'><h4>&sect;2.5.7: Cpp_Lex_Data</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>struct Cpp_Lex_Data { /* non-public internals */ } ;</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>Cpp_Lex_Data represents the state of the lexer so that the system may be resumable
got 4cpp ready to ship
2016-09-07 03:39:19 +00:00
and the user can manage the lexer state and decide when to resume lexing with it. To create
documentation for relex system, and some cleanup on buffer rendering
2016-09-20 00:51:28 +00:00
a new lexer state call cpp_lex_data_init.<br><br>
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
The internals of the lex state should not be treated as a part of the public API.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_lex_data_init_doc'>cpp_lex_data_init</a></div></div><hr><div id='Cpp_Lex_Result_doc' style='margin-bottom: 1cm;'><h4>&sect;2.5.8: Cpp_Lex_Result</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>enum Cpp_Lex_Result;</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>Cpp_Lex_Result is returned from the lexing engine to indicate why it stopped lexing.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Values</i></b></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>LexResult_Finished</span> = 0</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>This indicates that the system got to the end of the file and will not accept more input.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>LexResult_NeedChunk</span> = 1</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>This indicates that the system got to the end of an input chunk and is ready to receive the
new code manipulating experiments, fix to relex rule
2016-09-14 23:39:33 +00:00
next input chunk.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>LexResult_NeedTokenMemory</span> = 2</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>This indicates that the output array ran out of space to store tokens and needs to be
compiling in windows; building lexer library docs
2016-09-21 13:07:08 +00:00
replaced or expanded before continuing.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>LexResult_HitTokenLimit</span> = 3</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>This indicates that the maximum number of output tokens as specified by the user was hit.<br><br></div></div></div></div><hr><div id='Cpp_Relex_Data_doc' style='margin-bottom: 1cm;'><h4>&sect;2.5.9: Cpp_Relex_Data</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>struct Cpp_Relex_Data { /* non-public internals */ } ;</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>Cpp_Relex_Data represents the state of the relexer so that the system may be resumable.
documentation for relex system, and some cleanup on buffer rendering
2016-09-20 00:51:28 +00:00
To create a new relex state call cpp_relex_init.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'><a href='#cpp_relex_init_doc'>cpp_relex_init</a></div></div><hr></div></body></html>