<htmllang="en-US"><head><title>4coder API Docs</title><style>body{background:#FAFAFA;color:#0D0D0D;}h1,h2,h3,h4{color:#309030;margin:0;}h2{margin-top:6mm;}h3{margin-top:5mm;margin-bottom:5mm;}h4{font-size:1.1em;}a{color:#309030;text-decoration:none;}a:visited{color:#A0C050;}a:hover{background:#E0FFD0;}ul{list-style:none;padding:0;margin:0;}</style></head>
<h2id='section_introduction'>§1 Introduction</h2><div><p>This is the documentation for the 4cpp lexer version 1.0. The documentation is the newest piece of this lexer project so it may still have problems. What is here should be correct and mostly complete.</p><p>If you have questions or discover errors please contact <spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'>editor@4coder.net</span> or to get help from community members you can post on the 4coder forums hosted on handmade.network at <spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'>4coder.handmade.network</span></p></div>
<h2id='section_lexer_library'>§2 Lexer Library</h2><h3>§2.1 Lexer Intro</h3><div>The 4cpp lexer system provides a polished, fast, flexible system that takes in C/C++ and outputs a tokenization of the text data. There are two API levels. One level is setup to let you easily get a tokenization of the file. This level manages memory for you with malloc to make it as fast as possible to start getting your tokens. The second level enables deep integration by allowing control over allocation, data chunking, and output rate control.<br><br>To use the quick setup API you simply include 4cpp_lexer.h and read the documentation at <ahref='#cpp_lex_file_doc'>cpp_lex_file</a>.<br><br>To use the the fancier API include 4cpp_lexer.h and read the documentation at <ahref='#cpp_lex_step_doc'>cpp_lex_step</a>. If you want to be absolutely sure you are not including malloc into your program you can define FCPP_FORBID_MALLOC before the include and the "step" API will continue to work.<br><br>There are a few more features in 4cpp that are not documented yet. You are free to try to use these, but I am not totally sure they are ready yet, and when they are they will be documented.</div><h3>§2.2 Lexer Function List</h3><ul><li><ahref='#cpp_get_token_doc'>cpp_get_token</a></li><li><ahref='#cpp_lex_step_doc'>cpp_lex_step</a></li><li><ahref='#cpp_lex_data_init_doc'>cpp_lex_data_init</a></li><li><ahref='#cpp_lex_data_temp_size_doc'>cpp_lex_data_temp_size</a></li><li><ahref='#cpp_lex_data_temp_read_doc'>cpp_lex_data_temp_read</a></li><li><ahref='#cpp_lex_data_new_temp_doc'>cpp_lex_data_new_temp</a></li><li><ahref='#cpp_make_token_array_doc'>cpp_make_token_array</a></li><li><ahref='#cpp_free_token_array_doc'>cpp_free_token_array</a></li><li><ahref='#cpp_resize_token_array_doc'>cpp_resize_token_array</a></li><li><ahref='#cpp_lex_file_doc'>cpp_lex_file</a></li></ul><h3>§2.3 Lexer Types List</h3><ul><li><ahref='#Cpp_Token_Type_doc'>Cpp_Token_Type</a></li><li><ahref='#Cpp_Token_doc'>Cpp_Token</a></li><li><ahref='#Cpp_Token_Flag_doc'>Cpp_Token_Flag</a></li><li><ahref='#Cpp_Token_Array_doc'>Cpp_Token_Array</a></li><li><ahref='#Cpp_Get_Token_Result_doc'>Cpp_Get_Token_Result</a></li><li><ahref='#Cpp_Lex_Data_doc'>Cpp_Lex_Data</a></li><li><ahref='#Cpp_Lex_Result_doc'>Cpp_Lex_Result</a></li></ul><h3>§2.4 Lexer Function Descriptions</h3><divid='cpp_get_token_doc'style='margin-bottom: 1cm;'><h4>§2.4.1: cpp_get_token</h4><divstyle='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>Cpp_Get_Token_Result cpp_get_token(<divstyle='margin-left: 4mm;'>Cpp_Token_Array *token_array_in,<br>int32_t pos<br></div>)</div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><divstyle='font-weight: 600;'>token_array</div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The array of tokens from which to get a token.</div></div></div><div><divstyle='font-weight: 600;'>pos</div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The position, measured in bytes, to get the token for.</div></div></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Return</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'>A Cpp_Get_Token_Result struct is returned containing the index
of a token and a flag indicating whether the pos is contained in the token
or in whitespace after the token.</div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'>This call performs a binary search over all of the tokens looking
for the token that contains the specified position. If the position
is in whitespace between the tokens, the returned token index is the
index of the token immediately before the provided position. The returned
index can be -1 if the position is before the first token.<br><br></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'><ahref='#Cpp_Get_Token_Result_doc'>Cpp_Get_Token_Result</a></div></div><hr><divid='cpp_lex_step_doc'style='margin-bottom: 1cm;'><h4>§2.4.2: cpp_lex_step</h4><divstyle='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>Cpp_Lex_Result cpp_lex_step(<divstyle='margin-left: 4mm;'>Cpp_Lex_Data *S_ptr,<br>char *chunk,<br>int32_t size,<br>int32_t full_size,<br>Cpp_Token_Array *token_array_out,<br>int32_t max_tokens_out<br></div>)</div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><divstyle='font-weight: 600;'>S_ptr</div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The lexer state. Go to the Cpp_Lex_Data section to see how to initialize the state.</div></div></div><div><divstyle='font-weight: 600;'>chunk</div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The first or next chunk of the file being lexed.</div></div></div><div><divstyle='font-weight: 600;'>size</div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The number of bytes in the chunk including the null terminator if the chunk ends in a null terminator.
If the chunk ends in a null terminator the system will interpret it as the end of the file.</div></div></div><div><divstyle='font-weight: 600;'>full_size</div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>If the final chunk is not null terminated this parameter should specify the length of the
file in bytes. To rely on an eventual null terminator use HAS_NULL_TERM for this parameter.</div></div></div><div><divstyle='font-weight: 600;'>token_array_out</div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The token array structure that will receive the tokens output by the lexer.</div></div></div><div><divstyle='font-weight: 600;'>max_tokens_out</div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The maximum number of tokens to be output to the token array. To rely on the
max built into the token array pass NO_OUT_LIMIT here.</div></div></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'>This call is the primary interface of the lexing system. It is quite general so it can be used in
a lot of different ways. I will explain the general rules first, and then give some examples of common
ways it might be used.<br><br>
First a lexing state, Cpp_Lex_Data, must be initialized. The file to lex must be read into N contiguous chunks
to make sure it returns LexResult_Finished to you:<br><br>
<br><br><divstyle='font-family: "Courier New", Courier, monospace; text-align: left;margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #EFEFDF; padding: 0.25em;'>Cpp_Token_Array lex_file(char *file_name){<br> File_Data file = read_whole_file(file_name);<br> <br> char *temp = (char*)malloc(4096); // hopefully big enough<br> Cpp_Lex_Data lex_state = cpp_lex_data_init(temp); <br> <br> Cpp_Token_Array array = {0};<br> array.tokens = (Cpp_Token*)malloc(1 <<20);//hopefullybigenough<br> array.max_count = (1 <<20)/sizeof(Cpp_Token);<br> <br> Cpp_Lex_Result result = <br> cpp_lex_step(&lex_state, file.data, file.size, file.size,<br> &array, NO_OUT_LIMIT);<br> Assert(result == LexResult_Finished);<br> <br> free(temp);<br> <br> return(array);<br>}<br></div></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'><ahref='#Cpp_Lex_Data_doc'>Cpp_Lex_Data</a></div></div><hr><divid='cpp_lex_data_init_doc'style='margin-bottom: 1cm;'><h4>§2.4.3: cpp_lex_data_init</h4><divstyle='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>Cpp_Lex_Data cpp_lex_data_init(<divstyle='margin-left: 4mm;'>char *mem_buffer<br></div>)</div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><divstyle='font-weight: 600;'>tb</div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The memory to use for initializing the lex state's temp memory buffer.</div></div></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Return</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'>A brand new lex state ready to begin lexing a file from the beginning.</div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'>Creates a new lex state in the form of a Cpp_Lex_Data struct and returns the struct.
The system needs a temporary buffer that is as long as the longest token. 4096 is usually
enough but the buffer is not checked, so to be 100% bullet proof it has to be the same length
as the file being lexed.<br><br></div></div><hr><divid='cpp_lex_data_temp_size_doc'style='margin-bottom: 1cm;'><h4>§2.4.4: cpp_lex_data_temp_size</h4><divstyle='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>int32_t cpp_lex_data_temp_size(<divstyle='margin-left: 4mm;'>Cpp_Lex_Data *lex_data<br></div>)</div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><divstyle='font-weight: 600;'>lex_data</div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The lex state from which to get the temporary buffer size.</div></div></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'>This call gets the current size of the temporary buffer in the lexer state so
that you can move to a new temporary buffer by copying the data over.<br><br></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'><ahref='#cpp_lex_data_temp_read_doc'>cpp_lex_data_temp_read</a></div><divstyle='margin-left: 5mm; margin-right: 5mm;'><ahref='#cpp_lex_data_new_temp_doc'>cpp_lex_data_new_temp</a></div></div><hr><divid='cpp_lex_data_temp_read_doc'style='margin-bottom: 1cm;'><h4>§2.4.5: cpp_lex_data_temp_read</h4><divstyle='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>void cpp_lex_data_temp_read(<divstyle='margin-left: 4mm;'>Cpp_Lex_Data *lex_data,<br>char *out_buffer<br></div>)</div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><divstyle='font-weight: 600;'>lex_data</div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The lex state from which to read the temporary buffer.</div></div></div><div><divstyle='font-weight: 600;'>out_buffer</div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The buffer into which the contents of the temporary buffer will be written.
The size of the buffer must be at least the size as returned by cpp_lex_data_temp_size.</div></div></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'>This call reads the current contents of the temporary buffer.<br><br></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'><ahref='#cpp_lex_data_temp_size_doc'>cpp_lex_data_temp_size</a></div><divstyle='margin-left: 5mm; margin-right: 5mm;'><ahref='#cpp_lex_data_new_temp_doc'>cpp_lex_data_new_temp</a></div></div><hr><divid='cpp_lex_data_new_temp_doc'style='margin-bottom: 1cm;'><h4>§2.4.6: cpp_lex_data_new_temp</h4><divstyle='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>void cpp_lex_data_new_temp(<divstyle='margin-left: 4mm;'>Cpp_Lex_Data *lex_data,<br>char *new_buffer<br></div>)</div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><divstyle='font-weight: 600;'>lex_data</div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The lex state that will receive the new temporary buffer.</div></div></div><div><divstyle='font-weight: 600;'>new_buffer</div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The new temporary buffer that has the same contents as the old temporary buffer.</div></div></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'>This call can be used to set a new temporary buffer for the lex state. In cases where you want to
discontinue lexing, store the state, and resume later. In such a situation it may be necessary for you
to free the temp buffer that was originally used to make the lex state. This call allows you to supply
a new temp buffer when you are ready to resume lexing.<br><br>
However the new buffer needs to have the same contents the old buffer had. To ensure this you have to
use cpp_lex_data_temp_size and cpp_lex_data_temp_read to get the relevant contents of the temp buffer
before you free it.<br><br></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'><ahref='#cpp_lex_data_temp_size_doc'>cpp_lex_data_temp_size</a></div><divstyle='margin-left: 5mm; margin-right: 5mm;'><ahref='#cpp_lex_data_temp_read_doc'>cpp_lex_data_temp_read</a></div></div><hr><divid='cpp_make_token_array_doc'style='margin-bottom: 1cm;'><h4>§2.4.7: cpp_make_token_array</h4><divstyle='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>Cpp_Token_Array cpp_make_token_array(<divstyle='margin-left: 4mm;'>int32_t starting_max<br></div>)</div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><divstyle='font-weight: 600;'>starting_max</div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The number of tokens to initialize the array with.</div></div></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Return</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'>An empty Cpp_Token_Array with memory malloc'd for storing tokens.</div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'>This call allocates a Cpp_Token_Array with malloc for use in other
than the current size of the array the operation is ignored.</div></div></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'>This call allocates a new memory chunk and moves the existing tokens in the array
over to the new chunk.<br><br></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'><ahref='#cpp_make_token_array_doc'>cpp_make_token_array</a></div></div><hr><divid='cpp_lex_file_doc'style='margin-bottom: 1cm;'><h4>§2.4.10: cpp_lex_file</h4><divstyle='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>void cpp_lex_file(<divstyle='margin-left: 4mm;'>char *data,<br>int32_t size,<br>Cpp_Token_Array *token_array_out<br></div>)</div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Parameters</i></b></div><div><divstyle='font-weight: 600;'>data</div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The file data to be lexed in a single contiguous block.</div></div></div><div><divstyle='font-weight: 600;'>size</div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The number of bytes in data.</div></div></div><div><divstyle='font-weight: 600;'>token_array_out</div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The token array where the output tokens will be pushed.
This token array must be previously allocated with cpp_make_token_array</div></div></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'>Lexes an entire file and manages the interaction with the lexer system so that
parsing to determine the full nature of the token.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TOKEN_AMPERSAND</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>This is an 'ambiguous' token type because it requires
parsing to determine the full nature of the token.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TOKEN_TILDE</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>This is an 'ambiguous' token type because it requires
parsing to determine the full nature of the token.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TOKEN_PLUS</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>This is an 'ambiguous' token type because it requires
parsing to determine the full nature of the token.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TOKEN_MINUS</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>This is an 'ambiguous' token type because it requires
parsing to determine the full nature of the token.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TOKEN_INCREMENT</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>This is an 'ambiguous' token type because it requires
parsing to determine the full nature of the token.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TOKEN_DECREMENT</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>This is an 'ambiguous' token type because it requires
parsing to determine the full nature of the token.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TOKEN_SCOPE</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TOKEN_POSTINC</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>This type is for parser use, it is not output by the lexer.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TOKEN_POSTDEC</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>This type is for parser use, it is not output by the lexer.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TOKEN_FUNC_STYLE_CAST</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>This type is for parser use, it is not output by the lexer.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TOKEN_CPP_STYLE_CAST</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TOKEN_CALL</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>This type is for parser use, it is not output by the lexer.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TOKEN_INDEX</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>This type is for parser use, it is not output by the lexer.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TOKEN_DOT</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TOKEN_ARROW</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TOKEN_PREINC</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>This token is for parser use, it is not output by the lexer.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TOKEN_PREDEC</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>This token is for parser use, it is not output by the lexer.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TOKEN_POSITIVE</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>This token is for parser use, it is not output by the lexer.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TOKEN_NEGAITVE</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>This token is for parser use, it is not output by the lexer.<br><br></div></div></div><div><div><spanstyle='fon
It is the primary output of the lexing system.<br><br></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Fields</i></b></div><div><divstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>type</span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The type field indicates the type of the token.
All tokens have a type no matter the circumstances.<br><br></div></div></div><div><divstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>start</span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The start field indicates the index of the first character
of this token's lexeme.<br><br></div></div></div><div><divstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>size</span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The size field indicates the number of bytes in this token's lexeme.<br><br></div></div></div><div><divstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>state_flags</span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The state_flags should not be used outside of the lexer's implementation.<br><br></div></div></div><div><divstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>flags</span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The flags field contains extra useful information about the token.<br><br></div></div></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'><ahref='#Cpp_Token_Flag_doc'>Cpp_Token_Flag</a></div></div><hr><divid='Cpp_Token_Flag_doc'style='margin-bottom: 1cm;'><h4>§2.5.3: Cpp_Token_Flag</h4><divstyle='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>enum Cpp_Token_Flag;</div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'>The Cpp_Token_Flags are used to mark up tokens with additional information.<br><br></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Values</i></b></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TFLAG_PP_DIRECTIVE</span> = 0x1</span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>Indicates that the token is a preprocessor directive.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TFLAG_PP_BODY</span> = 0x2</span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>Indicates that the token is on the line of a preprocessor directive.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TFLAG_MULTILINE</span> = 0x4</span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>Indicates that the token spans across multiple lines. This can show up
on line comments and string literals with back slash line continuation.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TFLAG_IS_OPERATOR</span> = 0x8</span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>Indicates that the token is some kind of operator or punctuation like braces.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>CPP_TFLAG_IS_KEYWORD</span> = 0x10</span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>Indicates that the token is a keyword.<br><br></div></div></div></div><hr><divid='Cpp_Token_Array_doc'style='margin-bottom: 1cm;'><h4>§2.5.4: Cpp_Token_Array</h4><divstyle='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>struct Cpp_Token_Array {<br><divstyle='margin-left: 8mm;'>Cpp_Token * tokens;<br>int32_t count;<br>int32_t max_count;<br></div>};<br></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'>Cpp_Token_Array is used to bundle together the common elements
of a growing array of Cpp_Tokens. To initialize it the tokens field should
point to a block of memory with a size equal to max_count*sizeof(Cpp_Token)
and the count should be initialized to zero.<br><br></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Fields</i></b></div><div><divstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>tokens</span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The tokens field points to the memory used to store the array of tokens.<br><br></div></div></div><div><divstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>count</span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The count field counts how many tokens in the array are currently used.<br><br></div></div></div><div><divstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>max_count</span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The max_count field specifies the maximum size the count field may grow to before
the tokens array is out of space.<br><br></div></div></div></div><hr><divid='Cpp_Get_Token_Result_doc'style='margin-bottom: 1cm;'><h4>§2.5.5: Cpp_Get_Token_Result</h4><divstyle='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>struct Cpp_Get_Token_Result {<br><divstyle='margin-left: 8mm;'>int32_t token_index;<br>int32_t in_whitespace;<br></div>};<br></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'>Cpp_Get_Token_Result is the return result of the cpp_get_token call.<br><br></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Fields</i></b></div><div><divstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>token_index</span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The token_index field indicates which token answers the query. To get the token from
the source array <br><br><divstyle='font-family: "Courier New", Courier, monospace; text-align: left;margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #EFEFDF; padding: 0.25em;'>array.tokens[result.token_index]<br></div></div></div></div><div><divstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>in_whitespace</span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>The in_whitespace field is true when the query position was actually in whitespace
after the result token.<br><br></div></div></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'><ahref='#cpp_get_token_doc'>cpp_get_token</a></div></div><hr><divid='Cpp_Lex_Data_doc'style='margin-bottom: 1cm;'><h4>§2.5.6: Cpp_Lex_Data</h4><divstyle='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>struct Cpp_Lex_Data { /* non-public internals */ } ;</div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'>Cpp_Lex_Data represents the state of the lexer so that the system may be resumable
and the user can manage the lexer state and decide when to resume lexing with it. To create
a new lexer state that has not begun doing any lexing work call cpp_lex_data_init.<br><br>
The internals of the lex state should not be treated as a part of the public API.<br><br></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>See Also</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'><ahref='#cpp_lex_data_init_doc'>cpp_lex_data_init</a></div></div><hr><divid='Cpp_Lex_Result_doc'style='margin-bottom: 1cm;'><h4>§2.5.7: Cpp_Lex_Result</h4><divstyle='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>enum Cpp_Lex_Result;</div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><divstyle='margin-left: 5mm; margin-right: 5mm;'>Cpp_Lex_Result is returned from the lexing engine to indicate why it stopped lexing.<br><br></div><divstyle='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Values</i></b></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>LexResult_Finished</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>This indicates that the system got to the end of the file and will not accept more input.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>LexResult_NeedChunk</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>This indicates that the system got to the end of an input chunk and is ready to receive the
next input chunk.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>LexResult_NeedTokenMemory</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>This indicates that the output array ran out of space to store tokens and needs to be
replaced or expanded before continuing.<br><br></div></div></div><div><div><spanstyle='font-family: "Courier New", Courier, monospace; text-align: left;'><spanstyle='font-weight: 600;'>LexResult_HitTokenLimit</span></span></div><divstyle='margin-bottom: 6mm;'><divstyle='margin-left: 5mm; margin-right: 5mm;'>This indicates that the maximum number of output tokens as specified by the user was hit.<br><br></div></div></div></div><hr></div></body></html>
