Note this does not solve all issues because we are tokenizing contents of strings of which we do not know they contains SQL
when the string is actually not SQL and contains $ the tokenizer gets confused.
Extra lines before and after query are removed. Whitespace at end of line
is removed. SQL comments are converted to cpp style comments and are outside
the string literal.
To achieve this the function now uses the SQLLexer to know what is comment.
This also required the additional capability in the lexer to also return whitespace and newline tokens.
Also a few bugs in the lexer were fixed.
