(32/33) 580 - Fix #2732 - syntax highlighting broken if non-visible lines change

This patch makes `LineManager` set the `firstInvalidLineContext` upon insertion or removal of buffer content, instead of just moving it around. This results in the necessary reparsing of changed non-visible (and possibly subsequent) lines in `JEditBuffer.markTokens` which fixes [#2732]

Since the involved code is somewhat complex, here is a lengthy discussion:

### The error

The error can be reproduced with the wrap modes "none" and "hard" and any edit mode that contains multi-line SPANs . When a SPAN is started above the visible lines the syntax highlighting will not be properly updated until the start of the SPAN becomes visible.

(e.g. with the BeanShell Console and python edit mode one could insert / remove a multi-line string start: `buffer.insert(0, "'''")` / `buffer.remove(0,3)`)

### What's going on

After the insertion of the comment characters the visible lines are updated and repainted. This is (in part) being handled by `org.gjt.sp.jedit.textarea.ChunkCache` which calls `org.gjt.sp.jedit.buffer.JEditBuffer.markTokens`.
(This can be observed with the help of `TOKEN_MARKER_DEBUG` and `CHUNK_CACHE_DEBUG` in `org.gjt.sp.jedit.Debug`)

`JEditBuffer` uses a (private) `LineManager` instance to store line contexts (e.g. whether a line is inside a comment SPAN). `LineManager` already has a `firstInvalidLineContext` that is being used only by `JEditBuffer.markTokens` (if the edit mode is context sensitive): `markTokens` reparses all lines from the `firstInvalidLineContext` up to the requested `lineIndex` (if applicable).

Currently `firstInvalidLineContext` is set (only):

* by `JEditBuffer.markTokens` to the next unmarked line (if the line context has actually changed)
* to 0 via `JEditBuffer.loadText` which is being called upon instantiation and (re-)loading
* to 0 by `JEditBuffer.setTokenMarker`

`LineManager` itself updates `firstInvalidLineContext` in its `contentInserted` / `contentRemoved` methods (which are being called by their counterparts in `JEditBuffer`) so that it still points to the same line.

### What's wrong

Whenever content is inserted or removed from a buffer it warrants an update of the syntax highlighting of the affected lines (as anything can start a SPAN, depending on the edit mode). Therefore the `firstInvalidLineContext` should be re-set to the first affected line in `LineManager.contentInserted` / `.contentRemoved` (just like the `firstInvalidFoldLevel`)

### Why doesn't this happen all the time

Usually changes occur within the visible lines, which are (almost) unconditionally reparsed and redrawn so this error does not affect usual editing.

In soft wrap mode the `ChunkCache.getLineSubregionCount` (which gets called to determine the number of soft-wrapped lines of a physical line) causes a reparse of the changed lines and thus sets the `firstInvalidLineContext` if necessary. The error could also be "fixed" by removing the leading `if(!textArea.softWrap) return 1;` (line 279-280) which is probably not a good idea since this would obscure what's actually going on.

### History

I traced the code in question back to commit [r4549] where it is introduced as "various minor optimizations". (At this time, 2003, in `org.gjt.sp.jedit.buffer.OffsetManager` as `lastValidLineContext`.) I tried to find any reasoning for the code and could neither find it nor come up with anything myself. As far as I understand it this error has always been present.

Comments