Previous Next Tracker index See it online !

(191/212) 492 - Page large files in/out of memory

It is a real problem trying to edit a very large file with jedit. Files of 1 GB or more are almost impossible to edit, since the file needs at least 2 times the size of the file in maximum Java heap memory to edit. Even a file between 500 MB to 1 GB is difficult to edit as the end-user must increase the maximum Java heap memory to at least about 1200 MB to edit it. Files of 2 GB or more are impossible to edit because of the limitations of 32-bit heap memory allocation.

Jedit should be able to edit a file of any size, even if the size is greater than that of virtual memory. If a file is too large for the maximum amount of Java heap memory allocated Jedit should be able to edit the file by paging in whatever portion of the file is necessary to be viewed within any given Jedit buffer showing a portion of the file. The inability of jedit to edit any file size means that a programmer must use another editor to open files too large for Jedit to handle. This should be fixed in Jedit so that any size file can be edited by the editor. Even if paging in portions of a very large file slows down moving through the file a user of Jedit still should to edit a file of any allowable operating system size, and the end-user will accept slowdowns for very large files as long as editing the file does work correctly.

Submitted	eldienerlee - 2014-12-29 01:23:43.465000	Assigned
Priority	5	Labels
Status	open	Group	v5.2
Resolution	None

Comments

2014-12-29 04:54:46.305000 daleanson	I agree. I'd looked into the paging idea a few years ago, and it seemed difficult to implement. It would be worth a revisit, I think.
2014-12-30 04:38:53.319000 eldienerlee	I recently had to use another editor on very large files because JEdit could not edit them. I would prefer to use JEdit only. If you could revisit this issue it would be worthwhile to me while I admit I do not know if there are other users who might need to edit very large files with JEdit. Just the fact, however, that a user might have to readjust the maximum Java heap memory to edit large files seems to me a deficiency of JEdit. Whereas if JEdit were able to edit any size file, even within a paging system which would slow down editing, I believe that would be beneficial in principle to all end-users.
2014-12-30 16:31:05.254000 ezust	Up to now, I've also been using different editor when I need to edit really huge files. Adding this to an editor that was not designed for paging already is really hard. If I were you, I'd offer a bounty to someone to do it. And in that spirit, I will personally offer $100 for this particular feature. If others offer an additional bounty, then perhaps that will incentivize someone to actually do it.
2014-12-30 16:34:32.353000 ezust	Ticket moved from /p/jedit/bugs/3899/
2014-12-30 16:34:52.201000 ezust	- summary: Editing very large files is impossible with JEdit --> Page large files in/out of memory - Group: minor bug --> v5.2
2015-01-03 03:17:15.663000 daleanson	A big problem with implementing this now is the plugins that depend on the buffer being fully loaded, like Sidekick, PMD, Beauty, and possibly others (TaskList and Minimap maybe?) Many of the Sidekick parsers simply grab all the text from the buffer and parse it in one go to build the sidekick tree. This is a big performance hit on large files. Same thing with syntax highlighting. I'm thinking this might be better, or at least easier, to implement in a plugin, one that loads independently of VFS and is decoupled from all existing plugins.
2015-01-03 06:58:46.699000 ezust	I think plugins like sidekick, PMD, and beauty will have to be disabled for buffers that are only loading some pages and not the whole buffer.
2015-01-03 07:01:42.180000 ezust	and in order for bracket matching to work, the whole file must be scanned and indexed.
2015-01-03 10:17:14.666000 kerik-sf	On a related note, parsers often simply require a Reader instance. So they don't need the whole content in memory, simply a chunk at a time. But I could not find a straightforward way to access the buffer as a stream. I've written a simple [CharSequenceReader](https://sourceforge.net/p/jedit/svn/HEAD/tree/plugins/XML/trunk/xml/CharSequenceReader.java) in the XML plugin to build on top of the Segments. Before: ~~~~~~~~~~~~ String bufferContents = buffer.getText(startOffset, len); Reader r = new StringReader(bufferContents); ~~~~~~~~~~~~ After: ~~~~~~~~~~~~ CharSequence bufferContents = buffer.getSegment(startOffset,len); Reader r = new CharSequenceReader(bufferContents); ~~~~~~~~~~~~ Another more clever implementation of Reader on top of JEditBuffer could be devised, what do you think ?
2015-01-03 13:01:49.124000 kerik-sf	There's the same request for neovim: https://github.com/neovim/neovim/issues/614 They don't seem too eager to implement it either.
2015-01-28 22:38:26.041000 daleanson	Very preliminary, but I checked in a new plugin named "BigDoc" in to the jEdit svn plugin repository that uses nio file/memory mapping. Right now it just opens a file for viewing, but it seems to handle very large text files pretty well. It's based on the really old JEditTextArea, so it doesn't use the Buffer classes at all. Syntax highlighting for xml files works, so does bracket matching. Next will be to add in the ability to actually edit and save the file.