測試說明
創(chuàng)新互聯(lián)建站服務(wù)項(xiàng)目包括寧蒗網(wǎng)站建設(shè)、寧蒗網(wǎng)站制作、寧蒗網(wǎng)頁制作以及寧蒗網(wǎng)絡(luò)營銷策劃等。多年來,我們專注于互聯(lián)網(wǎng)行業(yè),利用自身積累的技術(shù)優(yōu)勢、行業(yè)經(jīng)驗(yàn)、深度合作伙伴關(guān)系等,向廣大中小型企業(yè)、政府機(jī)構(gòu)等提供互聯(lián)網(wǎng)行業(yè)的解決方案,寧蒗網(wǎng)站推廣取得了明顯的社會(huì)效益與經(jīng)濟(jì)效益。目前,我們服務(wù)的客戶以成都為中心已經(jīng)輻射到寧蒗省份的部分城市,未來相信會(huì)繼續(xù)擴(kuò)大服務(wù)區(qū)域并繼續(xù)獲得客戶的支持與信任!
寫2G文件,分批次寫入,每批次寫入128MB;
分別在Win7系統(tǒng)(3G內(nèi)存,雙核,32位,T系列處理器)和MacOS系統(tǒng)(8G內(nèi)存,四核,64位,i7系列處理器)下運(yùn)行測試。理論上跟硬盤類型和配置也有關(guān)系,這里不再貼出了。
測試代碼
package rwbigfile; import java.io.ByteArrayInputStream; import java.io.File; import java.io.IOException; import java.io.RandomAccessFile; import java.lang.reflect.Method; import java.nio.ByteBuffer; import java.nio.MappedByteBuffer; import java.nio.channels.Channels; import java.nio.channels.FileChannel; import java.nio.channels.FileChannel.MapMode; import java.nio.channels.ReadableByteChannel; import java.security.AccessController; import java.security.PrivilegedAction; import util.StopWatch; /** * NIO寫大文件比較 * @author Will * */ public class WriteBigFileComparison { // data chunk be written per time private static final int DATA_CHUNK = 128 * 1024 * 1024; // total data size is 2G private static final long LEN = 2L * 1024 * 1024 * 1024L; public static void writeWithFileChannel() throws IOException { File file = new File("e:/test/fc.dat"); if (file.exists()) { file.delete(); } RandomAccessFile raf = new RandomAccessFile(file, "rw"); FileChannel fileChannel = raf.getChannel(); byte[] data = null; long len = LEN; ByteBuffer buf = ByteBuffer.allocate(DATA_CHUNK); int dataChunk = DATA_CHUNK / (1024 * 1024); while (len >= DATA_CHUNK) { System.out.println("write a data chunk: " + dataChunk + "MB"); buf.clear(); // clear for re-write data = new byte[DATA_CHUNK]; for (int i = 0; i < DATA_CHUNK; i++) { buf.put(data[i]); } data = null; buf.flip(); // switches a Buffer from writing mode to reading mode fileChannel.write(buf); fileChannel.force(true); len -= DATA_CHUNK; } if (len > 0) { System.out.println("write rest data chunk: " + len + "B"); buf = ByteBuffer.allocateDirect((int) len); data = new byte[(int) len]; for (int i = 0; i < len; i++) { buf.put(data[i]); } buf.flip(); // switches a Buffer from writing mode to reading mode, position to 0, limit not changed fileChannel.write(buf); fileChannel.force(true); data = null; } fileChannel.close(); raf.close(); } /** * write big file with MappedByteBuffer * @throws IOException */ public static void writeWithMappedByteBuffer() throws IOException { File file = new File("e:/test/mb.dat"); if (file.exists()) { file.delete(); } RandomAccessFile raf = new RandomAccessFile(file, "rw"); FileChannel fileChannel = raf.getChannel(); int pos = 0; MappedByteBuffer mbb = null; byte[] data = null; long len = LEN; int dataChunk = DATA_CHUNK / (1024 * 1024); while (len >= DATA_CHUNK) { System.out.println("write a data chunk: " + dataChunk + "MB"); mbb = fileChannel.map(MapMode.READ_WRITE, pos, DATA_CHUNK); data = new byte[DATA_CHUNK]; mbb.put(data); data = null; len -= DATA_CHUNK; pos += DATA_CHUNK; } if (len > 0) { System.out.println("write rest data chunk: " + len + "B"); mbb = fileChannel.map(MapMode.READ_WRITE, pos, len); data = new byte[(int) len]; mbb.put(data); } data = null; unmap(mbb); // release MappedByteBuffer fileChannel.close(); } public static void writeWithTransferTo() throws IOException { File file = new File("e:/test/transfer.dat"); if (file.exists()) { file.delete(); } RandomAccessFile raf = new RandomAccessFile(file, "rw"); FileChannel toFileChannel = raf.getChannel(); long len = LEN; byte[] data = null; ByteArrayInputStream bais = null; ReadableByteChannel fromByteChannel = null; long position = 0; int dataChunk = DATA_CHUNK / (1024 * 1024); while (len >= DATA_CHUNK) { System.out.println("write a data chunk: " + dataChunk + "MB"); data = new byte[DATA_CHUNK]; bais = new ByteArrayInputStream(data); fromByteChannel = Channels.newChannel(bais); long count = DATA_CHUNK; toFileChannel.transferFrom(fromByteChannel, position, count); data = null; position += DATA_CHUNK; len -= DATA_CHUNK; } if (len > 0) { System.out.println("write rest data chunk: " + len + "B"); data = new byte[(int) len]; bais = new ByteArrayInputStream(data); fromByteChannel = Channels.newChannel(bais); long count = len; toFileChannel.transferFrom(fromByteChannel, position, count); } data = null; toFileChannel.close(); fromByteChannel.close(); } /** * 在MappedByteBuffer釋放后再對(duì)它進(jìn)行讀操作的話就會(huì)引發(fā)jvm crash,在并發(fā)情況下很容易發(fā)生 * 正在釋放時(shí)另一個(gè)線程正開始讀取,于是crash就發(fā)生了。所以為了系統(tǒng)穩(wěn)定性釋放前一般需要檢 * 查是否還有線程在讀或?qū)? * @param mappedByteBuffer */ public static void unmap(final MappedByteBuffer mappedByteBuffer) { try { if (mappedByteBuffer == null) { return; } mappedByteBuffer.force(); AccessController.doPrivileged(new PrivilegedAction<Object>() { @Override @SuppressWarnings("restriction") public Object run() { try { Method getCleanerMethod = mappedByteBuffer.getClass() .getMethod("cleaner", new Class[0]); getCleanerMethod.setAccessible(true); sun.misc.Cleaner cleaner = (sun.misc.Cleaner) getCleanerMethod .invoke(mappedByteBuffer, new Object[0]); cleaner.clean(); } catch (Exception e) { e.printStackTrace(); } System.out.println("clean MappedByteBuffer completed"); return null; } }); } catch (Exception e) { e.printStackTrace(); } } public static void main(String[] args) throws IOException { StopWatch sw = new StopWatch(); sw.startWithTaskName("write with file channel's write(ByteBuffer)"); writeWithFileChannel(); sw.stopAndPrint(); sw.startWithTaskName("write with file channel's transferTo"); writeWithTransferTo(); sw.stopAndPrint(); sw.startWithTaskName("write with MappedByteBuffer"); writeWithMappedByteBuffer(); sw.stopAndPrint(); } }
測試結(jié)果(Y軸是耗時(shí)秒數(shù))
內(nèi)存映射文件使用經(jīng)驗(yàn)
MappedByteBuffer需要占用“雙倍”的內(nèi)存(對(duì)象JVM堆內(nèi)存和Direct Byte Buffer內(nèi)存),可以通過-XX:MaxDirectMemorySize參數(shù)設(shè)置后者最大大小
不要頻繁調(diào)用MappedByteBuffer的force()方法,因?yàn)檫@個(gè)方法會(huì)強(qiáng)制OS刷新內(nèi)存中的數(shù)據(jù)到磁盤,從而只能獲得些微的性能提升(相比IO方式),可以用后面的代碼實(shí)例進(jìn)行定時(shí)、定量刷新
如果突然斷電或者服務(wù)器突然Down,內(nèi)存映射文件數(shù)據(jù)可能還沒有寫入磁盤,這時(shí)就會(huì)丟失一些數(shù)據(jù)。為了降低這種風(fēng)險(xiǎn),避免用MappedByteBuffer寫超大文件,可以把大文件分割成幾個(gè)小文件,但不能太?。ǚ駝t將失去性能優(yōu)勢)
ByteBuffer的rewind()方法將position屬性設(shè)回為0,因此可以重新讀取buffer中的數(shù)據(jù);limit屬性保持不變,因此可讀取的字節(jié)數(shù)不變
ByteBuffer的flip()方法將一個(gè)Buffer由寫模式切換到讀模式
ByteBuffer的clear()和compact()可以在我們讀完ByteBuffer中的數(shù)據(jù)后重新切回寫模式。不同的是clear()會(huì)將position設(shè)置為0,limit設(shè)為capacity,換句話說Buffer被清空了,但Buffer內(nèi)的數(shù)據(jù)并沒有被清空。如果Buffer中還有未被讀取的數(shù)據(jù),那調(diào)用clear()之后,這些數(shù)據(jù)會(huì)被“遺忘”,再寫入就會(huì)覆蓋這些未讀數(shù)據(jù)。而調(diào)用compcat()之后,這些未被讀取的數(shù)據(jù)仍然可以保留,因?yàn)樗鼘⑺羞€未被讀取的數(shù)據(jù)拷貝到Buffer的左端,然后設(shè)置position為緊隨未讀數(shù)據(jù)之后,limit被設(shè)置為capacity,未讀數(shù)據(jù)不會(huì)被覆蓋
定時(shí)、定量刷新內(nèi)存映射文件到磁盤
import java.io.File; import java.io.IOException; import java.io.RandomAccessFile; import java.nio.MappedByteBuffer; import java.nio.channels.FileChannel; public class MappedFile { // 文件名 private String fileName; // 文件所在目錄路徑 private String fileDirPath; // 文件對(duì)象 private File file; private MappedByteBuffer mappedByteBuffer; private FileChannel fileChannel; private boolean boundSuccess = false; // 文件最大只能為50MB private final static long MAX_FILE_SIZE = 1024 * 1024 * 50; // 最大的臟數(shù)據(jù)量512KB,系統(tǒng)必須觸發(fā)一次強(qiáng)制刷 private long MAX_FLUSH_DATA_SIZE = 1024 * 512; // 最大的刷間隔,系統(tǒng)必須觸發(fā)一次強(qiáng)制刷 private long MAX_FLUSH_TIME_GAP = 1000; // 文件寫入位置 private long writePosition = 0; // 最后一次刷數(shù)據(jù)的時(shí)候 private long lastFlushTime; // 上一次刷的文件位置 private long lastFlushFilePosition = 0; public MappedFile(String fileName, String fileDirPath) { super(); this.fileName = fileName; this.fileDirPath = fileDirPath; this.file = new File(fileDirPath + "/" + fileName); if (!file.exists()) { try { file.createNewFile(); } catch (IOException e) { e.printStackTrace(); } } } /** * * 內(nèi)存映照文件綁定 * @return */ public synchronized boolean boundChannelToByteBuffer() { try { RandomAccessFile raf = new RandomAccessFile(file, "rw"); this.fileChannel = raf.getChannel(); } catch (Exception e) { e.printStackTrace(); this.boundSuccess = false; return false; } try { this.mappedByteBuffer = this.fileChannel .map(FileChannel.MapMode.READ_WRITE, 0, MAX_FILE_SIZE); } catch (IOException e) { e.printStackTrace(); this.boundSuccess = false; return false; } this.boundSuccess = true; return true; } /** * 寫數(shù)據(jù):先將之前的文件刪除然后重新 * @param data * @return */ public synchronized boolean writeData(byte[] data) { return false; } /** * 在文件末尾追加數(shù)據(jù) * @param data * @return * @throws Exception */ public synchronized boolean appendData(byte[] data) throws Exception { if (!boundSuccess) { boundChannelToByteBuffer(); } writePosition = writePosition + data.length; if (writePosition >= MAX_FILE_SIZE) { // 如果寫入data會(huì)超出文件大小限制,不寫入 flush(); writePosition = writePosition - data.length; System.out.println("File=" + file.toURI().toString() + " is written full."); System.out.println("already write data length:" + writePosition + ", max file size=" + MAX_FILE_SIZE); return false; } this.mappedByteBuffer.put(data); // 檢查是否需要把內(nèi)存緩沖刷到磁盤 if ( (writePosition - lastFlushFilePosition > this.MAX_FLUSH_DATA_SIZE) || (System.currentTimeMillis() - lastFlushTime > this.MAX_FLUSH_TIME_GAP && writePosition > lastFlushFilePosition) ) { flush(); // 刷到磁盤 } return true; } public synchronized void flush() { this.mappedByteBuffer.force(); this.lastFlushTime = System.currentTimeMillis(); this.lastFlushFilePosition = writePosition; } public long getLastFlushTime() { return lastFlushTime; } public String getFileName() { return fileName; } public String getFileDirPath() { return fileDirPath; } public boolean isBundSuccess() { return boundSuccess; } public File getFile() { return file; } public static long getMaxFileSize() { return MAX_FILE_SIZE; } public long getWritePosition() { return writePosition; } public long getLastFlushFilePosition() { return lastFlushFilePosition; } public long getMAX_FLUSH_DATA_SIZE() { return MAX_FLUSH_DATA_SIZE; } public long getMAX_FLUSH_TIME_GAP() { return MAX_FLUSH_TIME_GAP; } }
以上就是本文的全部內(nèi)容,希望對(duì)大家的學(xué)習(xí)有所幫助,也希望大家多多支持創(chuàng)新互聯(lián)。
文章標(biāo)題:JavaNIO寫大文件對(duì)比(win7和mac)
網(wǎng)站鏈接:http://bm7419.com/article24/jcshce.html
成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián),為您提供Google、網(wǎng)站策劃、網(wǎng)站內(nèi)鏈、App開發(fā)、手機(jī)網(wǎng)站建設(shè)、品牌網(wǎng)站建設(shè)
聲明:本網(wǎng)站發(fā)布的內(nèi)容(圖片、視頻和文字)以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主,如果涉及侵權(quán)請(qǐng)盡快告知,我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場,如需處理請(qǐng)聯(lián)系客服。電話:028-86922220;郵箱:631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載,或轉(zhuǎn)載時(shí)需注明來源: 創(chuàng)新互聯(lián)