HDFS hflush vs hsync

hflush:  This API flushes all outstanding data (i.e. the current unfinished packet) from the client into the OS buffers on all DataNode replicas.

hsync: This API flushes the data to the DataNodes, like hflush(), but should also force the data to underlying physical storage via fsync (or equivalent). Note that only the current block is flushed to the disk device.

[1] https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java

9 Ağustos 2016

Posted In: dfsoutputstream, hadoop, hdfs, hflush, hsync

How to really persist your file in Java

Use FileChannel.force(boolean) or FileDescriptor.sync() to force data to be persistent on disk. Either of them can work. FileChannel.force use FileDispacther.force[1] and it calls fdatasync or fsync in Java 8. 

When you use OutputStream.flush, it does not guarantee the data to be written to disk, just flush it to OS. Better to use FileOutputStream.getChannel().force(true) or FileOutputStream.getFD().sync() to guarantee the persistency, performance might not be good.

Special Thanks to Yongkun. He wrote very good blog post. [2]

[1] http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/solaris/native/sun/nio/ch/FileDispatcherImpl.c#l141

[2] http://yongkunphd.blogspot.com/2013/12/how-fsync-works-in-java.html

9 Ağustos 2016

Posted In: fdatasync, FileChannel, fsync, java, OutputStream

Bloom Filters

Good tutorial for Bloom Filter understanding: http://billmill.org/bloomfilter-tutorial/

Bloom filters use case is following:

You have very large data sets that typically don’t fit in memory and you want to check your element it contains or not contains. Obviously It works very well for not contains detection.

if the bloom filter gives a hit: the item is probably inside
if the bloom filter gives a miss: the item is certainly not inside

How can I use in Java. Guava Provide a library for Bloom Filter:


m denotes the number of bits in the Bloom filter (bitSize) 

n denotes the number of elements inserted into the Bloom filter (maxKeys)

k represents the number of hash functions used (nbHash) 

e represents the desired false

positive rate for the bloom (err) If we fix the error rate (e) and know the number of entries, then the optimal bloom size 

m = -(nln(err) / (ln(2)^2) ~= nln(err) / ln(0.6185)

The probability of false positives is minimized when k = m/n ln(2).

6 Ağustos 2016

Posted In: bloomfilter, data structures, probabilistic data structure

Twitter Auto Publish Powered By : XYZScripts.com