Hbase oldWALs directory: what is it, when is it cleaned and who use its ?

The oldWALs folder gets cleaned regularly by a chore in master. When a WAL file is not needed any more for recovery purposes (when HBase can guaratee HBase has flushed all the data in the WAL file), it is moved to the oldWALs
folder for archival. The log stays there until all other references to the
WAL file are finished. There is currently two services which may keep the
files in the archive dir. First is a TTL process, which ensures that the
WAL files are kept at least for 10 min. This is mainly for debugging. You
can reduce this time by setting hbase.master.logcleaner.ttl configuration
property in master. It is by default 600000. The other one is replication.
If you have replication setup, the replication processes will hang on to
the WAL files until they are replicated. Even if you disabled the
replication, the files are still referenced.

Source: http://mail-archives.apache.org/mod_mbox/hbase-user/201502.mbox/%3CCAMUu0w9aOVBo7kGULiM9tXrULirqs9fm-3ra3pQccYpW_17uOw@mail.gmail.com%3E

12 Mart 2015

Posted In: hbase, oldWALs, replication, WAL

Hbase oldWALs directory: what is it, when is it cleaned and who use its ?

The oldWALs folder gets cleaned regularly by a chore in master. When a WAL file is not needed any more for recovery purposes (when HBase can guaratee HBase has flushed all the data in the WAL file), it is moved to the oldWALs
folder for archival. The log stays there until all other references to the
WAL file are finished. There is currently two services which may keep the
files in the archive dir. First is a TTL process, which ensures that the
WAL files are kept at least for 10 min. This is mainly for debugging. You
can reduce this time by setting hbase.master.logcleaner.ttl configuration
property in master. It is by default 600000. The other one is replication.
If you have replication setup, the replication processes will hang on to
the WAL files until they are replicated. Even if you disabled the
replication, the files are still referenced.

Source: http://mail-archives.apache.org/mod_mbox/hbase-user/201502.mbox/%3CCAMUu0w9aOVBo7kGULiM9tXrULirqs9fm-3ra3pQccYpW_17uOw@mail.gmail.com%3E

12 Mart 2015

Posted In: hbase, oldWALs, replication, WAL

The replication is turned off HBase, why is oldWALs directory still holded ?

Disabled replication will still hold on to the WAL files because, because it has a guarantee to not lose data between disable and enable. You can remove_peer, which frees up the WAL files to be eligible for deletion. When you re-add replication peer again, the replication will start from the current status, versus if you re-enable a peer, it will continue from where it left.

12 Mart 2015

Posted In: apache hbase, hbase, replication, WAL

The replication is turned off HBase, why is oldWALs directory still holded ?

Disabled replication will still hold on to the WAL files because, because it has a guarantee to not lose data between disable and enable. You can remove_peer, which frees up the WAL files to be eligible for deletion. When you re-add replication peer again, the replication will start from the current status, versus if you re-enable a peer, it will continue from where it left.

12 Mart 2015

Posted In: apache hbase, hbase, replication, WAL

WP Twitter Auto Publish Powered By : XYZScripts.com