Life saving Bash Settings

I saw this settings on a tweet flood. I did not want to lose them. I copied all of content in here. Thanks @kadirsert 

If you are continuously running shell commands on more than one Linux terminal, probably you want all of the shell (mostly bash) prompts to remember any command from any terminal.

With the following environmental variables to in the .bashrc file, you can do it so

Ignoring duplicate entries: 
export HISTCONTROL=ignoredups:erasedups

large history size:
export HISTSIZE=102400

This is for a big history file:
export HISTFILESIZE=100000

Appending commands to history file:
shopt -s histappend

Last but not least, this is for saving and reloading the history after each command is run:
export PROMPT_COMMAND=“history -a; history -c; history -r; $PROMPT_COMMAND”

Source: https://twitter.com/kadirsert/status/1092694397441978369

5 Şubat 2019

Posted In: bash, bashrc, linux, shell

Life saving Bash Settings

I saw this settings on a tweet flood. I did not want to lose them. I copied all of content in here. Thanks @kadirsert 

If you are continuously running shell commands on more than one Linux terminal, probably you want all of the shell (mostly bash) prompts to remember any command from any terminal.

With the following environmental variables to in the .bashrc file, you can do it so

Ignoring duplicate entries: 
export HISTCONTROL=ignoredups:erasedups

large history size:
export HISTSIZE=102400

This is for a big history file:
export HISTFILESIZE=100000

Appending commands to history file:
shopt -s histappend

Last but not least, this is for saving and reloading the history after each command is run:
export PROMPT_COMMAND=“history -a; history -c; history -r; $PROMPT_COMMAND”

Source: https://twitter.com/kadirsert/status/1092694397441978369

5 Şubat 2019

Posted In: bash, bashrc, linux, shell

How to fix PostgreSQL error “duplicate key violates unique constraint”

If you get this message when trying to insert data into a PostgreSQL database:


ERROR: duplicate key violates unique constraint

That likely means that the primary key sequence in the table you’re working with has somehow become out of sync, likely because of a mass import process (or something along those lines). Call it a “bug by design”, but it seems that you have to manually reset the a primary key index after restoring from a dump file. At any rate, to see if your values are out of sync, run these two commands:


SELECT MAX(the_primary_key) FROM the_table;

SELECT nextval('the_primary_key_sequence');

If the first value is higher than the second value, your sequence is out of sync. Back up your PG database (just in case), then run thisL


SELECT setval('the_primary_key_sequence', (SELECT MAX(the_primary_key) FROM the_table)+1);

That will set the sequence to the next available value that’s higher than any existing primary key in the sequence.

Reference: https://hcmc.uvic.ca/blogs/index.php?blog=22&p=8105&more=1&c=1&tb=1&pb=1

29 Ağustos 2017

Posted In: postgresql

How to fix PostgreSQL error “duplicate key violates unique constraint”

If you get this message when trying to insert data into a PostgreSQL database:


ERROR: duplicate key violates unique constraint

That likely means that the primary key sequence in the table you’re working with has somehow become out of sync, likely because of a mass import process (or something along those lines). Call it a “bug by design”, but it seems that you have to manually reset the a primary key index after restoring from a dump file. At any rate, to see if your values are out of sync, run these two commands:


SELECT MAX(the_primary_key) FROM the_table;

SELECT nextval('the_primary_key_sequence');

If the first value is higher than the second value, your sequence is out of sync. Back up your PG database (just in case), then run thisL


SELECT setval('the_primary_key_sequence', (SELECT MAX(the_primary_key) FROM the_table)+1);

That will set the sequence to the next available value that’s higher than any existing primary key in the sequence.

Reference: https://hcmc.uvic.ca/blogs/index.php?blog=22&p=8105&more=1&c=1&tb=1&pb=1

29 Ağustos 2017

Posted In: postgresql

How Spark SQL evaluate our Expressions

There are different evaluation paths in Spark SQL:

  • Interpreted. Expressions are evaluated using an eval(…) method. Plans are evaluated using iterators (volcano model). 
  • Expression Codegenerated. This means that all expressions are evaluated using a code generated function. Plans are evaluated using iterators.
  • Wholestafe Codegenerated. All expressions and most plans are evaluated using code generation.

For more information: https://issues.apache.org/jira/browse/SPARK-17728

9 Mart 2017

Posted In: apache, apache spark, apache spark sql, dataframe, scala, spark, spark sql, sql

Java happens before rules for Concurrency

The rules for happens-before are:

Program order rule. Each action in a thread happens-before every action in that thread that comes later in the program order.
Monitor lock rule. An unlock on a monitor lock happens-before every subsequent lock on that same monitor lock.
Volatile variable rule. A write to a volatile field happens-before every subsequent read of that same field.
Thread start rule. A call to Thread.start on a thread happens-before every action in the started thread.
Thread termination rule. Any action in a thread happens-before any other thread detects that thread has terminated, either by successfully return from Thread.join or by Thread.isAlive returning false.
Interruption rule. A thread calling interrupt on another thread happens-before the interrupted thread detects the interrupt (either by having InterruptedException thrown, or invoking isInterrupted or interrupted).
Finalizer rule. The end of a constructor for an object happens-before the start of the finalizer for that object.

5 Eylül 2016

Posted In: concurrency, happens-before, java, lock, thread, volatile

Java happens before rules for Concurrency

The rules for happens-before are:

Program order rule. Each action in a thread happens-before every action in that thread that comes later in the program order.
Monitor lock rule. An unlock on a monitor lock happens-before every subsequent lock on that same monitor lock.
Volatile variable rule. A write to a volatile field happens-before every subsequent read of that same field.
Thread start rule. A call to Thread.start on a thread happens-before every action in the started thread.
Thread termination rule. Any action in a thread happens-before any other thread detects that thread has terminated, either by successfully return from Thread.join or by Thread.isAlive returning false.
Interruption rule. A thread calling interrupt on another thread happens-before the interrupted thread detects the interrupt (either by having InterruptedException thrown, or invoking isInterrupted or interrupted).
Finalizer rule. The end of a constructor for an object happens-before the start of the finalizer for that object.

5 Eylül 2016

Posted In: concurrency, happens-before, java, lock, thread, volatile

HDFS hflush vs hsync

hflush:  This API flushes all outstanding data (i.e. the current unfinished packet) from the client into the OS buffers on all DataNode replicas.

hsync: This API flushes the data to the DataNodes, like hflush(), but should also force the data to underlying physical storage via fsync (or equivalent). Note that only the current block is flushed to the disk device.

[1] https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java

9 Ağustos 2016

Posted In: dfsoutputstream, hadoop, hdfs, hflush, hsync

WP Twitter Auto Publish Powered By : XYZScripts.com