Skip to content

2017

Docker Usage

Run a MySQL Container

Text Only
docker run --name={name} -p {host port}:3306 -e MYSQL_ROOT_PASSWORD={mysql root password} -e MYSQL_ROOT_HOST=localhost -d mysql/mysql-server:{mysql version} --config-entry-key=value
  • Check container logs docker logs {name}
  • Restore from dump files cat {dump file} | docker exec -i {name} mysql -u{user} -p{pass} {database}
  • Enter MySQL console docker exec -it {name} mysql -u{user} -p{pass}

Using HTTP/2 Client in Java 9

To use HTTP/2 in Java 9, it's required to declare the dependency of module jdk.incubator.httpclient. Here is the Java 9 module overview summary.

Assume the root package is com.jdk9.http.

  • To use module jdk.incubator.httpclient module, create module-info.java file with content:
Java
module com.jdk9.http {
    requires jdk.incubator.httpclient;
}

PostgreSQL backup & restore example

Backup

pg_dump --file ${backup file} --port "5432" --format=c --blobs --schema ${schema} ${database} -U ${user}

Restore

pg_restore -U ${user} --exit-on-error --verbose --dbname=${database} ${backup file}

PostgreSQL Basic Configurations After Installation

Allow all connections

Edit /etc/postgresql/${version}/main/pg_hba.conf, add host all all 0.0.0.0/0 md5 in section IPv4 local connections

Listen on all IP addresses

Edit /etc/postgresql/${version}/main/postgres.conf, in section Connection Settings, uncomment listen_addresses = 'localhost', change the localhost to *

Create a superuser

Run in shell, 1. su - postgres 2. createuser -P -s -e <username>, will prompt to input the password for the user

Spark Cluster Configuration

Following the Hadoop configurations.

In $SPARK_HOME/conf:

  • Configure spark-env.sh

cp spark-env.sh.template spark-env.sh and add following:

Text Only
PYSPARK_DRIVER_PYTHON=python3
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export SPARK_MASTER_HOST=$Master IP
export SPARK_WORKER_PORT=20001

Invoke Script Functions and Methods via JavaScript Engine

List supported script engines

Java
ScriptEngineManager manager = new ScriptEngineManager();
for(ScriptEngineFactory factory : manager.getEngineFactories()){
    System.out.printf("Name: %s%n" +
                    "\tVersion: %s%n" +
                    "\tLanguage name: %s%n" +
                    "\tLanguage version: %s%n" +
                    "\tExtensions: %s%n" +
                    "\tMime types: %s%n" +
                    "\tNames: %s%n",
            factory.getEngineName(),
            factory.getEngineVersion(),
            factory.getLanguageName(),
            factory.getLanguageVersion(),
            factory.getExtensions(),
            factory.getMimeTypes(),
            factory.getNames());
}

Linux Sed Command

Basic Usage

  • replace a string in a file

    sed -i "s/<string to be replaced>/<string will be replaced to>/g" $file

    You also can escape special character with a backslash