Hive Setup on Debian

2014/10/04 Saturday 11:45

A fine day , but nothing to do, no plans to go out.
So thought of uploading the manual for hive setup.

Having manual handy is quite helpful, when you have any idea to implement, but you are stuck because of environment. It takes time to search for setup manuals, then installing the stuff.
And when you find its finished, you get some issues of xxx not installed, yyy not present.
And then there starts the series of problems wherein you either give up your idea, or get so much involved in setting up things that you almost forget what was the main thing with which you started all this. There are few highly motivated people who do the setup and then implement their idea.

Well lets see how to install hive right from the scratch.

1) Install Virtual Box

I prefer to install on virtual box, because it is quite useful to understand how to setup from beginning, right from empty OS.

So first download the virtual box depending on your OS.
https://www.virtualbox.org/wiki/Downloads

2) Install Debian on Virtual Box

Download Debian ISO image
http://ftp.acc.umu.se/debian-cd/7.6.0/

Mount this Debian ISO image in your virtual box and install Debian OS on virtual box.
After installing, start your OS.

If you want, you can share the network of Virtual Box and your local PC. Also install SSH, so that you can connect from your local PC to your virtual debian os via any terminal.

3) Setting up JAVA

Download Java JDK
http://www.oracle.com/technetwork/java/javase/downloads/index.html

-----
## check OS version
file /sbin/init

## remove all old versions of java

apt-get purge openjdk-\*

## download the corresponding java 

cd /home/xxx/

wget "java download URL"

## extract the contents

## replace this file with yours

tar -xzvf "jdk-7u65-linux-x64.tar.gz" 

ls -la jdk-7u65-linux-x64

## move it to common location

mkdir -pv /usr/local/java/

mv  jdk-7u65-linux-x64  /usr/local/java/

## setup env variables

vi /etc/profile

----

JAVA_HOME=/usr/local/java/jdk1.7.0_65

PATH=$PATH:$HOME/bin:$JAVA_HOME/bin

JRE_HOME=/usr/local/java/jdk1.7.0_65

PATH=$PATH:$HOME/bin:$JRE_HOME/bin

export JAVA_HOME

export JRE_HOME

export PATH

----

## import the env variables

source /etc/profile

## update the symbolic links for java

update-alternatives --install "/usr/bin/java" "java" "/usr/local/java/jdk1.7.0_65/bin/java" 1

update-alternatives --install "/usr/bin/javac" "javac" "/usr/local/java/jdk1.7.0_65/bin/javac" 1

update-alternatives --install "/usr/bin/javaws" "javaws" "/usr/local/java/jdk1.7.0_65/bin/javaws" 1

update-alternatives --set java /usr/local/java/jdk1.7.0_65/bin/java

update-alternatives --set javac /usr/local/java/jdk1.7.0_65/bin/javac

update-alternatives --set javaws /usr/local/java/jdk1.7.0_65/bin/javaws

## confirm your java version

java -version

4) Setting up Hadoop (pre-requirement for Hive)

Download the corresponding hadoop

http://www.apache.org/dyn/closer.cgi/hadoop/common/

cd /home/xxx/

wget "hadoop download URL"

## extract the contents

## replace this file with yours

tar -xzvf "hadoop-2.5.1.tar.gz" 

ls -la hadoop-2.5.1

## move it to common location

mkdir -pv /usr/local/hadoop/

mv  hadoop-2.5.1  /usr/local/hadoop/

## setup env variables

vi /etc/profile

----

HADOOP_PREFIX=/usr/local/hadoop/hadoop-2.5.1

PATH=$PATH:$HOME/bin:$HADOOP_PREFIX/bin

export HADOOP_PREFIX

export PATH

----

## confirm your hadoop version

hadoop version

5) Setting up Hive

Download the corresponding hive

http://www.apache.org/dyn/closer.cgi/hive/

cd /home/xxx/

wget "hive download URL"

## extract the contents

## replace this file with yours

tar -xzvf "apache-hive-0.13.1-bin.tar.gz" 

ls -la apache-hive-0.13.1-bin

## move it to common location

mkdir -pv /usr/local/hive/

mv  apache-hive-0.13.1-bin  /usr/local/hive/

## setup env variables

vi /etc/profile

----

HIVE_HOME=/usr/local/hive/apache-hive-0.13.1-bin

PATH=$PATH:$HOME/bin:$HIVE_HOME/bin

export HIVE_HOME

export PATH

----

## confirm your hive version

hive

And there you go, your basic setup is complete.

Now implement anything.

For reference of commands, I think this Hive confluence is quite useful.

https://cwiki.apache.org/confluence/display/Hive/GettingStarted

2014/10/04 Saturday 1:11

Bash Vulnerability Shellshock

It was still office hours when I just got information about the bash vulnerability.
It was quite interesting to see this year that , the things which we believed to be the the safest one are gradually falling out.

There was a SSH heartbleed first and then the Bash Vulnerability now.
I was interested in this one particular, because it involves shell stuff.

The way to test this vulnerability is just execute

------

env x='() { :;}; echo vulnerable' bash -c "echo hhaha"

------

When i saw this piece of command for first time, it was mind boggling.

So lets check it piece by piece.

env : This command in bash is used for environment related operations.

Reference :

https://www.gnu.org/software/coreutils/manual/html_node/env-invocation.html

Basically we can set up exact copy of current working environment and execute another program in separately without affecting working environment.

If you want to add some new variables in the new environment, it can be done by setting it with 'env'

env x="test"

Executing this command will copy the existing environment variables and add new variable x into it.

Also it returns the list of variables.

I think the attacker uses some kind of parsing trick of bash, wherein the attack command is set as an environment variable.

env x='() { :;}; echo vulnerable;'

>> this sets the environment variable x as 

-----

_=/usr/bin/env

x=() { :;}; echo vulnerable;

-----

So now the bash part,

Once you have your crafted environment variable ready, you just need to open up new bash shell where these are set. With this, the command embedded in the variable gets executed.

-----

$ env x='() { :;}; echo vulnerable;' bash -c "echo haha" 

vulnerable

haha

-----

One more thing which i tried was to replace this echo command with the env command itself to check the variables status at that point.

But to my surprise, I got memory error.

------

$ env x='() { :;}; env;' bash -c "echo haha" 

>> Segmentation fault: 11

------

Check

tail /var/log/system.log

or crashLog 

I will update this page as when I find any interesting thing, 

So please keep in touch.

Setting up basic mail server with gmail account on debian

Sending mail via Gmail from ssmtp on debian !!!

ssh ${server}
su

# aptitude install sendmail
aptitude install ssmtp

vi /etc/ssmtp/ssmtp.conf
--------------------------
#
# Config file for sSMTP sendmail
#
# The person who gets all mail for userids < 1000
# Make this empty to disable rewriting.
root=xxxx@gmail.com

# The place where the mail goes. The actual machine name is required no
# MX records are consulted. Commonly mailhosts are named mail.domain.com
#mailhub=mail
mailhub=smtp.gmail.com:587

# Where will the mail seem to come from?
#rewriteDomain=

# The full hostname
hostname=myHostName

# Are users allowed to set their own From: address?
# YES - Allow the user to specify their own From: address
# NO - Use the system generated From: address
FromLineOverride=YES

AuthUser=xxxx
AuthPass=xxxx
UseSTARTTLS=YES
#UseTLS=YES
--------------------------

tail -f /var/log/mail.log
--------------------------
May 19 07:06:11 Lenovo-PC sSMTP[9764]: Creating SSL connection to host
May 19 07:06:12 Lenovo-PC sSMTP[9764]: SSL connection using RSA_ARCFOUR_SHA1
May 19 07:06:15 Lenovo-PC sSMTP[9764]: Sent mail for root@xxxx (221 2.0.0 closing connection zv3sm61423449pab.20 - gsmtp) uid=0 username=root outbytes=494

--------------------------

echo "Test Mail" | mail -s "Test" "xxxx@gmail.com"


######## extra settings ########

vi /etc/ssmtp/revaliases
--------------------------
# sSMTP aliases
#
# Format: local_account:outgoing_address:mailhub
#
# Example: root:your_login@your.domain:mailhub.your.domain[:port]
# where [:port] is an optional port number that defaults to 25.
root:xxx@gmail.com:smtp.gmail.com:587
anyUser:xxx@gmail.com:smtp.gmail.com:587
--------------------------