Sunday 14 July 2013

Useful linux commands

OS Details

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 10.04.4 LTS
Release: 10.04
Codename: lucid

System Information

$ uname -a
Linux p3307963.pubip.serverbeach.com 2.6.32-46-generic #108-Ubuntu SMP Thu Apr 11 15:56:25 UTC 2013 x86_64 GNU/Linux


$ uname -v

x86_64 (means system is 64 bit machine, otherwise its 32 bit machine)


User specific resource limit get/set

$ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Above command is very very important. Many a times TCP/IP socket connection grows a lot, if you sense that kind of problem, too many connections at a time, try to change "open files" limit to higher value, 20000 is  good enough. How to change the limit, you should read this article How to set ulimit in ubuntu/debian linux systems

Which user has opened how many files in sort order

$ lsof | awk '{if(NR>1) print $3}' | sort | uniq -c | sort -nr
   1256 root
    655 nishal
     16 www-data
      4 syslog
      4 ntp
      4 daemon

Check listening ports
$ netstat -nlp

change the open file limit in debian/ubuntu/linux system

For every user there is a resource limit configuration in linux. If it is not specified, it picks the default values, you can check them by command 
$ ulimit -a

Change Open file limit, ulimit on Debian/Ubuntu/Linux systems

------
Now to change it first of all pam limits by default is not loaded in ubuntu

$ vi /etc/pam.d/su
Un-comment the following line
#session    required   pam_limits.so
to
session    required   pam_limits.so

Now
$ vi /etc/security/limits.conf

and add the following lines to the end of the file (before the line # End of file)
*       soft  nofile   16000                                                      
*       hard nofile  64000

Save the file and quit vi.

Now To bring this in affect, you must restart the machine.

Below two lines will change the limit only for mysql user
mysql               soft    nofile          10240
mysql               hard   nofile          10240

Usually we need to change the "open file" limit from 1024(which is default) to higher value. But before that you should check which user is opening how many files.
$ lsof | awk '{if(NR>1) print $3}' | sort | uniq -c | sort -nr
   1256 root
    655 nishal
     16 www-data
      4 syslog
      4 ntp
      4 memcache
      4 daemon

The above result clearly says that root user has opened 1256 files, in such cases default limit of 1024 will start creating IO wait issue, connection timeout issue. To fix them, above article is beautiful.

------------

How to change Open file limit, ulimit on CentOS/Fedora/Red Hat


Command to check
ulimit -n
ulimit -a


1. vi sysctl.conf  and add this line  fs.file-max = 65536

2. vi /etc/security/limits.conf
*          soft     nproc          16384
*          hard     nproc          65535
*          soft     nofile         16384
*          hard     nofile         65535

3. Restart the server

4. ulmiit -n
16384


Mantis is not sending Email, How to fix?

I installed mantis for bug tracking system and it was done quite quick, may be in an hour. But the miserable part is, it was not sending email notification for any update, and it took more than couple of hour and still not fix... wtf is that.

OK, Chill !!!! I got the solution


Configuring the email settings is challenging task for most of the users start working with Mantis. The confusion is partially caused because PHP (so therefore Mantis) does not give you a very precise description about why it can not deliver emails. But you should try to see the log file /var/log/apache2/error.log and see if you can get some valuable information to proceed further.


In Most of the cases the configuration is very easy, and the good part is Mantis does maintain the mail related configuration variable as global ones.
So just need to change them and it should start working.


1. open your config_inc.php (located in: /var/www/mantis/config_inc.php)
2. copy the following code to the file:

$g_allow_signup    = ON;  //allows the users to sign up for a new account
$g_enable_email_notification = ON; //enables the email messages
$g_phpMailer_method = PHPMAILER_METHOD_SMTP; // this is most important
$g_smtp_host = 'smtp.gmail.com';
$g_smtp_connection_mode = 'tls';
$g_smtp_port = 587;
$g_smtp_username = 'youraccount@gmail.com'; //replace it with your gmail address
$g_smtp_password = '*********'; //replace it with your gmail password
$g_administrator_email = 'admin@example.com'; //this will be your administrator email address

3. go to your Mantis homepage (http://www.example.com/mantis)
4. click sign up for a new account
5. create a dummy account with your gmail address
6. press Signup
7. check your mail

--------------
Suppose you want to use other modes of smtp_connection, you can read about them at here http://www.mantisbt.org/docs/master-1.2.x/en/administration_guide/admin.config.email.html

If you want to use your own SMTP server without TLS or SSL conneciton, it would be something like.
$g_smtp_host = 'smtp.example.com';
$g_smtp_connection_mode = '';
$g_smtp_port = 25;
$g_smtp_username = 'youraccount@example'; //replace it with the email which can access your mail server
$g_smtp_password = '*********'; //replace it with your email account password

--------------
 /**
         * select the method to mail by:
         * PHPMAILER_METHOD_MAIL - mail()
         * PHPMAILER_METHOD_SENDMAIL - sendmail
         * PHPMAILER_METHOD_SMTP - SMTP
         * @global int $g_phpMailer_method
         */
        $g_phpMailer_method             = PHPMAILER_METHOD_MAIL;

Above code is present in "/mantis_home/config_defaults_inc.php" file.
As you see above mantis provide various ways of sending email, default is mail() command of linux, better is is make it "PHPMAILER_METHOD_SMTP", so that you can send email using any email server.

In the same file you can change from email also which will be shown in email. For that you need change the constant 
$g_from_email = "mantis@example.com"
$g_from_name = "Mantis Nishal Bug Tracker"

Note : For any further help, use this link http://www.mantisbt.org/forums/viewtopic.php?t=15398&f=3

#mantis not sending email notifications
#mantis email system not working
#mantis is not sending email
#mantis "sh: /usr/sbin/sendmail: not found"

Tuesday 2 July 2013

Using compression, gizp, mod_deflate, amazon cloudfront issue with gzip

If you are not using compression, you are missing something great feature. Please enable it, you'll find a big difference in page load time.

How does gzip work over the HTTP ?


When browsers make a request to a server, they send a Accept-Encoding header.
For most of the browser they will send Accept-Encoding: gzip, deflate. The server then knows that this browser accepts data compressed using gzip or deflate. Now, the server sees Accept-Encoding: gzip, deflate, sends the response compressed as gzip and marks it with the response header Content-Encoding: gzip.

The server can also optionally send another header Vary: Accept-Encoding. This tells proxies to vary the object in the proxy cache based on the Accept-Encoding header. The result is that the proxy will have a compressed and uncompressed version of the file in cache (and maybe even three: uncompressed, gzip compressed, deflate compressed). Failing to provide the Vary header may result in the wrong encoding going to an incompatible browser. The Vary header was introduced in HTTP/1.1


How to use gzip/deflate?


With Apache, it comes with a module mod_deflate. 

In any standard installation of Apache it comes with mod deflate. If it is not enabled, you can enable using.
$ a2nmod defalte
And reload conf or restart apache. After enabling, check the configuration
/etc/apache2/mods-enabled/deflate.conf

<IfModule mod_deflate.c>
          # these are known to be safe with MSIE 6
          AddOutputFilterByType DEFLATE text/html text/plain text/xml

          # everything else may cause problems with MSIE 6
          AddOutputFilterByType DEFLATE text/css
          AddOutputFilterByType DEFLATE application/x-javascript application/javascript application/ecmascript
          AddOutputFilterByType DEFLATE application/rss+xml
</IfModule>

Nginx comes by default with gzip module

edit the fle /etc/nginx/nginx.conf

          gzip on;
          gzip_disable "msie6";
          gzip_vary on;
          gzip_proxied any;
          gzip_comp_level 6;
          gzip_buffers 16 8k;
          gzip_http_version 1.1;
          gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

Issues with Amazon aws cloudfront and nginx and compression

If you are using cloudfront as a CDN to deliver the static content, then using nginx, you'll suddenly see that cloudfront is serving unzipped static contents.

The reason is cloudfront uses HTTP 1.0 to make request to origin server, but in nginx there is directive "gzip_http_version" which is set to 1.1, just check the above configuration. So what you need to do it make it 1.0

gzip_http_version 1.0;

That will enable the compression for cloudfront requests as well.

Monday 1 July 2013

Install latest version of Nginx

$ sudo -s
$ echo "deb http://ppa.launchpad.net/nginx/$nginx/ubuntu lucid main" > /etc/apt/sources.list.d/nginx-$nginx-lucid.list
$ apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C300EE8C
$ apt-get update

At this step you might get an error in upgrading
"W: GPG error: http://nginx.org lucid Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY ABF5BD827BD9BF62"

So try this
$ wget http://nginx.org/packages/keys/nginx_signing.key
$ cat nginx_signing.key | sudo apt-key add -
OK
$ apt-get update

Now Install
$apt-get install nginx

$/usr/sbin/nginx -V