Man page ASCII output tags

Simple way to output man page in ASCII only format (no any special symbols):

groff -P\-c -mandoc -Tascii file.1|col -bx > file.txt

file.txt is formatted as you see man output on your screen.

Small shell tricks

Directories tree view

Set it to your .bashrc and run as tree [DIR].

tree() {
find "${1:-.}" -type d -print 2>/dev/null|awk '!/\.$/ {for (i=1;i<NF;i++){d=length($i);if ( d < 5  && i != 1 )d=5;printf("%"d"s","|")}print "---"$NF}'  FS='/'
}

Patch selected file only

If you have big patch file with many files there, to patch only one (filename.py in example):

filterdiff -i '*filename.py' patch.diff | patch -pN

and to list all files in patch file use lsdiff or filterdiff --list.

Test HEAD HTTP with curl

curl -i -X HEAD http://...

Paging in FAR

view:<command
edit:<command

and to enable scrolling start with far -w.

Run alternate shell commands

This script run one of available command:

#!/bin/sh

trycmd() {
    a0=$1; shift
    for c in $*; do $c $a0 2>/dev/null && break; done
}

# usage: run one of: pipx|pip0|pip1|pip3|pip with arg: list
trycmd list pipx pip0 pip1 pip3 pip

Frozen keyboard in PyCharm (Linux)

Bug in forever buggy Java:

ibus restart

Remove passphrase from SSH key

openssl rsa -in ~/.ssh/id_rsa -out ~/.ssh/id_rsa_new
cp ~/.ssh/id_rsa ~/.ssh/id_rsa.backup
rm ~/.ssh/id_rsa
cp ~/.ssh/id_rsa_new ~/.ssh/id_rsa
chmod 400 ~/.ssh/id_rsa

Problem with RPC/NFS mount after update CentOS 7

After some digging I finally found that the reason for this is a buggy systemd unit file in RHEL/CentOS 7.

This is a bug in nfs-server.service and he fix is to copy nfs-server.service to /etc/systemd/system and replace all occurrences of "rpcbind.target" with "rpcbind.service".

See this bug for reference: https://bugzilla.redhat.com/show_bug.cgi?id=1171603

To allow connecting to CentOS

If there is trouble try to stop iptables:

service iptables start iptables flush service iptables stop

Script to run jobs suite remotely via screen

Often in needed to run some jobs remotely, for example, tests on remote test environment. I used different custom "daemon"-like scripts for this, monitoring the jobs/tests, but another and simple solution for Linux is to use screen tool. It supports sessions, detaching, so is good for such kind of tasks. Here is the script which runs multiple jobs on remote host (setted via $RMHOST env. var). Its syntax is:

  -cls
  -run JOB-NUMBER 'JOB-COMMAND'
  -runall < FILE-WITH-COMMANDS
  -submit < FILE-WITH-COMMANDS
Used env. vars: \$USER (login), \$RMHOST (hostname)

so to run it you must:

• set $RMHOST env. var
• create file with commands which will be ran remotely: each command is on own line and can have any $ variables for substitution: they will be substituted locally
• then call script as V1=a V2=b script -submit < created_file. This will substitute V1 and V2 in created_file and send resulting file over scp to remote host $RMHOST (you must have configured public-key access to host), then will send itself and call itself remotely with command -runall.

After it you can go to host, call screen -r to see your created session, to switch between windows (each job is ran on separate window). You will have local $LOG file with decscription of running commands and remote log file per job with info about what and when was run (job-<DATE>-<JOB_NUMBER>.log). Yes, each job has number/index related to line number of job suite file. If you want you can include generating of screen log file too (with -L option IMO).

While you are on remote host, you can run this script (which was scp-ed) with -cls command: it will remove all log files there.

That's it.

#!/bin/sh
SCR=`basename "$0"`
SESSION=${USER}_screen
LOG=rmjobs.log

_wrapjob() {
    _wrapjob_f=job-`date +%s`-$1.log
    echo "Started $1 at `date`: $2" > $_wrapjob_f
    echo '===============================================================================' >> $_wrapjob_f
    echo >> $_wrapjob_f
    $2|tee -a $_wrapjob_f
}

_runjob() {
    [ $1 = "0" ] && {
        # add -L to both if screelog.* logs are needed
        screen -dmS $SESSION -t "job$1" sh -c "~/$SCR -run $1 \"$2\";sh"
    } || {
        screen -S $SESSION -p 0 -X screen -t "job$1" sh -c "~/$SCR -run $1 \"$2\";sh"
    }
}

case "$1" in
    -cls) rm -f screenlog.*; rm -f job*.log;;
    -run) _wrapjob "$2" "$3";;
    -runall)
        i=0
        rm -f $LOG 2>/dev/null
        while read -r c; do
            c=`echo $c|tr -d '[:cntrl:]'`
            echo $c|egrep '^-' >/dev/null && continue
            _runjob $i "$c"
            echo "Ran $c;" >> $LOG
            i=`expr $i + 1`
            sleep 2
        done;;
    -submit)
        rm -f .jobs-suite.tmp 2>/dev/null
        while read -r c; do
            echo $c|egrep '^-' >/dev/null && continue
            eval "echo $c" >> .jobs-suite.tmp
        done
        scp "$0" "$USER@$RMHOST:$SCR"
        scp ".jobs-suite.tmp" $USER@$RMHOST:".jobs-suite.tmp"
        ssh -n -l $USER $RMHOST "sh $SCR -runall < .jobs-suite.tmp";;
    *)
        echo "Syntax:"
        echo "  -cls"
        echo "  -run JOB-NUMBER 'JOB-COMMAND'"
        echo "  -runall < FILE-WITH-COMMANDS"
        echo "  -submit < FILE-WITH-COMMANDS"
        echo "Used env. vars: \$USER (login), \$RMHOST (hostname)"
        exit 1;;
esac
:

Download all files by mime type

Download files of some types from URL:

wget -A ps,djvu -m -p -E -k -K -np -nH -L -l <DEEP> -e robots=off URL

all *.ps, *.djvu will be saved in own directories, but without host folder.

Perforce scripts

This is useful Perforce old my scripts-helpers.

Common utils:

# Returns number of changelists, prints list of changelists
find_p4cl() {
    p4 changes -u $USER -s pending|awk '
        BEGIN { found=0 }
        /^Change / {
            p4cl[found, 0] = $2
            p4cl[found, 1] = $4
            w = match($0, /\*pending\* /)
            if (w) d = substr($0, w + length("*pending* "))
            else d = ""
            p4cl[found, 2] = d
            found++
        }
        END {
            if (found) {
                print "Available ChangeLists:"
                print "======================"
                for (i=0; i<found; i++) {
                    printf "  %s on %s: %s...\n",
                        p4cl[i, 0],
                        p4cl[i, 1],
                        p4cl[i, 2]
                }
            }
            exit found
        }
    '
}

# Input from user changelist (show default value if it one only).
# If no input, returns default
read_p4cl() {
    local changelists _p4cl p4clhint
    changelists=`find_p4cl`
    if [ $? -eq 1 ]; then
        _p4cl=`echo "$changelists"|awk 'NR==3{print $1}'`
        p4clhint=" [${_p4cl}]"
    else
        _p4cl=""
        p4clhint=""
    fi
    echo "$changelists"
    echo
    echo -n "Enter P4 ChangeList number${p4clhint}: "; read p4cl; : ${p4cl:=$_p4cl}
}

# if not Perforce workspace then report it
check_p4ws() {
    CF=${P4CONFIG:-.p4config}
    [ -f "$CF" ] || { echo 'Seems not P4 workspace!' >&2; return 1; }
    return 0
}

List of midified files without switch it to "Edit" mode (devs often forget to do it :)

#!/bin/sh
# List of locally modified files without put them into ChangeList

. .p4utils
check_p4ws || exit 1
p4 reconcile -e -n

Review sedning:

#!/bin/sh
# Send ChangeList review request

. .p4utils
check_p4ws || exit 1
read_p4cl
echo "Posting review now. Wait, please..."
post-review $p4cl

Logging library for Unix Shell

This library is similar to Log library for Python or Java. Here its code:

#!/bin/sh

# Simple logging facility for Unix /bin/sh
# ========================================
#
# Configuration
# -------------
#
#   _LOGGING_DATEFMT, _LOGGING_FMT variables are used for format of datetime
#   and log record
#
#   * _LOGGING_DATEFMT : uses the same format as date(1) and Python Lib
#
#   * _LOGGING_FMT : uses format similar to Python logging Lib
#
#       Available fields for _LOGGING_FMT are:
#         asctime, filename, levelname, levelno, message, name, pathname,
#         process, processName, user
#
# Example
# -------
#
#     getLogger prg1
#     getLogger prg2
#     
#     addLogHandler prg1 FileHandler /dev/stderr
#     addLogHandler prg2 ConsoleHandler
#     addLogHandler prg2 FileHandler /var/log/somefile
#     addLogHandler prg2 FileHandler /var/log/somefile_copy
#
#     setLogLevel prg1 DEBUG
#
#     prg1_debug DebugMessage
#     prg1_error ErrorMessage
#     echo InfoMessage|prg2_info
#
#   will produces on STDERR console:
#     ...
#     2014-07-18T17:09:52+0300 prg1 DEBUG DebugMessage
#     ...
#
#   will produces on console:
#     ...
#     2014-07-18T17:09:52+0300 prg1 ERROR ErrorMessage
#     2014-07-18T17:09:52+0300 prg2 INFO InfoMessage
#     ...
#
#   and in /var/log/somefile:
#     ...
#     2014-07-18T17:09:52+0300 prg2 INFO InfoMessage
#     ...
#
#   and in /var/log/somefile_copy:
#     ...
#     2014-07-18T17:09:52+0300 prg2 INFO InfoMessage
#     ...

# TODO: log levels per handlers


############################## Settings #####################################

# Global log record format and date format:
_LOGGING_DATEFMT="%Y-%m-%dT%H:%M:%S%z"
_LOGGING_FMT="%asctime %name %levelname %message"



################################### Code #####################################

_log_SCRIPT=`readlink -f $0`

_log_getLogAttr() {
  name=$1; attr=${1}_$2
  eval echo $`echo $attr`
}

################################### Handlers #################################

# SYNOPSIS: ...
#
# Nothing to do, all args are ignored
_log_NullHandler() {
  :
}

# SYNOPSIS: $name $level $msg
#
# Logs $msg with $level via logger(1) utility
_log_SyslogHandler() {
  local name level msg syslog_level
  name="$1"; level="$2"; msg="$3"
  syslog_level=`_log_syslogLevel $level`
  logger -p "local0.${syslog_level}" -t $name "$msg"
}

# SYNOPSIS: $name $level $msg
#
# Logs message $msg on console
_log_ConsoleHandler() {
  local name level msg
  name="$1"; level="$2"; msg="$3"
  echo "$msg"
}

# SYNOPSIS: $subject $address $msg
#
# If mail(1) exists, call it to send $msg via EMail
_log_MailHandler() {
  command -v mail >/dev/null 2>&1 || return 1
  local subj addr name level msg
  subj="$1"; addr="$2"; name="$3"; level="$4"; msg="$5"
  echo "$msg"|mail -s "${subj}${level}: $name" $addr
}

# SYNOPSIS: $filename $msg
#
# Append message $msg to file $filename
_log_FileHandler() {
  local filename name level msg
  filename="$1"; name="$2"; level="$3"; msg="$4"
  touch "$filename"
  echo "$msg" >> "$filename"
}

_log_syslogLevel() {
  case $1 in
    CRITICAL)
      echo crit;;
    FATAL)
      echo panic;;
    ERROR)
      echo error;;
    WARNING)
      echo warning;;
    INFO)
      echo info;;
    DEBUG)
      echo debug;;
    *)
      echo notice;;
  esac
}

_log_levelNo() {
  case $1 in
    CRITICAL|FATAL)
      echo 50;;
    ERROR)
      echo 40;;
    WARNING)
      echo 30;;
    INFO)
      echo 20;;
    DEBUG)
      echo 10;;
    *)
      echo 0;;
  esac
}

_log() {
  local name level msg handlers h closure_args skip no_closure_args levelno enabled_levelno
  name="$1"; level="$2"; msg="$3"

  levelno=`_log_levelNo $level`
  enabled_levelno=`_log_getLogAttr $name level`
  [ $levelno -lt $enabled_levelno ] && return

  handlers=`_log_getLogAttr $name handlers`
  handlers=`echo "$handlers"|sed 's/__COMMA__/\\ /g'`
  for h in $handlers; do
    echo $skip|grep $h >/dev/null && continue
    closure_args=`_log_getLogAttr $name ${h}_handler_args`
    closure_args=`echo "$closure_args"|sed 's/__COMMA__/\\ /g'`
    no_closure_args=1
    for ca in $closure_args; do
      eval "$h $ca $name $level \"$msg\""
      no_closure_args=0
    done
    [ $no_closure_args ] && eval "$h $name $level \"$msg\""
    skip="$h $skip"
  done
}

# SYNOPSIS: $name $h [$args...]
#
# Adds handler $h for logger $name with possible arguments
# (see handlers' implementations for concrete args)
addLogHandler() {
  local name handler oldhandlers exthandler args oldargs
  name=$1; exthandler="$2"; handler=_log_"$exthandler"; shift 2; args="$*"
  oldhandlers=`_log_getLogAttr $name handlers`
  oldargs=`_log_getLogAttr $name ${handler}_handler_args`
  # __COMMA__ is used to separate items in lists
  eval "${name}_handlers=${handler}__COMMA__${oldhandlers:-_log_NullHandler}"
  eval "${name}_${handler}_handler_args=\"$args\"__COMMA__$oldargs"
}

# SYNOPSIS: $name | $name $longname
#
# No return and no output but setups logger for this $name. In 2nd form $longname
# is used in message substitution
getLogger() {
  local asctime filename levelname levelno message name pathname process processName _lv user longname

  asctime=$_LOGGING_DATEFMT
  filename=`getScriptName`
  message='${1:-`tee`}'
  name="$1"
  longname=${2:-$name}
  pathname=`echo "$_log_SCRIPT"|sed 's/\\//\\\\\//g'`
  process='$$'
  processName=`ps -A -opid,comm | awk -v PID=$$ '$1 == PID { print $2 }'`
  processName=`echo "$processName"|sed 's/\\//\\\\\//g'`
  user="$USER"

  for _lv in debug info warning error fatal critical; do
    levelname=`echo -n $_lv|tr '[:lower:]' '[:upper:]'`
    levelno=`_log_levelNo $levelname`

    fmt=`echo $_LOGGING_FMT | \
      sed 's/%asctime/\`date +$_LOGGING_DATEFMT\`/g' | \
      sed "s/%filename/$filename/g" | \
      sed "s/%levelname/$levelname/g" | \
      sed "s/%levelno/$levelno/g" | \
      sed "s/%message/$message/g" | \
      sed "s/%pathname/$pathname/g" | \
      sed "s/%processName/$processName/g" | \
      sed "s/%process/$process/g" | \
      sed "s/%user/$user/g" | \
      sed "s/%name/$longname/g"`
    eval "${name}_${_lv} () { _log $name $levelname \"$fmt\"; }"
  done
  setLogLevel $name INFO
}

# SYNOPSIS: $name $level
#
# Sets the threshold for $name logger to $level (INFO, ERROR, etc.). Logging
# messages which are less severe than $level will be ignored.
# Default is INFO
setLogLevel() {
  local name level levelno
  name="$1"; level=`echo -n $2|tr '[:lower:]' '[:upper:]'`
  levelno=`_log_levelNo $level`
  eval "${name}_level=$levelno"
}

# SYNOPSIS:
#
# Echos name of script
getScriptName() {
  echo `basename "$_log_SCRIPT"`
}

Example of usage:

#!/bin/sh
. ./logging.sh

getLogger prg1 `getScriptName`
addLogHandler prg1 ConsoleHandler
addLogHandler prg1 FileHandler ./lll1.log
addLogHandler prg1 FileHandler ./lll2.log
setLogLevel prg1 DEBUG

prg1_debug Debugggg
prg1_error Errrrror
echo 'Message-INFO!!!!
1111
2222'|prg1_info

Android build simplifier

Add this alias to ~/.bashrc:

alias amake='_(){ rm -f ./build.log; . ./build/envsetup.sh; lunch $1; make -j8 2>&1|tee ./build.log; }; _'

and now you can avoid envsetup.sh, lunch running and get build log file :) Better is to run:

$ amake something-userdebug

from Android source directory.
Or the same but as explicit function (put it, for example in ~/.zshrc):

function amake() {
/bin/bash -s stdin << EOF
rm -f ./build.log
. ./build/envsetup.sh
lunch $1
make -j8 2>&1|tee ./build.log
EOF
}

Android/smartphone hardware/software info

If you wanna get info (hardware configuration, software versions, memory info, etc.) you can do:

1. "tap" top bar with "tray icons" down and then "Settings"/"About" and any info item what you see there;
2. second way is more informative, but you should install additional FREE Application, ES File Explorer which is very usable (to manage all your files and folders even system - full control!). After install it, open application, go to root folder ("/") then "click" on "proc" folder and now you can open any file containing system info in standard Linux "/proc" (select "Open As Text" then with "ES Note Editor" after clicking on a file). You can see there interesting files like "cpuinfo", "version" and so on...

Converting PDF to JPG/PNG/PPM/TIFF

If you have big PDF with big resolution, then you can convert PDF to image file with pdftoppm:

pdftoppm -jpeg a.pdf a.jpg

Or use -png (see man for other options) to produce PNG

Crossplatform software delays

This is the simple macros for different delays usuable on Unix and Windows:

#ifndef _DELAYS_H
#define _DELAYS_H

#include 
#include 

/* Converting between seconds and CPU ticks (clock_t):
 *      MS2TICKS() converts milliseconds to ticks
 *      US2TICKS() converts microseconds to ticks
 *      TICKS2MS() converts ticks to milliseconds
 *      TICKS2US() converts ticks to microseconds
 */
#define MS2TICKS(MS) ((MS)*(CLOCKS_PER_SEC)/1000)
#define US2TICKS(US) ((US)*(CLOCKS_PER_SEC)/1000000)
#define TICKS2MS(T)  ((T)/((CLOCKS_PER_SEC)*1000))
#define TICKS2US(T)  ((T)/((CLOCKS_PER_SEC)*1000000))

/*
 * delay(MS)         sleeping on MS milliseconds
 */
#if defined(WIN32) || defined(_WIN32) || defined(__WIN32)
#  include 
#  define delay(MS) Sleep(MS)
#else
#  if defined(HAVE_USLEEP)
#    define delay(MS) usleep((MS)*1000)
#  elif defined(HAVE_SLEEP)
#    include 
#    define delay(MS) sleep((MS)/1000) /* granularity only ONE SECOND! */
#  else
#    error Can not found any sleep_clocks() implementation ideas
# endif
#endif

/*
 * pthread_delay(MS) sleeping thread on MS milliseconds
 */
#if defined(HAVE_PTHREAD_DELAY_NP)
#  define pthread_delay(MS) \
     do { \
             struct timespec __timespec={(MS)/1000,((MS)%1000)*1000000000}; \
             pthread_delay_np(&__timespec); \
     } \
     while(0)
#else
#  define pthread_delay(MS) delay(MS)
#endif

/*
 * unow(PNOW) store in var pointed by PNOW microseconds since Epoch
 */
#define unow(PNOW) \
{ \
        struct timeval __tv;  \
        gettimeofday(&__tv, NULL); \
        *(PNOW) = (__tv.tv_sec * 1000000) + __tv.tv_usec; \
}

#endif /*!_DELAYS_H*/

If HAVE_PTHREAD_DELAY_NP will be defined before including of this file, then pthread_delay() will be wrapper for pthread_delay_np() - see Pthreads for Win32 headers.

Linux box on Windows

It's frequent task for programmer: developing of application for alien OS. If you use Windows as host system and need Linux box, there are several free solutions: VmWare Player, VirtualBox, QEmu for your pleasure :)

But after done with virtual machine, you should to create some "bridge" to data exchange with guest OS (ssh/scp, samba/nfs, ftp, shared folder...). When you use VirtualBox, for example, you should install additional software to guest OS. But there is simpler solution: qemu with virtual FAT folders.

Download from QEmu on Windows ZIP file with qemu (yes, it's portable, and is only about 4Mb in ZIP!). Uncompress.

Suppose, we need only developer's box: no X, only GNU toolchain.

Let's try Archlinux, see: www.archlinux.org (download "CORE" ISO image).
Then create image - space for VM:

qemu-img create linux.img 1G

1G is enough room. This will be done quickly.
Then let's boot from ArchLinux CD image:

qemu -m 256 -hda c:\vm\linux.img -cdrom c:\vm\archlinux-2010.05-core-i686.iso -boot d -L c:\qemu\bios

We suppose user uncompress qemu to c:\qemu and puts images to c:\vm. After boot you should install ArchLinux - it's easy, but long :)
OK, after installation you can run Linux box with sharing some folder from host OS. Create first some folder, for example, c:\vm\shared. Then run qemu:

qemu -m 256 -hda c:\vm\linux.img -L c:\qemu\bios -hdb fat:rw:/vm/shared

QEmu creates virtual drive (hdb) from host folder "c:\vm\shared", and you can mount it in Linux box:

su
mkdir /shared
mount -f vfat /dev/sdb1 /shared

"Read-Write" option in qemu is dangerous, so more preferable is to run qemu without "write" mode. For example, in current (0.13.0) version of Windows qemu build there is error: if you add some text to file in /shared, this file will be deleted (on host, yeh:). Use it for read-only access, and never modify shared folder on host when it is mounted in the guest OS.

Have a fun!