Macbook Keyboard Tips for Windows users

Original Post - https://www.macworld.co.uk/how-to/mac/keyboard-shortcuts-mac-users-copy-paste-option-shortcuts-3504584/

The three most important keys on your Mac can be found to the left and right of the spacebar (for right and left handed use). Unfortunately these three keys seem to cause more confusion than any others.

Using the Option or Alt key on a Mac

There is a great deal of confusion over what Apple referrs to as the Option key. If you'are using a UK keyboard chances are this is called the Alt key so it's no wonder most people don't know where it is.
The Alt (aka Option) key can be found between Control and Command. It has an icon that looks like a slope and a dip with a line above it.
Chances are the first time you hear mention of Option you are following a tutorial and trying to fix something on your Mac. The Alt key is the one you use if you wish to select a boot partition when starting the computer, you also press it when typing certain characters on your keyboard, such as # (Alt-3) or ¢ (Alt-4).
Here's an overview of the hidden characters that you can type using Alt (the keys might be a bit different if you aren't using a UK keyboard as we are).

You may be wondering whether you can use the Alt key ctrl-alt-delete to shut down an unresponsive Mac. Force quitting on a Mac is slightly different to on a PC, here's how to ctrl-alt-delete on a Mac, aka Force Quit on a Mac.

How to type €, #, … and © on a Mac

  • Alt-2 - Euro sign (€)
  • Alt-3 - Hash sign (#) (Sometimes called the 'pound' sign)
  • Alt-: - ellipsis (…)
  • Alt-G – copyright ©

You can also use Alt/Option to do the following:
  • Control-Alt-Command-Power Button - Quit all apps
  • Alt-Shift-Command-Q - Log out of your user account on your Mac
  • Alt-Delete - delete the word to the left of the curser
  • Alt–Left Arrow - move the curser to the beginning of the previous word, add Shift to this to highlight the text
  • Alt–Right Arrow - move the curser to the end of the next word, add Shift to this to highlight the text
  • If you are selecting large sections of text, you can do so by moving the curser to the end of the section you wish to select and pressing Alt-Shift-Up Arrow until all the text is selected. (This only works in some apps).
  • Similarly, Alt-Shift-Down Arrow lets you highlight the text below the curser
  • Alt-Command-F will open the Find and Replace feature if your application has it
  • Alt-Command-T will show or hide the toolbar
  • Alt-Command-C is the key combo to use if you wish to copy a style, or copy the formatting settings to the clipboard
  • And Alt-Command-V will paste those formatting settings onto the text you wish to change
  • Alt-Shift-Command-V will paste and match style - so that the text you paste in has the same style as the text around it, rather
  • than the style bought over from the place you copied it from
  • Alt-Command-D will show or hide the Dock at the bottom of your screen
  • In the Finder, Alt-Command-L is a handy shortcut to open the Downloads folder
  • Also in the Finder, pressing Alt-Command-P will show the path so you can see the precise location of what you are looking at
  • Alt-Command-S will show or hide the Sidebar in the Finder
  • Alt-Command-N will start a new Smart Folder in the Finder
  • If you select a few files in the Finder, you can press Alt-Command-Y to see a full-screen slideshow of those files
  • A shortcut to the Display preferences is to press Alt-Brightness Up (or Brightness Down, aka F1 or F2)
  • While you can open Mission Control preferences by pressing Alt-Mission Control (F3)
  • To duplicate/copy an item in the Finder or on your Desktop, press Alt while dragging it.
  • To create an Alias (a shortcut to a file) you press Alt and Command together while dragging the file from the location in the
  • Finder to another location, an arrow sign will appear indicating that this is a link to the file rather than a copy of it

Using the Command key on a Mac

If you thought that the jumbling of Alt and Option was baffling, there's even more opportuity for confusion when it comes to the Command key. The Command key (cmd) has a legacy that leads to confusion - many older Mac users will refer to it as the Apple key, because in the past there used to be an Apple logo on it, but this logo stopped appearing a while ago when if was decided that there were a few too many Apple logos on Apple products.
The logo you will still find on this key looks like a squiggly square, or a four petalled flower. It was designed by Susan Kare for the original iMac (and based on the Scandinavian icon for place of interest).

The Command (cmd) key works in a similar way to the Control key on a PC. On a Mac you use the Command key where on a PC you would use Control (or Ctrl).
If you were wondering why Ctrl-B didn't make your text bold, chances are you were previously a PC user and didn't realise that Command is the new Control. You might find this useful: How to move from PC to Mac: Complete guide to switching to a Mac from a PC.
Here are a few of the key combinations that use Command:
  • Command-B - Bold
  • Command-I - Italic
  • Command-Z - Undo
  • Command-Q - Quit
  • Command-W - Close window
  • Command-P - Print
  • Shift-Command-P - Page setup (for checking how it will print)
  • Command-S - Save
  • Shift-Command-S - Save As or duplicate the document
  • Command-A - select all

How to copy and paste on a Mac

  1. Select the text you wish to copy - a quick way to do this is to place your mouse pointer over a word and click twice. Once you have the text selected you can drag your mouse across or up, or down, to select more words. Alternatively, if you are selecting a number of words or sentences, or paragraphs, you can click at the beginning of the section, then press the Shift key, and click at the end of the section.
  2. Press Command-C to copy the text (or Command-X if you want to 'cut' the text from where it is currently)
  3. Go to where you wish to Paste the text in and press Command-V
  • Command-C = Copy
  • Command-X = Cut
  • Command-V = Paste

There are lots more useful key combinations that use Command including:
  • Command-F - Find 
  • Command-G - Find again
  • Command-H - Hide the windows of the app you are using
  • Command-M - Minimise the current window and send it to the Dock
  • Command-N - open a New document
  • Command-W - close the current window
  • Command-Space Bar - open the Spotlight search window
  • Command-Tab - to switch between open apps
  • Shift-Command-3 to take a screenshot of the screen (more about taking screenshots on a Mac here: How to take a screenshot on a Mac)
  • Command-Comma (,) - Open preferences for the app you are using
  • Command-T show or hide Fonts window
  • Command-Left Arrow -  move the curser to the beginning of the line
  • Command-Right Arrow -  move the curser to the end of the line
  • Command-Up Arrow -  move the curser to the beginning of the document
  • Command-Down Arrow -  move the curser to the end of the document (Press shift to select the text between the insertion point and the destination in each of these scenarios)
  • Command–Left Curly Bracket - Align Left
  • Command–Right Curly Bracket - Align Right
  • Shift-Command-| - Centre
  • Shift-Command-Minus sign - Decrease font size
  • Shift-Command-Plus sign - Increase font size
  • Shift-Command-Question mark - Open Help menu
  • If you are in the Finder or in a web browser, or any other app that supports Tabs, Command-T will open a new tab.
In the Finder you could try the following:
  • Command-D - Duplicate the file
  • Command-E - Eject the volumne
  • Command-F - Search
  • Command-I - Get Info
  • Shift-Command-D - Open the Desktop folder
  • Shift-Command-F - Open the All My Files folder
  • Shift-Command-H - Open the Home folder
  • Shift-Command-G - Open a Go To folder window
  • Shift-Command-I - Open your iCloud Drive
  • Command-K - Connect to the server
  • Shift-Command-K - Browse the network 
  • Command-L - Make an alias
  • Shift-Command-O - Open the Documents folder
  • Shift-Command-R - Shortcut to the AirDrop window
  • Command-Delete - sends the selected item to the Trash
  • Shift-Command-Delete - Empty the Trash (add the Alt key if you don't want to see the confirmation dialogue)

Using the Control key on a Mac

With the Command key doing the job of the Control key on a PC, what is the point of Control on a Mac keyboard, you may be wondering.
Content continues below
The most common use of Control is to mimic the right click on a mouse or when using the mouse pad (since some Apple mice don't have the right click option).
If you want more tips about How to right-click on a Mac read this.

There are many more uses for Control when used with other key combinations, for example:
  • Control-Command-Power button will restart your Mac
  • Control-Shift-Power button - puts your display to sleep
  • Control–Option–Command–Power button - quits all your apps and shuts your Mac
  • Control-H Delete the character on the left
  • Control-D Delete the character on the right
  • Control-K Delete the text from where your curser is to the end of the line
  • Control-A Move to the beginning of the line (more here: How to find End and Home on a Mac keyboard)
  • Control-E Move to the end of a line or paragraph
  • Control-F Move forward one character
  • Control-B Move backward one character
You can also use the Control key to add a document or folder to the Dock: Go to the Finder and select the item you wish to add to the Dock (or search for it using Spotlight: Cmd-Space, or select it on your Desktop). Then press Control-Shift-Command-T.

What do the F keys do on a Mac?

There are a few other Apple specific keys (depending on your keyboard):
F1/F2 - Brightness Up and Down
F3 – Mission Control (for an overview of all running applications, grouping windows from the same application, and your Spaces)
F4 - Is a shortcut to all the Apps you have on your Mac

F10/F11/F12 – Sound

You can set other F keys to do Mission Control actions. Go to System Preferences > Mission Control and add unused F keys to do functions such as Show Desktop or Dashboard.

How to type letters with accents on a Mac

Some letters can be typed with accents on top, like this é, ä, ö. This is easy to do on a Mac:
  1. Hold the letter down on the keyboard until a bubble menu with all the different options appears
  2. Each accent option has a number below it, tap the number on the keyboard to turn the letter into that accented version, or click on the accented letter with your mouse

How to type special characters, Emoji and maths symbols on a Mac

Emoji Characters on a Mac
You can use the Character Viewer to find special characters, Emoji and maths symbols.
Press Command-Control-Space and by default you will see the Emoji characters. To see the Character Viewer, with  special characters from any font on your Mac, click on the Character Viewer icon in the right corner of the window, or in the menu bar next to the time and date. 
Once you have the Character Viewer open, use the sidebar to view different categories, such as Currency Symbols or Maths Symbols and double-click any item in the main window to insert it into your document.
You can also search for any option using the Search field in the top-right. Enter a term like "cat" to find all the symbols that are cat-like.
The Character Viewer is placed permanently above all other windows, so you can continue typing in your app and view the Character Viewer on top of its document. You can switch between a small and large Character Viewer using the icon to the right of the Search Field.
It is also possible to add the Character Viewer as a Menu bar icon, this enables you to quickly access it from any app. Open System Preferences, choose Keyboard  > Keyboard and select the Show Keyboard & Character Viewers in Menu Bar option. Now you can click on the Character Viewer icon in the Menu bar and choose Show Character Viewer.

How do I add emoji on a Mac?

As we said above, just press Command-Control-Space and you will see a collection of Emoji you can use.
If you have the Character Viewer open you will find a section called Emoji in the sidebar.
Often it's easier to use the Search field in Character Viewer to find Emoji characters.
If you want to read more about using emoji this may be useful: How to use emoji

How to view shortcuts on a Mac

Use the Keyboard Viewer to learn special characters
One neat trick to learning keyboard shortcuts on a Mac is to use the Keyboard Viewer. Enable the Show Keyboard & Character Viewers in Menu Bar option (in System Preferences > Keyboard > Keyboard).
Now click on the Character Viewer icon in the Menu bar and choose Show Keyboard Viewer. A visual representation of the keyboard appears, and as you press keys they will be highlighted. If you hold down the Alt and Shift keys the Keyboard Viewer shows all the special characters on each key. You can use this to learn the special characters on each key.

Keyboard combinations for shutting down or Sleep your Mac

  • Ctrl-Eject = Show the restart / sleep / shutdown dialog
  • Shift-Control-Eject = Will put your displays to sleep
  • Command-Alt-Eject = Will put the computer to sleep
  • Command-Control-Eject = Save/Quit all applications then restarts Mac
  • Command-Alt-Control-Eject = Quit all applications then shuts down the Mac
  • Command-Shift-Q = Log out of your OS X user account (you'll be asked to confirm action)
  • Command-Shift-Alt-Q = Log out of your OS X user account immediately (you won't be asked to confirm action)
  • Command-Alt-esc = Force Quit
  • Command-shift-Alt-esc (for three seconds) = Force Quit the front-most application
Read next: How to lock a Mac

How to use the Application Switcher

Another handy key combo is the one that brings up the Application switcher. This is a handy way to move between different applications you have open.
  • Command-Tab = Move to the next most recently used application from your open applications
  • Command-Shift-Tab = Move backward through a list of open applications (sorted by recent use)
  • Command-~ (Tilde) = Move backward through a list of open applications (only when Application switcher is active)

List of Packages to Install for Anaconda


Anaconda - Package manager used to install packages and run python as a user.

This means that you can run things on server without sudo priveliges.


Steps to installation

Download Anaconda

Run bash <Anaconda.sh>

List of packages to Install are as follows

conda install -c groakat lame
conda install -c conda-forge -y sox libvorbis
conda install -c jjhelmus tensorflow
conda install -c conda-forge matplotlib
conda install numpy pyyaml mkl setuptools cmake cffi
conda install -c conda-forge tqdm
conda install -c conda-forge libsvm
conda install -c geneko pp
conda install pytorch torchvision cuda80 -c soumith

How to use a scheduler - Torque - QSUB


Check the status of your jobs
qstat

qstat - lists all the jobs on the system.

Check status of jobs for a user
qstat -u <username>

Get all details about a particular job
qstat -f <job id>


Delete a job from queue
qdel <job id> 

Numpy Tricks


How to print/display the number in full in an array?
import numpy as np
np.set_printoptions(threshold=np.nan)

Firefox 57 aka Firefox Quantum - The best browser so far from Firefox



Wow, I am amazed at the speed of this amazing browser - Firefox Quantum.

Firefox Quantum has a concept of loading the CSS engine and other changes which makes the browser a breeze to use with.



Firefox Quantum is releasing in November 2017. But why wait - Go ahead and try this browser out here - https://www.mozilla.org/en-US/firefox/channel/desktop/#beta

And for all languages the website links is here - https://www.mozilla.org/en-US/firefox/beta/all/

You will certainly consider leaving Google Chrome after trying the browser out!!!!


Reference:- http://www.zdnet.com/article/mozilla-firefox-57-is-so-fast-were-calling-it-firefox-quantum/

Install Octave with GUI 4.2.1 in Mac OS - Sierra or High Sierra


Hi everyone,

Today, we are going to review how to install Octave 4.2.1 in Mac OS Sierra or High Sierra.

Octave installation with GUI using MacPorts is successful.

How to Install MacPorts - [Refer steps on official MacPorts page ] - https://www.macports.org/install.php

Installation of Octave :-

sudo port selfupdate

sudo port install octave

Use Octave with GUI


What didnt work : -

I tried installing Octave using HomeBrew and it didnt work with GUI. Normal Octave using command line works.

Octave installation which Did not work on GUI :-

brew tap homebrew/science

brew update && brew upgrade

brew install octave

I also tried using

brew reinstall octave --with-qt --with-fltk --with-gui

Which also didnt load using GUI

Commands Reference - http://wiki.octave.org/Octave_for_macOS

Sox of Silence - Original post - http://digitalcardboard.com/blog/2009/08/25/the-sox-of-silence/

The SoX of Silence


SoX is, by their own definition, the Swiss Army knife of audio manipulation.
And no doubt it’s full of fun with slicing and dicing and playback and recording and filtering and effects capabilities.
But SoX is a command line tool, which means obscure syntax and parameters in order to get things done.
I’ve been trying off and on for months to try to understand the silence filter from within SoX, which allows one to remove silence from the beginning, middle, or end of the audio. Sounds, simple, doesn’t it?  Well, it should be.
silence [-l] above-periods [duration threshold[d|%] [below-periods duration threshold[d|%]]
Removes silence from the beginning, middle, or end of the audio. Silence is anything below a specified threshold.
The above-periods value is used to indicate if audio should be trimmed at the beginning of the audio. A value of zero indicates no silence should be trimmed from the beginning. When specifying an non-zero above-periods, it trims audio up until it finds non-silence. Normally, when trimming silence from beginning of audio the above-periods will be 1 but it can be increased to higher values to trim all audio up to a specific count of non-silence periods. For example, if you had an audio file with two songs that each contained 2 seconds of silence before the song, you could specify an above-period of 2 to strip out both silence periods and the first song.
When above-periods is non-zero, you must also specify a duration and threshold. Duration indications the amount of time that non-silence must be detected before it stops trimming audio. By increasing the duration, burst of noise can be treated as silence and trimmed off.
Threshold is used to indicate what sample value you should treat as silence. For digital audio, a value of 0 may be fine but for audio recorded from analog, you may wish to increase the value to account for background noise.
When optionally trimming silence from the end of the audio, you specify a below-periods count. In this case, below-period means to remove all audio after silence is detected. Normally, this will be a value 1 of but it can be increased to skip over periods of silence that are wanted. For example, if you have a song with 2 seconds of silence in the middle and 2 second at the end, you could set below-period to a value of 2 to skip over the silence in the middle of the audio.
For below-periods, duration specifies a period of silence that must exist before audio is not copied any more. By specifying a higher duration, silence that is wanted can be left in the audio. For example, if you have a song with an expected 1 second of silence in the middle and 2 seconds of silence at the end, a duration of 2 seconds could be used to skip over the middle silence.
Unfortunately, you must know the length of the silence at the end of your audio file to trim off silence reliably. A work around is to use the silence effect in combination with the reverse effect. By first reversing the audio, you can use the above-periods to reliably trim all audio from what looks like the front of the file. Then reverse the file again to get back to normal.
To remove silence from the middle of a file, specify a below-periods that is negative. This value is then treated as a positive value and is also used to indicate the effect should restart processing as specified by the above-periods, making it suitable for removing periods of silence in the middle of the audio.
The option -l indicates that below-periods duration length of audio should be left intact at the beginning of each period of silence. For example, if you want to remove long pauses between words but do not want to remove the pauses completely.
The period counts are in units of samples. Duration counts may be in the format of hh:mm:ss.frac, or the exact count of samples. Threshold numbers may be suffixed with d to indicate the value is in decibels, or % to indicate a percentage of maximum value of the sample value (0% specifies pure digital silence).
The following example shows how this effect can be used to start a recording that does not contain the delay at the start which usually occurs between `pressing the record button’ and the start of the performance:
rec parameters filename other-effects silence 1 5 2%
Huh?
So lets try to clarify some of the mess from the man page.  First a couple of important notes:
  • When specifying duration, use a trailing zero for whole numbers of seconds (ie, 1.0 instead of 1 to specify 1 second). If you don’t, SoX assumes you’re specifying a number of samples.  Who on earth would want to specify samples instead seconds? You got me. Alternatively, you can specify durations of time in the format hh:mm:ss.frac.
  • Use at 0.1% at a minimum for an audio threshold. Even though 0% is supposed to be pure digital silence, with my test file I couldn’t get silence to trim unless I used a threshold larger than 0%. If you’d like, you can specify the threshold in decibels using d (such as -96d or -55d).
  • The realistic values for the above-period parameter are 0 and 1 and values for the below-period parameter are pretty much just -1 and 1. The documentation states that values larger than 1 can be used, but it only really makes sense for files with consistent audio breaks. Just trust me, it’s weird. I’ll get into what those values actually mean in the examples.
Now onto some examples! I’ll be showing you visually what happens to a sound file when we apply the various parameters to the silence filter.
I generated a test sound file with 60 seconds of white noise and then silenced various parts of the clip, leaving me with an audio file that looks like this:
SoX Silence Example (Original File)

Example 1: Trimming silence at the beginning

sox in.wav out1.wav silence 1 0.1 1%
The above-period parameter is first after the silence parameter, and for the sake of this article, it should be set to 1 if you want to use the filter. This example roughly translates to: trim silence (anything less than 1% volume) until we encounter sound lasting more than 0.1 seconds in duration. The output of this command produces the following:
sox in.wav out1.wav silence 1 0.1 1%
We’ve lopped off the silence at the beginning of the clip. For simplicity’s sake, we’ll refer to the 1% threshold as silence from now on.

Example 2: Ignoring noise bursts

sox in.wav out2.wav silence 1 0.3 1%
By changing the duration parameter to 0.3, we tell SoX to ignore the burst of noise at the beginning of the example clip. This produces the following:
sox in.wav out2.wav silence 1 0.3 1%
We can ignore short pops and clicks in audio by adjusting this duration parameter.

Example 3: Stopping recording when no sound detected

sox in.wav out3.wav silence 1 0.3 1% 1 0.3 1%
Now we introduce the below-period parameter it’s respective sub-parameters.  Just like the above-period parameter, just set it to 1 and call it good.  The command above translates to: trim silence until we detect at least 0.3 seconds of noise, and then trim everything after we detect at least 0.3 seconds of silence.
sox in.wav out3.wav silence 1 0.3 1% 1 0.3 1%
This returns a file with just the first 4 seconds of noise (note that we ignore that 0.25 sec burst of noise at the beginning). Where’s the rest of the clip?  Well, it’s gone. Not super practical for post-production of audio, but can be useful when recording live audio, so that SoX stops when it doesn’t encounter sound for a certain number of seconds.
So an aside: if you’re looking to trim silence from the beginning and the end of a audio file, you’ll need to utilize the reverse filter and a temp file like so:
sox in.wav temp.wav silence 1 0.1 1% reverse
sox temp.wav out.wav silence 1 0.1 1% reverse
Don’t forget to delete that temp.wav file when you’re done.
Jakob points out in the comments that you can trim silence from both ends in one fell swoop by chaining the effects like so:
sox in.wav out.wav silence 1 0.1 1% reverse silence 1 0.1 1% reverse

Example 4: Trimming all silence

sox in.wav out4.wav silence 1 0.1 1% -1 0.1 1%
By changing the below-period parameter to -1, we can trim instances of silence in the middle of the clip, by allowing the filter to restart after it detects noise of the specified duration.
sox in.wav out4.wav silence 1 0.1 1% -1 0.1 1%
In my example clip, it’s impossible to detect where the silence used to be, but with an actual podcast or other audio, it should be easier to tell.

Example 5: Ignoring short periods of silence

sox in.wav out5.wav silence 1 0.1 1% -1 0.5 1%
In similar fashion as Example 2, we can instruct SoX to ignore small moments of silence (1/2 second in this example).
sox in.wav out5.wav silence 1 0.1 1% -1 0.5 1%
When trimming silence from podcasts and the like, this prevents you from removing moments when someone stops to take a breath and making the conversation sound too rushed.

Example 6: Shortening long periods of silence

sox in.wav out6.wav silence -l 1 0.1 1% -1 2.0 1%
So what if you wanted to just shorten long moments of silence rather than remove them entirely?  Well, you need to add the -l parameter, but it needs to be placed first, before the other parameters for the filter effect. The example above results in trimming all silence longer than 2 seconds down to only 2 seconds long.
sox in.wav out6.wav silence -l 1 0.1 1% -1 2.0 1%
Note that SoX does nothing to bits of silence shorter than 2 seconds.

Example 7: Shortening long periods of silence and ignoring noise bursts

sox in.wav out7.wav silence -l 1 0.3 1% -1 2.0 1%
Finally, let’s tie it all together by trimming silence longer than 2 seconds down to 2 seconds long, but ignore noise such as pops and clicks amidst the moments of silence.
sox in.wav out7.wav silence -l 1 0.3 1% -1 2.0 1%
As a result you’ll see that we’ve cropped out the 0.25 seconds of noise at the beginning of the clip, but left the 0.5 seconds of noise in the middle.
For actual usage, you’ll probably want to specify something shorter than 0.3 seconds for the duration if you’re just trying to filter out pops and clicks.

Bonus Example 8: Splitting audio based on silence

sox in.wav out.wav silence 1 0.5 1% 1 5.0 1% : newfile : restart
Using SoX’s newfile pseudo-effect allows us to split an audio file based on periods of silence, and then calling restart starts the effects chain over from the beginning. In this example, SoX will split audio when it detects 5 or more seconds of silence. You’ll end up with output files named out001.wavout002.wav, and so on.

Final Thoughts

There you have it.  This is what I know about the silence filter effect in SoX.  Example 7–where we trim some but not all of the silence and ignore pops and clicks–is ultimately what I was trying to figure out when writing this article, but I figure the other examples have got to be a good reference for somebody me.
The above and below-period values are still mostly a mystery to me.  I may address them in another post, but for now, I’m just going to use this as a cheat sheet in case I forget.
And don’t forget to use the trailing zero when specifying whole seconds. Even while writing this I forgot multiple times.
I welcome thoughts, ideas, comments, and corrections. Please.

Find number of files in all subdirectories


Here's a simple command in bash to do the same.

for f in *; do [ -d ./"$f" ] && find ./"$f" -maxdepth 1 -exec echo \; | wc -l && echo $f; done

Works with all file names. Enjoy :)


Below commands are picked from the stackoverflow post:-
http://unix.stackexchange.com/questions/4105/how-do-i-count-all-the-files-recursively-through-directories


List folder with filecount

find -maxdepth 1 -type d | sort | while read -r dir; do n=$(find "$dir" -type f | wc -l); printf "%4d : %s\n" $n "$dir"; done

for f in */*; do echo "$f -> $(ls $f | wc -l)"; done

List folder with Zero File Count

find -maxdepth 1 -type d | sort | while read -r dir; do n=$(find "$dir" -type f | wc -l); if [ $n -gt 0 ]; then printf "%4d : %s\n" $n "$dir"; fi; done

List folder with subfolder count

find -maxdepth 1 -type d | sort | while read -r dir; do n=$(find "$dir" -type d | wc -l); let n--; printf "%4d : %s\n" $n "$dir"; done

List folder with nonzero subfolder count 

find -maxdepth 1 -type d | sort | while read -r dir; do n=$(find "$dir" -type d | wc -l); let n--; if [ $n -gt 0 ]; then printf "%4d : %s\n" $n "$dir"; fi; done

List Empty folder

find -maxdepth 1 -type d | sort | while read -r dir; do n=$(find "$dir" | wc -l); let n--; if [ $n -eq 0 ]; then printf "%4d : %s\n" $n "$dir"; fi; done

List non-empty folder with content count 

find -maxdepth 1 -type d | sort | while read -r dir; do n=$(find "$dir" | wc -l); let n--; if [ $n -gt 0 ]; then printf "%4d : %s\n" $n "$dir"; fi; done

How to use awk tool.

Hi all,

Today, through this post I am going to talk about some of the basics of the awk tool and real life case scenarios where this tool is very useful.

awk -F, '{sum+=NF} END {print sum}' <csv file> - Counts the total number of comma separated values in the csv file.

Awk command to calculate sum of floating point numbers in a column stored in a file.

awk '{sum+=sprintf("%f",$1)}END{printf "%.6f\n",sum}' <filename>
OR
LC_NUMERIC="C" awk '{sum += $1} END {print sum}' <filename>


How to add line numbers using awk command?

awk '{printf("%1d,%s\n", NR,$0)}' filename > filename_numbered

How to use awk to print from the 1st column to the last column?

awk '{out=""; for(i=2;i<=NF;i++){out=out" "$i}; print out}'


Print first few characters of a variable with awk?

 awk '{print substr($1,2,11)}' file

This will print second to 11th character in the variable $1. 

Kaggle download data

There might be instances where you might need to download a Kaggle dataset to another machine, possibly in an amazons ec2 instance.
In order to download the data, you need to be logged in.
One solution is to export your cookies and tell wget to use your cookies when downloading the data
This is a chrome extension you can use to export your cookies.
And here is the command in wget to download the data
wget -x -c --load-cookies cookies.txt -P data -nH --cut-dirs=5 https://www.kaggle.com/c/diabetic-retinopathy-detection/download/trainLabels.csv.zip
So, the above command will use your exported cookies to download the file to a folder called data

SQL Database Basic Commands


1. Create a SQLite Database (and a Table)

First, let us understand how create a SQLite database with couple of tables, populate some data, and view those records.

The following example creates a database called company.db. This also creates an employee table with 3 columns (id, name and title), and a department table in the company.db database. We’ve purposefully missed the deptid column in the employee table. We’ll see how to add that later.

# sqlite3 company.db
sqlite> create table employee(empid integer,name varchar(20),title varchar(10));
sqlite> create table department(deptid integer,name varchar(20),location varchar(10));
sqlite> .quit
Note: To exit from the SQLite commandline “sqlite>” prompt, type “.quit” as shown above.

A SQLite database is nothing but a file that gets created under your current directory as shown below.

# ls -l company.db
-rw-r--r--. 1 root root 3072 Sep 19 11:21 company.db
2. Insert Records

The following example populates both employee and department table with some sample records.

You can execute all the insert statements from the sqlite command line, or you can add those commands into a file and execute the file as shown below.

First, create a insert-data.sql file as shown below.

# vi insert-data.sql
insert into employee values(101,'John Smith','CEO');
insert into employee values(102,'Raj Reddy','Sysadmin');
insert into employee values(103,'Jason Bourne','Developer');
insert into employee values(104,'Jane Smith','Sale Manager');
insert into employee values(105,'Rita Patel','DBA');

insert into department values(1,'Sales','Los Angeles');
insert into department values(2,'Technology','San Jose');
insert into department values(3,'Marketing','Los Angeles');
The following will execute all the commands from the insert-data.sql in the company.db database

# sqlite3 company.db < insert-data.sql
3. View Records

Once you’ve inserted the records, view it using select command as shown below.

# sqlite3 company.db
sqlite> select * from employee;
101|John Smith|CEO
102|Raj Reddy|Sysadmin
103|Jason Bourne|Developer
104|Jane Smith|Sale Manager
105|Rita Patel|DBA

sqlite> select * from department;
1|Sales|Los Angeles
2|Technology|San Jose
3|Marketing|Los Angeles
4. Rename a Table

The following example renames department table to dept using the alter table command.

sqlite> alter table department rename to dept;
5. Add a Column to an Existing Table

The following examples adds deptid column to the existing employee table;

sqlite> alter table employee add column deptid integer;
Update the department id for the employees using update command as shown below.

update employee set deptid=3 where empid=101;
update employee set deptid=2 where empid=102;
update employee set deptid=2 where empid=103;
update employee set deptid=1 where empid=104;
update employee set deptid=2 where empid=105;
Verify that the deptid is updated properly in the employee table.

sqlite> select * from employee;
101|John Smith|CEO|3
102|Raj Reddy|Sysadmin|2
103|Jason Bourne|Developer|2
104|Jane Smith|Sale Manager|1
105|Rita Patel|DBA|2
6. View all Tables in a Database

Execute the following command to view all the tables in the current database. The folowing example shows that there are two tables in the current database.

sqlite> .tables
dept      employee
7. Create an Index

The following example creates an unique index called empidx on the empid field of employee table.

sqlite> create unique index empidx on employee(empid);
Once an unique index is created, if you try to add another record with an empid that already exists, you’ll get an error as shown below.

sqlite> insert into employee values (101,'James Bond','Secret Agent',1);
Error: constraint failed
8. Create a Trigger

For this example, first add a date column called “updatedon” on employee table.

sqlite> alter table employee add column updatedon date;
Next, create a file that has the trigger definition. The following trigger will update the “updatedon” date column with the current timestamp whenever you perform an update on this table.

# vi employee_update_trg.sql
create trigger employee_update_trg after update on employee
begin
  update employee set updatedon = datetime('NOW') where rowid = new.rowid;
end;
Create the trigger on the company.db database as shown below.

# sqlite3 company.db < employee_update_trg.sql
Now anytime you update any record in the employee table, the “updatedon” date column will be updated with the current timestamp as shown below. The following example updates the “updatedon” timestamp for empid 104 through trigger.

# sqlite3 company.db
sqlite> update employee set title='Sales Manager' where empid=104;

sqlite> select * from employee;
101|John Smith|CEO|3|
102|Raj Reddy|Sysadmin|2|
103|Jason Bourne|Developer|2|
104|Jane Smith|Sales Manager|1|2012-09-15 18:29:28
105|Rita Patel|DBA|2|
9. Create a View

The following example creates a view called “empdept”, which combines fields from both employee and dept table.

sqlite> create view empdept as select empid, e.name, title, d.name, location from employee e, dept d where e.deptid = d.deptid;
Now you can execute select command on this view just like a regular table.

sqlite> select * from empdept;
101|John Smith|CEO|Marketing|Los Angeles
102|Raj Reddy|Sysadmin|Technology|San Jose
103|Jason Bourne|Developer|Technology|San Jose
104|Jane Smith|Sales Manager|Sales|Los Angeles
105|Rita Patel|DBA|Technology|San Jose
After creating a view, if you execute .tables, you’ll also see the view name along with the tables.

sqlite> .tables
dept      empdept   employee
10. SQLite Savepoint, Rollback, Commit

Currently dept table has the following 3 records.

sqlite> select * from dept;
1|Sales|Los Angeles
2|Technology|San Jose
3|Marketing|Los Angeles
Now, create a savepoint called “major”, and perform some transactions on the dept table. As you see below, we’ve added two records, deleted one record, after creating a savepoint called “major”.

sqlite> savepoint major;
sqlite> insert into dept values(4,'HR','Los Angeles');
sqlite> insert into dept values(5,'Finance','San Jose');
sqlite> delete from dept where deptid=1;
sqlite> select * from dept;
2|Technology|San Jose
3|Marketing|Los Angeles
4|HR|Los Angeles
5|Finance|San Jose
Now for some reason, if we don’t want the above transactions, we can rollback the changes to a particular savepoint. In this example, we are rolling back all the changes we’ve made after the “major” savepoint.

sqlite> rollback to savepoint major;
sqlite> select * from dept;
1|Sales|Los Angeles
2|Technology|San Jose
3|Marketing|Los Angeles
If you don’t want your savepoints anymore, you can erase it using release command.

sqlite> release savepoint major;
11. Additional Date Functions

By default, the date columns values displayed in UTC time. To display in the local time, use the datetime command on the date column as shown below.

sqlite> select empid,datetime(updatedon,'localtime') from employee;
104|2012-09-15 11:29:28
You can also use strftime to display the date column in various output.

sqlite> select empid,strftime('%d-%m-%Y %w %W',updatedon) from employee;
104|19-09-2012 3 38
The following are the possible modifers you can use in the strftime function.

%d day of month: 00
%f fractional seconds: SS.SSS
%H hour: 00-24
%j day of year: 001-366
%J Julian day number
%m month: 01-12
%M minute: 00-59
%s seconds since 1970-01-01
%S seconds: 00-59
%w day of week 0-6 with Sunday==0
%W week of year: 00-53
%Y year: 0000-9999
%% %
12. Dropping Objects

You can drop all the above created objects using the appropriate drop command as shown below.

Since we are dropping objects for testing purpose, copy the company.db to a test.db and try these commands on the test.db

# cp company.db test.db

# sqlite3 test.db
sqlite> .tables
dept      empdept   employee

sqlite> drop index empidx;
sqlite> drop trigger employee_update_trg;
sqlite> drop view empdept;
sqlite> drop table employee;
sqlite> drop table dept;
All the tables and views from the test.db are now deleted.

sqlite> .tables
sqlite>
Note: When you drop a table all the indexes and triggers for that table are also dropped.

13. Operators

The following are the possible operators you can use in SQL statements.

||
* / %
+ –
<< >> & |
< >=
= == != <> IS IS NOT IN LIKE GLOB MATCH REGEXP
AND OR
For example:

sqlite> select * from employee where empid >= 102 and empid  select * from dept where location like 'Los%';
1|Sales|Los Angeles
3|Marketing|Los Angeles
14. Explain Query Plan

Execute “explain query plan”, to get information about the table that is getting used in a query or view. This is very helpful when you are debugging a complex query with multiple joins on several tables.

sqlite> explain query plan select * from empdept;
0|0|TABLE employee AS e
1|1|TABLE dept AS d
For a detailed trace, just execute “explain” followed by the query to get more performance data on the query. This is helpful for debugging purpose when the query is slow.

sqlite> explain select empid,strftime('%d-%m-%Y %w %W',updatedon) from employee;
0|Trace|0|0|0||00|
1|Goto|0|12|0||00|
2|OpenRead|0|2|0|4|00|
3|Rewind|0|10|0||00|
4|Column|0|0|1||00|
5|String8|0|3|0|%d-%m-%Y %w %W|00|
6|Column|0|3|4||00|
7|Function|1|3|2|strftime(-1)|02|
8|ResultRow|1|2|0||00|
9|Next|0|4|0||01|
10|Close|0|0|0||00|
11|Halt|0|0|0||00|
12|Transaction|0|0|0||00|
13|VerifyCookie|0|19|0||00|
14|TableLock|0|2|0|employee|00|
15|Goto|0|2|0||00|
15. Attach and Detach Database

When you have multiple database, you can use attach command to execute queries across database.

For example, if you have two database that has the same table name with different data, you can create a union query across the database to view the combined records as explained below.

In this example, we have two company database (company1.db and company2.db). From the sqlite prompt, attach both these database by giving alias as c1 and c2 as shown below.

# sqlite3
sqlite> attach database 'company1.db' as c1;
sqlite> attach database 'company2.db' as c2;
Execute “.database” command which will display all the attached databases.

sqlite> .database
seq  name             file
---  ---------------  ------------------
0    main
2    c1               /root/company1.db
3    c2               /root/company2.db
Now, you can execute a union query across these databases to combine the results.

sqlite> select empid, name, title from c1.employee union select empid, name, title from c2.employee;
101|John Smith|CEO
102|Raj Reddy|Sysadmin
103|Jason Bourne|Developer
104|Jane Smith|Sales Manager
105|Rita Patel|DBA
201|James Bond|Secret Agent
202|Spider Man|Action Hero
After attaching a database, from the current sqlite session, if you want to detach it, use detach command as shown below.

sqlite> detach c1;

sqlite> .database
seq  name             file
---  ---------------  -----------------
0    main
2    c2               /root/company2.db

Original Post : - http://www.thegeekstuff.com/2012/09/sqlite-command-examples/

Install NVIDIA and CUDA in Linux version without any problems.


Hi all,

If you are reading this post means you wish to run CUDA environment and has  tried installing CUDA with less success or running cuda for the first time.
Dont worry I have got all covered in this blog post.

To Install CUDA and run NVIDIA drivers follow the below steps: -

  1. Find the version of Linux installed using uname -r command or cat /etc/*-release command
  2. Find already existing default drivers from NVIDIA - sudo apt-cache search nvidia
  3. Install the building dependencies first using sudo apt-get install linux-headers-$(uname -r)
  4. Install the nvidia driver For example use always one driver version before latest to be on the safe side. So if you plan to install nvidia-352 driver use the command : - sudo apt-get install nvidia-352
  5. Ensure that novaeu driver is black listed.
  6. Reboot the pc using command sudo reboot
  7. Now, launch the terminal and type nvidia-smi => this should display an output something like nvidia driver version and the jobs/process running as no process running in the output in terminal
  8. Congratulations, now you have nvidia driver installed properly
  9. Download CUDA 8.0RC. In order to download the installers, you need to login or register(free) first. Choose Linux -> x86_64 -> Ubuntu -> 16.04 -> runfile(local). Download both Base Installer and Patch 1. Run command:
  10. Download link is
    wget http://developer.download.nvidia.com/compute/cuda/*
    sudo sh cuda_7.5.27_linux.run
  11. Set up the development environment by modifying the PATH and LD_LIBRARY_PATH variables, also add them to the end of the .bashrc file 
  12. export PATH=/usr/local/cuda-7.5/bin${PATH:+:${PATH}} 
  13. export LD_LIBRARY_PATH=/usr/local/cuda-7.5/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH} 
  14. Source the bashrc file using source ~/.bashrc 
  15. Now type nvcc -V to check that Cuda is installed or not
  16. Install cudnn library : - Download cudnn library from NVIDIA website:  -  https://developer.nvidia.com/cudnn
  17. tar -zxvf cudnn-7.5-linux-x64-v5.1.tgz
  18. cd cuda/
  19. sudo cp include/* /usr/local/cuda-7.5/include/
  20. sudo cp lib64/* /usr/local/cuda-7.5/lib64/
  21. sudo chmod a+r /usr/local/cuda-7.5/lib64/libcudnn*
  22. Check whether cudnn is installed now or not
    1. cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2 
  23. If the above command does not give any error - congratulations you have installed cudnn. 
Stay tuned for further updates

List of useful websites every researcher must know about


Hi all,

I am sure that everyone who does research is wondering which are the websites I should be know about to be more involved in the research community.

Publishing papers is one skill and encouraging researchers to read your published papers requires some level of work and making your work as public.
After all, a research paper which is not being read has a low impact as compared to papers which are read and one that has practical importance in the community.

Here's a list of some of the websites researchers should know about: -

Search for relevant Papers - Literature Survey :
  1. Google Scholar : - https://scholar.google.co.in/  
  2. IEEE Xplore - http://ieeexplore.ieee.org/Xplore/home.jsp
  3. Academic Search Engine - Refseek - http://www.refseek.com/
  4. iSeek - http://education.iseek.com/iseek/home.page
  5. ACM papers - http://dl.acm.org/
  6. Microsoft Academic Research - http://academic.research.microsoft.com/
  7. Open Science Framework - https://osf.io/
  8. IRIS AI - Link research papers -  https://iris.ai/
  9. Life Sciences Papers Search Engine - pubget.com  
  10. Search Engine for Innnovation - https://en.expernova.com/
  11. Scopus - https://www.scopus.com/home.uri  
  12. Semantic Scholar - https://www.semanticscholar.org/
  13. Science Direct - http://www.sciencedirect.com/
  14. Visual Arxiv Paperscape - http://paperscape.org/
  15. ResearchPad - http://researchpad.co/
  16. CiteSeerX - http://citeseerx.ist.psu.edu/
  17. Paperity - http://paperity.org/
  18. JSTOR - http://www.jstor.org/
  19. AcademicIndex - http://www.academicindex.net/
  20. Machine learning papers - JMLR - http://www.jmlr.org/
  21. Base - https://www.base-search.net
  22. MIT research - http://web.mit.edu/research/
  23. Google Research - https://research.google.com/
  24. Arxiv - Sanity Check - http://www.arxiv-sanity.com/
  25. MIT research open library access - https://dspace.mit.edu
Preprint Servers : -
  1. Arxiv : -  https://arxiv.org/
  2. Biology Preprint - http://biorxiv.org/
  3. HAL - https://hal.archives-ouvertes.fr/
  4. AcademicKarma - http://academickarma.org/ 

Research Profile : -
  1. Research Gate : - https://www.researchgate.net/
  2. Academia Edu : - https://www.academia.edu/
  3. Sciforum - http://sciforum.net/
  4. Researcher ID - http://www.researcherid.com
  5. ORCID - http://orcid.org/
  6. Scholar Bridge - http://www.scholarbridge.com/  
  7. Vivo - http://vivoweb.org/
  8. Mysciencework - https://www.mysciencework.com/
  9. Labroots - https://www.labroots.com/

Online Useful Tools : -
  1. Drawings and figures - draw.io - https://www.draw.io/
  2. Overleaf - Latex Coding - https://www.overleaf.com/
  3. ShareLatex - https://www.sharelatex.com/
  4. Quip - https://quip.com/
  5. Google Docs - https://docs.google.com
  6. Google Charts - https://developers.google.com/chart/
  7. Hivebench - https://www.hivebench.com/
  8. Plotly - https://plot.ly/
  9. Datahero - https://datahero.com/
  10. CiteULike - http://www.citeulike.org/
DataSet Search and Download: -
  1. Zenodo - https://zenodo.org/
  2. Academic Torrents - http://academictorrents.com/
  3. Mendeley Data - https://data.mendeley.com/
  4. Dataverse - http://dataverse.org/
  5. Databrary - https://nyu.databrary.org/
  6. Science Media - http://www.science-media.org/
  7. Registry of research data repository - http://www.re3data.org/
  8. Kaggle Datasets - https://kaggle.com/datasets
Paper Ratings:-

Reduced Price Paper Access: -
  1. DeepDyve - https://www.deepdyve.com/
  2. Hinari [Health Journals] - http://extranet.who.int/hinari/en/journals.php 
  3. Research4life - http://www.research4life.org/
Nice website collection: -

Check this page back as I will keep updating the list.
Upvote on google plus if useful and share the post with other researchers.

Thank you for your time.