Why This Guide Matters

• Time Savings: Eliminate 100+ hours of research and trial-and-error
• Strategic Decisions: Choose tools that scale with your growth
• Cost Optimization: Balance open-source vs commercial solutions
• Future-Proofing: Select technologies with strong community momentum
• Integration Success: Build cohesive ecosystems rather than tool silos

1. Programming Languages

Programming languages form the foundation of your entire tech stack. The right choice impacts everything from development speed to performance to hiring capabilities.

Key Trends

• Rust and TypeScript show strongest growth momentum
• Python dominates data science and ML ecosystems
• JavaScript ecosystem continues fragmenting (Node.js, Deno, Bun)
• Go increasingly preferred for cloud-native applications

Recommendations

• Web Development: TypeScript, JavaScript, Go
• Data Science: Python, R, Julia
• Mobile: Swift (iOS), Kotlin (Android), Dart (Flutter)
• Systems Programming: Rust, C++, Go
• Enterprise: Java, C#, TypeScript

2. Front End Development

Front-end development has evolved dramatically from simple HTML/CSS to complex component-based architectures with build tools, state management, and optimization requirements.

Key Trends

• React maintains dominance but faces competition from Vue.js and Svelte
• TypeScript adoption exceeds 80% for new enterprise projects
• Build tools: Vite rapidly displaces Webpack for new projects
• CSS frameworks: Utility-first approach (Tailwind) beats component libraries

Recommendations

• Large Enterprise: Angular, React + TypeScript
• Startups/SMEs: Vue.js, Svelte, React
• Performance Critical: Svelte, Solid.js, Qwik
• SEO-Focused: Next.js, Astro, Remix
• Rapid Prototyping: Alpine.js, htmx

3. Back End Development

Backend technologies power your application logic, data processing, and API integrations. The choice impacts scalability, security, and development speed.

Key Trends

• APIs: GraphQL gaining ground but REST remains dominant
• Runtimes: Bun challenging Node.js for performance
• Python: Django vs FastAPI decision for new projects
• JavaScript: Express.js vs Nest.js architectural preferences

Recommendations

• REST APIs: Express.js, FastAPI, Django REST Framework
• GraphQL: Apollo Server, tRPC, NestJS GraphQL
• Microservices: Fastify, Starlette, Go-based services
• Real-time: Firebase, Socket.io, WebSockets
• Serverless: Vercel Functions, AWS Lambda, Firebase Functions

4. Full Stack Solutions

Full-stack solutions provide end-to-end development capabilities, from database to deployment, often with integrated tooling and reduced complexity.

Key Trends

• Headless CMS: Gaining momentum over traditional CMS
• Website Builders: No-code solutions competing with custom development
• E-commerce: Shopify dominance vs custom solutions
• API-first: Separation of frontend/backend concerns

Recommendations

• JavaScript Ecosystem: MERN, MEVN, MEAN (choose based on frontend preference)
• Content-Heavy: WordPress, Webflow, headless CMS with frontend frameworks
• E-commerce: Shopify (rapid) vs custom (flexible)
• API Design: REST (simple) vs GraphQL (flexible) vs gRPC (performance)
• Rapid Development: No-code builders vs custom development

5. Mobile Development

Mobile development encompasses native platforms, cross-platform solutions, and progressive web apps. The choice affects reach, performance, and development complexity.

Key Trends

• React Native vs Flutter: Major cross-platform battle continues
• Progressive Web Apps: Gaining capabilities, closing gap with native
• Native Development: Still preferred for performance-critical apps
• Low-code Mobile: Emerging category for simple applications

Mobile development recommendations

• Native Performance: Swift (iOS), Kotlin (Android)
• Code Reuse: React Native (JavaScript), Flutter (Dart)
• Rapid Development: Expo (React Native), Ionic (Web tech)
• Enterprise: Native development + .NET MAUI (C# / .NET)
• Simple Apps: Progressive Web Apps + React Native

6. Databases

Database selection impacts performance, scalability, data modeling, and development complexity. The landscape now includes traditional relational, NoSQL, and vector databases for AI applications.

Key Trends

• Vector Databases: Essential for AI/ML applications
• Serverless: PlanetScale, Neon leading the shift
• Multi-model: Databases supporting multiple query patterns
• PostgreSQL: Becoming the default choice for new projects

Databases recommendations

• Traditional Applications: PostgreSQL, MySQL, SQLite
• AI/ML Applications: Pinecone, Weaviate, PostgreSQL with pgvector
• Real-time Analytics: Elasticsearch, DuckDB, ClickHouse
• Distributed Systems: CockroachDB, Cassandra
• Mobile/IoT: SQLite, DuckDB
• Rapid Development: Supabase, PlanetScale, Neon
• Object Mapping: Prisma (TypeScript), SQLAlchemy (Python)

7. Data Formats

Data formats determine how information is stored, transmitted, and processed. The right choice impacts performance, compatibility, and development complexity.

Format selection criteria

• Human Readability: JSON, YAML, CSV vs binary formats
• Performance: Binary formats (Parquet, Avro) vs text-based
• Schema Evolution: Protocol Buffers, Avro vs static schemas
• Ecosystem Support: Community adoption and tool compatibility

Format recommendations

• API Communication: JSON (web), Protocol Buffers (microservices)
• Big Data Processing: Parquet, ORC, Avro
• Configuration Files: YAML, TOML, JSON
• Data Exchange: CSV (simple), JSON (structured), XML (enterprise)
• Python Serialization: Pickle (internal), JSON (external), Parquet (analytics)

8. DevOps

DevOps encompasses development workflows, infrastructure management, testing, and deployment automation. The right tools dramatically impact team productivity and system reliability.

Key Trends

• AI-Enhanced Development: GitHub Copilot, Cursor, Windsurf
• Git Platforms: GitHub dominates, but GitLab and self-hosted options growing
• Testing: Shift-left testing and AI-powered test generation
• Infrastructure: Terraform vs PaaS solutions

DevOps stack recommendations

• Version Control: Git + GitHub (most common) or GitLab (integrated CI/CD)
• Containerization: Docker + Kubernetes for production
• CI/CD: GitHub Actions (simple), GitLab CI/CD (integrated), Jenkins (custom)
• Testing: pytest + pytest-cov (Python), Jest (JavaScript), Selenium (E2E)
• Code Quality: Prettier + ESLint (JavaScript), Black + Ruff (Python)
• AI Development: GitHub Copilot (VS Code), Cursor (standalone), Windsurf (advanced)

9. Data Engineering

Data engineering encompasses data pipelines, orchestration, processing, storage, and infrastructure. This field has exploded with the growth of big data and machine learning.

Key Trends

• Cloud-Native: Shift from on-premise to managed services
• Real-time Processing: Streaming platforms replacing batch processing
• ML Operations: Integration of data pipelines with model lifecycle
• Data Mesh: Decentralized data ownership architecture

Data engineering stack recommendations

• Orchestration: Apache Airflow (open-source), Dagster (asset-based)
• Processing: Apache Spark (big data), dbt (analytics engineering)
• Streaming: Apache Kafka (self-hosted), Google Pub/Sub (managed)
• Monitoring: Prometheus + Grafana (open-source), Datadog (managed)
• Cloud Platform: AWS (comprehensive), GCP (ML-focused), Azure (enterprise)
• Deployment: Vercel (frontend), Railway (full-stack), Render (multi-platform)

10. Data Visualization

Data visualization tools transform raw data into actionable insights. The choice impacts audience understanding, interactivity, and development complexity.

Key Trends

• Self-Service BI: Business users increasingly create their own dashboards
• Python Visualization: Matplotlib/Seaborn vs Plotly ecosystem competition
• Real-time Dashboards: Growing demand for live data updates
• No-Code Solutions: Closing gap with custom development

Visualization tool recommendations

• Business Users: Power BI, Tableau, Looker (enterprise), Metabase (open-source), Apache Superset (open-source)
• Data Scientists: Streamlit (rapid), Dash (complex), Gradio (ML interfaces)
• Python Development: Matplotlib (basic), Seaborn (statistical), Plotly (interactive)
• Exploratory Analysis: Sweetviz, PyGWalker, Jupyter + standard libraries
• Custom Dashboards: Dash, Streamlit, or custom React/Vue + D3.js

11. Machine Learning

Machine learning encompasses frameworks, libraries, and platforms for building and deploying predictive models. The field continues evolving rapidly with new architectures and automation tools.

Key Trends

• Deep Learning: TensorFlow vs PyTorch competition continues
• AutoML: Democratizing ML for non-experts
• LLM Integration: Traditional ML evolving with language models
• MLOps: Focus on model lifecycle and deployment

ML framework recommendations

• Deep Learning: PyTorch (research), TensorFlow (production), JAX (performance)
• Traditional ML: scikit-learn (general), XGBoost/LightGBM (tabular data)
• AutoML: PyCaret (open-source), Dataiku (enterprise)
• Time Series: Prophet (forecasting), sktime (advanced)
• NLP: NLTK (traditional), Transformers (modern)
• Reinforcement Learning: Stable Baselines3, RLlib
• Data Analysis: pandas + NumPy + SciPy (foundational)
• Notebook Environments: JupyterLab (local development), Google Colab (cloud GPU), Kaggle Kernels (competitions), Binder (reproducible environments)

12. Large Language Models

The LLM ecosystem encompasses frameworks for building AI applications, model serving platforms, and orchestration tools. This field is rapidly evolving with new architectures and capabilities.

Key Trends

• Agent-Based AI: Multi-agent systems becoming mainstream
• Open Models: Hugging Face leading democratization
• Serving Infrastructure: vLLM, Ollama for local deployment
• Observability: Langfuse, Guardrails for production monitoring

LLM stack recommendations

• Application Development: LangChain (comprehensive), LlamaIndex (data-focused)
• Multi-Agent Systems: AutoGen, Crew.ai, AutoGen Studio (visual)
• Model Serving: vLLM (performance), Ollama (local), Agno (managed)
• Observability: Langfuse (open-source), Guardrails (safety)
• Model Discovery: Hugging Face (open models), OpenRouter (API aggregation)
• Data Integration: MCP (protocol standardization)

13. Other Essential Tools

These tools support development workflows, design processes, project management, and productivity. While not directly programming-related, they’re essential for efficient development.

Key Trends

• Knowledge Management: Notion vs Obsidian vs Logseq competition
• Design Tools: Figma dominance in collaborative design
• Automation: n8n vs Zapier vs Make for workflow automation
• Security: Password managers and VPNs becoming standard

Essential tools recommendations

• Knowledge Management: Obsidian (privacy), Notion (collaboration), Logseq (open-source)
• Design: Figma (collaborative), Canva (non-designers), Adobe (professionals)
• Automation: n8n (self-hosted), Zapier (integrations), Make (visual)
• Project Management: Linear (modern), Jira (enterprise), Trello (simple)
• Security: Bitwarden (open-source), 1Password (features), NordVPN (privacy)

2026 Technology Trends

Rising Technologies

• Rust: Systems programming with memory safety
• TypeScript: Default for enterprise JavaScript projects
• Vector Databases: Essential for AI applications
• AI-Enhanced Development: GitHub Copilot, Cursor, Windsurf
• Serverless Platforms: Vercel, Railway, Neon, PlanetScale
• Low-Code Automation: n8n, Zapier, Make

Declining Tools

• jQuery: Legacy in modern web development
• Traditional CMS: Replaced by headless CMS and frontend frameworks
• Monolithic Architectures: Microservices and serverless preferred
• Manual Deployments: CI/CD automation now standard

Integration Trends

• API-First Design: Frontend and backend separation
• Microservices: Domain-driven service architecture
• Event-Driven: Kafka, event sourcing, CQRS patterns
• Observability: OpenTelemetry standardization
• GitOps: Infrastructure as code with automated deployment

Conclusion

The technology landscape in 2026 offers unprecedented choice and power, but also complexity. Success comes from strategic tool selection, integration planning, and continuous learning. The technology choices you make today will impact your success tomorrow. Use this guide to make informed decisions, build robust systems, and develop sustainable development practices.

Key Takeaways

1. Ecosystem Matters More Than Individual Tools: Choose technologies that work well together rather than isolated “best-of-breed” solutions.

2. Balance Current Needs with Future Growth: Select tools that scale with your success and adapt to changing requirements.

3. Invest in Learning: The most valuable asset is team knowledge and ability to adapt to new technologies.

4. Monitor Trends, Don’t Chase: Be aware of emerging technologies but adopt based on actual needs, not hype.

5. Build Observability In: Choose tools with strong monitoring and debugging capabilities from the start.

This guide represents comprehensive analysis of 200+ tools across 13 categories.

Last Updated: November 2025

Definition

Vim is a free and open-source text editor program, popular on UNIX systems.

Cheat sheet

You can download a printable Vim command line cheat sheet here.

Some definitions

The Robot Operating System (ROS) is a collection of tools and libraries that helps you build complex and robust robot applications. There are various distributions of ROS, for example ROS Hydro, ROS Indigo, ROS Kinetic, ROS Melodic. ROS 2 was released in 2021.

A Node is process that performs computation and that can communicate with other nodes.

Topics are buses over which nodes exchange messages. Nodes can publish messages to a topic and/or subscribe to a topic to receive messages.

A Message is a data structure (supporting integer, floats, boolean, arrays, etc…) that is used by nodes to communicate with each other.

A Publisher is a Node that writes information TO a Topic.

A Subscriber is a node that writes information FROM a Topic.

A Service is a client/server system that allows Nodes to communicate data to each other. Its an alternative to Topics.

Gazebo is an open-source physics simulator integrated in ROS. It can set up a world and run robotics simulations.

Rviz is a visualization software, that allows you to view Gazebo data (if you are simulating) or real world data (if you are using the real robot).

A Package is a directory where a ROS program is stored.

Catkin is the official build system of ROS.

Cheat sheet

You can download a printable ROS cheat sheet here (Source).

ROS Packages

• Launch a ROS package

roslaunch <package_name> <launch_file>

For example:

# PACKAGE
roslaunch turtlebot_teleop keyboard_teleop.launch
roslaunch bb_8_teleop keyboard_teleop.launch

# PUBLISHER
roslaunch publisher_example move.launch
roslaunch publisher_example stop.launch

# SERVICE
roslaunch service_demo service_launch.launch
rosservice call /service_demo "{}"

# ACTION
roslaunch action_demo action_launch.launch
roslaunch action_demo_client client_launch.launch

• List ROS packages

rospack list
rospack list | grep <package_name>

• Refresh the package list

rospack profile

• List first order dependencies

rospack depends <package_name>

• List all dependencies

rospack depends <package_name>

• Create a package

roscd
cd ..
cd src/
catkin_create_pkg <pkg_name> <pkg_dependencies>
cd ..
catkin_make

For example,

catkin_create_pkg my_package rospy

• Install a package from a Github repository

cd ~/catkin_ws/src

# clone repo
git clone https://github.com/<...>.git

# check for missing dependencies
rosdep update
rosdep check --from-paths . --ignore-src --rosdistro melodic

# install missing dependencies (if needed)
rosdep install --from-paths . --ignore-src --rosdistro melodic -y

# build package
cd ~/catkin_ws/
catkin_make

• Create a Python programme inside a ROS package

roscd <package_name>
cd src/
vim <python_file_name.py>
chmod +x <python_file_name.py>
roscd <package_name>
mkdir launch
vim launch/<launch_file.launch>

• Compile all packages in catkin directory

roscd
catkin_make
. ~/catkin_ws/devel/setup.bash      # Source the setup file
source /opt/ros/melodic/setup.bash  # Source environment setup file
echo $ROS_PACKAGE_PATH              # Print the package path

• Compile only one package

roscd
catkin_make --only-pkg-with-deps <package_name>

ROS nodes

• List all the nodes currently running

rosnode list

For example,

roscore
rosrun turtlesim turtlesim_node # in another terminal
rosnode list                    # in another terminal  

• Run a node within a package

rosrun <package_name> <node_name>

For example:

rosrun turtlesim turtlesim_node
rosrun rviz rviz                   # Run Rviz (debugging)
rosrun rqt_plot rqt_plot
rosrun rqt_graph rqt_graph         # Visualise graph (node connections)

• Print info about a node

rosnode info /<node_name>

• Test connectivity to node

rosnode ping

• List nodes running on a particular machine or list machines

rosnode machine

• Kill a running node

rosnode kill

• Purge registration information of unreachable nodes

rosnode cleanup

ROS parameters

• List ROS parameters

rosparam list

• Get parameter value

rosparam get <parameter_name>

• Assign a value to a parameter

rosparam set <parameter_name> <value>

• Load parameters from files

rosparam load

• Dump parameters to file

rosparam dump

• Delete parameter

rosparam delete

ROS Topics

• List active topics (with verbose option)

rostopic list
rostopic list -v

• Print the output messages of a topic to the screen in real time

rostopic echo /<topic_name>

• Publish message with specified value to a topic

rostopic pub <topic_name> <message_type> <value>

• Display bandwidth used by topic

rostopic bw

• Display delay of topic from timestamp in header

rostopic delay

• Find topics by type

rostopic find

• Display publishing rate of topic

rostopic hz

• Print topic or field type

rostopic type <topic_name>

ROS messages

• Show message description (and its structure)

rosmsg show <message>
rosmsg info <message>

• This is useful to check the structure of the messages

roscd std_msgs/msg/
roscd geometry_msgs/msg/
roscd nav_msgs/msg/

• List all messages

rosmsg list

• Display message called md5sum

rosmsg md5

• List messages in a package

rosmsg package

• List packages that contain messages

rosmsg packages

Gazebo

• start Gazebo

gazebo

• run Gazebo server

gzserver

• run Gazebo client

gzclient

• kill Gazebo server

killall gzserver

Other commands

• Start ROS core

roscore

• List ROS environment variables

export | grep ROS

• Go to the Catkin workspace

roscd

• Go to “package_name”

roscd <package_name>

Launch file structure

<launch>
<!-- My Package launch file -->
<node pkg="<package_name>" type="<python_file_name.py>" name="<node_name>" output="screen">
</node>
</launch>

Example of publisher node that write messages to a ROS topic

#! /usr/bin/env python

import rospy          # Import the Python library for ROS
from std_msgs.msg import Int32 # Import the Int32 message from the std_msgs package   

rospy.init_node('topic_publisher') # Initiate a Node named 'topic_publisher'
pub = rospy.Publisher('/counter', Int32, queue_size=1) # Create a Publisher object, that will publish on the /counter topic + messages of type Int32

rate = rospy.Rate(2) # Set a publish rate of 2 Hz
count = Int32() # Create a var of type Int32
count.data = 0 # Initialize 'count' variable

while not rospy.is_shutdown(): # Create a loop that will go until someone stops the program execution
    pub.publish(count) # Publish the message within the 'count' variable
    count.data += 1 # Increment 'count' variable
    rate.sleep() # Make sure the publish rate maintains at 2 Hz

Example of subscriber node that read messages to a ROS topic

#! /usr/bin/env python

import rospy
from std_msgs.msg import Int32

def callback(msg): # Define a function called 'callback' that receives a parameter named 'msg'
    print msg.data # Print the value 'data' inside the 'msg' parameter

rospy.init_node('topic_subscriber') # Initiate a Node called 'topic_subscriber'

# Create a Subscriber object that will listen
# to the /counter topic and will call the
# 'callback' function each time it reads
# something from the topic
sub = rospy.Subscriber('/counter', Int32, callback) 

rospy.spin()

Definition

Markdown is an easy-to-read, easy-to-write syntax for formatting plain text into HTML code. It is commonly used to write README.md files in Git repositories.

Here are some comprehensive Markdown reference guides for basic syntax and advanced features.

Heading

H1

H2

H3

Bold

bold text

Italic

italicized text

Blockquote

blockquote

Ordered List

1. First item
2. Second item
3. Third item

Unordered List

• First item
• Second item
• Third item

Code

code

Horizontal Rule

Link

Markdown Guide

Image

Table

Fenced Code Block

{
  "firstName": "John",
  "lastName": "Smith",
  "age": 25
}

Footnote

Here’s a sentence with a footnote. ¹

Definition List

term

definition

Strikethrough

The world is flat.

Task List

• Write the press release
• Update the website
• Contact the media

Some definitions

• Linux is an open-source operating system. It comes under different “flavours” or distributions such as Ubuntu, Arch Linux, Fedora, etc…
• Bash - for Bourne Again SHell - is Linux’ command line interpreter (or shell) that provides a command line interface inside a terminal emulator.

Cheat sheet

You can download a printable Linux command line cheat sheet here from Cheatography.

1. Find files

• Find files match a pattern

find -name "query"

• Find files match a pattern, case insensitive

find -iname "query" 

• Find all files that don’t contain the pattern

find -not -name "query_to_avoid"

• Find all files in current directory

find . -type f

• Find all directories in current directory

find . -type d

• Find all symbolic links in current directory

find . -type l 

• Find all items ending in “.conf” in current directory

find . -name *.conf

• Find all files in the current directory starting with “pro”

find . -name pro* 

• Find all items containing “yolo” in current directory

find . -name *yolo*

• Find pattern ‘tensorflow’ in python files in the current directory

find . -name *.py | grep -rnw tensorflow 

Option explained:

• -r or -R : recursive
• -n : show line number
• -w : match the whole word

• -l (lower-case L) : give the file name of matching files (optional).

• Find pattern in file recursively

grep -rnw '/path/to/somewhere/' -e 'pattern'

grep -r 'pattern'

• Move file with extension recursively to a directory

find /thisdir -type f -name '*.ogg' -exec mv -i {} /somedir \;

2. Backup installed packages

• Write list of packages

sudo dpkg --get-selections "*" > my_packages.txt

• Install list of packages from txt file

sudo dpkg --set-selections < my_packages.txt
sudo apt-get -u dselect-upgrade

3. Command line history

• Search command line history (shortcut)

CTRL + r

• View command line history

history

4. Copy to/from SSH server

• When logged in locally, copy from local to remote

scp /file/to/send username@remote:/where/to/put
# OR
rsync -avzh /file/to/send username@remote:/where/to/put

where the rsync options are:

• -a : archive mode, which preserves permissions, ownership, and modification times
• -v : enables verbose
• -z : enables compression during transfer

• -h : outputs numbers in human-readable format

• When logged in locally, from remote to local

scp username@remote:/file/to/send /where/to/put
# OR
rsync -avzh username@remote:/file/to/send /where/to/put

• When logged in remotely, copy from remote to local

sftp username@remote
get -r "remote/dir/path" "local/dir/path"

• When logged in remotely, copy from local to remote

sftp username@remote
put -r "local/dir/path" "remote/dir/path"

(note: in Ubuntu, you may encounter this error: “Couldn’t canonicalise: No such file or directory”. Solution: create a directory on the remote first using mkdir)

• Launch a job on an SSH server and be able to log off without killing the job

nohup python my_script.py >& log.run &

5. File ownership / make executable

• Change directory ownership

chown -R user:group /directory/of/interest

• Make script executable for all users

chmod a+x "file name"

6. System specs

• Disk usage of the entire filesystem

df -h --total
df -h

• Disk usage of current directory

du -sh .

• Show disk usage of the 40 biggest subfolders in the current directory (ranked)

du -hsx * | sort -rh | head -n 40

• Show disk usage of all the subfolders in the current directory one by one

du -hsx *

• Show disk usage of the biggest subfolders in the current directory (ranked), with hidden directories

du -sch .[!.]* * | sort -rh

• In Ubuntu, this nice tool can explore the disk usage of your entire system quickly

sudo apt-get install ncdu
ncdu

• View system specs (and export them to a html file)

sudo lshw
sudo lshw -html > specs.html

• Check Linux Kernel

uname -a

• Check Ubuntu version

lsb_release -a

• Print total, free and used RAM memory

free -h

• List CPU and processor info

lscpu

• Print generic hardware information

hwinfo

• Print PCI buses, including graphics card, network adapter

lspci

• List block devices (storage and partitions)

lsblk

• List devices connected via USB

ls /dev

• View running processes

top  
htop

• Print the process ID of a running task

pidof firefox

7. Kill a running job

kill 'process ID'

Note, you can find process ID using the ‘top’ command or using pidof.

8. Use SLURM (job scheduler for computing clusters)

• Submit job

sbatch job.sh

• View queue

squeue -u "user name"  
squeue

9. Synchronize files

This removes files from destination that don’t exist in source.

rsync -azvh source/ destination/ --delete

10. Manipulate PDF documents with pdftk

• Cut pdf pages

pdftk full-pdf.pdf cat 12-15 output outfile_p12-15.pdf

• Merge pdf pages

pdftk file1.pdf file2.pdf file3.pdf cat output newfile.pdf

11. Make Gifs and videos from images using ffmpeg

• Create video from images

ffmpeg -i input_0%d.png -vcodec libx264 ouput.mp4
ffmpeg -i input_%d.png -filter:v "setpts=10.0*PTS" -vcodec libx264 ouput.mp4
ffmpeg -start_number 1 -i input%d.png -filter:v "setpts=2*PTS" -filter:v "crop=490:360:15:15" -r 48 -c:v libx264 -crf 0 -profile:v high444 -preset slow -b:v 15M ouput.mp4

• Create gif from video

ffmpeg -t 30 -i input.mp4 -vf "fps=10,scale=1280:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" -loop 0 output.gif
ffmpeg -t 30 -i input.mp4 -vf "fps=24,scale=300:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" -loop 0 output.gif

Create gif from images using Convert (from ImageMagik)

convert -resize 50% -delay 20 -loop 0 input_{0..99}.jpg output.gif
convert -resize 50% -delay 20 -loop 0 *.jpg output.gif
convert -resize 50% -delay 20 -loop 0 `ls -v` output.gif

Create gif from images using Gimp (GUI, potentially more robust than Convert)

1. Select File Menu > Open as Layers > Select all images you want to be in the GIF > Open

2. Select Filters from main Menu > Animation > Click Optimize for GIF

3. Save GIF Select File > click Export as > Select File Type as gif > Select ‘As Animation’ > Select ‘Loop Forever’

12. Useful shortcuts

• Open a terminal new widow

CTRL + ALT + T

• Open a new terminal tab

CTRL + SHIFT + T

• Close the current terminal tab

CTRL + SHIFT + W

• Kill the current running process

CTRL + C

• Exit Python interpreter / log out from SSH

CTRL + D 

• Copy-paste in terminal with a mouse
  1. select text
  2. move cursor where you want to paste
  3. paste with a middle mouse click
• Insert the euro sign (€)

CTRL+SHIFT+u
20ac
"enter"

• Put task in background

CTRL + C 

• Bring it back to the foreground

fg

• Nicely formated ls command

ls -lrt

• Download from website

wget http://www.website-name.com
curl http://www.website-name.com

• Execute the previous command with sudo

  sudo !!
  

• Login as root

  sudo -i
  

• Logout as root (exit)

  CTRL + D
  

Some definitions

Git is a version-control tool for tracking changes in source code during software development.

A commit records a snapshot of your changes and associate it with a message and a commit id.

A branch is a separate version of the main repository that allows the developper to test a new feature for example without affecting the main repository.

Staging means adding a file to the Git index before commiting it.

Remote repositories are versions of your source code that are hosted on the Internet, mainly to share the code or collaborate with others. Examples of Git-based source code repository hosting service include Github, Bitbucket or Gitlab.

You can download a printable Git command line cheat sheet here (Source).

_^Source

Create a repository

• create a new local repository

git init my_project_name

• Download from an existing repository

git clone my_url

Observe the repository

• list files not yet commited

git status

• Show full change history

git log

• Show change history for file/directory including diffs

git log -p [file/directory]

• Show the changes to files not yet staged

git diff

• Show the changes between two commit ids

git diff commit1 commit2

Working with branches

• List all local branches

git branch

• Create a new branch

git branch new_branch

• Switch to a branch and update working directory

git checkout my_branch

• Create a new branch and switch to this branch

git checkout -b new_branch

• Delete a branch

git branch -d my_branch

• Merge branch_a into branch_b

git checkout branch_b
git merge branch_a

Resolve merging conflicts

git mergetool --tool=emerge

It will open 3 windows: version a on top left, version b on top right and final version at the bottom.

• press n for next change
• press a or b to choose which version I want to keep
• press q to quit and save

Make a change

• Stage all changed files in current directory

git add .

• Commit all your tracked files to versioned history

git commit -am "commit message"

• Unstages file, keeping the file changes

git reset [file]

• Revert everything to the last commit

git reset --hard

Synchronize with remote repository

• Get the latest changes from origin (no merge)

git fetch

• Fetch the latest changes from origin and merge

git pull

• Push local changes to the origin

git push

Add an existing SSH key to the agent

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519

Fix remote repo not empty

When I want to push to a remote repository that is not empty (for example if it was initialised with a license or readme file).

git fetch origin main:tmp
git rebase tmp
git push origin HEAD:main
git branch -D tmp

What is Docker?

Docker is a tool used to deploy software across different platform without running into compatibility issues.

A Docker image is a lightweight, standalone, executable package of software that includes everything needed to run an application (code, libraries, settings, etc…). On a Linux system, Docker images are stored in /var/lib/docker.

A running instance of an image is called a container. A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another.

Cheat sheet

You can download a printable Linux command line cheat sheet here from Cheatography.

Another handy cheat sheet can be found here.

• Check Docker installation

docker version
docker run hello-world

• List docker images

docker images

• List dangling Docker images

docker images -f dangling=true

• Remove dangling images and build caches, all stopped containers and all networks not used by at least 1 container (useful to free some space).

docker system prune

• Remove docker image by ID

docker rmi -f IMAGE_ID

• Download a Docker image

docker pull IMAGE_NAME

• List running containers

docker ps
docker container ls

• List all containers

docker ps -a

• Stop a container

docker stop CONTAINER_ID

• Remove a container

docker rm CONTAINER_ID

• Stop all the containers

docker stop $(docker ps -a -q)

• Remove all the containers

docker rm $(docker ps -a -q)

• Run a docker image

docker run -it --rm IMAGE_ID

• Options of the run command:

-u $(id -u):$(id -g)       # assign a user and a group ID
--gpus all                 # allow GPU support
-it                        # run an interactive container inside a terminal
--rm                        # automatically remove the container after exiting
--name my_container        # give it a friendly name
-v ~/docker_ws:/notebooks  # share a directory between the host and the container
-p 8888:8888               # define port 8888 to connect to the container (for Jupyter notebooks)

• Open a new terminal in a running container

docker exec -it CONTAINER_ID bash

Create a new Docker image

There are 2 main methods.

Method 1 : Using a Dockerfile

1. Create a file called Dockerfile on your host machine.

    # getting base Ubuntu image
    FROM ubuntu
   
    # file author / maintainer
    LABEL maintainer="your_email_address"
   
    # update the repository sources list
    RUN apt-get update
   
    # print Hello World
    CMD ["echo", "Hello World"]
   

2. Build the image.

    docker build -t myimage:0.1 .
   

3. Run the image.

    docker run myimage:0.1
   

Method 2 : Commit a Docker image from a running container

Modify a running container and run this in another terminal.

docker commit CONTAINER_ID my_new_image

Some definitions

Conda is an open source package manager tool and environment management system. It is mostly used for created isolated Python environments but it can package and distribute software for any language.

Anaconda is a free and open-source Python distribution for scientific computing. It includes conda and plus hundreds of popular Python packages such as numpy, scipy, matplotlib, pandas, etc…

Miniconda is a lighter alternative to Anaconda that just include conda and its dependencies but no Python packages.

A virtual environment is a named, isolated, working copy of Python that that maintains its own files, directories, and paths so that you can work with specific versions of libraries or Python itself without affecting other Python projects. Virtual environments make it easy to cleanly separate different projects and avoid problems with different dependencies and version requirements across components.

Please see the official page for more information.

You can download the official Conda cheat sheet here.

Other environment managers

virtualenv is a tool to create isolated environments for Python only.

venv is a tool to create isolated environments for Python 3 only.

Cheat sheet

• Create environment

conda create -n myenv
conda create -n myenv python=3.6      # with a specific python version
conda create -n myenv scipy           # with a specific python package
conda env create -f environment.yml   # from a .yml file

• List environments

conda env list

• Activate environment

conda activate myenv

• Deactivate environment

conda deactivate

• View a list of the packages in an environment

conda list
conda list -n myenv

• Export a YML environment file

conda env export > environment.yml

• Remove an environment

conda env remove --name myenv

• Check conda version

conda info

• Update conda

conda update conda

• Platypus
• MOEA
• PyGMO
• PaGMO
• Inspyred
• DEAP
• EMOO
• jMetal
• PYMOO

Features comparison

Algorithms comparison

We will look at how a genetic algorithm can be used to find the minimum of a function and implement it in Python.

Some definitions

A genetic algorithm is an optimisation heuristic inspired from the mechanism of Darwin’s natural evolution. It leverages the process of natural selection to select individuals that are fittest to survive to the next generation.

• Individual (= chromosome): one possible solution to the problem
• Design parameters (= genes): the parameters to optimise. Each individual is defined by a list of its design parameters.
• Population: a group of individuals or designs to be optimised
• Generation: a population at a specific interation
• Objective function (= fitness function): the function to minimise
• Design space: set of all the possible design parameters values
• Objective space: set of all the objective function value
• Parents: individuals belonging to an older generation
• Offspring: individual belonging to a younger generation
• Crossover: it consists in combining the genes of two parents to create an offspring.
• Mutation: it consists in altering a few genes of a parent to create an offspring.

In its most basic form, a genetic algorithm comprises the following stages:

1. Initialisation of the population
2. Selection of the best parents
3. Reproduction (by mutation and/or by crossover) to generate offsprings
4. Evaluation of the population
5. Termination according to criterion

Implementation

We will implement a simple genetic algorithm in Python. The objective is to find the minimum of this objective function with 2 design parameters: \(Z(x, y) = x\sin(x)*y\cos(y)\). The source code can be found here.