AwsJunkie

Access Amazon S3 Tables from Local using Apache Spark Docker Container | Demo

Wed, 22 Jan 2025 00:00:00 GMT

Recently, Amazon S3 has introduced S3 Tables, a fully managed Apache Iceberg tables for storing tabular data at scale. It is specifically optimized for analytics workloads to deliver up to 3x faster query throughput and up to 10x higher transactions per second compared to self-managed tables (Iceber

Effortless Automation: Rapidly Generate AWS CloudFormation Templates and CDK Apps for Your Existing Resources in Minutes!

Wed, 07 Feb 2024 00:00:00 GMT

CloudFormation IaC generator can generate AWS CloudFormation template and AWS CDK apps for your existing AWS resources in minutes without writing any code. It saves several weeks of manual effort and enables you to quickly adopt Infrastructure as Code (IaC) for your AWS workloads. The process invol

Idempotency Support for Route Table and Network ACL Creation in Amazon VPC!

Mon, 05 Feb 2024 00:00:00 GMT

Amazon VPN now supports idempotency for route table and network ACL creation. You can incorporate a retry mechanism in your creation workflow without creating duplicate resources. When creating route tables and network ACLs using the AWS CLI or API, you can now specify a clientToken parameter (--cl

Explore the Power of Account-Level Subscription Filters in Amazon CloudWatch Logs!

Thu, 01 Feb 2024 00:00:00 GMT

CloudWatch logs from all or multiple log groups can now be delivered to the desired destination (Amazon Kinesis Data Stream, Amazon Kinesis Data Firehose, or AWS Lambda) using just one account-level subscription filter. ℹ️There is no option in CloudWatch Dashboard UI. We need to use aws logs put-ac

[new capability] Amazon EC2 supports disabling Amazon Machine Image (or AMI)

Mon, 23 Oct 2023 00:00:00 GMT

Housekeeping is always an important activity. Otherwise, soon, it will become an operational and management overhead. Fortunately, Amazon EC2 now supports disabling an Amazon Machine Image (or AMI). Let’s take a closer look. Imagine there are hundreds of AMIs in your Catalog. Some are outdated, som

How to Run Bash Scripts on AWS Lambda?

Mon, 25 Sep 2023 00:00:00 GMT

If you want to run bash scripts on AWS Lambda, create a Lambda function with a custom runtime. Here are the steps to follow: Create a Lambda function with the "Author from scratch" option and "Use default bootstrap on Amazon Linux 1" as runtime. 🏖️The custom runtime option - "Provide your own boo

Multivalue answer (MVA) support for DNS queries of Amazon S3 endpoints

Tue, 19 Sep 2023 00:00:00 GMT

With multivalue answer (MVA) support for DNS queries of Amazon S3 endpoints, you can achieve enhanced reliability and higher aggregate throughput for your connections to S3. It is available at no cost to you. A single DNS query of an S3 endpoint (e.g. s3.us-east-1.amazonaws.com) will provide 8 S3 I

Automatically set up connectivity between Amazon RDS or Aurora database and AWS Lambda in minutes

Fri, 25 Aug 2023 00:00:00 GMT

The Amazon RDS console now offers the feature of automatically setting up connectivity between Amazon RDS or Aurora database and AWS Lambda in minutes. This feature helps to eliminate manual tasks such as configuring VPC, security groups, subnets, ingress/egress rules, and RDS proxy and boost produc

How to Access Amazon S3 Bucket like a Local File System

Mon, 21 Aug 2023 00:00:00 GMT

Mountpoint for Amazon S3 is now generally available and ready for production workloads. This open-source simple file client is optimized for read-heavy workloads that require high throughput and focused on high-performance access to large data sets. In this post, we'll see how to effortlessly acces

Easy housekeeping with the new capability of Amazon EventBridge Scheduler

Fri, 18 Aug 2023 00:00:00 GMT

To create, trigger, and manage scheduled events and tasks, we use Amazon EventBridge Scheduler (serverless offering). Over time, we accumulate a large number of unused finished schedules, which count towards your account's quota (1 million schedules by default). To meet the demands of many enterpris

Use "Public IP Insights" [new feature] to optimize usage of chargeable public IPv4 address

Wed, 16 Aug 2023 00:00:00 GMT

Public IPv4 addresses will no longer be free. Effective February 1, 2024, Amazon will charge you 0.005$ per public IPv4 address per hour. You will likely soon use "Public IP insights" (a new feature) to optimize the usage of public IPv4 addresses and associated costs to prevent unexpected bills. It

Better, Faster and Cheaper with AWS Lambda's new Python 3.11 runtime

Mon, 31 Jul 2023 00:00:00 GMT

If you are still running your AWS Lambda workloads on Python 3.10 or earlier versions, migrate to Python 3.11 for better performance and lower cost. AWS Lambda has added support for Python 3.11. On an average, Python 3.11 is 25% faster than 3.10 Faster Startup and Faster Runtime are the two main f

Detect recursive loop and stop run-away AWS Lambda function

Fri, 21 Jul 2023 00:00:00 GMT

A run-away AWS Lambda function may hog all available concurrency of your account and cause unexpected charges in the AWS bill. But Lambda can now detect and stop certain types of recursive or infinite loops. In this post, we'll explore more on this feature with a video demo. 🖋️Currently, this fea

How to use GitHub Action with AWS CodeBuild ?

Tue, 18 Jul 2023 00:00:00 GMT

With the release of GitHub Action support, AWS CodeBuild has expanded its scope of possibilities. You can leverage pre-built actions or third-party solutions available in GitHub’s marketplace. There is no additional cost for running GitHub Action Runner. What is GitHub Action? "Automate, custo

Test and Debug AWS CodeBuild Builds Locally using AWS CodeBuild agent

Mon, 17 Jul 2023 00:00:00 GMT

If you are still using AWS CodeBuild managed service for testing and debugging, then you have the scope to save cost and expedite the build-test-debug cycle loop using AWS CodeBuild local agent. Troubleshooting and fixing issues in BuildSpec file is so easy on the local development machine. In this

AWS CodeBuild - GitHub Actions: COMMAND_EXECUTION_ERROR Message: Error while executing command | [error]Docker pull failed with exit code 1

Mon, 10 Jul 2023 00:00:00 GMT

For GitHub Docker container actions, "build project must have privileged mode enabled and be run by the default Docker user (root)." Otherwise, build will fail with following error logs. [Container] 2023/07/11 02:09:11 Entering phase PRE_BUILD [Container] 2023/07/11 02:09:11 Preparing action comma

How to Install and Configure Amazon DynamoDB local version 2.0 (downloadable)?

Fri, 07 Jul 2023 00:00:00 GMT

Amazon DynamoDB local not only speeds up development and testing iteration loop, it also saves cost of using DynamoDB webservices. Free downloadable version of Amazon DynamoDB (DynamoDBLocal.jar) can run on any operating systems or platform having JRE (Java Runtime Environment). "This latest ver

AWS S3: The difference between the request time and the current time is too large

Tue, 04 Jul 2023 00:00:00 GMT

If system clock is out of sync, we may encounter error message like "An error occurred (RequestTimeTooSkewed) when calling the ListBuckets operation: The difference between the request time and the current time is too large." ag@aj:~$ aws s3 ls An error occurred (RequestTimeTooSkewed) when calling

How to uninstall AWS CLI version 2?

Tue, 04 Jul 2023 00:00:00 GMT

If you already have AWS Command Line Interface (AWS CLI 2) and want to uninstall it from your system, then you will find all the steps to remove it. Find symbolic link for aws program. This will give you the --bin-dir abhijit@awsjunkie:~$ which aws /usr/local/bin/aws Find installation director

Mountpoint for Amazon S3 supports sequential write operations for creating new files

Tue, 04 Jul 2023 00:00:00 GMT

Mountpoint for Amazon S3 (an open source simple file client for mounting S3 bucket as a local file system) was launched on AWS Pi Day 2023 with no support for write operations. Now, it has added the support for sequential write operations to create new files. Let's quickly take a look. Installer i

Amazon S3 dual-layer server-side encryption with AWS KMS (DSSE-KMS)

Tue, 20 Jun 2023 00:00:00 GMT

Amazon S3 has recently added support for two independent layers of server-side encryption with AWS KMS (DSSE-KMS) for top-secret workloads. With this launch, Amazon S3 is the only cloud object storage service that supports multi layers of server-side encryption at object level. "DSSE-KMS is desi

How to setup CodeWhisperer with JupyterLab?

Sun, 21 May 2023 00:00:00 GMT

Amazon CodeWhisperer (AI coding companion) is now available as an extension for JupyterLab. If you outline the coding task as a comment, this generative AI extension will auto-suggest single-line, multiline or full-function code right within the JupyterLab. Here's a guide on how to set up the Cod

AWS Lambda SnapStart - Turbocharge Java Application Startup by up to 10x at no extra cost

Fri, 21 Apr 2023 00:00:00 GMT

Spring Framework 5.0 introduced Spring Cloud Function to aid the implementation of business logic as a function and to enable Spring Boot features on different serverless cloud providers (AWS Lambda, Azure, Google Cloud Platform, Oracle Fn platform and Apache OpenWhisk). Its "functional" style of be

How to run Amazon Linux 2023 (AL2023) locally on WSL?

Fri, 14 Apr 2023 00:00:00 GMT

Amazon Linux 2023 (AL2023) is the next generation of Amazon Linux for developing and running cloud applications in high-performance and security enhanced execution environment. This rpm-based Linux distribution is optimized for Amazon EC2. Amazon Linux 2023 is not available in the Microsoft Store.

Fixed: Temporary failure in name resolution in WSL2

Wed, 12 Apr 2023 00:00:00 GMT

DNS resolution not working in WSL2? Getting messages like "Temporary failure in name resolution" or "Temporary failure resolving..."? Issue: abhijit@AwsJunkie:~$ ping www.AwsJunkie.com ping: www.AwsJunkie.com: Temporary failure in name resolution abhijit@AwsJunkie:~$ sudo apt update Err:1 http

What is AWS Lambda's new Response Payload streaming feature?

Sat, 08 Apr 2023 00:00:00 GMT

AWS Lambda's new feature, Response Streaming, can enhance user experience, responsiveness, and search engine rankings of your web applications by lowering Time to First Byte (TTFB). Additionally, it supports a larger payload (soft limit of 20 MB) compared to a traditional buffered response (max 6 MB

AWS Console Toolbar - a new feature that brings both AWS Management Console and AWS CloudShell in a single view

Mon, 03 Apr 2023 00:00:00 GMT

Last week, AWS launched a new feature called Console Toolbar that brings both AWS Management Console and AWS CloudShell in a single view. There is no additional charge for using CloudShell and Console Toolbar :-) What is AWS CloudShell? AWS CloudShell is a pre-authenticated ready to use browser-

How to install awslocal ( LocalStack AWS CLI) on Ubuntu running on WSL2 (Windows Subsystem for Linux)?

Sun, 02 Apr 2023 00:00:00 GMT

If you are already enjoying LocalStack ("A fully functional local cloud stack") for building awesome AWS cloud and serverless applications locally, then awslocal will add extra Saccharin to sweeten your development experience for sure. What is awslocal? awslocal is a wrapper around the AWS CLI to

How to install LocalStack (AWS cloud service emulator) on WSL2 to develop cloud and serverless apps offline?

Mon, 27 Mar 2023 00:00:00 GMT

LocalStack is an AWS cloud service emulator that can be used as a local cloud sandbox for development, prototyping and testing. It is "A fully functional local cloud stack". You can efficiently build AWS cloud and serverless applications locally without connecting to AWS cloud. Therefore, it will he

How to install Node.js on Ubuntu 22.04 distro running on WSL2 (Windows Subsystem for Linux)?

Sun, 26 Mar 2023 00:00:00 GMT

In this post, we'll see how to install Node.js (open-source, cross-platform JavaScript runtime environment) on Ubuntu 22.04 distro running in WSL2 (Windows Subsystem for Linux). Option #1 - Install Node.js with APT package Manager from the Official Repositories NodeJs is already available in defa

Mountpoint for Amazon S3 -Build & Demo

Thu, 16 Mar 2023 00:00:00 GMT

On AWS Pi Day 2023, Amazon announced Mountpoint for Amazon S3, an open-source simple file client for mounting S3 buckets as a local file system. It is optimized for read-heavy workloads that require high throughput and is mainly focused on high-performance access to large data sets. So naturally, it

Does a "stopped" EC2 instance cost any money?

Tue, 14 Mar 2023 00:00:00 GMT

There is no charge for a stopped EC2 instance or any data transfer fees. Associated Elastic IP (EIP) addresses and attached Amazon Elastic Block Store (EBS) volumes will incur charges. If you don't need Elastic IP addresses, disassociate and release. As an EBS snapshot is relatively cheaper than a

Install AWS Command Line Interface (AWS CLI) on Windows Subsystem for Linux (WSL 2)

Thu, 23 Feb 2023 00:00:00 GMT

AWS Command Line Interface (AWS CLI) is used to interact with different AWS (Amazon Web Services) product and services in command-line shell. AWS CLI version 2 is the latest one. In this article, we'll see how to download and install AWS CLI on Windows Subsystem for Linux (WSL 2). Video: Update

Install AWS Serverless Application Model Command Line Interface (AWS SAM CLI) on Windows Subsystem for Linux (WSL 2)

Thu, 23 Feb 2023 00:00:00 GMT

AWS SAM command line interface (AWS SAM CLI) tool is must have for building AWS serverless applications. In this post, we'll see how to install latest version of AWS SAM CLI on Windows Subsystem for Linux (WSL 2). Video: Prerequisites: AWS Command Line Interface (AWS CLI 2) is recommended for S

Fix: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Sat, 04 Feb 2023 00:00:00 GMT

Issue: sudo service docker start says "Starting Docker: docker" but sudo service docker status shows "Docker is not running" and most docker commands throw following message "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?". abhijit@AwsJunkie:~$ sud

Install Docker Engine and Docker Compose on Windows Subsystem for Linux (WSL 2)

Fri, 03 Feb 2023 00:00:00 GMT

If you already have Docker Desktop installed or want to use docker on both Windows and WSL2, then Docker Desktop is better choice. But I personally prefer to install and run the "most-loved Tool" docker inside Windows Subsystem for Linux (WSL 2). This keeps the development environment isolated and t

Install JupyterLab or Jupyter Notebook in Windows Subsystem for Linux (WSL2)

Wed, 01 Feb 2023 00:00:00 GMT

Web-based interactive development environment JupyterLab (the next generation of Jupyter Notebook ) is very popular in the field of data science and machine learning. Windows Subsystem for Linux (WSL2) is well known for super-fast boot time, less resource consumption and seamless integration of Linu

Check or Upgrade/Downgrade Windows Subsystem for Linux (WSL) Version

Sun, 02 Oct 2022 00:00:00 GMT

This is a short post on how to check or upgrade/downgrade the version of Windows Subsystem for Linux (WSL) running on Windows 10 or Windows 11 host. To check WSL version set to installed Linux distro, execute wsl -l -v or wsl --list --verbose C:\Users\AwsJunkie>wsl --list --verbose NAME

Setup Proxy for CURL in Ubuntu running on WSL (Windows Subsystem for Linux)

Sun, 11 Sep 2022 00:00:00 GMT

If Windows host machine is running behind any proxy (e.g. corporate proxy in office network), then it is required to pass/configure Proxy server and Proxy Authentication for cURL commands. Let's explore multiple ways to do it. CURL option "proxy" and "proxy-user" cURL command has option -x, --pr

Update WSL2 (Windows Subsystem for Linux) kernel on Windows

Sun, 11 Sep 2022 00:00:00 GMT

Windows Build version 20150 has introduced command line options to update WSL2 (Windows Subsystem for Linux) Kernel very conveniently. Let's try that. Run wmic os get BuildNumber in command line to check the build version of your Windows. C:\Users\AwsJunkie>wmic os get BuildNumber BuildNumber 220

How to Backup and Restore Linux Distro on WSL (Windows Subsystem for Linux)?

Sat, 03 Sep 2022 00:00:00 GMT

Nobody can deny the importance of backup & restore process for any environment. Specially, when multi-steps are involved in setup and configuration and we keep modifying the environment with time. Basically, if anything goes wrong for any reason, we don't want to end up wasting time by setting up th

How to Uninstall or Remove Linux Distro Installed on WSL (Windows Subsystem for Linux)?

Fri, 02 Sep 2022 00:00:00 GMT

Uninstallation of redundant or unused Linux Distros on Windows Subsystem for Linux (WSL) is an important activity to reclaim space and maintain hygiene of the system. It is just one command to remove a distro from WSL. Video: * Run the following command in Windows Command Prompt to find the lis

How to Install Windows Subsystem for Linux (WSL2) on Windows 11?

Tue, 30 Aug 2022 00:00:00 GMT

Steps to install Windows Subsystem for Linux (WSL2) on Windows 10 works for Windows 11 too. But Microsoft made it super simple in Windows 11. Just one command that takes minutes to install a Linux distribution (Ubuntu by default). If you are curious about other 3 different options that I mentioned

How to Install Windows Subsystem for Linux (WSL2) on Windows 10?

Thu, 26 Aug 2021 00:00:00 GMT

Windows Subsystem for Linux (WSL) is an awesome feature of Windows 10. No more VMware or Oracle VirtualBox is required to run Linux on Windows. WSL is faster and less resource consuming, zero or less configuration compared to any other traditional Virtual Machines (VM). Installation and setup of Lin

How to change hostname on Ubuntu running on Windows WSL?

Tue, 13 Jul 2021 00:00:00 GMT

If you are running Ubuntu on Windows WSL (Windows Subsystem for Linux) and wondering how to change hostname permanently then follow all steps mentioned below. Once hostname is changed, if you get "unable to resolve host : Name or service not known" then follow the step #4 for the fix. V

Kid's Eye and Energy Safe Smart TV - Part 2

Thu, 09 Jul 2020 00:00:00 GMT

Sometimes I feel, kids do this just to irritate us. Or may be, we always have a prioritized list of problems in our subconscious mind. When we solve the top one, the next one pops up and gets highlighted. It is indeed never ending. Few extra hours of running TV will hardly make any difference in m

Kid's Eye Safe Smart TV - Part 1

Fri, 19 Jun 2020 00:00:00 GMT

Outside of home COVID-19. Inside at home COVID-16. Whole world is fighting against COVID-19. But for COVID-16, I am all alone. I am tired of saying "please stay away from the screen". So thought of this concept project to avoid brain infection at home. By the way, I call two of my kids (age 1 and 6

Internet enabled (IOT) smoke alarm with SMS notification using esp8266, MQ2, ThingSpeak and Twilio

Mon, 04 Sep 2017 00:00:00 GMT

ESP8266 based Internet enabled (IOT) smoke and fire alarm to send SMS notification (Twilio) and to trigger alarm with flashing red LED if it detects smoke and leakage of combustible gas using MQ2 gas sensor. Also ThingSpeak analytics is a great bonus.

IOT - Home Automation project using Geofencing, MQTT, ESP8266 and MycroPython

Mon, 08 May 2017 00:00:00 GMT

A home automation project that maps switching on/off of appliances with your entry/exit of a geofence (virtual perimeter for a real-world geographic area). Step into a cool room without having to wait for the air conditioning to start functioning? Watch the garage door open as you drive in.

Home Automation - Internet of Things - MQTT (MQ Telemetry Transport), ESP8266 and MycroPython

Wed, 22 Mar 2017 00:00:00 GMT

This is my first post on home automation. As a 'Hello World' project, switching on/off a bulb is a common use case. This post is not an exception. But to spice it up, we'll use MQTT (MQ Telemetry Transport) - a machine-to-machine (M2M)/Internet of Things (IOT) connectivity protocol and an android ap

Configure STA_IF interface of ESP8266 (MycroPython firmware) to connect with WIFI network automatically on boot

Tue, 14 Mar 2017 00:00:00 GMT

ESP8266 can act as both Station / WIFI client (STA_IF interface) and Access Point (AP_IF interface). AP_IF interface is active by default in MycroPython firmware on ESP826 . In the earlier post, we connected to the access point (AP mode) of ESP8266 and used WebREPL to remotely control switch on/off

Example: Remotely switch on/off and control brightness of an LED using MicroPython WebREPL

Wed, 08 Mar 2017 00:00:00 GMT

So far, we have flashed MicroPython firmware onto an ESP8266 and setup WebREPL to run programs over WIFI on the chip. Now it's time for a hands-on excercise. We'll remotely switch on/off and control brightness of an LED using MicroPython WebREPL.

How to setup WebREPL to connect to Python prompt (REPL) of ESP8266 over WIFI network?

Thu, 23 Feb 2017 00:00:00 GMT

WebREPL daemon auto-start is disabled by default in MycroPython on ESP8266 for security reason. Here, we'll enable the same, connect to the REPL over WIFI using locally installed WebREPL client and run a "Hello World" program remotely on the chip.

How to flash MicroPython firmware onto an ESP8266 ESP-12E chip using esptool?

Sat, 04 Feb 2017 00:00:00 GMT

By default, ESP8226 module comes with AT Firmware. In this post, we'll see how to flash MicroPython firmware onto an ESP8266 ESP-12E chip using esptool. Tools,Technologies and Components used in this article 1. ESPlorer IDE 2. MicroPython 3. macOS Sierra 4. Components: * ESP8266 ESP

How to uninstall OpenCV (Open Source Computer Vision) from Raspberry Pi - Raspbian Jessie?

Sun, 10 Jul 2016 00:00:00 GMT

To uninstall OpenCV (Open Source Computer Vision) from Raspberry Pi (Raspbian Jessie), follow the steps mentioned below in this article.

Install and Configure Oracle Java ME Embedded on Raspberry Pi

Tue, 17 Mar 2015 00:00:00 GMT

Raspberry Pi is a low cost, credit-card sized and fastest selling personal computer. In this article, we'll install and configure Oracle Java ME Embedded on a Raspberry Pi. What is a Raspberry Pi? The Raspberry Pi is a low cost, credit-card sized computer that plugs into a computer monitor or