1. Data Sources

The first component of a data pipeline is the data sources. These are the various systems and apps where your data is stored, such as databases, cloud applications, or web services. The data in these sources can come in many different formats, including structured data in a relational database, unstructured data in a NoSQL database, or semi-structured data in a log file.

2. Data Extraction

Once you’ve identified your data sources, the next step is to extract the data and bring it into a common format for processing. This is known as data extraction. The process of data extraction varies depending on the type of data source and the format of the data. For example, you might use SQL queries to extract data from a relational database, or API calls to extract data from a cloud application.

3. Data Transformation

Once the data has been extracted, the next step is to transform it into a structured format that can be loaded into the data warehouse. This is known as data transformation. The process of data transformation includes cleaning and normalizing the data, as well as transforming it into the required format. For example, you might convert dates into a standardized format, or map categorical variables to numerical values.

4. Data Loading

After the data has been transformed, the next step is to load it into the data warehouse or another storage system. This is known as data loading. The process of data loading varies depending on the type of storage system and the format of the data. For example, you might use SQL INSERT statements to load data into a relational database, or use a data ingestion tool to load data into a data lake.

5. Data Storage

The data warehouse or other storage system where the data is stored after it has been loaded is known as data storage. The data storage is typically a centralized repository for storing and managing large amounts of data from multiple sources, typically in a structured and organized manner. The type of data storage you use will depend on the requirements and objectives of your data pipeline, as well as the nature and complexity of the data being stored.

6. Data Processing

Once the data has been loaded into the data storage, the next step is to process it in order to prepare it for analysis and reporting. This is known as data processing. The process of data processing includes aggregating, filtering, and transforming the data, as well as performing other operations on the data to make it useful for analysis and reporting.

7. Data Analysis

The process of using data, analytics, and other tools to gain insights into the data and support decision-making is known as data analysis. Data analysis can involve running queries against the data, building predictive models, or generating visualizations and reports. The specific methods used in data analysis will depend on the requirements and objectives of your data pipeline, as well as the nature and complexity of the data being analyzed.

8. Data Visualization

Data visualization is the process of representing the data in a graphical format, such as charts, graphs, or maps, in order to make it easier to understand and interpret. Data visualization can be used to communicate key insights and trends in the data, and is an important part of the data analysis process.

9. Data Reporting

Data reporting is the process of generating reports or dashboards that summarize the data and provide insights into it. Data reports can be used to communicate key metrics and KPIs to stakeholders, or to track the performance of different parts of the business. Data reports can be generated on a scheduled basis, or on demand, and can be customized to meet the specific needs of the business.

10. Data Monitoring

Finally, it’s important to monitor the data pipeline to ensure that it is running smoothly and producing accurate results. Data monitoring involves tracking the performance and health of the different components of the data pipeline, and identifying and fixing any issues that arise. Data monitoring is critical for ensuring that the data pipeline is reliable and delivering accurate results, and can help to identify and resolve issues before they become major problems.

Understanding the key components and terminology of a data pipeline is important for anyone who wants to build and manage a data pipeline effectively. Whether you’re a data analyst, a data engineer, or a business analyst, understanding these key concepts will help you to build and manage data pipelines that deliver accurate and actionable results.

The post Understanding the Key Components of a Data Pipeline appeared first on Daniel Iser.

This is handled by resolvers typically, but in this scenario, to get all of the tooling speaking the same language, it requires changes to multiple configurations.

Following most guides online I was able to resolve one issue after another, so I figured I would share my experience, the problems I faced and the solutions that brought it together.

The usual recommendations start with adding something like a "paths" object to your tsconfig.json file unfortunately that doesn’t work out of the box on its own with your IDE or the greater wp-scripts build routines. It takes multiple changes to get it all respecting your custom aliases.

The examples below are based on real code, with files structured as

- src
  - components
    - controlled-tab-panel
      -index.tsx
  - settings-page
    - container
      - index.tsx

And we want to switch from

import TabPanel from '../../components/controlled-tab-panel';

To something simple such as

import TabPanel from '@components/controlled-tab-panel';

Problems this helps solve.

Typescript Error: Cannot find module ‘@components/controlled-tab-panel’ or its corresponding type declarations.ts(2307)

ESLint Issues in VS Code: Unable to resolve path to module ‘@components/controlled-tab-panel’.eslintimport/no-unresolved

Webpack builds fail: Module not found: Error: Can’t resolve ‘@components/controlled-tab-panel’

Get resolvers working in Typescript

First up get it working in Typescript. Edit your tsconfig.json , make sure “baseUrl” is set and add a “paths” object that includes aliases and their src path. Assuming the example setup described above.

{
	"compilerOptions": {
		"baseUrl": "./src",
		"paths": {
			"@components/*": [
				"components/*"
			]
		},
	}
}

Get wp-scripts/ Webpack builds working again.

If you haven’t already extend the default WP config with your own.

Modify your config accordingly like so.

// defaultConfig references imported base config from @wordpress/scripts
{
	resolve: {
		...defaultConfig.resolve,
		alias: {
			...defaultConfig.resolve.alias,
			// add as many aliases as you like!
			'@components': path.resolve( __dirname, 'src/components' ),
		},
	},
}

Fix eslint issues with the same resolvers

With the above, everything technically “works”, but the IDE (VS Code for me) still shows warnings from linting, and wp-scripts linter will throw those same issues.

The first thing you need to do is make your life easier, install eslint-import-resolver-webpack. I also have eslint-plugin-import set up before, so not sure if it was part of the overall solution or not. Further our config extends the plugin:@wordpress/eslint-plugin/recommended library.

Merge the following into your .eslintrc.js or similar file.

{

	settings: {

		'import/resolver': {
			webpack: {
				config: 'webpack.config.js',
			},
		}
	}
}

The above simply maps your current working webpack resolver config to also be used for eslint.

Hope that helps someone out there, took me hours to work out all the individual issues and solutions.

The post Getting module aliases working with @wordpress/scripts, Webpack, Typescript & ESLint appeared first on Daniel Iser.

Since most of us use Git via Github or similar to manage the actual code and day to day changes, we must use SVN from time to time to get a release out to the public.

I’ve put together the following cheat sheet for WordPress.org SVN usage that has some optimized commands for doing the necessary stuff very quickly.

I know there are some good Github actions and workflows to automate this, but some times it just helps to have a few quick CLI commands that work.

Hope you find them helpful.

Checkout

Checkout Trunk Only (Preferred)

If you don’t need to compare versioned tags or modify wordpress.org plugin page assets this is the ideal way to do it by only downloading the /trunk working branch.

To check out only the trunk working branch you will use the following methodology, replacing plugin-name with the correct plugin slug.

svn checkout --depth empty https://plugins.svn.wordpress.org/plugin-name/ plugin-name
svn update --set-depth infinity plugin-name/trunk

Checkout WordPress Page Assets Only

To modify banner images or plugin icons this will get the correct folder set up.

svn checkout --depth empty https://plugins.svn.wordpress.org/plugin-name/ plugin-name
svn update --set-depth infinity plugin-name/assets

Stage Changes

Remove Missing Files

Remove files that have been deleted since last release.

svn st | grep "!" | cut -d! -f2 | sed 's/^ *//' | xargs -n 500 -d "\n" -r svn rm

Add New Files

Add files that were created since last release.

svn st | grep ^? | cut -d? -f2 | sed 's/^ *//' | xargs -n 500 -d "\n" -r svn add

Fix Asset Content-Type

This only applies to the wordpress.org plugin /assets directory which needs to serve these files as images, not as binary data like the SVN repository does.

svn propset svn:mime-type image/png *.png
svn propset svn:mime-type image/jpeg *.jpg
svn propset svn:mime-type image/gif *.gif
svn propset svn:mime-type image/svg+xml *.svg
svn propset svn:mime-type image/svg+xml *.svgz

Push Changes

Commit Trunk Changes

Make sure to to run an svn st and double check everything looks right before proceeding. You would immediately follow this by tagging the release remotely, found below.

cd plugin-name/trunk
svn ci -m "Updating code to release vx.x.x"

# Example Output
> Enter your wordpress.org username
> Enter your wordpress.org password
> Sending        trunk/plugin-name.php
> Transmitting file data .................................
> Committing transaction...
> Committed revision 2687342.

Tagging Release Remotely

When working with only the /trunk branch locally this method allows first pushing the trunk updates, then remotely copying /trunk to a new versioned folder in the /tags directory on the remote SVN repository such as /tags/1.5.3.

svn copy https://plugins.svn.wordpress.org/plugin-name/trunk \
           https://plugins.svn.wordpress.org/plugin-name/tags/1.0.0 \
           -m "Tagging the 1.0.0 release"

# Example Output
> Enter your wordpress.org username
> Enter your wordpress.org password
> Committing transaction...
> Committed revision 2687342.

Quick Hacks

Update already tagged readme

Sometimes you need to update just the readme.txt file. This can be done easily with only the /trunk branch locally. Update plugin-name and 1.0.0 version according to your needs.

First modify the readme.txt file then use the following.

cd plugin-name/trunk
svn ci -m "Readme update" readme.txt

# Remove the readme in the latest tag folder.
svn rm -m "Remove outdated readme" \
		https://plugins.svn.wordpress.org/plugin-name/tags/1.0.0/readme.txt

# Copy new trunk/readme to the latest tag folder.
svn copy -m "Updating readme for existing tag" readme.txt \
		https://plugins.svn.wordpress.org/plugin-name/tags/1.0.0
           

The post Working with WordPress.org SVN Cheat Sheet appeared first on Daniel Iser.

If your lucky enough to have a NAS with abundant space, or are willing to add one, you can back up your games there.

This allows you to delete some when you need to make room without them being lost on the NAS, and if you want them back you simply copy that game back to your Steam library, then click the install button in Steam. It will scan the files and update only what is needed.

But copying games every time Steam pushes updates can be a tedious and time consuming task.

I wrote the following PowerShell script to simplify & automate this process. Once you set the paths it will auto sync all games in your library to the NAS.

It does this by looping over each game folder in your library and mirroring it to your NAS game library using the Robocopy.exe command.

We loop over each game separately as we are using mirror copy which would delete games on the NAS that didn’t exist locally if we did it on the library directly. But mirror mode is excellent at keeping a direct & up to date copy on the NAS without leaving extra files.

In the code below I also took advantage of the small speed boost for mapped network drives, so G:\ represents a share on my NAS specifically for game libraries.

Set it to run overnight and your all set. If you have a large library or slow network connection to the NAS this can take some time.

Side note, this could also be used to move games from an SSD to a slower HDD, if you wanted you could even modify it to add symbolic links and play games from the other folder.

The post Sync Steam Game Library to NAS using PowerShell appeared first on Daniel Iser.

In a recent Windows Insider update, along with new drivers from Nvidia, support for running GPU based tasks with Tensorflow pretty simple.

You can follow the documentation, or watch a video outlining it here. There were a few things that were out of date in that guide, so check out the notes below.

I’m here to cover pros, cons, pitfalls and just log the issues I faced and how I solved them in hopes to help others and prevent myself from wasting time in the future.

Once you have the base setup working I highly suggest exporting your WSL instance so you can reset to that point later.

What we’re gonna do:

1. Follow the Nvidia docs to set up Cuda support inside WSL2. This includes updating Windows, drivers, installing Cuda toolkit and Docker.
2. Install python & pip.
3. Install anaconda3 or miniconda3.

Install python & pip

For Ubuntu 20.04 Python3 is the default. If you are using Ubunto 18.04 or older you can use these instructions to update to Python3.

In my case I only need to install pip using the following:

sudo apt update
sudo apt install python3-pip

Tip: copy the following to your ~/.bashrc file to use python and pip instead of python3 and pip3.

# Enter `which python3` & `which pip3` into the command line to get the paths.
alias python='/usr/bin/python3' 
alias pip='/usr/bin/pip3'

Setup Issues & Guide Corrections

The following issues were things I either ran into or had previously. I have provided solutions and corrections where available.

Commands differ for Ubuntu 20.04

Specifically when adding apt-keys under the Setting up CUDA Toolkit section, be sure to change the ubuntu1804 text to ubuntu2004 to match your version.

Cuda Toolkit is now v11.2

Install Toolkit using apt-get install -y cuda-toolkit-11-2 instead.

Docker Installer Confusion

When running the docker installer it detects the /mnt/c/ProgramFiles copy of docker. It seems like its not gonna work, but if you just wait for the 20-40s sleep timers it does finish properly.

Disable Docker for Windows

If you already use Docker for Windows, be sure to go into its settings → Resources → WSL Integration and uncheck your Machine Learning WSL instance. This is just a precaution so its not trying to constantly install itself.

Outdated Jupyter Example

The Nvidia setup docs show an example for Jupyter Notebooks. Unfortunately it took me hours to realize it was very out of date. Over a year old in fact. This became apparent when some of the tutorial notebooks wouldn’t complete the first few steps due to missing functions such as subsplit.

Instead use the following:

docker run -it --gpus all -p 8888:8888 tensorflow/tensorflow:latest-gpu-jupyter

Docker doesn’t autostart

After playing for a while you will quickly become annoyed having to run sudo service docker start ever time. Follow these 2 guides to fix this:

1. Configure it to start automatically.
2. How to automatically start the Docker daemon on WSL2

The post Machine Learning with WSL2, Cuda GPU Acceleration & Docker appeared first on Daniel Iser.

You can find recent examples here.

SVG Filter Generator

https://codesandbox.io/s/v0l0j

Here you can see that as you add “Filters” and chain them to the “Result” node, the output code changes in the textarea field. It includes custom nodes for each filter, node meta as well as actual usage of the node graph as you build it.

Subflows – Flows Inside Nodes

https://codesandbox.io/s/szcsn

This one took me by surprise, once I saw it, the simplicity of it was pretty awesome. It uses the “Contextual Zoom” effect to have nodes that as you zoom in reveal their own subflow graph that can be managed within that context.

Go ahead and try it, zoom in on Node 2 below.

The post React Flow Example Resources appeared first on Daniel Iser.

Working as of 04/23/2021

After following a few guides (linked below) I wasn’t able to get Minikube running under WSL2. So this guide serves as a quick TL:DR that worked for me and was repeatable.

I will say that I have given my user password-less sudo access, which you can read about in my guide to setting up WSL for development.

Install WSL2 systemd & prerequisites

From the script by DamionGans

Make sure you have git installed.

sudo apt install git

Then clone the helper script which automates systemd setup in WSL containers.

git clone https://github.com/DamionGans/ubuntu-wsl2-systemd-script.git
cd ubuntu-wsl2-systemd-script/
bash ubuntu-wsl2-systemd-script.sh
# Enter your password and wait until the script has finished

Close the terminal and reopen a new one and verify its working.

systemctl

If you see a list of units, the script worked.

Install Minikube

Minikube requires conntrack packages to work properly, so install that prior to moving on.

sudo apt install -y conntrack

Finally install minikube itself, and move it to the bin folder so its globally available when you restart the bash.

# Download the latest version of Minikube
curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
# Make the binary executable
chmod +x ./minikube
# Move the binary to your executable path
sudo mv ./minikube /usr/local/bin/

minikube config set driver docker

Attach or Reattach Docker Desktop WSL Integration

Finally start minikube and run the dashboard

minikube start

minikube status

minikube dashboard

And now when you visit the link provided you should see a functional Kubernetes dashboard: http://127.0.0.1:40143/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/#/overview?namespace=default

Guides with more details on each of these steps:

• https://kubernetes.io/blog/2020/05/21/wsl-docker-kubernetes-on-the-windows-desktop/
• https://forum.snapcraft.io/t/running-snaps-on-wsl2-insiders-only-for-now/13033

The post Setup Minikube on WSL2 for Windows 10 appeared first on Daniel Iser.

1. Set up new server.
2. Install clean copy of WordPress
3. Set up test domain. (Optional)
4. Set up SSL on the new server.
5. Install WP Migrate DB Pro.
6. Clone old site to new server:
   • Migration service
   • WP Migrate DB Pro addons
7. Test thoroughly with temporary or test domain.
8. Modify local hosts file so that yourdomain.com points to the new servers IP and test again.
9. Put up 503 maintenance mode on old site.
10. Migrate data one last time.
11. Migrate DNS.

Tips

• Use CloudFlare as DNS changes are near instant.
• Use WP Migrate DB Pro to do last sync after testing/before dns change.

The post Migrating Ecommerce Sites without Data Loss appeared first on Daniel Iser.

How to export a WSL distro image

Exporting a WSL distro is a fantastic way to back it up, clone it, share it or just about anything else you can think of. Use the following commands to do it quickly from the PowerShell or Command Prompt terminals.

Arguments

Distro Name

Unique identity for each instance.

Distro File

Path and file name for the image that will be exported.

Syntax

# Syntax
wsl --export DistroName DistroFile
# Example where Ubuntu is the Distro Name and exporting the Ubuntu-20.04--Clean.tar image.
wsl --export Ubuntu D:\WSL\Ubuntu-20.04--Clean.tar

Back to top

How to import WSL distro image

Importing WSL distributions from tar image files is extremely fast & simple. Use the following commands to do it quickly from the PowerShell or Command Prompt terminals.

Arguments

Distro Name

Unique identity for each instance.

Install Path

Path to store the VM disk image.

Distro File

A tar file containing the distro image.

Syntax

# Syntax
wsl --import DistroName InstallPath DistroFile
# Example where Ubuntu is the Distro Name, we are storing it in D:\Ubuntu folder and importing the Ubuntu-20.04.tar image.
wsl --import Ubuntu D:\Ubuntu D:\WSL\Ubuntu-20.04.tar

Note: There is a known issue when importing a WSL distro in that it will not properly log in to your user account by default. The fix is simple enough.

Run sudo nano /etc/wsl.conf to edit the WSL config file. Add the following and change the username to match your own.

[user]
default=username

Back to top

How to remove an existing WSL distro

The following syntax is used in either the PowerShell or cmd command prompt.

Arguments

Distro Name

Unique identity for each instance.

Syntax

# Syntax
wsl --unregister DistroName
# Example where Ubuntu is the Distro Name.
wsl --unregister Ubunt

Back to top

Reset an existing WSL distribution

Resetting a distribution is a bit different as you don’t use the command line.

1. Open the Start Menu and search for Apps & features.

2. Search for your distro in the list, click it and choose the Advanced options link.

3. Scroll down and click the Reset button. The next time you run your WSL terminal for that distribution it will be a clean distro.

Back to top

The post Importing & Exporting WSL Linux Distros appeared first on Daniel Iser.

You are likely here because you have a WordPress plugin that manages user data in some way and your users are SCREAMING about GDPR Compliance (as they should be). This is currently the only guide to GDPR compliance for WordPress plugins until the WP Codex is updated.

This will help you get started with what support WordPress core is bringing in the near future. I will keep this updated as new features are offered.

There was a 3rd party GDPR standards plugin being developed but it got enough attention to get co-opted into the core at least in theory. It used class-based implementations in PHP, where core chose to go the more common WP way using hooks & filters.

The final details of how these functionalities will work for users and site owners are still in the works, but users will be able to request their data for export or removal, and site admins will be able to approve these requests in an automated fashion. Here is a quick rundown of what is known so far.

WP Core v4.9.6 will include several new Privacy functions functionalities including:

• Privacy Settings
• Privacy Policy Guide
• Personal Data Exporter
• Personal Data Eraser
• Helper Functions
• WooCommerce
• Guide Change Notes

There are a few other items coming such as general cookie consent messages, GDPR optin fields on all core forms and a few other things to make site owners lives simpler when dealing with compliance. Below I will detail each of the v4.9.6 functionalities and how to use them for your plugin.

Privacy Settings

A new settings page available at Settings -> Privacy has been added to the admin & network admin areas.

This will offer the ability to create or assign a page to hold your site or networks Privacy Policy.

This page also introduces a new user capability manage_privacy_options which allows limiting which site administrators have access to change these settings apart from manage_options.

Multisite Note: By default, only network admins will be able to set these options. There will be an additional option to change it from Network Wide Policy to Per Site Policy. Only when the network admin sets it to Per Site will the extra Settings -> Privacy page for every single site appear. These pages will be hidden otherwise.

Privacy Policy Guide – GDPR Privacy Policy

With the latest copy of core v4.9.6-rc.1 the Privacy Policy Guide is now a dedicated page. Before it was located above the content editor of your selected privacy page, but this led to some confusion due to the overwhelming business of the interface as seen below.

Now when editing the Privacy Page a notice like this will be shown instead, linking to a (currently hidden) inner admin page. You can access it manually by visiting the /wp-admin/tools.php?wp-privacy-policy-guide=1 on your domain.

This new page is broken into multiple sections with a nice table of contents at the top. The number of sections will vary both with the number of plugins and with time as more plugins add their own information. Each section excluding the Introduction also includes a Copy button to quickly grab it for pasting into your privacy policy text.

Each plugin will them be able to add its own section or sections, near mimicking what WordPress core has done where needed to inform the user of what information they need to add or disclose in their privacy policy as well as what section it should go under.

As of now these are the processes I have noticed during testing.

• When first installed a new page is generated for a privacy policy.
• New options in the admin allow changing this assigned & mandatory page.
• When you edit the privacy policy page at first it will have a basic template of sections & headings inserted into the editor. There will also be a notice with a link to the Privacy Policy Guide page.
• Any time a plugin changes the suggested text, plugin or theme is activated or deactivated, it will notify the admins that the privacy policy may need to be updated.
• Eventually (I believe) core will include agreeing to checkboxes and possibly a cookie notice that points to these policies.

Registering additional content for the Privacy Policy Generator in your WP plugin is pretty easy.

The example below shows how this would look.

Personal Data Exporter – GDPR Right of Access 

Data portability requirements mean that users will be able to request all data a site has stored about them and site admins will be able to approve and send them a zipped HTML file by email that includes all of their information.

This works by looping through a list of registered data exporters and rendering the info passed back. It is still in the early but very reliable stages of development, and already available in the WP nightly betas as of now.

Registering a custom Personal Data Exporter for your WordPress plugin is rather easy. The following is a basic example for a fictitious plugin that adds additional user profile information via user_meta. For a version that includes a pagination example such as how core handles comments, see here

Personal Data Eraser – GDPR Right to be Forgotten

The term “eraser” in this context refers to a process that will either anonymize or remove all data related to a user per their request. If the data cannot be removed for some reason an explanation should be given.

Erasers work in much the same way as the exporters. You first register a callback and label. Erasers do 2 things:

• They attempt to anonymize or remove the user’s data.
• When data cannot be removed, it returns a message indicating why that data could not be erased or anonymized.

As such an exporters return should always look like this.

return array(
    'items_removed' => false, // Boolean whether items were anonymized or removed.
    'items_retained' => false, // Boolean whether items were not able to be anonymized and were retained.
    'messages' => array(),
    'done' => true,
);

Note: items_retained refers to data that couldn’t be erased, for instance, a WooCommerce order that has not shipped yet.

Registering a custom Personal Data Eraser for your WordPress plugin is just as simple as the exporter. Here is an example based on our social profiles example above. Example of how core anonymizes comments found here

Helper functions to make it easier.

Function to anonymize any data type. Types include email, url, ip, date, text, longtext.

Since there is a new Privacy Policy page there are new helper functions to get that pages URL. I imagine there will shortly be conditionals such as is_privacy_policy_page() or similar.

get_privacy_policy_url();
the_privacy_policy_link( $before, $after );

There are also a couple functions to help anonymize data when possible rather than removing it.

/**
 * Helper to anonymize data.
 * 
 * @param $type string accepts email, url, ip, date, text, & longtext.
 * @param $value string value to be anonymized.
 *
 * @return string
 */
wp_privacy_anonymize_data( $type, $value );

/**
 * Helper to anonymize IP addresses.
 *
 * @param $ip_address string IP adddress to be anonymized.
 *
 * @return string
 */
wp_privacy_anonymize_ip( $ip_address )

WooCommerce

If you develop products that extend WooCommerce check out this guide for some tips on doing that quickly.

Changelog

The purpose of this log is to keep track of changes after publishing in case you want to know exactly what has changed since you began implementation.

• 05/29/2018 – Added link to WooCommerce specific developer information.
• 05/12/2018 – Added section for Privacy Settings page, new user capability & network admin options.
• 05/14/2018 – Updated section on Privacy Policy Guide to match v4.9.6-rc.1. Reorganized sections to be more logically ordered.

The post Practical Guide to GDPR compliance for WordPress plugin & theme developers. appeared first on Daniel Iser.