🌀 Gate IV – The Flux of Motion: Currents and Bridges

As you emerge from the Hall of Convergence, the ground beneath you trembles. Before you stretches the River of Time — a swirling torrent of frozen moments, captured in ancient video scrolls that flicker with ethereal motion. The Guardian of Flux appears, a spectral figure rippling like water:

“The world does not stand still, apprentice. It flows, it shifts, it deceives. To cross this river, you must trace the unseen currents — the optical flow — that binds past to present. Warp the waves, stabilize the storm, and reveal the hidden paths of movement. Only then shall the final gate open, restoring the full World of Vision.”

In this gate, you delve into optical flow: the apparent motion of pixels between frames. This is the essence of dynamic vision — understanding how scenes evolve over time. Succeed here, and you graduate as a full Visioneer, guardian of the reconstructed archive.

🎯 Mission Objectives

1. Implement Lucas-Kanade Flow: Compute u,v motion per pixel using local neighborhoods (no cv2).
2. Optical Flow Warping: Use LK flow to warp one frame onto another, simulating motion compensation (no cv2).

Pre-defined Code

Run these cells to load your enchanted video scrolls (8 short clips from the Archive). They'll take ~5 mins on first load.

# Mounting google drive
from google.colab import drive
drive.mount('/content/drive')

Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).

pip install scipy

Requirement already satisfied: scipy in /usr/local/lib/python3.12/dist-packages (1.16.3)
Requirement already satisfied: numpy<2.6,>=1.25.2 in /usr/local/lib/python3.12/dist-packages (from scipy) (2.0.2)

# Importing all the required libraries
import numpy as np
import cv2
import matplotlib.pyplot as plt
from matplotlib import style
from glob import glob
from natsort import natsorted

from tqdm import tqdm
import imageio
from IPython.display import Image
from scipy import ndimage
style.use('ggplot')

# Loading all the images from the drive
videos_images = []

for path in tqdm(natsorted(glob('/content/drive/My Drive/ES666CV/videos/D/*'))):
    sub_path = path + '/*'
    video_frames = []
    for files in natsorted(glob(sub_path)):
        video_frames.append(cv2.imread(files, 1))
    video_frames = np.array(video_frames)
    videos_images.append(video_frames)

print(f"Loaded {len(videos_images)} videos.")

# videos_images = np.array(video_frames) # Only possible because all images are of same size

100%|██████████| 8/8 [00:05<00:00,  1.58it/s]

Loaded 8 videos.

def show_image_grid(images, M, N, title='Title', figsize=8):
    # Assuming 'images' is a numpy array of shape (num_images, height, width, channels)
    if M==1:
        row_size = figsize
        col_size = figsize//4
    elif N==1:
        row_size = figsize//4
        col_size = figsize
    else:
        row_size, col_size = figsize, figsize

    fig, axes = plt.subplots(M, N, figsize=(row_size, col_size))

    if len(images.shape) < 4:
        images = np.expand_dims(images.copy(), axis=0)

    fig.suptitle(title)
    for i in range(M):
        for j in range(N):
            if M==1 and N==1:
                ax = axes
            elif M == 1 or N==1:
                ax = axes[max(i, j)]
            else:
                ax = axes[i, j]
            index = i * N + j
            if index < images.shape[0]:
                ax.imshow(cv2.cvtColor(images[index], cv2.COLOR_BGR2RGB))
            ax.axis('off')
    plt.tight_layout()
    plt.show()

def images_to_gif(frames, video_id):
    with imageio.get_writer(f"{video_id}.gif", mode="I", fps=24) as writer:
        for idx, frame in enumerate(frames):
            frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
            writer.append_data(frame)

def show_gif(video_id):
    with open(f"/content/{video_id}.gif",'rb') as f:
        display(Image(data=f.read(), format='png'))

# Preview the raw scrolls (videos 0-7)
for video_idx, video_frames in enumerate(tqdm(videos_images)):
    images_to_gif(video_frames, video_idx)
    show_gif(video_idx)

Task 1: Lucas-Kanade – Gripping the River's Currents

The Guardian of Flux gestures to the River of Time, its surface deceptively calm: "See how the waters shift unseen? To grip this flux, apprentice, implement the Lucas-Kanade rite—from scratch, peering into neighborhoods with convolve2d kernels. Trace u,v motion per pixel, assuming constancy in the flow. Visualize arrows on the grayscale scrolls of video 0 (frames 0→8), revealing the currents' direction."

Your Quest (3 pts): Craft the Lucas-Kanade optical flow algorithm. Compute gradients (Ix, Iy, It) with convolve2d kernels. Solve local systems for [u,v] in windows. Draw arrows on significant motion on normalized grayscale base.

Outputs: Arrowed grayscale GIF frames of flow over video.

The Guardian's Tomes (References as Ancient Scrolls)

• Optical Flow (Shi-Tomasi, Sparse LK/HS, Dense Farneback) - Part I – Rite of tracing.
• Implementing Lucas-Kanade in Python – Neighborhood forge.
• The Math Behind Optical Flow – Constancy's equation.

Forge the flow... 🌀

Lucas–Kanade Optical Flow: Theory & Mathematics

The Lucas–Kanade method assumes that the motion (optical flow) between two consecutive frames is small and constant within a local neighborhood.

---

1. Brightness Constancy Assumption

$ I(x, y, t) = I(x + u, y + v, t + 1) $ Expanding with a Taylor series and ignoring higher-order terms: $ I_x u + I_y v + I_t = 0 $ where

• $ I_x = \frac{\partial I}{\partial x} $ — gradient along x
• $ I_y = \frac{\partial I}{\partial y} $ — gradient along y
• $ I_t = \frac{\partial I}{\partial t} $ — temporal gradient between frames

---

2. Local Window Solution

For each pixel, consider a local window $ W $ and solve for $ (u, v) $: $ \sum_{(x, y) \in W} \begin{bmatrix} I_x^2 & I_x I_y \\ I_x I_y & I_y^2 \end{bmatrix} \begin{bmatrix} u \\ v \end{bmatrix}

• \sum_{(x, y) \in W} \begin{bmatrix} I_x I_t \\ I_y I_t \end{bmatrix} $

---

3. Least-Squares Solution

$ \begin{bmatrix} u \\ v \end{bmatrix} = (A^T A)^{-1} A^T b $ where
$ A = [I_x, I_y] $ and $ b = -I_t $.

from scipy.signal import convolve2d
import cv2
import numpy as np

def optical_flow(frames, video_id):
    # Convert all frames to normalized grayscale
    gray_frames = [cv2.cvtColor(f, cv2.COLOR_BGR2GRAY).astype(np.float32) / 255.0 for f in frames]

    # Define derivative kernels
    kernel_x = np.array([[-1, 1],
                         [-1, 1]], dtype=np.float32) * 0.25
    kernel_y = np.array([[-1, -1],
                         [1, 1]], dtype=np.float32) * 0.25
    kernel_t1 = np.ones((2, 2), dtype=np.float32) * 0.25
    kernel_t2 = -kernel_t1

    output_frames = []

    # Process consecutive frame pairs
    for i in range(len(gray_frames) - 1):
        I1, I2 = gray_frames[i], gray_frames[i + 1]

        # Compute gradients using convolution
        Ix = (convolve2d(I1, kernel_x, boundary='symm', mode='same') +
              convolve2d(I2, kernel_x, boundary='symm', mode='same'))
        Iy = (convolve2d(I1, kernel_y, boundary='symm', mode='same') +
              convolve2d(I2, kernel_y, boundary='symm', mode='same'))
        It = (convolve2d(I2, kernel_t1, boundary='symm', mode='same') +
              convolve2d(I1, kernel_t2, boundary='symm', mode='same'))

        # Initialize optical flow fields
        u, v = np.zeros_like(I1), np.zeros_like(I1)
        window_size = 5
        w = window_size // 2

        # Estimate flow vectors per local window
        for y in range(w, I1.shape[0] - w):
            for x in range(w, I1.shape[1] - w):
                Ix_local = Ix[y - w:y + w + 1, x - w:x + w + 1].flatten()
                Iy_local = Iy[y - w:y + w + 1, x - w:x + w + 1].flatten()
                It_local = It[y - w:y + w + 1, x - w:x + w + 1].flatten()

                A = np.vstack((Ix_local, Iy_local)).T
                b = -It_local

                ATA = A.T @ A
                ATb = A.T @ b

                # Solve only if matrix is well-conditioned
                if np.linalg.det(ATA) >= 1e-5:
                    nu = np.linalg.inv(ATA) @ ATb
                    u[y, x], v[y, x] = nu[0], nu[1]

        # Prepare visualization frame
        vis = cv2.normalize(I1, None, 0, 255, cv2.NORM_MINMAX).astype(np.uint8)
        vis = cv2.cvtColor(vis, cv2.COLOR_GRAY2BGR)

        # Draw motion vectors (red arrows)
        step = 8
        for y in range(0, vis.shape[0], step):
            for x in range(0, vis.shape[1], step):
                if np.hypot(u[y, x], v[y, x]) > 0.5:
                    end_x = int(x + u[y, x] * 5)
                    end_y = int(y + v[y, x] * 5)
                    cv2.arrowedLine(vis, (x, y), (end_x, end_y),
                                    (0, 0, 255), 1, tipLength=0.3)

        output_frames.append(vis)

    # Save visualization as GIF
    optical_flow_frame = output_frames
    images_to_gif(optical_flow_frame, f"{video_id}_optical")

for video_idx, video_frames in enumerate(tqdm(videos_images)):
    optical_flow(video_frames, video_idx)

100%|██████████| 8/8 [17:19<00:00, 129.88s/it]

for video_idx in tqdm(range(len(videos_images))):
    show_gif(str(video_idx)+'_optical')

Color and all frames(Extra)

from scipy import signal
import numpy as np
import cv2

def optical_flow_consecutive(im1, im2, window_size, threshold=1e-2):
    """
    Estimate dense optical flow (u, v) between two consecutive images using the Lucas-Kanade method.

    Steps:
    1. Compute spatial (Ix, Iy) and temporal (It) image gradients using convolution filters.
    2. For each pixel, collect gradients from its local neighborhood (window_size × window_size).
    3. Solve the least-squares system A·[u, v]^T = -b for motion vectors.
    4. Ensure numerical stability using Shi-Tomasi’s criterion (smallest eigenvalue > threshold).

    Args:
        im1 (np.ndarray): Frame at time t.
        im2 (np.ndarray): Frame at time t+1.
        window_size (int): Size of the local neighborhood window.
        threshold (float): Eigenvalue threshold for well-conditioned regions.
    """

    # Convert both frames to grayscale and normalize to [0,1]
    im1 = cv2.cvtColor(im1, cv2.COLOR_BGR2GRAY) / 255.0
    im2 = cv2.cvtColor(im2, cv2.COLOR_BGR2GRAY) / 255.0

    half_w = window_size // 2

    # Gradient kernels
    kernel_x = np.array([[-1., 1.], [-1., 1.]])
    kernel_y = np.array([[-1., -1.], [1., 1.]])
    kernel_t = np.array([[1., 1.], [1., 1.]])

    # Compute derivatives
    Ix = signal.convolve2d(im1, kernel_x, boundary='symm', mode='same')
    Iy = signal.convolve2d(im1, kernel_y, boundary='symm', mode='same')
    It = signal.convolve2d(im2, kernel_t, boundary='symm', mode='same') \
       + signal.convolve2d(im1, -kernel_t, boundary='symm', mode='same')

    # Initialize optical flow fields
    u = np.zeros_like(im1)
    v = np.zeros_like(im1)

    # Lucas-Kanade per-pixel estimation
    for i in range(half_w, im1.shape[0] - half_w):
        for j in range(half_w, im1.shape[1] - half_w):
            # Extract local window patches
            Ix_patch = Ix[i-half_w:i+half_w+1, j-half_w:j+half_w+1].flatten()
            Iy_patch = Iy[i-half_w:i+half_w+1, j-half_w:j+half_w+1].flatten()
            It_patch = It[i-half_w:i+half_w+1, j-half_w:j+half_w+1].flatten()

            # Construct system A·v = -b
            A = np.vstack((Ix_patch, Iy_patch)).T
            b = It_patch.reshape(-1, 1)

            # Check conditioning using minimum eigenvalue
            if np.min(np.linalg.eigvals(A.T @ A)) < threshold:
                continue

            # Solve for [u, v]
            flow = np.linalg.pinv(A) @ b
            u[i, j] = flow[0, 0]
            v[i, j] = flow[1, 0]

    return u, v, Ix, Iy, It

def draw_on_frame(frame, U, V, color=(0, 255, 0), scale=5, step_size=10, magnitude_threshold=1.0):
    """
    Draw flow vectors as arrows on an image using OpenCV.
    Arrows are drawn only for significant motion magnitudes.
    """
    line_color = color
    max_magnitude = np.sqrt(np.max(U**2 + V**2))

    for i in range(0, frame.shape[0], step_size):
        for j in range(0, frame.shape[1], step_size):
            u, v = U[i, j], V[i, j]
            magnitude = np.sqrt(u**2 + v**2)

            # Skip weak motions
            if magnitude > magnitude_threshold * max_magnitude:
                # Normalize and scale the arrow length
                u_scaled = (u / max_magnitude) * scale
                v_scaled = (v / max_magnitude) * scale
                cv2.arrowedLine(
                    frame,
                    (j, i),
                    (int(round(j + u_scaled)), int(round(i + v_scaled))),
                    line_color,
                    1,
                    tipLength=0.3
                )
    return frame

def overlay_flow_on_frame(frame, u, v, color, step=10, scale=1):
    """
    Overlay optical flow vectors on a frame using matplotlib's quiver plot.
    Returns a new frame with quiver arrows drawn on it.
    """
    # Prepare coordinate grids
    y, x = np.mgrid[0:u.shape[0], 0:u.shape[1]]

    # Downsample vectors for clarity
    x_ds = x[::step, ::step]
    y_ds = y[::step, ::step]
    u_ds = u[::step, ::step]
    v_ds = v[::step, ::step]

    # Convert to RGB for visualization
    frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    plt.figure(figsize=(10, 10))
    plt.imshow(frame_rgb)

    plt.quiver(
        x_ds, y_ds, u_ds, v_ds,
        color=color, angles='xy',
        scale_units='xy', scale=scale
    )
    plt.axis('off')
    plt.savefig("temp_overlay.png", bbox_inches='tight', pad_inches=0)
    plt.close()

    overlay_frame = cv2.imread("temp_overlay.png")
    return overlay_frame

def optical_flow(frames, video_id, scale=1):
    """
    Compute optical flow between consecutive frames and visualize as arrowed overlays.
    Creates and saves a GIF of all overlay frames.
    """
    optical_flow_frames = []

    for i in range(len(frames) - 1):
        im1, im2 = frames[i], frames[i + 1]
        u, v, _, _, _ = optical_flow_consecutive(im1, im2, 15)

        # Adjust visualization settings per video ID
        if video_id < 6 and video_id != 1:
            overlay = overlay_flow_on_frame(im1, u, v, "red", 5, 0.5)
        elif video_id == 1:
            overlay = overlay_flow_on_frame(im1, u, v, "red", 5, 0.2)
        elif video_id in [6, 7]:
            overlay = overlay_flow_on_frame(im1, u, v, "blue", 8, 0.5)
        elif video_id == 8:
            overlay = overlay_flow_on_frame(im1, u, v, "yellow", 5, 0.3)
        elif video_id == 9:
            overlay = overlay_flow_on_frame(im1, u, v, "blue", 10, 0.9)
        elif video_id == 10:
            overlay = overlay_flow_on_frame(im1, u, v, "red", 10, 0.8)
        else:
            overlay = overlay_flow_on_frame(im1, u, v, "red", 5, 0.5)

        optical_flow_frames.append(overlay)

    images_to_gif(optical_flow_frames, f"{video_id}_optical")

for video_idx, video_frames in enumerate(tqdm(videos_images)):
    optical_flow(video_frames, video_idx)

100%|██████████| 8/8 [1:06:56<00:00, 502.09s/it]

for video_idx in tqdm(range(len(videos_images))):
    show_gif(str(video_idx)+'_optical')

Quest Time (1pt): The Guardian pauses, the river's murmur soft: "You've felt the flow's first tug, apprentice. Sit by the water and answer these simple thoughts—let them sharpen your eyes."

• What story do the arrows tell? Do they all point one way, like a steady stream, or twist like wind? What might that mean for the scene?

• If arrows are few or short, what does it represent?

What story do the arrows tell?
The arrows show how things in the video are moving.

• If they all point in one direction, it means everything is moving the same way — like the camera is panning or the whole scene is shifting together.
• If they twist or go in different directions, it means different parts are moving differently — like people, animals, or objects moving on their own.

If arrows are few or short, what does it represent?
If the arrows are few or short, it means there is little or no movement.
The scene is mostly still or has very small motion, suggesting a calm or static moment in the video.

Task 2: Optical Flow Warping – Bridging the River's Waves

With the currents traced, the Guardian points to the chasm between frames: "The flow is known, but the river deceives—past and present drift apart. Bridge them with optical warping, apprentice: Remap the 1st frame's pixels along the unseen paths of your Lucas-Kanade rite, forging a vision that aligns with the 9th frame. From scratch, weave the warp, then compare the bridged image to the distant shore."

Your Quest (4pts): Select 1st and 9th frames from the videos. Use computed flow from Task 1. Implement warping from scratch: For each pixel in the 9th frame, trace back via-flow to sample from the 1st frame.

Outputs: Warped image ; Compare PSNR values of warped vs 9th frame of each video.

Guardian Tomes

• Motion and Optical Flow – Currents' path.
• Image Warping – Forging bridges.
• Forward and Backward Warping for Optical Flow-Based Frame Interpolation – Weaving rite.

Bridge the flux... 🌀

from scipy.signal import convolve2d
from scipy import ndimage
from scipy.ndimage import map_coordinates
from skimage.metrics import peak_signal_noise_ratio as psnr
import numpy as np
import cv2

def optical_flow_warping(frame_a, frame_b):
    """
    Warp frame_a toward frame_b using optical flow computed via the Lucas–Kanade method.
    Returns the warped image and prints the PSNR between the warped result and frame_b.
    """

    # --- Convert frames to normalized grayscale ---
    I1 = cv2.cvtColor(frame_a, cv2.COLOR_BGR2GRAY).astype(np.float32) / 255.0
    I2 = cv2.cvtColor(frame_b, cv2.COLOR_BGR2GRAY).astype(np.float32) / 255.0

    # --- Spatial and temporal derivative kernels ---
    kernel_x = np.array([[-1, 1], [-1, 1]]) * 0.25
    kernel_y = np.array([[-1, -1], [1, 1]]) * 0.25
    kernel_t1 = np.ones((2, 2)) * 0.25
    kernel_t2 = -kernel_t1

    # --- Compute derivatives ---
    Ix = convolve2d(I1, kernel_x, boundary='symm', mode='same') + convolve2d(I2, kernel_x, boundary='symm', mode='same')
    Iy = convolve2d(I1, kernel_y, boundary='symm', mode='same') + convolve2d(I2, kernel_y, boundary='symm', mode='same')
    It = convolve2d(I2, kernel_t1, boundary='symm', mode='same') + convolve2d(I1, kernel_t2, boundary='symm', mode='same')

    # --- Lucas–Kanade flow estimation (windowed least squares) ---
    window = 5
    W = np.ones((window, window))
    Ix2 = ndimage.convolve(Ix * Ix, W, mode='reflect')
    Iy2 = ndimage.convolve(Iy * Iy, W, mode='reflect')
    Ixy = ndimage.convolve(Ix * Iy, W, mode='reflect')
    Ixt = ndimage.convolve(Ix * It, W, mode='reflect')
    Iyt = ndimage.convolve(Iy * It, W, mode='reflect')

    det = Ix2 * Iy2 - Ixy**2
    det[det == 0] = 1e-10  # avoid divide-by-zero
    u = (-Iy2 * Ixt + Ixy * Iyt) / det
    v = (Ixy * Ixt - Ix2 * Iyt) / det

    # --- Backward warping: sample pixels from I1 using flow vectors ---
    h, w = I1.shape
    X, Y = np.meshgrid(np.arange(w), np.arange(h))
    Xb = np.clip(X + u, 0, w - 1)
    Yb = np.clip(Y + v, 0, h - 1)

    warped_gray = map_coordinates(I1, [Yb, Xb], order=1, mode='reflect')
    warped_image = cv2.cvtColor((warped_gray * 255).astype(np.uint8), cv2.COLOR_GRAY2BGR)

    # --- Compare similarity ---
    psnr_value = psnr(I2, warped_gray)
    print(f"PSNR: {psnr_value:.2f} dB")

    return warped_image

for video_idx, video_frames in enumerate(tqdm(videos_images)):
    i = 1
    warped_image = optical_flow_warping(video_frames[i], video_frames[i+8])
    show_image_grid(np.array([video_frames[i+8], warped_image]), 1, 2, 'Warp Image Comparison', figsize=16)

  0%|          | 0/8 [00:00<?, ?it/s]

PSNR: 25.45 dB

 12%|█▎        | 1/8 [00:00<00:02,  3.00it/s]

PSNR: 27.66 dB

 25%|██▌       | 2/8 [00:00<00:02,  2.81it/s]

PSNR: 23.10 dB

 38%|███▊      | 3/8 [00:01<00:01,  2.79it/s]

PSNR: 22.92 dB

 50%|█████     | 4/8 [00:01<00:01,  2.99it/s]

PSNR: 16.60 dB

 62%|██████▎   | 5/8 [00:01<00:01,  2.94it/s]

PSNR: 16.74 dB

 75%|███████▌  | 6/8 [00:02<00:00,  3.05it/s]

PSNR: 16.86 dB

 88%|████████▊ | 7/8 [00:02<00:00,  3.19it/s]

PSNR: 21.73 dB

100%|██████████| 8/8 [00:02<00:00,  3.02it/s]

Color(Extra)

def inverse_warp_image_from_scratch(frame, u, v, current_time, other_time, target_time):
    """
    Warp an image using inverse optical flow fields without external interpolation libraries.

    Args:
        frame (np.ndarray): Grayscale image to warp.
        u (np.ndarray): Horizontal optical flow field.
        v (np.ndarray): Vertical optical flow field.
        current_time (float): The time of the current frame.
        other_time (float): The time of the other frame (frame we are warping towards).
        target_time (float): The time of the target frame (interpolated frame).

    Returns:
        np.ndarray: Warped image.
    """
    h, w = frame.shape[:2]
    warped_image = np.zeros_like(frame, dtype=np.float32)

    time_scale = (target_time - current_time) / (other_time - current_time)

    # Iterate over all pixels in the target (warped) image
    for y in range(h):
        for x in range(w):
            # Compute the source (original) pixel location using inverse flow
            x_orig = x - u[y, x] * time_scale
            y_orig = y - v[y, x] * time_scale

            # Ensure that the original pixel location is within bounds
            if x_orig < 0 or x_orig >= w - 1 or y_orig < 0 or y_orig >= h - 1:
                continue

            x0, y0 = int(np.floor(x_orig)), int(np.floor(y_orig))
            x1, y1 = min(x0 + 1, w - 1), min(y0 + 1, h - 1)

            dx, dy = x_orig - x0, y_orig - y0

            # bilinear interpolation
            warped_image[y, x] = ((1 - dx) * (1 - dy) * frame[y0, x0] + dx * (1 - dy) * frame[y0, x1] + (1 - dx) * dy * frame[y1, x0] + dx * dy * frame[y1, x1])

    return np.clip(warped_image, 0, 255)

def optical_flow_warping(frame_a, frame_b, start_time, end_time, target_time):
    """
    Perform optical flow-based warping to interpolate a frame at target_time.

    Args:
        frame_a (np.ndarray): First frame.
        frame_b (np.ndarray): Second frame.
        start_time (float): Time of frame_a.
        end_time (float): Time of frame_b.
        target_time (float): Target time for the interpolated frame.

    Returns:
        np.ndarray: Warped interpolated frame.
    """
    # Compute optical flow from frame_a to frame_b (forward)
    U_front, V_front, _, _, _ = optical_flow_consecutive(frame_a, frame_b, 15)
    warped_frame_forward = inverse_warp_image_from_scratch(frame_a, U_front, V_front, start_time, end_time, target_time).astype(np.uint8)

    # Compute optical flow from frame_b to frame_a (backward)
    U_back, V_back, _, _, _ = optical_flow_consecutive(frame_b, frame_a, 15)
    warped_frame_backward = inverse_warp_image_from_scratch(frame_b, -U_back, -V_back, end_time, start_time, target_time).astype(np.uint8)

    return warped_frame_forward, warped_frame_backward

for video_idx, video_frames in enumerate(tqdm(videos_images)):
    i = 1
    warped_image, _ = optical_flow_warping(video_frames[i], video_frames[i+8], start_time=i, end_time = i + 8, target_time= i + 8)
    show_image_grid(np.array([video_frames[i+8], warped_image]), 1, 2, 'Warp Image Comparison', figsize=16)

Quest Time (1pt): The Guardian's voice echoes softly: "You've woven the bridge, apprentice—past pulled to present. Pause and ponder on a basic truth."

What is "backward warping," and why pull from source to target instead of pushing forward?

Backward warping means that for every pixel in the target image, we trace backward through the optical flow field to find where that pixel came from in the source image, and then sample its value from there.

We “pull” pixels from the source (using coordinates from the target) instead of “pushing” them forward because:

• In forward warping, multiple source pixels might land on the same target pixel or leave gaps (holes) where no source pixel maps.
• In backward warping, every target pixel gets a value by sampling from a valid source location — giving a complete, hole-free warped image.

---

In short:
Backward warping = “pulling” colors from the source into each target pixel.
It avoids holes and overlaps that occur when trying to “push” pixels forward.

🔮 The Final Curiosity Rite – Decoding the Hidden Gate

As the River of Time settles into calm reflection, the Guardian of Flux reveals one last mystery.
From the depths of the Archive, an image materializes — a sealed Gate unlike any before. Its surface hums faintly, patterns shifting between shadows and light.

“The journey ends where perception begins,” whispers the Guardian.
“Hidden within these pixels lies the last message of the Order. To read it, you must see beyond brightness and color — into the very bits of vision itself."

Your task (1pt): Decode the concealed message embedded within this Gate’s bit-planes. Extract the hidden text from the image’s lower bits using bitwise shift operators. The lower bits of image will reveal the final teaching of the Visioneers — a message only visible to those who have mastered perception itself.

Jot the decoded text below—the door creaks wider with your sight. Well earned, Visioneer; the veil thins... 🌀

encrypted_gate = cv2.imread('/content/drive/MyDrive/ES666CV/videos/Decode_Message/encoded.png')
encrypted_gate = cv2.cvtColor(encrypted_gate, cv2.COLOR_BGR2RGB)

# Display
plt.figure(figsize=(8, 6))
plt.imshow(encrypted_gate)
plt.axis('off')
plt.show()

import cv2
import numpy as np
import matplotlib.pyplot as plt

# Load and convert to RGB
encrypted_gate = cv2.imread('/content/drive/MyDrive/ES666CV/videos/Decode_Message/encoded.png')
encrypted_gate = cv2.cvtColor(encrypted_gate, cv2.COLOR_BGR2RGB)

# Step 1: Extract only the least significant bit (LSB) of each channel
lsb = encrypted_gate & 1  # keeps only last bit of each channel (R,G,B)

# Step 2: Combine bits to visualize hidden structure
# You can scale it up so it’s visible as 0 or 255
hidden_image = (lsb * 255).astype(np.uint8)

plt.figure(figsize=(8,6))
plt.imshow(hidden_image)
plt.axis('off')
plt.show()

# Step 3: If the hidden text is encoded in grayscale:
gray_hidden = cv2.cvtColor(hidden_image, cv2.COLOR_RGB2GRAY)
plt.figure(figsize=(8,6))
plt.imshow(gray_hidden, cmap='gray')
plt.axis('off')
plt.show()

---

🏁 Epilogue – The World Reconstructed

The final Gate fades open as the encoded message glows faintly on your screen.
Bit by bit, light turns into language — and you received the last teaching of the Order.

The River of Time falls silent. The Guardians withdraw into the mist.
You stand alone in the Grand Archive — now alive once more, its images breathing,
its motion restored, its fragments whole.

You have crossed the Four Gates of Vision. Each gate was more than an assignment — it was a rite of perception.
Through them, you have learned not merely to look, but to see:
to discover order in data, motion in stillness, and meaning in patterns.

“The Archive is whole again,” murmurs the Grand Archivist.
“You have not just processed images — you have perceived the world itself.”

The scrolls close. The Hall of Vision fades into light.
All that remains is reflection, motion, and truth — beyond pixels.

🌀 Thus ends the Chronicle of the Visioneers...Congratulations!